<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Ollama Num_parallel on RockB</title><link>https://baeseokjae.github.io/tags/ollama-num_parallel/</link><description>Recent content in Ollama Num_parallel on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Wed, 04 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/ollama-num_parallel/index.xml" rel="self" type="application/rss+xml"/><item><title>Two Agents, One Ollama: Measuring Why the Second Request Queues Instead of Speeding Up (2026)</title><link>https://baeseokjae.github.io/posts/ollama-concurrent-requests-latency-2026/</link><pubDate>Wed, 04 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/ollama-concurrent-requests-latency-2026/</guid><description>A CPU-only measurement of two simultaneous Ollama requests: the second one waits one full generation, and the response body proves it. Includes the one-line queue diagnostic, a 17-batch matrix for OLLAMA_NUM_PARALLEL 1/2/4, and why extra slots buy latency, not throughput.</description><content:encoded><![CDATA[<p>You start a second coding agent against the Ollama endpoint that already answers your first one. Nothing errors out. The second call simply sits there for two or three seconds, and when both finish your &ldquo;parallel&rdquo; run has taken longer than running the two prompts one after the other. The usual conclusion is that the model is slow, the quant is wrong, or the machine needs more RAM. On the host measured here, none of those were true: the second request was queued, and the HTTP response said so in its own timing fields.</p>
<p>This post is the mechanism, a diagnostic you can run against one response, and a measured verdict for CPU-only machines. Everything below comes from 17 recorded batches on a 2-core aarch64 box with Ollama 0.20.3 and <code>qwen2.5:0.5b</code>, run on 2026-11-04 between 00:33:20Z and 00:35:37Z. The raw records are published with the article so the numbers can be recomputed rather than trusted.</p>
<h2 id="what-ollama-does-with-the-second-request">What Ollama does with the second request</h2>
<p>Ollama&rsquo;s own documentation describes two levels of concurrency: several models resident at once, and, inside one model, several requests processed in parallel. The second level is what a subagent fan-out or a background embedding call runs into, and it is a per-model setting rather than a per-request one — <a href="https://docs.ollama.com/faq.md">the FAQ states that parallel request processing is configured when the model is loaded, and that a model is &ldquo;configured to allow parallel request processing&rdquo; only if memory allows</a>. When memory does not allow it, &ldquo;all new requests will be queued until the new model can be loaded&rdquo;, and &ldquo;queued requests will be processed in order&rdquo;.</p>
<p>The compiled defaults decide how often that path is taken. In the runtime&rsquo;s configuration source, <a href="https://raw.githubusercontent.com/ollama/ollama/main/envconfig/config.go">parallel request processing defaults to 1, the pending queue to 512, and the maximum number of resident models to 0, meaning automatic</a>. So a stock install handles exactly one generation per model at a time. The second request is not rejected and not run concurrently — it goes into the scheduler&rsquo;s pending channel, which is also where the &ldquo;server busy&rdquo; error comes from: <a href="https://raw.githubusercontent.com/ollama/ollama/main/server/sched.go">in the scheduler, a request whose runner is neither loaded nor reloadable is pushed to a pending channel, and only a full channel returns <code>ErrMaxQueue</code></a>. Up to 512 requests can therefore wait politely for a slot before anything fails.</p>
<p>That is the whole trap. A queued request returns HTTP 200 with a normal-looking answer. The only trace of the wait is time that belongs to no phase of the work.</p>
<h2 id="the-diagnostic-one-response-four-fields">The diagnostic: one response, four fields</h2>
<p><code>/api/generate</code> reports its own timing breakdown, in nanoseconds: <code>total_duration</code>, <code>load_duration</code>, <code>prompt_eval_duration</code>, and <code>eval_duration</code>, <a href="https://docs.ollama.com/api/generate.md">documented in the generate API reference</a>. For a request that was never queued, those three components add up to the total. For a request that waited, they do not, and the difference is the queue wait:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -s http://127.0.0.1:11434/api/generate -d <span style="color:#e6db74">&#39;{&#34;model&#34;:&#34;qwen2.5:0.5b&#34;,&#34;prompt&#34;:&#34;hello&#34;,&#34;stream&#34;:false}&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| python3 -c <span style="color:#e6db74">&#39;import json,sys; r=json.load(sys.stdin); t=r[&#34;total_duration&#34;]; \
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">q=t-(r[&#34;load_duration&#34;]+r[&#34;prompt_eval_duration&#34;]+r[&#34;eval_duration&#34;]); \
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">print(&#34;total_s=%.2f queue_s=%.2f queue_fraction=%.2f&#34; % (t/1e9, q/1e9, q/t))&#39;</span>
</span></span></code></pre></div><p>Two details make the difference between a working diagnostic and a misleading one. First, <code>load_duration</code> is not zero even when the model is already resident: on this host it was 0.20-0.30 s for a warm 0.5B model and 0.49-1.04 s on the 4-slot instance. Load plus prompt-eval overhead was 0.24-0.34 s on the default and 2-slot instances, 0.62-0.92 s in the warm 4-slot batch and 0.96-1.32 s in the second cold 4-slot batch, so subtracting only <code>eval_duration</code> from <code>total_duration</code> inflates apparent waiting by roughly a quarter to a third of a second at one or two slots and by up to about 1.3 s at four. (The first cold 4-slot batch is worse still — a prompt-cache miss put 4.64 s of prompt evaluation on three of its four requests — and is excluded from the reported table for that reason.) Second, the fraction matters more than the absolute value, because a long generation dilutes the same queue wait into a smaller share.</p>
<p>On the measured host the split was unambiguous. Solo requests in the default configuration left a residual of 0.05 s, about 2% of a 2.3-2.6 s total. In the same configuration, the second of two simultaneous requests showed a residual of 2.16, 2.27, 2.17 and 2.41 s across four batches — 40-51% of its total time — while the first request in that batch still finished with a 0.05 s residual.</p>
<h2 id="the-measurement-17-batches-three-configurations">The measurement: 17 batches, three configurations</h2>
<p>The runtime was Ollama 0.20.3, <a href="https://github.com/ollama/ollama/releases/tag/v0.20.3">published 2026-04-07 according to the GitHub release metadata</a>. The probe sends <code>POST /api/generate</code> with <code>stream: false</code>, <code>temperature: 0</code>, <code>seed: 42</code> and <code>num_predict: 64</code>, one fixed prompt, and every request produced <code>eval_count: 46</code>. Concurrent requests synchronise on a barrier so they leave together. Three server configurations were used: A is the pre-existing default server on port 11434; B is a dedicated instance on port 11500 with <code>OLLAMA_NUM_PARALLEL=2</code>; C is a dedicated instance on port 11501 with <code>OLLAMA_NUM_PARALLEL=4</code>. Both dedicated instances used the system model directory and <code>OLLAMA_MAX_QUEUE=1</code>, which is worth noting: with free slots available, nothing was rejected, so the single-queue-slot limit never fired.</p>
<table>
	<thead>
			<tr>
					<th>Config</th>
					<th>Concurrency</th>
					<th>Warm batches</th>
					<th>Per-request total (s)</th>
					<th>Per-request eval (s)</th>
					<th>Queue residual (s)</th>
					<th>Aggregate tok/s</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A default</td>
					<td>1</td>
					<td>3</td>
					<td>2.30-2.59</td>
					<td>2.00-2.27</td>
					<td>0.05</td>
					<td>17.7-20.0</td>
			</tr>
			<tr>
					<td>A default</td>
					<td>2</td>
					<td>4</td>
					<td>2.40-2.62 (first) / 4.49-5.69 (second)</td>
					<td>2.02-2.26 (first) / 1.96-3.13 (second)</td>
					<td>0.05 (first) / 2.16-2.41 (second)</td>
					<td>16.2-20.4</td>
			</tr>
			<tr>
					<td>B <code>NUM_PARALLEL=2</code></td>
					<td>1</td>
					<td>2</td>
					<td>2.29-2.48</td>
					<td>1.99-2.17</td>
					<td>0.05</td>
					<td>18.5-20.1</td>
			</tr>
			<tr>
					<td>B <code>NUM_PARALLEL=2</code></td>
					<td>2</td>
					<td>3</td>
					<td>4.08-4.60</td>
					<td>3.67-4.13</td>
					<td>0.09-0.17</td>
					<td>20.0-22.0</td>
			</tr>
			<tr>
					<td>C <code>NUM_PARALLEL=4</code></td>
					<td>4</td>
					<td>1</td>
					<td>8.70-8.94</td>
					<td>7.81-7.90</td>
					<td>0.12-0.34</td>
					<td>20.6</td>
			</tr>
	</tbody>
</table>
<p>Four batches were excluded as cold and are marked <code>cold=true</code> in the per-batch summary: the very first solo batch (4.93 s, of which 4.61 s was <code>eval_duration</code> at 10.0 tok/s; <code>load_duration</code> was 0.22 s, no higher than the warm solo rows), one 2-slot batch on config B where a cold prompt cache pushed <code>prompt_eval_duration</code> to 1.58 s, and the two earliest 4-slot batches (19.2 s and 16.4 s): the first carries the 4.64 s prompt-eval miss on three requests described above, and the second spent 14.8 s of its 16.4 s in <code>eval_duration</code> at 3.1 tok/s with <code>load_duration</code> 0.73-1.04 s. They ran at 9.6 and 11.2 aggregate tok/s against 20.6 in the warm batch. Cold exclusion is stated because it changes the numbers: the excluded 4-slot batches are twice as slow as the one warm batch reported.</p>
<h2 id="what-the-numbers-say">What the numbers say</h2>
<p>Under the default configuration, the second request waited about as long as one full generation. Its total was 4.49-5.69 s against 2.30-2.59 s for a solo request, and the wait it added — 2.16-2.41 s — is almost exactly the eval time of the batch-mate holding the slot (2.02-2.26 s). Per-request token rates mostly held: in three of the four batches the queued request generated at 22.0-23.4 tok/s against 20.3-22.8 for that batch-mate. The exception is rep 1, where the queued request ran at 14.7 tok/s (eval_duration 3.13 s) against 21.7 for the first, even though the first request&rsquo;s own latency was unchanged at 2.46 s. That is still a queue, not CPU contention: if the two generations had been sharing two cores for their whole duration, the first request&rsquo;s latency would have moved, and it did not.</p>
<p>Setting <code>OLLAMA_NUM_PARALLEL=2</code> removed the wait and replaced it with slower generations. Both requests came back in 4.08-4.60 s with <code>eval_duration</code> 3.67-4.13 s, against 2.00-2.27 s solo, and the queue residual fell to 0.09-0.17 s. Per-request throughput halved from roughly 20-23 tok/s to 11.1-12.5 tok/s. On two cores, two concurrent generations each run at about half speed, which is what a shared CPU should do.</p>
<p>Aggregate throughput barely moved. Warm batches gave a median of 19.1 tok/s solo on config A, 19.8 tok/s for config A at concurrency 2, 22.0 tok/s for config B at concurrency 2, and 20.6 tok/s for the single warm config C batch at concurrency 4. The best-case ratio that could be argued, 22.0 / 19.8 = 1.11x, sits inside a run-to-run spread of 16.2-22.0 tok/s, and far below the 1.5x improvement that would have made extra slots worth recommending on this hardware. Doubling or quadrupling the slot count changed latency per request by roughly 1.8-3.9x and throughput by nothing measurable.</p>
<p>The practical consequence is the counter-intuitive one: for two prompts on a CPU-only host, fanning out to two agents takes as long as — or slightly longer than — calling them one after the other, because the queue serialises them anyway when only one slot exists, and splits the cores when two exist. Parallelism on a CPU host is not free capacity; it is a different shape of the same capacity.</p>
<h2 id="what-it-costs-in-memory">What it costs in memory</h2>
<p>Slots are not free. The FAQ states that parallel request processing increases the context size by the number of parallel requests, and the server log confirms the arithmetic: the load line reports <code>Parallel:2 ... KvSize:8192</code> for the 2-slot instance and <code>Parallel:4 ... KvSize:16384</code> for the 4-slot instance, against a 4,096-token default context. <code>KvSize</code> is the total KV allocation across slots, not a per-slot value.</p>
<p><code>/api/ps</code> shows the result in reported model memory: <code>size</code> 442,191,360 bytes for the default server and 492,523,008 bytes with two slots — 50,331,648 bytes, or 48.0 MiB, more, a rise of 11.4%. One detail that catches readers out: <code>context_length</code> stayed at 4096 in both <code>/api/ps</code> captures, so a per-request context that halved is not what the API reports. The KV allocation doubles — <code>KvSize:8192</code> for two slots against the 4,096-token default context — while the total reported <code>size</code> rises by 11.4%. The reported per-slot context does not change. If you are sizing a second slot, do the arithmetic from context times slot count, as in <a href="/posts/local-llm-vram-kv-cache-budget-2026/">the KV-cache budget walkthrough for a local coding model</a>, rather than trusting the single <code>context_length</code> field.</p>
<p>Every one of these captures reports <code>size_vram: 0</code>. This host has no GPU, and nothing here should be read as a statement about GPU behaviour, where a second slot may overlap compute across SMs and actually raise throughput.</p>
<h2 id="what-to-do-instead">What to do instead</h2>
<p>Serialise, then batch. If the second call is an embedding step or a summarisation pass that does not need to interleave with the first, run it after the agent&rsquo;s tool call rather than beside it — you lose nothing on a CPU host, because the scheduler was serialising it anyway whenever the slot was busy.</p>
<p>If you do want concurrent slots, enable them deliberately and budget the memory, then verify with the residual diagnostic that requests are actually overlapping rather than queueing. A residual near zero with doubled <code>eval_duration</code> means the slots are working as intended; a residual equal to one solo generation means you are still at one slot, whatever the environment variable says, which is the usual symptom of setting it on the client instead of the server process.</p>
<p>Move to a GPU when throughput, not latency, is the goal. The 1.11x aggregate result here is a statement about two CPU cores, not about <code>OLLAMA_NUM_PARALLEL</code> in general — and even on a GPU, <a href="/posts/vllm-vs-ollama-production-2026/">the queue-versus-batching distinction is what decides whether a serving stack helps</a>.</p>
<h2 id="what-this-is-not">What this is not</h2>
<p>It is not a GPU benchmark, a model comparison, or a vLLM comparison. One 0.5B Q4_K_M model, 46 generated tokens, two cores: long prompts and prefill-bound workloads were not measured, and <code>OLLAMA_MAX_LOADED_MODELS</code> behaviour — a second <em>model</em> being swapped in — was not tested at all. The 503 path behind <code>OLLAMA_MAX_QUEUE</code> is documented in the source linked above but was not reproduced live, because every request in this matrix eventually received a slot. The queue-versus-compute split is derived from Ollama&rsquo;s own duration fields; no external profiler or debug-level scheduler trace corroborates it.</p>
<p>If your local agent is underperforming, scheduler queueing is one of several causes. Context truncation is the other common one on the same endpoint, covered in <a href="/posts/ollama-silent-context-truncation-2026/">why a local model silently answers with less context than you sent</a>. If you leave Ollama for llama.cpp&rsquo;s server, the equivalent knob is <code>-np</code>, and its splitting behaviour differs from Ollama&rsquo;s scheduler — see <a href="/posts/llama-server-context-overflow-diagnosis-2026/">diagnosing llama-server context overflow</a>. For the environment-variable reference and the OpenAI-compatible endpoint, the <a href="/posts/ollama-api-local-llm-integration-developer-guide-2026/">Ollama API integration guide</a> lists the same three settings; the tuning block in <a href="/posts/aider-ollama-local-coding-2026/">the Aider + Ollama setup</a> is the one this post measures. And if you are choosing a model for a CPU-only host in the first place, <a href="/posts/best-ollama-models-coding-2026/">the coding-model shortlist for CPU inference</a> is the better starting point than any concurrency tuning.</p>
<h2 id="reproduce-it">Reproduce it</h2>
<p>The measurement is reproducible on a 2-core CPU box in under three minutes: 17 batches took 137 s including model load. The files below are the exact inputs and outputs of the run described above, published so every figure in this article can be recomputed from the raw response bodies.</p>
<table>
	<thead>
			<tr>
					<th>File</th>
					<th>What it is</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/probe.py">probe.py</a></td>
					<td>stdlib-only probe: barrier-synchronised concurrent <code>POST /api/generate</code>, records status and all timing fields</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/run.sh">run.sh</a></td>
					<td>the exact commands: host capture, <code>/api/ps</code> before and after, the three server configurations, per-cell warm-up</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/results.jsonl">results.jsonl</a></td>
					<td>17 batches, one JSON object per batch, every request&rsquo;s raw nanosecond fields</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/summary.csv">summary.csv</a></td>
					<td>per-batch derived rows: aggregate tok/s, per-request medians, queue residual, cold flag</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/method.md">method.md</a></td>
					<td>request shape, derived formulas, what was deliberately not measured; the authoritative <code>cold</code> flag is in <code>summary.csv</code></td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/host.txt">host.txt</a></td>
					<td><code>ollama --version</code>, <code>nproc</code>, <code>free -m</code>, kernel and architecture</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/ps_A.txt">ps_A.txt</a> / <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/ps_B.txt">ps_B.txt</a></td>
					<td><code>/api/ps</code> after each configuration: <code>size</code>, <code>size_vram</code>, <code>context_length</code></td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/server_b_lines.txt">server_b_lines.txt</a> / <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/server_c_lines.txt">server_c_lines.txt</a></td>
					<td>the server config and load lines showing <code>OLLAMA_NUM_PARALLEL</code>, <code>Ollama version 0.20.3</code>, <code>Parallel</code> and <code>KvSize</code></td>
			</tr>
	</tbody>
</table>
<p>To rerun on your own hardware, start a second server on a spare port against the same model directory rather than restarting the one your editor is using, point <code>probe.py</code> at it, and compare the residual fraction for concurrency 1 and 2. If the second request&rsquo;s residual is near zero and its <code>eval_duration</code> roughly doubles, your slots are working; if the residual is about one solo generation, they are not. Note that <code>summary.csv</code> reports the per-batch median across the requests in that batch, so a two-request batch with one queued request shows a residual median of roughly half the queued request&rsquo;s own wait — read the per-request fields in <code>results.jsonl</code> when you want the wait of the request that actually queued.</p>
]]></content:encoded></item><item><title>Two Agent Sessions, One Local Model Server: what concurrency does to latency, context and tool calls</title><link>https://baeseokjae.github.io/posts/local-llm-concurrent-agent-sessions-2026/</link><pubDate>Mon, 02 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/local-llm-concurrent-agent-sessions-2026/</guid><description>Ollama ships one parallel slot per model, so a second agent session waits behind the first: four concurrent requests serialised to 4.0x the single-request wall time on a 2-vCPU host, and OLLAMA_NUM_PARALLEL=4 made each request 4.4-8.0x slower against the default server&amp;#39;s single-request baseline (3.8-7.0x against its own) instead of faster. Measured latency, the documented context rules, and what to log when a session stalls.</description><content:encoded><![CDATA[<p>Your first coding-agent session answers in a couple of seconds. You open a second one against the same local server, and now the first session&rsquo;s next request stalls for many seconds, or your client aborts with a read timeout while the server log shows no error at all. Restart the server and run one session: everything is fast again.</p>
<p>Nothing is wrong with the model or with either agent. Both sessions are sending requests to one server process that was never configured to work on more than one at a time, and the queue between them is invisible to your client. This article shows the documented defaults that decide how many requests a local server will actually run at once, measures what the second and fourth session do to wall time on a small CPU host, and lists what to record so you can prove the cause instead of guessing.</p>
<h2 id="the-shared-resource-is-the-server-not-the-model-weights">The shared resource is the server, not the model weights</h2>
<p>Two agents pointed at <code>http://localhost:11434</code> are two HTTP clients against one Ollama process. That process loads the model once and then decides how to schedule the requests it receives. Ollama&rsquo;s own documentation splits concurrency into two levels: how many <em>models</em> may be resident at once, and how many <em>parallel requests</em> a single model will process at the same time — <a href="https://docs.ollama.com/faq.md">Ollama FAQ, concurrency section</a>. The second number is the one that decides whether your two sessions overlap or take turns.</p>
<p>The FAQ states the rule directly: &ldquo;For a given model, if there is sufficient available memory when the model is loaded, it is configured to allow parallel request processing&rdquo;, and a model that is loaded with only one request processing at a time will make the second caller wait. Requests that cannot be served immediately are queued in arrival order — &ldquo;Queued requests will be processed in order&rdquo; — and only rejected when the queue itself is full (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>).</p>
<h2 id="ollamas-default-is-one-slot-per-model-not-four">Ollama&rsquo;s default is one slot per model, not four</h2>
<p>Three server settings decide the shape of the queue, and all three have defaults you probably never set (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>) — the values below are quoted from that page, which is the same text this run read on 2026-11-02:</p>
<table>
	<thead>
			<tr>
					<th>Setting</th>
					<th>Documented default</th>
					<th>What it changes</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>OLLAMA_NUM_PARALLEL</code></td>
					<td>1</td>
					<td>Parallel requests the model processes at once; &ldquo;Required RAM will scale by <code>OLLAMA_NUM_PARALLEL</code> * <code>OLLAMA_CONTEXT_LENGTH</code>&rdquo;</td>
			</tr>
			<tr>
					<td><code>OLLAMA_MAX_LOADED_MODELS</code></td>
					<td>3 (CPU) or 3 x GPUs</td>
					<td>How many distinct models may stay resident</td>
			</tr>
			<tr>
					<td><code>OLLAMA_MAX_QUEUE</code></td>
					<td>512</td>
					<td>Requests held while busy before new ones are rejected</td>
			</tr>
	</tbody>
</table>
<p>Check what your own server is running with before you theorise about it. An empty <code>env | grep '^OLLAMA'</code> means every value is a default, and the model row from <code>ollama ps</code> prints the context the server believes it is serving plus how long it will stay loaded:</p>
<pre tabindex="0"><code>$ env | grep &#39;^OLLAMA&#39;        # no output = all defaults, so OLLAMA_NUM_PARALLEL=1
$ ollama ps                   # NAME  ID  SIZE  PROCESSOR  CONTEXT  UNTIL
</code></pre><p>The per-request context is a separate knob from those server variables: <code>num_ctx</code> is the &ldquo;Context length size (number of tokens)&rdquo; in the <a href="https://docs.ollama.com/api/generate.md">Generate API reference</a>, and <code>options.num_ctx</code> is what an agent client sends with each call. A client that sets a generous context per request is not asking for a generous number of slots.</p>
<h2 id="what-the-probe-measured-on-one-cpu-host">What the probe measured on one CPU host</h2>
<p>The measurement below is deliberately narrow: one machine, one small model, no GPU, and one fixed prompt. It answers &ldquo;what does a second session cost on a host like mine&rdquo;, and it says nothing about throughput on hardware with more cores.</p>
<table>
	<thead>
			<tr>
					<th>Item</th>
					<th>Value</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Host</td>
					<td>2 vCPU aarch64 (Neoverse-N1), 11 GiB RAM, no GPU</td>
			</tr>
			<tr>
					<td>Server</td>
					<td><code>ollama version is 0.20.3</code></td>
			</tr>
			<tr>
					<td>Model</td>
					<td><code>qwen2.5:0.5b</code>, 397 MB, Q4_K_M</td>
			</tr>
			<tr>
					<td>Request</td>
					<td>one non-streaming <code>POST /api/generate</code>, fixed 59-token prompt, <code>num_ctx=2048</code>, <code>num_predict=64</code>, <code>temperature=0</code>, <code>seed=7</code>, <code>keep_alive=5m</code></td>
			</tr>
			<tr>
					<td>Load</td>
					<td>N concurrent clients released from a single barrier, so every request enters the server in the same instant</td>
			</tr>
			<tr>
					<td>Config A</td>
					<td>system server, no <code>OLLAMA_*</code> variables (defaults, therefore one slot)</td>
			</tr>
			<tr>
					<td>Config B</td>
					<td>second server on <code>127.0.0.1:11435</code> started as <code>OLLAMA_HOST=127.0.0.1:11435 OLLAMA_NUM_PARALLEL=4 OLLAMA_CONTEXT_LENGTH=2048 ollama serve</code></td>
			</tr>
	</tbody>
</table>
<p>Wall time is measured client-side around the whole call, which is exactly what your agent&rsquo;s HTTP client sees. Every request in every cell returned HTTP 200 with <code>eval_count=64</code>.</p>
<table>
	<thead>
			<tr>
					<th>Config</th>
					<th>N</th>
					<th>Observed wall times (ms)</th>
					<th>Slowest vs the 3256.9 ms default N=1 row</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>1</td>
					<td>3256.9; 3209.0</td>
					<td>1.0x</td>
			</tr>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>2</td>
					<td>3223.4, 6235.9</td>
					<td>1.9x</td>
			</tr>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>4</td>
					<td>4163.3, 7127.3, 10138.3, 13131.4</td>
					<td>4.0x</td>
			</tr>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>4 (repeat)</td>
					<td>3504.4, 6513.7, 9455.4, 12418.3</td>
					<td>3.8x</td>
			</tr>
			<tr>
					<td>B: <code>OLLAMA_NUM_PARALLEL=4</code></td>
					<td>4</td>
					<td>25586.7, 26199.3, 26211.4, 26212.7</td>
					<td>8.0x</td>
			</tr>
			<tr>
					<td>B: <code>OLLAMA_NUM_PARALLEL=4</code></td>
					<td>4 (repeat)</td>
					<td>14003.7, 14316.3, 14320.7, 14325.4</td>
					<td>4.4x</td>
			</tr>
			<tr>
					<td>B: <code>OLLAMA_NUM_PARALLEL=4</code></td>
					<td>1</td>
					<td>3753.9</td>
					<td>1.15x</td>
			</tr>
	</tbody>
</table>
<p>The last column divides by one denominator throughout: the 3256.9 ms N=1 row of configuration A, the default server. That makes the last three rows a comparison between two servers, not a before/after on one. Measured against the four-slot server&rsquo;s own single-request row (3753.9 ms), the same two four-slot cells are 7.0x and 3.8x — the same conclusion, a smaller factor, and the number that isolates what the slot setting did on that process.</p>
<p>Two shapes are worth reading out of that table. Under the default configuration the four requests finish at roughly 3.0-second intervals — 4163, 7127, 10138, 13131 ms in the first repeat, 2964 to 3011 ms apart, and 3504, 6514, 9455, 12418 ms in the second, 2942 to 3009 ms apart — which is about one single-request duration each, the signature of one slot serving four requests back to back rather than four requests progressing together. The N=2 cell shows the same thing at smaller scale: one request returned in 3223 ms, the second in 6236 ms, because it started by waiting for the first to finish.</p>
<p>With four slots the shape inverts, and it does not improve. In the first repeat all four requests finished within 630 ms of each other but each took 25.6 to 26.2 seconds, roughly twice as long as the slowest queued request under one slot. In the second repeat the same cell landed at 14.0 to 14.3 seconds. The single-request baseline on that second server was 3753.9 ms, so four slots bought parallelism without buying speed: the four requests shared the same two cores, and every one of them ran slower than it would have alone.</p>
<p>The honest confidence statement: most cells have two repeats, not twenty, and the N=2 cell and the second server&rsquo;s N=1 baseline have one. The spread between repeats is large — the same four-slot cell produced a 26-second band and a 14-second band — so treat these as order-of-magnitude figures for a two-core box, not as p95 latencies. They are not a throughput benchmark and they do not transfer to GPU serving.</p>
<h2 id="why-the-client-sees-a-timeout-rather-than-a-503">Why the client sees a timeout rather than a 503</h2>
<p>The obvious hypothesis — &ldquo;the second session gets rejected&rdquo; — does not hold at the sizes a developer actually runs. <code>OLLAMA_MAX_QUEUE</code> defaults to 512 requests, so the server holds queued work instead of refusing it unless you are far past two or three agent sessions (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>). In the probe, 21 requests across all cells produced zero non-200 statuses and zero error bodies; the only observable effect of concurrency was elapsed time.</p>
<p>That is worse than an error, because the failure moves into your client. A queued request looks identical to a slow one until your own read timeout fires. An agent framework with a 30-second HTTP timeout on a host where a queued request can take 26 seconds will abort, retry, and — if the retried tool call carried a side effect — do the same work twice. If you have seen duplicate writes from a &ldquo;flaky&rdquo; local agent, the retry-on-timeout path is worth auditing alongside the queue behaviour in <a href="https://baeseokjae.github.io/posts/agent-tool-retry-duplicate-side-effects-2026/">agent tool retries and duplicate side effects</a>.</p>
<p>The practical consequence: on a one-slot server, your effective concurrency limit is set by the client timeout, not by the server&rsquo;s capacity. Two sessions are fine if a single request takes 3 seconds and your timeout is 30. Four sessions with a longer prompt are not, because the fourth request inherits three queue waits and only the last one looks slow.</p>
<h2 id="context-multiplied-by-slots-here-per-slot-there">Context: multiplied by slots here, per-slot there</h2>
<p>The memory side of concurrency is documented, and it is not what most people assume. Ollama does not divide one context window among parallel requests — it multiplies: &ldquo;Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation&rdquo; (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>). Combined with the stated RAM rule (<code>OLLAMA_NUM_PARALLEL</code> x <code>OLLAMA_CONTEXT_LENGTH</code>), that means raising slots on a memory-tight box can turn a working setup into an allocation failure at load time, before any request is served.</p>
<p>The reported field is a trap. A server started with <code>OLLAMA_NUM_PARALLEL=4</code> and <code>OLLAMA_CONTEXT_LENGTH=2048</code> answered <code>/api/ps</code> with <code>context_length: 2048</code> for a model it had just loaded, even though the vendor rule says four slots allocate four times that context. Reading <code>ollama ps</code> CONTEXT as &ldquo;the memory I am using&rdquo; will therefore understate a multi-slot server; the recorded response is in the published evidence below.</p>
<p>llama.cpp&rsquo;s server takes the opposite configuration shape. Slots are counted explicitly with <code>-np, --parallel N</code> (&ldquo;number of server slots, default: -1, -1 = auto&rdquo;), and per-slot context is a separate flag: <code>--kv-unified-per-slot N</code>, &ldquo;context limit per parallel slot; when set without -c/&ndash;ctx-size, the shared KV pool is sized to n_parallel*N&rdquo; (<a href="https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md">llama.cpp server README</a>). So on llama-server you choose both the slot count and the per-slot window, while on Ollama the per-request <code>num_ctx</code> is multiplied by the slot count. Either way, the number that matters for memory is slots x context, not context.</p>
<p>Before you raise slots, do the arithmetic for the model you actually run — the KV-cache sizing formula and a measured check of it are in <a href="https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/">sizing KV cache and context for a local coding model</a>. A 4-slot server is a 4x cache budget on the same weights.</p>
<h2 id="a-decision-table-for-two-to-four-sessions">A decision table for two to four sessions</h2>
<table>
	<thead>
			<tr>
					<th>Situation</th>
					<th>What to configure</th>
					<th>Why</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Two or three agent sessions on a 2-core CPU box</td>
					<td>Leave <code>OLLAMA_NUM_PARALLEL=1</code>; raise the client&rsquo;s read timeout well past a single request&rsquo;s worst case</td>
					<td>Four slots on two cores measured 4.4-8.0x slower per request than the default server&rsquo;s 3256.9 ms single-request row, and 3.8-7.0x slower against the four-slot server&rsquo;s own 3753.9 ms row; queueing is cheaper than contention at this core count</td>
			</tr>
			<tr>
					<td>Spare cores and spare memory</td>
					<td>Try <code>OLLAMA_NUM_PARALLEL=N</code> with <code>OLLAMA_CONTEXT_LENGTH</code> set explicitly, then measure your own p50</td>
					<td>The vendor rule multiplies context by slots, so both time and memory change together; nothing here predicts your hardware</td>
			</tr>
			<tr>
					<td>Sessions that must never block each other</td>
					<td>Separate servers on separate ports with separate models, or separate hosts</td>
					<td>Two processes cannot contend for the same slot, and each keeps its own context budget</td>
			</tr>
			<tr>
					<td>llama-server instead of Ollama</td>
					<td>Set <code>-np</code> explicitly, and set <code>--kv-unified-per-slot</code> when you want a hard per-session window</td>
					<td>Auto slots hide how the shared KV pool is divided; explicit values make the memory budget auditable</td>
			</tr>
	</tbody>
</table>
<p>The middle row is the one to be sceptical about: the only first-party evidence in this article for raising slots is negative, on two cores. GPU serving divides work differently and was not measured here, so nothing above is a claim about what slots do on a GPU.</p>
<h2 id="what-to-log-when-a-session-stalls">What to log when a session stalls</h2>
<p>If a session dies while another is running, the question is not &ldquo;which agent is broken&rdquo; but &ldquo;what did this request wait for&rdquo;. Record these five items per request and the answer is usually visible in one table:</p>
<ul>
<li>Client-side wall time around the whole HTTP call — the number your timeout compares against.</li>
<li>HTTP status and the verbatim response body on failure; a 200 with a body is a different bug from a 503.</li>
<li><code>done_reason</code> and <code>eval_count</code> from the response — a <code>length</code> stop means your own <code>num_predict</code> cap was hit, not that the server truncated anything.</li>
<li>Start timestamps of every concurrent session, so queue waits can be attributed rather than assumed.</li>
<li>The server&rsquo;s own row (<code>ollama ps</code> CONTEXT and UNTIL, or the llama-server slot log) taken while the requests are in flight, not after they finish.</li>
</ul>
<p>None of that requires instrumenting the model. It does mean your agent&rsquo;s HTTP layer must log the call rather than only its own decision loop.</p>
<p>One thing this article does <strong>not</strong> establish: whether concurrency truncates tool-call arguments. No tool-calling payload was sent in this probe, no malformed tool call appeared in any cell, and no shortened context was observed — every response carried the same 316 characters and the same <code>eval_count</code>. If you are debugging a tool call that arrives with a cut-off JSON argument, the surfaces to inspect are the raw assistant message before your parser, the <code>done_reason</code>, and the per-request <code>num_ctx</code>, but this measurement gives you no evidence that a second session causes it.</p>
<h2 id="files-method-and-how-to-rerun-it">Files, method, and how to rerun it</h2>
<p>The probe is a single stdlib Python script: it releases N clients from one barrier, records wall time, status and the verbatim error body per request, and flushes partial results after every cell so a deadline cannot lose data. These files are first-party and published with the article:</p>
<table>
	<thead>
			<tr>
					<th>File</th>
					<th>What it is</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/probe.py">probe.py</a></td>
					<td>The driver: barrier-synchronised cells, per-request records, deadline and partial flush</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/probe-default.json">probe-default.json</a></td>
					<td>Raw records for config A (defaults, one slot), 2026-11-02T08:03:10Z to 08:03:49Z</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/probe-par4.json">probe-par4.json</a></td>
					<td>Raw records for config B (<code>OLLAMA_NUM_PARALLEL=4</code>), 08:04:54Z to 08:05:38Z</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/ps-check.json">ps-check.json</a></td>
					<td>The <code>/api/ps</code> response quoted above, with the server environment that produced it</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/method.md">method.md</a></td>
					<td>Host, versions, run order, and what was deliberately not measured</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/raw-records-manifest.json">raw-records-manifest.json</a></td>
					<td>SHA-256 and byte counts of the research-stage files, so a rerun can be compared byte for byte; <code>ps-check.json</code> was recorded afterwards in the write run and its hash is in the review record rather than this manifest</td>
			</tr>
	</tbody>
</table>
<p>To repeat the default-configuration cells on your own box, keep the script in a scratch directory of your choosing and give it a deadline in epoch seconds:</p>
<pre tabindex="0"><code>OLLAMA_PROBE_BASE=http://127.0.0.1:11434 OLLAMA_PROBE_LABEL=my-host \
OLLAMA_PROBE_OUT=&#34;$TMPDIR/probe-mine.json&#34; \
OLLAMA_PROBE_DEADLINE=$(( $(date +%s) + 150 )) \
python3 probe.py
</code></pre><p>For the four-slot cell, start a second server with your own model and flags, point <code>OLLAMA_PROBE_BASE</code> at it, and run <code>python3 probe.py n4</code>, which replays N=4 twice and then N=1 as the baseline. Compare the second client&rsquo;s wall time against the first: if the N=2 pair finishes at the same time, your server has two or more usable slots; if the second finishes at roughly twice the first, it has one.</p>
<h2 id="what-this-does-and-does-not-establish">What this does and does not establish</h2>
<p>Measured on one host: with the shipped default of one slot, four concurrent agent-shaped requests serialised and the last one took 4.0x the single-request wall time; setting <code>OLLAMA_NUM_PARALLEL=4</code> on the same two-core box made each request 4.4x to 8.0x slower than the default server&rsquo;s 3256.9 ms single-request row (3.8x to 7.0x against that server&rsquo;s own 3753.9 ms baseline) rather than faster; every request still returned HTTP 200.</p>
<p>Not measured, and not claimed: GPU serving of any kind, a real coding model generating hundreds of tokens, tool-calling payloads, throughput in tokens per second, memory high-water marks, and any comparison between Ollama and llama.cpp beyond their documented flag semantics. The absolute latencies here describe a two-vCPU host running a 397 MB model with a 64-token generation cap; your numbers will differ, and the shape — queueing at the default, contention when slots exceed cores — is what this article asks you to check rather than assume. For the wider comparison of engines and their parallel-request handling, see <a href="https://baeseokjae.github.io/posts/vllm-vs-ollama-production-2026/">vLLM vs Ollama in production</a> and <a href="https://baeseokjae.github.io/posts/local-ai-model-serving-frameworks-2026/">local model serving frameworks</a>; for the multi-agent side, <a href="https://baeseokjae.github.io/posts/local-ai-agents-guide-2026/">running local AI agents</a> covers what each session does with the model it gets.</p>
]]></content:encoded></item></channel></rss>