<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Cpu-Only Llm Inference on RockB</title><link>https://baeseokjae.github.io/tags/cpu-only-llm-inference/</link><description>Recent content in Cpu-Only Llm Inference on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 08 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/cpu-only-llm-inference/index.xml" rel="self" type="application/rss+xml"/><item><title>Agent Loops That Alternate Local Models: Measuring Ollama keep_alive and Cold-Start Stalls (2026)</title><link>https://baeseokjae.github.io/posts/ollama-keep-alive-model-switch-latency-2026/</link><pubDate>Sun, 08 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/ollama-keep-alive-model-switch-latency-2026/</guid><description>Twelve recorded responses on a two-core CPU-only Ollama host: keep_alive=0 reloaded on every switch, keep_alive=5m still evicted when OLLAMA_MAX_LOADED_MODELS=1, and keep_alive=-1 kept both models resident. Includes the load_duration/total_duration diagnostic and the recorded /api/ps residency states.</description><content:encoded><![CDATA[<p>Your agent finishes a coding turn, then calls a vision model to read a screenshot, then goes back to the coder. Each hand-off costs a pause the client reports as model latency: eight seconds, fifty seconds, eight seconds. The obvious diagnosis is that the model is slow, the quantization is wrong, or the machine needs more RAM. On the host measured here, none of those explained it. The model was being loaded from disk, again, on every switch, and the HTTP response said so in its own timing fields.</p>
<p>This article is the mechanism, a diagnostic you can run against a single response, and the measured residency behaviour of three <code>keep_alive</code> configurations on one CPU-only machine. Every number below comes from responses recorded on 2026-11-06 between 13:24:31Z and 13:33:36Z against Ollama 0.20.3, and the raw records ship with the post so you can recompute them instead of trusting a table.</p>
<h2 id="the-pause-that-looks-like-model-latency">The pause that looks like model latency</h2>
<p>An Ollama server holds a model in memory for a while after answering, then unloads it. The FAQ states the default window plainly: <a href="https://docs.ollama.com/faq.md">models are kept in memory for 5 minutes before being unloaded</a>. <code>keep_alive</code> on <code>/api/generate</code> and <code>/api/chat</code> overrides that window, and the same page lists four accepted forms: a duration string such as <code>10m</code>, a number of seconds, any negative number, which keeps the model loaded, and <code>0</code>, which unloads the model immediately after generating a response.</p>
<p>Two models in one loop interact with two separate limits. The first is how long each model stays resident. The second is <a href="https://docs.ollama.com/faq.md">OLLAMA_MAX_LOADED_MODELS, &ldquo;the maximum number of models that can be loaded concurrently provided they fit in available memory&rdquo;, defaulting to 3 for CPU inference</a>. When the number of models your loop touches exceeds the resident limit, or when the second model does not fit alongside the first, something has to be evicted — and an evicted model is a cold model on its next call, no matter what <code>keep_alive</code> says.</p>
<p>That is the trap: <code>keep_alive=5m</code> reads like &ldquo;keep my model warm&rdquo; and behaves like it only until a second model needs the slot.</p>
<h2 id="one-response-contains-both-phases">One response contains both phases</h2>
<p><code>/api/generate</code> reports its own timing breakdown in nanoseconds, and the field descriptions are specific: <a href="https://docs.ollama.com/api/generate.md"><code>total_duration</code> is &ldquo;time spent generating the response&rdquo;, <code>load_duration</code> is &ldquo;time spent loading the model&rdquo;, <code>prompt_eval_duration</code> covers prompt evaluation and <code>eval_duration</code> covers token generation</a>. The same page documents <a href="https://docs.ollama.com/api/generate.md"><code>keep_alive</code>: &ldquo;model keep-alive duration (for example 5m or 0 to unload immediately)&rdquo;</a>.</p>
<p>So one response is enough to classify a slow call. If <code>load_duration</code> is a large fraction of <code>total_duration</code>, you paid for a load. If it is near zero while the wall clock was still long, the time went somewhere else — a queued request behind another generation, which is a different problem with a different fix (covered in <a href="/posts/ollama-concurrent-requests-latency-2026/">the concurrency measurement on this site</a>).</p>
<p>The ratio is the diagnostic:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -s http://127.0.0.1:11434/api/generate <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  -d <span style="color:#e6db74">&#39;{&#34;model&#34;:&#34;qwen2.5:0.5b&#34;,&#34;prompt&#34;:&#34;Reply with the single word ok.&#34;,&#34;stream&#34;:false}&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| python3 -c <span style="color:#e6db74">&#39;import json,sys; r=json.load(sys.stdin); t=r[&#34;total_duration&#34;]; l=r[&#34;load_duration&#34;]; \
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">print(f&#34;load {l/1e9:.2f}s of {t/1e9:.2f}s = {l/t:.0%}&#34;)&#39;</span>
</span></span></code></pre></div><p><code>load 6.63s of 7.72s = 86%</code> is a load-dominated call. It means the model was not in memory when the request arrived, and no amount of prompt trimming or quantization tweaking will remove that second of wall clock.</p>
<p>To see <em>what</em> is resident rather than infer it, <a href="https://docs.ollama.com/api/ps.md"><code>/api/ps</code> lists the models currently running, with an expiry timestamp and the VRAM they occupy</a>. That endpoint is what turns a guess about eviction into an observation. The probe below calls it before and after every request.</p>
<h2 id="what-was-measured">What was measured</h2>
<p>Three configurations ran, out of four scripted, alternating a 0.5B text model and a 2.1B vision model on one endpoint, with <code>num_predict=64</code>, <code>temperature=0</code>, <code>seed=42</code> and the prompt <code>Reply with the single word ok.</code> Each configuration got a freshly started <code>ollama serve</code> on the same port — 11435 by default — stopped between phases, so that <code>OLLAMA_MAX_LOADED_MODELS</code> was re-applied and nothing stayed warm from a previous configuration.</p>
<table>
	<thead>
			<tr>
					<th>Config</th>
					<th>keep_alive</th>
					<th>OLLAMA_MAX_LOADED_MODELS</th>
					<th>Calls recorded</th>
					<th>Status</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A</td>
					<td><code>0</code></td>
					<td>3</td>
					<td>6</td>
					<td>complete</td>
			</tr>
			<tr>
					<td>B</td>
					<td><code>5m</code></td>
					<td>1</td>
					<td>4</td>
					<td>complete</td>
			</tr>
			<tr>
					<td>D</td>
					<td><code>-1</code></td>
					<td>3</td>
					<td>2</td>
					<td>stopped at the attempt deadline</td>
			</tr>
			<tr>
					<td>C</td>
					<td><code>5m</code></td>
					<td>3</td>
					<td>0</td>
					<td>not run</td>
			</tr>
	</tbody>
</table>
<p>The host is a two-core aarch64 machine with 11,927 MB total RAM and no swap, running Ollama 0.20.3 with CPU inference only — <code>/api/ps</code> reported <code>size_vram: 0</code> for every resident model, so nothing here describes GPU behaviour. Config C, the &ldquo;warm second call with room for two&rdquo; baseline, never executed: each <code>qwen3-vl:2b</code> load costs 29–63 seconds on this box, and the run&rsquo;s deadline arrived first. The gap is stated again under limits, because it is the difference between &ldquo;pinning helps&rdquo; and &ldquo;pinning removes the load stall&rdquo;, and only the first was measured.</p>
<h2 id="config-a-keep_alive0-reloads-on-every-single-call">Config A: keep_alive=0 reloads on every single call</h2>
<p>Six calls, alternating models, unload immediately after each response.</p>
<table>
	<thead>
			<tr>
					<th>#</th>
					<th>Model</th>
					<th>total_duration</th>
					<th>load_duration</th>
					<th>load share</th>
					<th>Resident after</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>0</td>
					<td>qwen2.5:0.5b</td>
					<td>7.72 s</td>
					<td>6.63 s</td>
					<td>86%</td>
					<td>qwen2.5:0.5b</td>
			</tr>
			<tr>
					<td>1</td>
					<td>qwen3-vl:2b</td>
					<td>72.09 s</td>
					<td>46.32 s</td>
					<td>64%</td>
					<td>qwen3-vl:2b</td>
			</tr>
			<tr>
					<td>2</td>
					<td>qwen2.5:0.5b</td>
					<td>9.18 s</td>
					<td>8.09 s</td>
					<td>88%</td>
					<td>qwen2.5:0.5b</td>
			</tr>
			<tr>
					<td>3</td>
					<td>qwen3-vl:2b</td>
					<td>49.87 s</td>
					<td>39.38 s</td>
					<td>79%</td>
					<td>qwen3-vl:2b</td>
			</tr>
			<tr>
					<td>4</td>
					<td>qwen2.5:0.5b</td>
					<td>9.93 s</td>
					<td>8.13 s</td>
					<td>82%</td>
					<td>qwen2.5:0.5b</td>
			</tr>
			<tr>
					<td>5</td>
					<td>qwen3-vl:2b</td>
					<td>65.89 s</td>
					<td>38.47 s</td>
					<td>58%</td>
					<td>qwen3-vl:2b</td>
			</tr>
	</tbody>
</table>
<p>Six of six calls carried a non-zero <code>load_duration</code>. The text model&rsquo;s load share stayed between 82% and 88% (median 86%); the vision model&rsquo;s between 58% and 79% (median 64%). Each <code>keep_alive=0</code> response unloaded its own model after answering, so that model&rsquo;s next turn started cold: the <code>/api/ps</code> capture taken immediately after a response listed the model that had just answered, while every <code>resident_before</code> at the following switch listed only the other model and never both. Six calls, six loads.</p>
<p>Note the absolute numbers rather than only the shares. Loading the 0.5B model cost 6.6–8.1 seconds here; loading the 2.1B vision model cost 38.5–46.3 seconds. The share varies because the denominator includes generation, and the vision model&rsquo;s generation phase swung from 8.3 to 24.4 seconds across three calls on an otherwise idle two-core machine. Comparing load shares <em>across</em> two models with different output lengths confuses the two phases; comparing a single model across configurations does not.</p>
<h2 id="config-b-keep_alive5m-does-not-survive-an-eviction-limit-of-1">Config B: keep_alive=5m does not survive an eviction limit of 1</h2>
<p>Same loop, <code>keep_alive=5m</code>, but <code>OLLAMA_MAX_LOADED_MODELS=1</code>.</p>
<table>
	<thead>
			<tr>
					<th>#</th>
					<th>Model</th>
					<th>total_duration</th>
					<th>load_duration</th>
					<th>load share</th>
					<th>Resident before → after</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>0</td>
					<td>qwen2.5:0.5b</td>
					<td>6.23 s</td>
					<td>5.12 s</td>
					<td>82%</td>
					<td>none → qwen2.5:0.5b</td>
			</tr>
			<tr>
					<td>1</td>
					<td>qwen3-vl:2b</td>
					<td>58.76 s</td>
					<td>29.07 s</td>
					<td>49%</td>
					<td>qwen2.5:0.5b → qwen3-vl:2b</td>
			</tr>
			<tr>
					<td>2</td>
					<td>qwen2.5:0.5b</td>
					<td>9.65 s</td>
					<td>4.88 s</td>
					<td>51%</td>
					<td>qwen3-vl:2b → qwen2.5:0.5b</td>
			</tr>
			<tr>
					<td>3</td>
					<td>qwen3-vl:2b</td>
					<td>58.11 s</td>
					<td>39.14 s</td>
					<td>67%</td>
					<td>qwen2.5:0.5b → qwen3-vl:2b</td>
			</tr>
	</tbody>
</table>
<p>Four of four calls loaded. Call 2 is the one that matters: the 0.5B model had been requested with <code>keep_alive=5m</code> two calls earlier, the 5-minute window had not elapsed, and it was still reloaded for 4.88 seconds — because call 1 had evicted it. Every <code>resident_before</code> list contains the other model and every <code>resident_after</code> list contains only the model that just answered. The keep-alive window was never the binding constraint; the resident slot count was.</p>
<p>One caveat on the default: the FAQ says the CPU default is 3, so <code>OLLAMA_MAX_LOADED_MODELS=1</code> is a deliberately tightened setting, used here to isolate the eviction rule with two small models on a small host. The rule it demonstrates is general — if the number of models your loop alternates exceeds the resident limit, <code>keep_alive</code> cannot save you — but the specific value 1 should not be read as a stock install. A recommendation to raise the limit to 2, as some guides make, only helps if both models actually fit; see the residency note below.</p>
<h2 id="config-d-keep_alive-1-keeps-both-models-resident">Config D: keep_alive=-1 keeps both models resident</h2>
<p>Two calls, then the deadline.</p>
<table>
	<thead>
			<tr>
					<th>#</th>
					<th>Model</th>
					<th>total_duration</th>
					<th>load_duration</th>
					<th>load share</th>
					<th>Resident after</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>0</td>
					<td>qwen2.5:0.5b</td>
					<td>15.49 s</td>
					<td>9.67 s</td>
					<td>62%</td>
					<td>qwen2.5:0.5b</td>
			</tr>
			<tr>
					<td>1</td>
					<td>qwen3-vl:2b</td>
					<td>114.64 s</td>
					<td>62.77 s</td>
					<td>55%</td>
					<td>qwen3-vl:2b, qwen2.5:0.5b</td>
			</tr>
	</tbody>
</table>
<p>Call 1 is the observation that configs A and B never produced: the probe&rsquo;s <code>resident_after</code> list for that call contains <strong>both</strong> models. The pinning request was honoured — <code>keep_alive=-1</code> kept the text model resident while a second model loaded alongside it. The record in <code>results.jsonl</code> carries the two names and <code>size_vram: 0</code> for each; the 2,947,025,696-byte (2.75 GiB) weight size is the <code>qwen3-vl:2b</code> entry in the <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/ps_A_ka0_max3.json">config-A</a> and <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/ps_B_ka5m_max1.json">config-B</a> <code>/api/ps</code> captures for the same digest. No post-phase <code>/api/ps</code> capture was saved for config D, so co-residency here is the per-call resident list, not a snapshot of the co-resident total.</p>
<p>What was <em>not</em> measured is the payoff. Call 2 would have been the first warm call in the whole matrix — the 0.5B model answering with no load phase — and it never ran. So the honest verdict is narrower than the usual advice: pinning both models changes the residency state as claimed, and this run does not contain a warm-call timing that proves the saved seconds. Anyone quoting a speed-up from <code>keep_alive=-1</code> on this evidence would be quoting something that was not recorded.</p>
<h2 id="the-decision-rule-this-supports">The decision rule this supports</h2>
<p>Given a working pair of models and one endpoint:</p>
<ol>
<li>Run the loop twice with the diagnostic above. If <code>load_duration/total_duration</code> is consistently below roughly 0.1 while the wall clock is still long, loading is not your problem — look at queueing or at the client&rsquo;s own retries, not at <code>keep_alive</code>.</li>
<li>If the share is high and <code>/api/ps</code> between calls shows only one model, you are being evicted. Decide which of the two causes applies by checking whether both models fit: compare each model&rsquo;s reported <code>size</code> in <code>/api/ps</code> against free memory, and remember that the <a href="/posts/local-llm-vram-kv-cache-budget-2026/">KV cache is allocated per model on top of the weights</a>.</li>
<li>If they fit with headroom, pin them with <code>keep_alive=-1</code> and confirm with <code>/api/ps</code> that both entries appear. That removes the reload; it does not by itself make generation faster.</li>
<li>If they do not fit, <code>keep_alive</code> is the wrong lever. Batch the loop by model — all coder calls, then all vision calls — so each switch is paid once instead of once per turn, or give the second model its own endpoint so each server holds one resident model and neither evicts the other.</li>
<li>If you are choosing the pair itself, prefer the smaller model on the constrained side: the load cost scales with the weights, and 39 seconds against 7 seconds on the same machine is the difference between a usable loop and an unusable one.</li>
</ol>
<p>The residency check is the part worth making routine. <code>keep_alive</code> tells the server what you want; <code>/api/ps</code> tells you what you got, and the two disagreed in every configuration measured here.</p>
<h2 id="reproduce-it">Reproduce it</h2>
<p>The runner and probe ship with this article and use only the Python standard library:</p>
<ul>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/run.sh"><code>run.sh</code></a> — runs configurations B, D and C in that order, starting a fresh server per configuration on the same port and stopping it between phases so <code>OLLAMA_MAX_LOADED_MODELS</code> is re-applied, and captures <code>/api/ps</code> on both sides of each phase.</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/probe.py"><code>probe.py</code></a> — alternates the two models, records <code>total_duration</code>, <code>load_duration</code>, <code>prompt_eval_duration</code>, <code>eval_duration</code>, <code>eval_count</code>, HTTP status, the resident list before and after each call, and the derived <code>load_share</code>.</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/results.jsonl"><code>results.jsonl</code></a> — configs B and D as recorded, one JSON object per call.</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/results_configA.jsonl"><code>results_configA.jsonl</code></a> — the six config-A records from the original separate invocation of the same probe; the file also holds one extra cold-open B call that the tables do not use. <code>run.sh</code> does not run config A, so use the invocation below for it.</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/ps_A_ka0_max3.json"><code>ps_A_ka0_max3.json</code></a> and <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/ps_B_ka5m_max1.json"><code>ps_B_ka5m_max1.json</code></a> — the <code>/api/ps</code> captures taken before and after those phases.</li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-keep-alive-model-switch-latency-2026/env.txt"><code>env.txt</code></a> — core count, <code>ollama --version</code>, <code>free -m</code> and swap state at run time.</li>
</ul>
<p>Run <code>./run.sh</code> from a directory containing both scripts; it writes to <code>results.jsonl</code> in that directory and starts servers on port 11435 by default. It covers configurations B, D and C only. For the six config-A records, start a server with <code>OLLAMA_MAX_LOADED_MODELS=3</code> — <code>run.sh</code>&rsquo;s <code>start_server 3</code> shows the environment it uses — and run <code>python3 probe.py http://127.0.0.1:11435 A_ka0_max3 0 6 results_configA.jsonl</code>: same label, <code>keep_alive=0</code> and six alternating calls as the shipped file. Every row of the tables above can be read straight out of those JSONL files, including the <code>resident_before</code> and <code>resident_after</code> arrays that carry the eviction and pinning evidence.</p>
<h2 id="limits">Limits</h2>
<ul>
<li>Everything here is CPU-only inference on a two-core host with no swap. GPU hosts load models in a fraction of the time and the load share will be much smaller; the diagnostic still applies, the numbers do not transfer.</li>
<li>Each configuration started from a fresh server, so config-to-config timings also include cold page cache. Within-configuration comparisons are the sound ones; do not read config D&rsquo;s 114.64-second vision call as evidence that pinning made generation slower.</li>
<li>Config C (5m with room for two models) never ran, and config D stopped after two calls. The matrix is therefore A complete, B complete, D partial — with the warm pinned call, which is the whole selling point of pinning, unmeasured. Config D also has no post-phase <code>/api/ps</code> capture, so its co-residency rests on the probe&rsquo;s <code>resident_after</code> list.</li>
<li>The two models differ in output length for the same <code>num_predict</code> (the vision model returned 64 tokens with no visible text, consistent with reasoning tokens), so cross-model load shares are not directly comparable. Absolute <code>load_duration</code> values are.</li>
<li>One model pair, one quantization each, one host. Other pairs will change the absolute load cost.</li>
<li>The run used local hardware and no paid provider; attributable monetary cost is unknown, not zero.</li>
</ul>
<p>The narrow takeaway: a multi-second pause on every model switch is usually a load, the response proves it in one field, and keep-alive only keeps a model resident while the server is allowed to keep it. If you never check <code>/api/ps</code>, you are tuning a window that an eviction already closed.</p>
]]></content:encoded></item><item><title>Two Agents, One Ollama: Measuring Why the Second Request Queues Instead of Speeding Up (2026)</title><link>https://baeseokjae.github.io/posts/ollama-concurrent-requests-latency-2026/</link><pubDate>Wed, 04 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/ollama-concurrent-requests-latency-2026/</guid><description>A CPU-only measurement of two simultaneous Ollama requests: the second one waits one full generation, and the response body proves it. Includes the one-line queue diagnostic, a 17-batch matrix for OLLAMA_NUM_PARALLEL 1/2/4, and why extra slots buy latency, not throughput.</description><content:encoded><![CDATA[<p>You start a second coding agent against the Ollama endpoint that already answers your first one. Nothing errors out. The second call simply sits there for two or three seconds, and when both finish your &ldquo;parallel&rdquo; run has taken longer than running the two prompts one after the other. The usual conclusion is that the model is slow, the quant is wrong, or the machine needs more RAM. On the host measured here, none of those were true: the second request was queued, and the HTTP response said so in its own timing fields.</p>
<p>This post is the mechanism, a diagnostic you can run against one response, and a measured verdict for CPU-only machines. Everything below comes from 17 recorded batches on a 2-core aarch64 box with Ollama 0.20.3 and <code>qwen2.5:0.5b</code>, run on 2026-11-04 between 00:33:20Z and 00:35:37Z. The raw records are published with the article so the numbers can be recomputed rather than trusted.</p>
<h2 id="what-ollama-does-with-the-second-request">What Ollama does with the second request</h2>
<p>Ollama&rsquo;s own documentation describes two levels of concurrency: several models resident at once, and, inside one model, several requests processed in parallel. The second level is what a subagent fan-out or a background embedding call runs into, and it is a per-model setting rather than a per-request one — <a href="https://docs.ollama.com/faq.md">the FAQ states that parallel request processing is configured when the model is loaded, and that a model is &ldquo;configured to allow parallel request processing&rdquo; only if memory allows</a>. When memory does not allow it, &ldquo;all new requests will be queued until the new model can be loaded&rdquo;, and &ldquo;queued requests will be processed in order&rdquo;.</p>
<p>The compiled defaults decide how often that path is taken. In the runtime&rsquo;s configuration source, <a href="https://raw.githubusercontent.com/ollama/ollama/main/envconfig/config.go">parallel request processing defaults to 1, the pending queue to 512, and the maximum number of resident models to 0, meaning automatic</a>. So a stock install handles exactly one generation per model at a time. The second request is not rejected and not run concurrently — it goes into the scheduler&rsquo;s pending channel, which is also where the &ldquo;server busy&rdquo; error comes from: <a href="https://raw.githubusercontent.com/ollama/ollama/main/server/sched.go">in the scheduler, a request whose runner is neither loaded nor reloadable is pushed to a pending channel, and only a full channel returns <code>ErrMaxQueue</code></a>. Up to 512 requests can therefore wait politely for a slot before anything fails.</p>
<p>That is the whole trap. A queued request returns HTTP 200 with a normal-looking answer. The only trace of the wait is time that belongs to no phase of the work.</p>
<h2 id="the-diagnostic-one-response-four-fields">The diagnostic: one response, four fields</h2>
<p><code>/api/generate</code> reports its own timing breakdown, in nanoseconds: <code>total_duration</code>, <code>load_duration</code>, <code>prompt_eval_duration</code>, and <code>eval_duration</code>, <a href="https://docs.ollama.com/api/generate.md">documented in the generate API reference</a>. For a request that was never queued, those three components add up to the total. For a request that waited, they do not, and the difference is the queue wait:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -s http://127.0.0.1:11434/api/generate -d <span style="color:#e6db74">&#39;{&#34;model&#34;:&#34;qwen2.5:0.5b&#34;,&#34;prompt&#34;:&#34;hello&#34;,&#34;stream&#34;:false}&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| python3 -c <span style="color:#e6db74">&#39;import json,sys; r=json.load(sys.stdin); t=r[&#34;total_duration&#34;]; \
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">q=t-(r[&#34;load_duration&#34;]+r[&#34;prompt_eval_duration&#34;]+r[&#34;eval_duration&#34;]); \
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">print(&#34;total_s=%.2f queue_s=%.2f queue_fraction=%.2f&#34; % (t/1e9, q/1e9, q/t))&#39;</span>
</span></span></code></pre></div><p>Two details make the difference between a working diagnostic and a misleading one. First, <code>load_duration</code> is not zero even when the model is already resident: on this host it was 0.20-0.30 s for a warm 0.5B model and 0.49-1.04 s on the 4-slot instance. Load plus prompt-eval overhead was 0.24-0.34 s on the default and 2-slot instances, 0.62-0.92 s in the warm 4-slot batch and 0.96-1.32 s in the second cold 4-slot batch, so subtracting only <code>eval_duration</code> from <code>total_duration</code> inflates apparent waiting by roughly a quarter to a third of a second at one or two slots and by up to about 1.3 s at four. (The first cold 4-slot batch is worse still — a prompt-cache miss put 4.64 s of prompt evaluation on three of its four requests — and is excluded from the reported table for that reason.) Second, the fraction matters more than the absolute value, because a long generation dilutes the same queue wait into a smaller share.</p>
<p>On the measured host the split was unambiguous. Solo requests in the default configuration left a residual of 0.05 s, about 2% of a 2.3-2.6 s total. In the same configuration, the second of two simultaneous requests showed a residual of 2.16, 2.27, 2.17 and 2.41 s across four batches — 40-51% of its total time — while the first request in that batch still finished with a 0.05 s residual.</p>
<h2 id="the-measurement-17-batches-three-configurations">The measurement: 17 batches, three configurations</h2>
<p>The runtime was Ollama 0.20.3, <a href="https://github.com/ollama/ollama/releases/tag/v0.20.3">published 2026-04-07 according to the GitHub release metadata</a>. The probe sends <code>POST /api/generate</code> with <code>stream: false</code>, <code>temperature: 0</code>, <code>seed: 42</code> and <code>num_predict: 64</code>, one fixed prompt, and every request produced <code>eval_count: 46</code>. Concurrent requests synchronise on a barrier so they leave together. Three server configurations were used: A is the pre-existing default server on port 11434; B is a dedicated instance on port 11500 with <code>OLLAMA_NUM_PARALLEL=2</code>; C is a dedicated instance on port 11501 with <code>OLLAMA_NUM_PARALLEL=4</code>. Both dedicated instances used the system model directory and <code>OLLAMA_MAX_QUEUE=1</code>, which is worth noting: with free slots available, nothing was rejected, so the single-queue-slot limit never fired.</p>
<table>
	<thead>
			<tr>
					<th>Config</th>
					<th>Concurrency</th>
					<th>Warm batches</th>
					<th>Per-request total (s)</th>
					<th>Per-request eval (s)</th>
					<th>Queue residual (s)</th>
					<th>Aggregate tok/s</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A default</td>
					<td>1</td>
					<td>3</td>
					<td>2.30-2.59</td>
					<td>2.00-2.27</td>
					<td>0.05</td>
					<td>17.7-20.0</td>
			</tr>
			<tr>
					<td>A default</td>
					<td>2</td>
					<td>4</td>
					<td>2.40-2.62 (first) / 4.49-5.69 (second)</td>
					<td>2.02-2.26 (first) / 1.96-3.13 (second)</td>
					<td>0.05 (first) / 2.16-2.41 (second)</td>
					<td>16.2-20.4</td>
			</tr>
			<tr>
					<td>B <code>NUM_PARALLEL=2</code></td>
					<td>1</td>
					<td>2</td>
					<td>2.29-2.48</td>
					<td>1.99-2.17</td>
					<td>0.05</td>
					<td>18.5-20.1</td>
			</tr>
			<tr>
					<td>B <code>NUM_PARALLEL=2</code></td>
					<td>2</td>
					<td>3</td>
					<td>4.08-4.60</td>
					<td>3.67-4.13</td>
					<td>0.09-0.17</td>
					<td>20.0-22.0</td>
			</tr>
			<tr>
					<td>C <code>NUM_PARALLEL=4</code></td>
					<td>4</td>
					<td>1</td>
					<td>8.70-8.94</td>
					<td>7.81-7.90</td>
					<td>0.12-0.34</td>
					<td>20.6</td>
			</tr>
	</tbody>
</table>
<p>Four batches were excluded as cold and are marked <code>cold=true</code> in the per-batch summary: the very first solo batch (4.93 s, of which 4.61 s was <code>eval_duration</code> at 10.0 tok/s; <code>load_duration</code> was 0.22 s, no higher than the warm solo rows), one 2-slot batch on config B where a cold prompt cache pushed <code>prompt_eval_duration</code> to 1.58 s, and the two earliest 4-slot batches (19.2 s and 16.4 s): the first carries the 4.64 s prompt-eval miss on three requests described above, and the second spent 14.8 s of its 16.4 s in <code>eval_duration</code> at 3.1 tok/s with <code>load_duration</code> 0.73-1.04 s. They ran at 9.6 and 11.2 aggregate tok/s against 20.6 in the warm batch. Cold exclusion is stated because it changes the numbers: the excluded 4-slot batches are twice as slow as the one warm batch reported.</p>
<h2 id="what-the-numbers-say">What the numbers say</h2>
<p>Under the default configuration, the second request waited about as long as one full generation. Its total was 4.49-5.69 s against 2.30-2.59 s for a solo request, and the wait it added — 2.16-2.41 s — is almost exactly the eval time of the batch-mate holding the slot (2.02-2.26 s). Per-request token rates mostly held: in three of the four batches the queued request generated at 22.0-23.4 tok/s against 20.3-22.8 for that batch-mate. The exception is rep 1, where the queued request ran at 14.7 tok/s (eval_duration 3.13 s) against 21.7 for the first, even though the first request&rsquo;s own latency was unchanged at 2.46 s. That is still a queue, not CPU contention: if the two generations had been sharing two cores for their whole duration, the first request&rsquo;s latency would have moved, and it did not.</p>
<p>Setting <code>OLLAMA_NUM_PARALLEL=2</code> removed the wait and replaced it with slower generations. Both requests came back in 4.08-4.60 s with <code>eval_duration</code> 3.67-4.13 s, against 2.00-2.27 s solo, and the queue residual fell to 0.09-0.17 s. Per-request throughput halved from roughly 20-23 tok/s to 11.1-12.5 tok/s. On two cores, two concurrent generations each run at about half speed, which is what a shared CPU should do.</p>
<p>Aggregate throughput barely moved. Warm batches gave a median of 19.1 tok/s solo on config A, 19.8 tok/s for config A at concurrency 2, 22.0 tok/s for config B at concurrency 2, and 20.6 tok/s for the single warm config C batch at concurrency 4. The best-case ratio that could be argued, 22.0 / 19.8 = 1.11x, sits inside a run-to-run spread of 16.2-22.0 tok/s, and far below the 1.5x improvement that would have made extra slots worth recommending on this hardware. Doubling or quadrupling the slot count changed latency per request by roughly 1.8-3.9x and throughput by nothing measurable.</p>
<p>The practical consequence is the counter-intuitive one: for two prompts on a CPU-only host, fanning out to two agents takes as long as — or slightly longer than — calling them one after the other, because the queue serialises them anyway when only one slot exists, and splits the cores when two exist. Parallelism on a CPU host is not free capacity; it is a different shape of the same capacity.</p>
<h2 id="what-it-costs-in-memory">What it costs in memory</h2>
<p>Slots are not free. The FAQ states that parallel request processing increases the context size by the number of parallel requests, and the server log confirms the arithmetic: the load line reports <code>Parallel:2 ... KvSize:8192</code> for the 2-slot instance and <code>Parallel:4 ... KvSize:16384</code> for the 4-slot instance, against a 4,096-token default context. <code>KvSize</code> is the total KV allocation across slots, not a per-slot value.</p>
<p><code>/api/ps</code> shows the result in reported model memory: <code>size</code> 442,191,360 bytes for the default server and 492,523,008 bytes with two slots — 50,331,648 bytes, or 48.0 MiB, more, a rise of 11.4%. One detail that catches readers out: <code>context_length</code> stayed at 4096 in both <code>/api/ps</code> captures, so a per-request context that halved is not what the API reports. The KV allocation doubles — <code>KvSize:8192</code> for two slots against the 4,096-token default context — while the total reported <code>size</code> rises by 11.4%. The reported per-slot context does not change. If you are sizing a second slot, do the arithmetic from context times slot count, as in <a href="/posts/local-llm-vram-kv-cache-budget-2026/">the KV-cache budget walkthrough for a local coding model</a>, rather than trusting the single <code>context_length</code> field.</p>
<p>Every one of these captures reports <code>size_vram: 0</code>. This host has no GPU, and nothing here should be read as a statement about GPU behaviour, where a second slot may overlap compute across SMs and actually raise throughput.</p>
<h2 id="what-to-do-instead">What to do instead</h2>
<p>Serialise, then batch. If the second call is an embedding step or a summarisation pass that does not need to interleave with the first, run it after the agent&rsquo;s tool call rather than beside it — you lose nothing on a CPU host, because the scheduler was serialising it anyway whenever the slot was busy.</p>
<p>If you do want concurrent slots, enable them deliberately and budget the memory, then verify with the residual diagnostic that requests are actually overlapping rather than queueing. A residual near zero with doubled <code>eval_duration</code> means the slots are working as intended; a residual equal to one solo generation means you are still at one slot, whatever the environment variable says, which is the usual symptom of setting it on the client instead of the server process.</p>
<p>Move to a GPU when throughput, not latency, is the goal. The 1.11x aggregate result here is a statement about two CPU cores, not about <code>OLLAMA_NUM_PARALLEL</code> in general — and even on a GPU, <a href="/posts/vllm-vs-ollama-production-2026/">the queue-versus-batching distinction is what decides whether a serving stack helps</a>.</p>
<h2 id="what-this-is-not">What this is not</h2>
<p>It is not a GPU benchmark, a model comparison, or a vLLM comparison. One 0.5B Q4_K_M model, 46 generated tokens, two cores: long prompts and prefill-bound workloads were not measured, and <code>OLLAMA_MAX_LOADED_MODELS</code> behaviour — a second <em>model</em> being swapped in — was not tested at all. The 503 path behind <code>OLLAMA_MAX_QUEUE</code> is documented in the source linked above but was not reproduced live, because every request in this matrix eventually received a slot. The queue-versus-compute split is derived from Ollama&rsquo;s own duration fields; no external profiler or debug-level scheduler trace corroborates it.</p>
<p>If your local agent is underperforming, scheduler queueing is one of several causes. Context truncation is the other common one on the same endpoint, covered in <a href="/posts/ollama-silent-context-truncation-2026/">why a local model silently answers with less context than you sent</a>. If you leave Ollama for llama.cpp&rsquo;s server, the equivalent knob is <code>-np</code>, and its splitting behaviour differs from Ollama&rsquo;s scheduler — see <a href="/posts/llama-server-context-overflow-diagnosis-2026/">diagnosing llama-server context overflow</a>. For the environment-variable reference and the OpenAI-compatible endpoint, the <a href="/posts/ollama-api-local-llm-integration-developer-guide-2026/">Ollama API integration guide</a> lists the same three settings; the tuning block in <a href="/posts/aider-ollama-local-coding-2026/">the Aider + Ollama setup</a> is the one this post measures. And if you are choosing a model for a CPU-only host in the first place, <a href="/posts/best-ollama-models-coding-2026/">the coding-model shortlist for CPU inference</a> is the better starting point than any concurrency tuning.</p>
<h2 id="reproduce-it">Reproduce it</h2>
<p>The measurement is reproducible on a 2-core CPU box in under three minutes: 17 batches took 137 s including model load. The files below are the exact inputs and outputs of the run described above, published so every figure in this article can be recomputed from the raw response bodies.</p>
<table>
	<thead>
			<tr>
					<th>File</th>
					<th>What it is</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/probe.py">probe.py</a></td>
					<td>stdlib-only probe: barrier-synchronised concurrent <code>POST /api/generate</code>, records status and all timing fields</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/run.sh">run.sh</a></td>
					<td>the exact commands: host capture, <code>/api/ps</code> before and after, the three server configurations, per-cell warm-up</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/results.jsonl">results.jsonl</a></td>
					<td>17 batches, one JSON object per batch, every request&rsquo;s raw nanosecond fields</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/summary.csv">summary.csv</a></td>
					<td>per-batch derived rows: aggregate tok/s, per-request medians, queue residual, cold flag</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/method.md">method.md</a></td>
					<td>request shape, derived formulas, what was deliberately not measured; the authoritative <code>cold</code> flag is in <code>summary.csv</code></td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/host.txt">host.txt</a></td>
					<td><code>ollama --version</code>, <code>nproc</code>, <code>free -m</code>, kernel and architecture</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/ps_A.txt">ps_A.txt</a> / <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/ps_B.txt">ps_B.txt</a></td>
					<td><code>/api/ps</code> after each configuration: <code>size</code>, <code>size_vram</code>, <code>context_length</code></td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/server_b_lines.txt">server_b_lines.txt</a> / <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-concurrent-requests-latency-2026/server_c_lines.txt">server_c_lines.txt</a></td>
					<td>the server config and load lines showing <code>OLLAMA_NUM_PARALLEL</code>, <code>Ollama version 0.20.3</code>, <code>Parallel</code> and <code>KvSize</code></td>
			</tr>
	</tbody>
</table>
<p>To rerun on your own hardware, start a second server on a spare port against the same model directory rather than restarting the one your editor is using, point <code>probe.py</code> at it, and compare the residual fraction for concurrency 1 and 2. If the second request&rsquo;s residual is near zero and its <code>eval_duration</code> roughly doubles, your slots are working; if the residual is about one solo generation, they are not. Note that <code>summary.csv</code> reports the per-batch median across the requests in that batch, so a two-request batch with one queued request shows a residual median of roughly half the queued request&rsquo;s own wait — read the per-request fields in <code>results.jsonl</code> when you want the wait of the request that actually queued.</p>
]]></content:encoded></item></channel></rss>