<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Coding Agent Latency on RockB</title><link>https://baeseokjae.github.io/tags/coding-agent-latency/</link><description>Recent content in Coding Agent Latency on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 08 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/coding-agent-latency/index.xml" rel="self" type="application/rss+xml"/><item><title>Turn-Time Budget for a Local Coding Agent: what a measured tokens/second figure buys you in tool-call turns</title><link>https://baeseokjae.github.io/posts/local-agent-turn-time-budget-2026/</link><pubDate>Sun, 08 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/local-agent-turn-time-budget-2026/</guid><description>One turn is prefill over the prompt plus decode over the output plus tool overhead. Using llama-bench&amp;#39;s printed CPU rows for 7B Q4_0 at one thread, a 512-token prompt and 256-token output cost 82.98 s of prefill and 63.21 s of decode: 148.19 s per turn, 1.23 h for thirty turns, and a decode rate of at least 0.497 tok/s to fit a 600 s client timeout. A stdlib harness measures your own host and inverts the equation.</description><content:encoded><![CDATA[<p>Your local model answers a chat question in a couple of seconds and feels fine. You wire it into an agent that needs thirty tool calls to finish one refactor, and around the eighth request the client gives up with a read timeout, or the patch comes back cut in half because generation stopped mid-function. Nothing in the server log looks like an error. Restart the session with a shorter conversation and it completes.</p>
<p>The question that actually decides the fix is arithmetic: given the rate this machine produces, how long is one turn, how long is thirty turns, and what decode rate does the client timeout require? This article answers that with a formula, a worked example computed from llama.cpp&rsquo;s own printed benchmark rows, and a small harness you run against your own endpoint. It publishes no tokens-per-second table of its own, and the reason is at the end: the one place this project could run the measurement was too noisy to produce a number worth printing.</p>
<h2 id="one-tokenssecond-figure-is-the-wrong-unit">One tokens/second figure is the wrong unit</h2>
<p>A turn is not one number. It is two different operations with different costs. llama.cpp&rsquo;s benchmark tool states the split in its own flag list: prompt processing (<code>pp</code>) is &ldquo;processing a prompt in batches&rdquo;, and text generation (<code>tg</code>) is &ldquo;generating a sequence of tokens&rdquo;, measured as separate tests (<a href="https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/llama-bench/README.md">llama.cpp llama-bench README</a>). Prefill scales with how many tokens you send; decode scales with how many tokens you ask for back. They do not move together: in the CPU rows quoted further down, one thread gives 6.17 tok/s of prefill against 4.05 tok/s of decode, while 32 threads gives 59.00 against 16.41, so the ratio between the two is a property of the host and the batch size rather than of the model. This is also why a model-selection page that declines to rank models by speed is right to refuse (<a href="/posts/best-ollama-models-coding-2026/">best Ollama models for coding</a>): a rate without a host, a prompt length and an output length is not a property of the model.</p>
<p>Ollama exposes the same split per request. Its generate endpoint returns <code>prompt_eval_count</code> (&ldquo;Number of input tokens in the prompt&rdquo;), <code>prompt_eval_cached_count</code> (&ldquo;Number of prompt tokens read from the cache&rdquo;), <code>prompt_eval_duration</code> (&ldquo;Time spent evaluating uncached prompt tokens in nanoseconds&rdquo;), <code>eval_count</code> (&ldquo;Number of output tokens generated in the response&rdquo;) and <code>eval_duration</code> (&ldquo;Time spent generating tokens in nanoseconds&rdquo;) (<a href="https://docs.ollama.com/api/generate.md">Ollama API reference, generate</a>). That is everything needed to derive two rates for one real request instead of trusting a headline figure someone measured on a different machine:</p>
<table>
	<thead>
			<tr>
					<th>Field in the response</th>
					<th>What it counts</th>
					<th>Rate you derive</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>prompt_eval_count</code></td>
					<td>total input tokens: cached plus newly evaluated</td>
					<td>—</td>
			</tr>
			<tr>
					<td><code>prompt_eval_cached_count</code></td>
					<td>the cached subset of those tokens (optional in the response)</td>
					<td>—</td>
			</tr>
			<tr>
					<td><code>prompt_eval_duration</code></td>
					<td>nanoseconds spent evaluating the uncached remainder</td>
					<td>prefill tok/s = (count − cached) ÷ (duration ÷ 1e9)</td>
			</tr>
			<tr>
					<td><code>eval_count</code></td>
					<td>tokens actually generated</td>
					<td>—</td>
			</tr>
			<tr>
					<td><code>eval_duration</code></td>
					<td>nanoseconds spent generating them</td>
					<td>decode tok/s = count ÷ (duration ÷ 1e9)</td>
			</tr>
	</tbody>
</table>
<p>Three caution notes are built into those definitions. First, <code>prompt_eval_count</code> is the <em>whole</em> prompt, cached tokens included, while <code>prompt_eval_duration</code> times only the ones the server actually evaluated — so dividing the count by the duration overstates prefill on any prefix-cache hit. Ollama&rsquo;s own metrics printer subtracts the cached count before dividing (<code>uncached := max(0, m.PromptEvalCount-cached)</code> then <code>uncached / duration</code> in <a href="https://raw.githubusercontent.com/ollama/ollama/main/api/types.go"><code>api/types.go</code></a>), and the server builder shows why the two numbers disagree: the count is <code>CacheN + PromptN</code> while the duration is <code>PromptMS</code> (<a href="https://raw.githubusercontent.com/ollama/ollama/main/llm/llama_server.go"><code>llm/llama_server.go</code></a>). Second, <code>prompt_eval_cached_count</code> is optional, so a missing value has to be treated as 0 — the no-cache case — which is what the harness below does. Third, <code>prompt_eval_duration</code> excludes network and client time. All three are reasons to derive the prefill numerator as a subtraction and to add overhead explicitly rather than hope the server accounted for it.</p>
<h2 id="the-turn-equation">The turn equation</h2>
<p>For a single stream, with no queueing:</p>
<pre tabindex="0"><code>turn_s = prompt_tokens / prefill_tps + output_tokens / decode_tps + tool_overhead_s
</code></pre><p>Three terms, three different levers. <code>tool_overhead_s</code> is everything between the model finishing and the next request starting: tool execution, file reads, transport, retry backoff. On a local host it is often the smallest term, and that is exactly why it misleads people — it is the term that hides the retry multiplier later.</p>
<p>Worked example. The inputs are llama-bench&rsquo;s own printed sample rows for <code>llama 7B mostly Q4_0</code>, backend <code>CPU</code>, 1 thread: <code>pp 64</code> at 6.17 ± 0.07 tok/s and <code>tg 16</code> at 4.05 ± 0.02 tok/s (<a href="https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/llama-bench/README.md">llama-bench README table</a>). They are upstream sample output, not a measurement made for this article and not a description of any reader&rsquo;s machine. The scenario is declared: a 512-token prompt, a 256-token output, 2 s of tool overhead per turn.</p>
<table>
	<thead>
			<tr>
					<th>Term</th>
					<th>Arithmetic</th>
					<th>Seconds</th>
					<th>Share of turn</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Prefill</td>
					<td>512 ÷ 6.17</td>
					<td>82.98</td>
					<td>56%</td>
			</tr>
			<tr>
					<td>Decode</td>
					<td>256 ÷ 4.05</td>
					<td>63.21</td>
					<td>43%</td>
			</tr>
			<tr>
					<td>Tool overhead</td>
					<td>declared</td>
					<td>2.00</td>
					<td>1%</td>
			</tr>
			<tr>
					<td><strong>Turn</strong></td>
					<td>sum</td>
					<td><strong>148.19</strong></td>
					<td>100%</td>
			</tr>
	</tbody>
</table>
<p>Prefill is more than half of that turn. That is the finding most tokens/second headlines erase: a page that quotes one number cannot tell you whether the dominant cost is reading your prompt or writing the answer.</p>
<h2 id="what-thirty-turns-cost">What thirty turns cost</h2>
<p>Multiply, assuming a prompt that does not grow:</p>
<table>
	<thead>
			<tr>
					<th>Turns</th>
					<th>Task time</th>
					<th>Hours</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>10</td>
					<td>1,481.9 s</td>
					<td>0.41</td>
			</tr>
			<tr>
					<td>30</td>
					<td>4,445.8 s</td>
					<td>1.23</td>
			</tr>
			<tr>
					<td>50</td>
					<td>7,409.6 s</td>
					<td>2.06</td>
			</tr>
	</tbody>
</table>
<p>The assumption is the problem. An agent loop appends every tool result to the conversation, so the prompt grows each turn and the task time grows faster than the turn count suggests. Holding decode and overhead fixed at the values above, a 1,024-token prompt costs 231.17 s per turn and a 4,096-token prompt costs 729.07 s per turn. All of the extra 580.9 s went to prefill: 4,096 ÷ 6.17 = 663.86 s of it.</p>
<h2 id="inverting-the-equation">Inverting the equation</h2>
<p>The useful form runs backwards. If the client&rsquo;s per-request budget is <code>T</code>, the prompt costs <code>P</code> seconds to prefill and overhead is <code>H</code>, the slowest decode rate that still fits one turn is:</p>
<pre tabindex="0"><code>min_decode_tps = output_tokens / (T - P - H)
</code></pre><p>At the same inputs: 256 ÷ (600 − 82.98 − 2) = <strong>0.497 tok/s</strong>. The 600 s figure is a real default, not an illustration — openai-python ships <code>DEFAULT_TIMEOUT = httpx2.Timeout(timeout=600, connect=5.0)</code> and <code>DEFAULT_MAX_RETRIES = 2</code> (<a href="https://raw.githubusercontent.com/openai/openai-python/main/src/openai/_constants.py">openai-python <code>_constants.py</code></a>). Two retries mean one slow turn can occupy up to three times its own latency: 3 × 148.19 = 444.6 s for a turn that was already inside budget.</p>
<p>There is a harder limit above the decode floor. Prefill alone consumes the whole 600 s budget at 6.17 × 600 = 3,702 prompt tokens. Past that point the minimum decode rate stops existing: at a 4,096-token prompt the turn is 729.07 s, and no amount of decode speed saves it. The same arithmetic at 8,192 prompt tokens gives 1,327.7 s of prefill — before the first output token is generated. This is the failure mode that looks like a model problem and is actually a context problem, and it is why <a href="/posts/local-llm-vram-kv-cache-sizing-2026/">deciding whether the model fits your card</a> is only step one: fitting is not the same as finishing a turn.</p>
<h2 id="where-the-time-goes-and-the-order-to-fix-it">Where the time goes, and the order to fix it</h2>
<p>Given a measured prefill and decode rate, the terms rank predictably:</p>
<ol>
<li><strong>Prompt length.</strong> It multiplies prefill, and prefill is the largest term in the worked example. Sending whole files instead of retrieved spans is the usual cause.</li>
<li><strong>Output tokens.</strong> It multiplies decode. A capped <code>num_predict</code> turns a 63.21 s decode term into 31.60 s at 128 tokens, and the turn into 116.59 s.</li>
<li><strong>Retries.</strong> Two retries turn one over-budget turn into three, which is how a task that should take 1.23 h becomes a session that looks hung.</li>
<li><strong>The timeout itself.</strong> Raising it hides the problem until the next context growth, and it is the only lever that costs nothing to describe and costs the most to rely on.</li>
</ol>
<p>If prefill alone exceeds the request budget, shortening the output is the wrong first fix. The turn stays over budget no matter how few tokens you ask for. Cut context, or move the model off CPU, or raise <code>T</code> — in that order of cheapness.</p>
<h2 id="measure-your-own-host">Measure your own host</h2>
<p>The harness is one stdlib file, <code>turn_budget.py</code>, with a three-prompt fixture and a test file. It does not download models and does not set server variables; it talks to an endpoint you already have running:</p>
<pre tabindex="0"><code>python3 turn_budget.py measure --model &lt;your-tag&gt; --jsonl &#34;$TMPDIR/runs.jsonl&#34; --repeats 3
python3 turn_budget.py summarise --jsonl &#34;$TMPDIR/runs.jsonl&#34;
python3 turn_budget.py compute --jsonl &#34;$TMPDIR/runs.jsonl&#34; \
    --turns 30 --output-tokens 256 --tool-overhead 2 --timeout 600
</code></pre><p>Three design rules matter more than the code. First, rates are derived from the per-request counters above — prefill from <code>prompt_eval_count</code> minus <code>prompt_eval_cached_count</code>, over the uncached-only <code>prompt_eval_duration</code> — never from a headline figure or from the raw count. Second, every record is written and <code>fsync</code>ed before the next request starts — a run killed at its cap still leaves usable evidence, which is exactly how an earlier attempt of this project lost seven of nine records. Third, the summary applies a spread gate: repeats that disagree by more than 10% are reported as <code>rejected_noisy</code> and the medians are not printed as a baseline. <code>compute</code> refuses a rejected run file for the same reason.</p>
<p>Run the arithmetic without a server, or reproduce the numbers above with no endpoint at all:</p>
<pre tabindex="0"><code>python3 turn_budget.py example
python3 turn_budget.py compute --prefill-tps 6.17 --decode-tps 4.05 \
    --prompt-tokens 512 --output-tokens 256 --tool-overhead 2 --turns 30 --timeout 600
</code></pre><p>The 24 unit tests pin every figure in this article: 82.98 s of prefill, 63.21 s of decode, 148.19 s per turn, 4,445.8 s over thirty turns, the 0.497 tok/s floor, 3,702 prompt tokens at the timeout, and the round trip where feeding the minimum decode rate back into the turn equation returns exactly 600 s. They also pin the cache accounting — a 64-token prompt with 32 tokens served from the prefix cache has to produce the same rate as a 32-token uncached prompt, a missing <code>prompt_eval_cached_count</code> has to be treated as 0, and a fully cached prompt has to be refused instead of reported. They run with <code>python3 test_turn_budget.py</code>, need no network, and use a synthetic response to exercise the JSONL path including a failed request.</p>
<h2 id="the-negative-result-behind-the-missing-baseline">The negative result behind the missing baseline</h2>
<p>This project tried to publish a measured baseline and could not. The attempt ran on a shared 2-vCPU, 11 GB sandbox against <code>qwen3-vl:2b</code> (<code>Q4_K_M</code>, digest <code>0635d9d857d497aeadba3d7d27485746c50554446f9f6ec01ef39788221adbe</code>) with three fixtures and three repeats inside a declared 110 s run budget. Two of nine records completed; both were recovered from the script&rsquo;s stdout after its own 130 s kill cap ended the run — the method note documents the two limits separately. Decode came out at 1.615 and 3.165 tok/s — a 96% spread — and prefill at 3.781 and 228.859 tok/s, the slower one including cold model load. That fails the same 10% gate the shipped script enforces, so those figures are recorded as a rejected run and appear nowhere in this article as a baseline or a hardware comparison.</p>
<p>The 228.859 tok/s figure is also cache-affected and is not a prefill measurement. It was computed the old way, by dividing <code>prompt_eval_count</code> — which counts cached tokens as well as newly evaluated ones — by the uncached-only <code>prompt_eval_duration</code>; both records show a 32-token <code>prompt_eval_count</code> for the same fixture, and neither captured <code>prompt_eval_cached_count</code>, so how much of that 32 was served from the prefix cache cannot be recovered. The corrected formula can only put the true uncached rate at or below 228.859 tok/s, and how far below depends on a cached count nobody recorded — which is the whole reason the run is published as rejected rather than quoted.</p>
<p>The useful part of that failure is the shape of it: the same fixture and token counts took 79.8 s and 16.9 s on the same host minutes apart. A single run on a machine with co-resident services is not a measurement, and the gate exists because the obvious alternative — average the two and print 2.39 tok/s — would have produced a confident, wrong number.</p>
<h2 id="what-this-cannot-tell-you">What this cannot tell you</h2>
<p>The worked example rests on CPU rows printed in a README with no CPU model named, at a <code>pp</code> batch of 64 tokens and a 16-token <code>tg</code> run, so it supports &ldquo;CPU prefill can dominate a long-context turn&rdquo; and nothing more precise. No GPU was measured for this article and no cross-hardware table is offered; GPU rates, batch sizes and backend flags change both terms by orders of magnitude, and a comparison built from mixed sources would be worse than no comparison. A different tokenizer changes <code>prompt_tokens</code> for the same text, so a reader who copies the 512-token scenario with different content will get a different prefill term — derive token counts from the response, not from word counts.</p>
<h2 id="reader-deliverables">Reader deliverables</h2>
<p>The harness, fixture, tests and the worked-example output are published files in this site&rsquo;s repository, not a private directory:</p>
<ul>
<li>Measurement harness — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/turn_budget.py">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/turn_budget.py</a></li>
<li>Three-prompt fixture — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/fixtures/prompts.json">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/fixtures/prompts.json</a></li>
<li>Arithmetic tests — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/test_turn_budget.py">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/test_turn_budget.py</a></li>
<li>Test run record (command, exit code, 24 tests, no network) — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/test-run.json">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/test-run.json</a></li>
<li>Worked-example output, as printed by the script — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/worked-example.json">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/worked-example.json</a></li>
<li>Rejected run records from the failed baseline attempt — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/runs.jsonl">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/runs.jsonl</a></li>
<li>Method and host notes for that attempt — <a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/method.md">https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-agent-turn-time-budget-2026/method.md</a></li>
</ul>
<p>None of these files contain credentials, hostnames or personal data; they are text the script reads and writes. The fixture is invented test prose, and the run records carry the model digest, request options and derived rates only.</p>
<h2 id="if-you-only-remember-one-thing">If you only remember one thing</h2>
<p>The number to measure twice is not tokens per second. It is prompt tokens in, output tokens out, and a timeout — because <code>turn_s = prompt_tokens / prefill_tps + output_tokens / decode_tps + overhead</code> has an inverse, and the inverse is the only version that tells you whether your agent will finish. If your measured prefill puts the prompt alone past the client&rsquo;s budget, the model&rsquo;s decode speed is irrelevant; a second concurrent session cannot fix that either, as the <a href="/posts/local-llm-concurrent-agent-sessions-2026/">single-stream concurrency measurements</a> show, and neither can <a href="/posts/local-llm-rig-cost-payback-2026/">the payback calculation</a> that consumes this page&rsquo;s rate as an input.</p>
<h2 id="sources">Sources</h2>
<ul>
<li>Ollama API reference, generate endpoint (<code>prompt_eval_count</code>, <code>prompt_eval_cached_count</code>, <code>prompt_eval_duration</code>, <code>eval_count</code>, <code>eval_duration</code>) — <a href="https://docs.ollama.com/api/generate.md">https://docs.ollama.com/api/generate.md</a></li>
<li>Ollama server source, <code>llm/llama_server.go</code> (<code>promptEvalCount()</code> returns <code>CacheN + PromptN</code>; <code>PromptEvalCachedCount</code> is <code>CacheN</code>; <code>PromptEvalDuration</code> is <code>PromptMS</code>) — <a href="https://raw.githubusercontent.com/ollama/ollama/main/llm/llama_server.go">https://raw.githubusercontent.com/ollama/ollama/main/llm/llama_server.go</a></li>
<li>Ollama API source, <code>api/types.go</code> (metrics printer computes the prompt eval rate as <code>(PromptEvalCount - cached) / PromptEvalDuration</code>) — <a href="https://raw.githubusercontent.com/ollama/ollama/main/api/types.go">https://raw.githubusercontent.com/ollama/ollama/main/api/types.go</a></li>
<li>llama.cpp llama-bench README (<code>pp</code>/<code>tg</code> definitions and the printed CPU rows for 7B Q4_0 at 1 thread) — <a href="https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/llama-bench/README.md">https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/llama-bench/README.md</a></li>
<li>openai-python <code>_constants.py</code> (<code>DEFAULT_TIMEOUT</code> 600 s, <code>DEFAULT_MAX_RETRIES</code> 2) — <a href="https://raw.githubusercontent.com/openai/openai-python/main/src/openai/_constants.py">https://raw.githubusercontent.com/openai/openai-python/main/src/openai/_constants.py</a></li>
</ul>
]]></content:encoded></item><item><title>Two Agent Sessions, One Local Model Server: what concurrency does to latency, context and tool calls</title><link>https://baeseokjae.github.io/posts/local-llm-concurrent-agent-sessions-2026/</link><pubDate>Mon, 02 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/local-llm-concurrent-agent-sessions-2026/</guid><description>Ollama ships one parallel slot per model, so a second agent session waits behind the first: four concurrent requests serialised to 4.0x the single-request wall time on a 2-vCPU host, and OLLAMA_NUM_PARALLEL=4 made each request 4.4-8.0x slower against the default server&amp;#39;s single-request baseline (3.8-7.0x against its own) instead of faster. Measured latency, the documented context rules, and what to log when a session stalls.</description><content:encoded><![CDATA[<p>Your first coding-agent session answers in a couple of seconds. You open a second one against the same local server, and now the first session&rsquo;s next request stalls for many seconds, or your client aborts with a read timeout while the server log shows no error at all. Restart the server and run one session: everything is fast again.</p>
<p>Nothing is wrong with the model or with either agent. Both sessions are sending requests to one server process that was never configured to work on more than one at a time, and the queue between them is invisible to your client. This article shows the documented defaults that decide how many requests a local server will actually run at once, measures what the second and fourth session do to wall time on a small CPU host, and lists what to record so you can prove the cause instead of guessing.</p>
<h2 id="the-shared-resource-is-the-server-not-the-model-weights">The shared resource is the server, not the model weights</h2>
<p>Two agents pointed at <code>http://localhost:11434</code> are two HTTP clients against one Ollama process. That process loads the model once and then decides how to schedule the requests it receives. Ollama&rsquo;s own documentation splits concurrency into two levels: how many <em>models</em> may be resident at once, and how many <em>parallel requests</em> a single model will process at the same time — <a href="https://docs.ollama.com/faq.md">Ollama FAQ, concurrency section</a>. The second number is the one that decides whether your two sessions overlap or take turns.</p>
<p>The FAQ states the rule directly: &ldquo;For a given model, if there is sufficient available memory when the model is loaded, it is configured to allow parallel request processing&rdquo;, and a model that is loaded with only one request processing at a time will make the second caller wait. Requests that cannot be served immediately are queued in arrival order — &ldquo;Queued requests will be processed in order&rdquo; — and only rejected when the queue itself is full (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>).</p>
<h2 id="ollamas-default-is-one-slot-per-model-not-four">Ollama&rsquo;s default is one slot per model, not four</h2>
<p>Three server settings decide the shape of the queue, and all three have defaults you probably never set (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>) — the values below are quoted from that page, which is the same text this run read on 2026-11-02:</p>
<table>
	<thead>
			<tr>
					<th>Setting</th>
					<th>Documented default</th>
					<th>What it changes</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>OLLAMA_NUM_PARALLEL</code></td>
					<td>1</td>
					<td>Parallel requests the model processes at once; &ldquo;Required RAM will scale by <code>OLLAMA_NUM_PARALLEL</code> * <code>OLLAMA_CONTEXT_LENGTH</code>&rdquo;</td>
			</tr>
			<tr>
					<td><code>OLLAMA_MAX_LOADED_MODELS</code></td>
					<td>3 (CPU) or 3 x GPUs</td>
					<td>How many distinct models may stay resident</td>
			</tr>
			<tr>
					<td><code>OLLAMA_MAX_QUEUE</code></td>
					<td>512</td>
					<td>Requests held while busy before new ones are rejected</td>
			</tr>
	</tbody>
</table>
<p>Check what your own server is running with before you theorise about it. An empty <code>env | grep '^OLLAMA'</code> means every value is a default, and the model row from <code>ollama ps</code> prints the context the server believes it is serving plus how long it will stay loaded:</p>
<pre tabindex="0"><code>$ env | grep &#39;^OLLAMA&#39;        # no output = all defaults, so OLLAMA_NUM_PARALLEL=1
$ ollama ps                   # NAME  ID  SIZE  PROCESSOR  CONTEXT  UNTIL
</code></pre><p>The per-request context is a separate knob from those server variables: <code>num_ctx</code> is the &ldquo;Context length size (number of tokens)&rdquo; in the <a href="https://docs.ollama.com/api/generate.md">Generate API reference</a>, and <code>options.num_ctx</code> is what an agent client sends with each call. A client that sets a generous context per request is not asking for a generous number of slots.</p>
<h2 id="what-the-probe-measured-on-one-cpu-host">What the probe measured on one CPU host</h2>
<p>The measurement below is deliberately narrow: one machine, one small model, no GPU, and one fixed prompt. It answers &ldquo;what does a second session cost on a host like mine&rdquo;, and it says nothing about throughput on hardware with more cores.</p>
<table>
	<thead>
			<tr>
					<th>Item</th>
					<th>Value</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Host</td>
					<td>2 vCPU aarch64 (Neoverse-N1), 11 GiB RAM, no GPU</td>
			</tr>
			<tr>
					<td>Server</td>
					<td><code>ollama version is 0.20.3</code></td>
			</tr>
			<tr>
					<td>Model</td>
					<td><code>qwen2.5:0.5b</code>, 397 MB, Q4_K_M</td>
			</tr>
			<tr>
					<td>Request</td>
					<td>one non-streaming <code>POST /api/generate</code>, fixed 59-token prompt, <code>num_ctx=2048</code>, <code>num_predict=64</code>, <code>temperature=0</code>, <code>seed=7</code>, <code>keep_alive=5m</code></td>
			</tr>
			<tr>
					<td>Load</td>
					<td>N concurrent clients released from a single barrier, so every request enters the server in the same instant</td>
			</tr>
			<tr>
					<td>Config A</td>
					<td>system server, no <code>OLLAMA_*</code> variables (defaults, therefore one slot)</td>
			</tr>
			<tr>
					<td>Config B</td>
					<td>second server on <code>127.0.0.1:11435</code> started as <code>OLLAMA_HOST=127.0.0.1:11435 OLLAMA_NUM_PARALLEL=4 OLLAMA_CONTEXT_LENGTH=2048 ollama serve</code></td>
			</tr>
	</tbody>
</table>
<p>Wall time is measured client-side around the whole call, which is exactly what your agent&rsquo;s HTTP client sees. Every request in every cell returned HTTP 200 with <code>eval_count=64</code>.</p>
<table>
	<thead>
			<tr>
					<th>Config</th>
					<th>N</th>
					<th>Observed wall times (ms)</th>
					<th>Slowest vs the 3256.9 ms default N=1 row</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>1</td>
					<td>3256.9; 3209.0</td>
					<td>1.0x</td>
			</tr>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>2</td>
					<td>3223.4, 6235.9</td>
					<td>1.9x</td>
			</tr>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>4</td>
					<td>4163.3, 7127.3, 10138.3, 13131.4</td>
					<td>4.0x</td>
			</tr>
			<tr>
					<td>A: defaults (1 slot)</td>
					<td>4 (repeat)</td>
					<td>3504.4, 6513.7, 9455.4, 12418.3</td>
					<td>3.8x</td>
			</tr>
			<tr>
					<td>B: <code>OLLAMA_NUM_PARALLEL=4</code></td>
					<td>4</td>
					<td>25586.7, 26199.3, 26211.4, 26212.7</td>
					<td>8.0x</td>
			</tr>
			<tr>
					<td>B: <code>OLLAMA_NUM_PARALLEL=4</code></td>
					<td>4 (repeat)</td>
					<td>14003.7, 14316.3, 14320.7, 14325.4</td>
					<td>4.4x</td>
			</tr>
			<tr>
					<td>B: <code>OLLAMA_NUM_PARALLEL=4</code></td>
					<td>1</td>
					<td>3753.9</td>
					<td>1.15x</td>
			</tr>
	</tbody>
</table>
<p>The last column divides by one denominator throughout: the 3256.9 ms N=1 row of configuration A, the default server. That makes the last three rows a comparison between two servers, not a before/after on one. Measured against the four-slot server&rsquo;s own single-request row (3753.9 ms), the same two four-slot cells are 7.0x and 3.8x — the same conclusion, a smaller factor, and the number that isolates what the slot setting did on that process.</p>
<p>Two shapes are worth reading out of that table. Under the default configuration the four requests finish at roughly 3.0-second intervals — 4163, 7127, 10138, 13131 ms in the first repeat, 2964 to 3011 ms apart, and 3504, 6514, 9455, 12418 ms in the second, 2942 to 3009 ms apart — which is about one single-request duration each, the signature of one slot serving four requests back to back rather than four requests progressing together. The N=2 cell shows the same thing at smaller scale: one request returned in 3223 ms, the second in 6236 ms, because it started by waiting for the first to finish.</p>
<p>With four slots the shape inverts, and it does not improve. In the first repeat all four requests finished within 630 ms of each other but each took 25.6 to 26.2 seconds, roughly twice as long as the slowest queued request under one slot. In the second repeat the same cell landed at 14.0 to 14.3 seconds. The single-request baseline on that second server was 3753.9 ms, so four slots bought parallelism without buying speed: the four requests shared the same two cores, and every one of them ran slower than it would have alone.</p>
<p>The honest confidence statement: most cells have two repeats, not twenty, and the N=2 cell and the second server&rsquo;s N=1 baseline have one. The spread between repeats is large — the same four-slot cell produced a 26-second band and a 14-second band — so treat these as order-of-magnitude figures for a two-core box, not as p95 latencies. They are not a throughput benchmark and they do not transfer to GPU serving.</p>
<h2 id="why-the-client-sees-a-timeout-rather-than-a-503">Why the client sees a timeout rather than a 503</h2>
<p>The obvious hypothesis — &ldquo;the second session gets rejected&rdquo; — does not hold at the sizes a developer actually runs. <code>OLLAMA_MAX_QUEUE</code> defaults to 512 requests, so the server holds queued work instead of refusing it unless you are far past two or three agent sessions (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>). In the probe, 21 requests across all cells produced zero non-200 statuses and zero error bodies; the only observable effect of concurrency was elapsed time.</p>
<p>That is worse than an error, because the failure moves into your client. A queued request looks identical to a slow one until your own read timeout fires. An agent framework with a 30-second HTTP timeout on a host where a queued request can take 26 seconds will abort, retry, and — if the retried tool call carried a side effect — do the same work twice. If you have seen duplicate writes from a &ldquo;flaky&rdquo; local agent, the retry-on-timeout path is worth auditing alongside the queue behaviour in <a href="https://baeseokjae.github.io/posts/agent-tool-retry-duplicate-side-effects-2026/">agent tool retries and duplicate side effects</a>.</p>
<p>The practical consequence: on a one-slot server, your effective concurrency limit is set by the client timeout, not by the server&rsquo;s capacity. Two sessions are fine if a single request takes 3 seconds and your timeout is 30. Four sessions with a longer prompt are not, because the fourth request inherits three queue waits and only the last one looks slow.</p>
<h2 id="context-multiplied-by-slots-here-per-slot-there">Context: multiplied by slots here, per-slot there</h2>
<p>The memory side of concurrency is documented, and it is not what most people assume. Ollama does not divide one context window among parallel requests — it multiplies: &ldquo;Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation&rdquo; (<a href="https://docs.ollama.com/faq.md">Ollama FAQ</a>). Combined with the stated RAM rule (<code>OLLAMA_NUM_PARALLEL</code> x <code>OLLAMA_CONTEXT_LENGTH</code>), that means raising slots on a memory-tight box can turn a working setup into an allocation failure at load time, before any request is served.</p>
<p>The reported field is a trap. A server started with <code>OLLAMA_NUM_PARALLEL=4</code> and <code>OLLAMA_CONTEXT_LENGTH=2048</code> answered <code>/api/ps</code> with <code>context_length: 2048</code> for a model it had just loaded, even though the vendor rule says four slots allocate four times that context. Reading <code>ollama ps</code> CONTEXT as &ldquo;the memory I am using&rdquo; will therefore understate a multi-slot server; the recorded response is in the published evidence below.</p>
<p>llama.cpp&rsquo;s server takes the opposite configuration shape. Slots are counted explicitly with <code>-np, --parallel N</code> (&ldquo;number of server slots, default: -1, -1 = auto&rdquo;), and per-slot context is a separate flag: <code>--kv-unified-per-slot N</code>, &ldquo;context limit per parallel slot; when set without -c/&ndash;ctx-size, the shared KV pool is sized to n_parallel*N&rdquo; (<a href="https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md">llama.cpp server README</a>). So on llama-server you choose both the slot count and the per-slot window, while on Ollama the per-request <code>num_ctx</code> is multiplied by the slot count. Either way, the number that matters for memory is slots x context, not context.</p>
<p>Before you raise slots, do the arithmetic for the model you actually run — the KV-cache sizing formula and a measured check of it are in <a href="https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/">sizing KV cache and context for a local coding model</a>. A 4-slot server is a 4x cache budget on the same weights.</p>
<h2 id="a-decision-table-for-two-to-four-sessions">A decision table for two to four sessions</h2>
<table>
	<thead>
			<tr>
					<th>Situation</th>
					<th>What to configure</th>
					<th>Why</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Two or three agent sessions on a 2-core CPU box</td>
					<td>Leave <code>OLLAMA_NUM_PARALLEL=1</code>; raise the client&rsquo;s read timeout well past a single request&rsquo;s worst case</td>
					<td>Four slots on two cores measured 4.4-8.0x slower per request than the default server&rsquo;s 3256.9 ms single-request row, and 3.8-7.0x slower against the four-slot server&rsquo;s own 3753.9 ms row; queueing is cheaper than contention at this core count</td>
			</tr>
			<tr>
					<td>Spare cores and spare memory</td>
					<td>Try <code>OLLAMA_NUM_PARALLEL=N</code> with <code>OLLAMA_CONTEXT_LENGTH</code> set explicitly, then measure your own p50</td>
					<td>The vendor rule multiplies context by slots, so both time and memory change together; nothing here predicts your hardware</td>
			</tr>
			<tr>
					<td>Sessions that must never block each other</td>
					<td>Separate servers on separate ports with separate models, or separate hosts</td>
					<td>Two processes cannot contend for the same slot, and each keeps its own context budget</td>
			</tr>
			<tr>
					<td>llama-server instead of Ollama</td>
					<td>Set <code>-np</code> explicitly, and set <code>--kv-unified-per-slot</code> when you want a hard per-session window</td>
					<td>Auto slots hide how the shared KV pool is divided; explicit values make the memory budget auditable</td>
			</tr>
	</tbody>
</table>
<p>The middle row is the one to be sceptical about: the only first-party evidence in this article for raising slots is negative, on two cores. GPU serving divides work differently and was not measured here, so nothing above is a claim about what slots do on a GPU.</p>
<h2 id="what-to-log-when-a-session-stalls">What to log when a session stalls</h2>
<p>If a session dies while another is running, the question is not &ldquo;which agent is broken&rdquo; but &ldquo;what did this request wait for&rdquo;. Record these five items per request and the answer is usually visible in one table:</p>
<ul>
<li>Client-side wall time around the whole HTTP call — the number your timeout compares against.</li>
<li>HTTP status and the verbatim response body on failure; a 200 with a body is a different bug from a 503.</li>
<li><code>done_reason</code> and <code>eval_count</code> from the response — a <code>length</code> stop means your own <code>num_predict</code> cap was hit, not that the server truncated anything.</li>
<li>Start timestamps of every concurrent session, so queue waits can be attributed rather than assumed.</li>
<li>The server&rsquo;s own row (<code>ollama ps</code> CONTEXT and UNTIL, or the llama-server slot log) taken while the requests are in flight, not after they finish.</li>
</ul>
<p>None of that requires instrumenting the model. It does mean your agent&rsquo;s HTTP layer must log the call rather than only its own decision loop.</p>
<p>One thing this article does <strong>not</strong> establish: whether concurrency truncates tool-call arguments. No tool-calling payload was sent in this probe, no malformed tool call appeared in any cell, and no shortened context was observed — every response carried the same 316 characters and the same <code>eval_count</code>. If you are debugging a tool call that arrives with a cut-off JSON argument, the surfaces to inspect are the raw assistant message before your parser, the <code>done_reason</code>, and the per-request <code>num_ctx</code>, but this measurement gives you no evidence that a second session causes it.</p>
<h2 id="files-method-and-how-to-rerun-it">Files, method, and how to rerun it</h2>
<p>The probe is a single stdlib Python script: it releases N clients from one barrier, records wall time, status and the verbatim error body per request, and flushes partial results after every cell so a deadline cannot lose data. These files are first-party and published with the article:</p>
<table>
	<thead>
			<tr>
					<th>File</th>
					<th>What it is</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/probe.py">probe.py</a></td>
					<td>The driver: barrier-synchronised cells, per-request records, deadline and partial flush</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/probe-default.json">probe-default.json</a></td>
					<td>Raw records for config A (defaults, one slot), 2026-11-02T08:03:10Z to 08:03:49Z</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/probe-par4.json">probe-par4.json</a></td>
					<td>Raw records for config B (<code>OLLAMA_NUM_PARALLEL=4</code>), 08:04:54Z to 08:05:38Z</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/ps-check.json">ps-check.json</a></td>
					<td>The <code>/api/ps</code> response quoted above, with the server environment that produced it</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/method.md">method.md</a></td>
					<td>Host, versions, run order, and what was deliberately not measured</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-concurrent-agent-sessions-2026/raw-records-manifest.json">raw-records-manifest.json</a></td>
					<td>SHA-256 and byte counts of the research-stage files, so a rerun can be compared byte for byte; <code>ps-check.json</code> was recorded afterwards in the write run and its hash is in the review record rather than this manifest</td>
			</tr>
	</tbody>
</table>
<p>To repeat the default-configuration cells on your own box, keep the script in a scratch directory of your choosing and give it a deadline in epoch seconds:</p>
<pre tabindex="0"><code>OLLAMA_PROBE_BASE=http://127.0.0.1:11434 OLLAMA_PROBE_LABEL=my-host \
OLLAMA_PROBE_OUT=&#34;$TMPDIR/probe-mine.json&#34; \
OLLAMA_PROBE_DEADLINE=$(( $(date +%s) + 150 )) \
python3 probe.py
</code></pre><p>For the four-slot cell, start a second server with your own model and flags, point <code>OLLAMA_PROBE_BASE</code> at it, and run <code>python3 probe.py n4</code>, which replays N=4 twice and then N=1 as the baseline. Compare the second client&rsquo;s wall time against the first: if the N=2 pair finishes at the same time, your server has two or more usable slots; if the second finishes at roughly twice the first, it has one.</p>
<h2 id="what-this-does-and-does-not-establish">What this does and does not establish</h2>
<p>Measured on one host: with the shipped default of one slot, four concurrent agent-shaped requests serialised and the last one took 4.0x the single-request wall time; setting <code>OLLAMA_NUM_PARALLEL=4</code> on the same two-core box made each request 4.4x to 8.0x slower than the default server&rsquo;s 3256.9 ms single-request row (3.8x to 7.0x against that server&rsquo;s own 3753.9 ms baseline) rather than faster; every request still returned HTTP 200.</p>
<p>Not measured, and not claimed: GPU serving of any kind, a real coding model generating hundreds of tokens, tool-calling payloads, throughput in tokens per second, memory high-water marks, and any comparison between Ollama and llama.cpp beyond their documented flag semantics. The absolute latencies here describe a two-vCPU host running a 397 MB model with a 64-token generation cap; your numbers will differ, and the shape — queueing at the default, contention when slots exceed cores — is what this article asks you to check rather than assume. For the wider comparison of engines and their parallel-request handling, see <a href="https://baeseokjae.github.io/posts/vllm-vs-ollama-production-2026/">vLLM vs Ollama in production</a> and <a href="https://baeseokjae.github.io/posts/local-ai-model-serving-frameworks-2026/">local model serving frameworks</a>; for the multi-agent side, <a href="https://baeseokjae.github.io/posts/local-ai-agents-guide-2026/">running local AI agents</a> covers what each session does with the model it gets.</p>
]]></content:encoded></item></channel></rss>