<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Prompt_eval_count on RockB</title><link>https://baeseokjae.github.io/tags/prompt_eval_count/</link><description>Recent content in Prompt_eval_count on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Thu, 12 Nov 2026 06:52:00 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/prompt_eval_count/index.xml" rel="self" type="application/rss+xml"/><item><title>Ollama Tool Definitions Are Not Free: The Measured Prompt-Token Cost of Every Tool You Register</title><link>https://baeseokjae.github.io/posts/ollama-tool-definition-prompt-token-cost-2026/</link><pubDate>Thu, 12 Nov 2026 00:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/ollama-tool-definition-prompt-token-cost-2026/</guid><description>On Ollama 0.20.3 with qwen3:0.6b a tools block adds about 58 fixed prompt tokens before the first definition and about 57 more per terse tool, against about 292 for a verbose one — so twelve terse tools spend 746 of a 4096-token default window before the user types. Eight measured cells, three identical repeats each, raw JSONL and the aggregation script published.</description><content:encoded><![CDATA[<p>You register twelve tools on a local agent, send a six-word message, and <code>prompt_eval_count</code> comes back at 764 instead of 18. Nothing is wrong: the definitions you sent are part of the prompt, and on a 4k default context that registry has already spent 746 of your 4096 tokens before the conversation starts — and it re-pays that cost on every turn.</p>
<p>This article measures the cost instead of estimating it. One model, one fixed message, one local server, tool sets of 0, 1, 3, 6 and 12 definitions in a deliberately terse form, and sets of 1, 3 and 6 in a deliberately verbose form, three repeats per cell, with the raw request and response records published next to the script.</p>
<p>The short answer: a terse definition costs <strong>about 57 prompt tokens</strong> on this model, a verbose one <strong>about 292</strong>, and a non-empty <code>tools</code> block costs a fixed <strong>~58 tokens</strong> of scaffolding on top of the chat template. The useful part is not the average — it is that both terms are linear, so you can compute your own registry&rsquo;s standing cost from your own schemas before running anything.</p>
<h2 id="what-prompt_eval_count-actually-counts">What <code>prompt_eval_count</code> actually counts</h2>
<p>Ollama reports the prompt size of each <code>/api/chat</code> response as <code>prompt_eval_count</code>, documented as the &ldquo;Number of tokens in the prompt&rdquo; (<a href="https://docs.ollama.com/api/chat.md">chat API reference</a>). That number is not the length of the message you typed; it is the length of the fully rendered prompt after the chat template is applied, which is why the no-tools baseline in this fixture is 18 tokens for a six-word message.</p>
<p>The <code>tools</code> field is documented as an &ldquo;Optional list of function tools the model may call during the chat&rdquo; (<a href="https://docs.ollama.com/api/chat.md#tools">chat API, tools parameter</a>), and each entry is a function definition carrying a name, a description and a JSON Schema <code>parameters</code> object (<a href="https://docs.ollama.com/capabilities/tool-calling.md">tool calling documentation</a>). What the documentation cannot tell you — because it is model- and template-specific — is how many tokens that object becomes once it has been rendered.</p>
<p>That gap is the whole problem. A schema is authored in JSON, billed in tokens, and the conversion rate depends on the tokenizer of whichever model you pulled. The argument shapes themselves are ordinary JSON Schema objects, so nested objects and <code>additionalProperties</code> behave exactly as the <a href="https://json-schema.org/understanding-json-schema/reference/object">JSON Schema object reference</a> describes — and, per the numbers below, as expensive as they are elaborate.</p>
<h2 id="method">Method</h2>
<p>The fixture is a single driver, <code>tool_schema_token_audit.py</code>, run against a local Ollama 0.20.3 server on a 2 vCPU / 11 GB container with no GPU. The model is <code>qwen3:0.6b</code> at Q4_K_M. Every request uses the same user message (&ldquo;Reply with the single word: ready.&rdquo;), <code>stream: false</code>, <code>temperature: 0</code>, <code>num_ctx: 8192</code> and <code>num_predict: 8</code>, and differs only in the <code>tools</code> array.</p>
<p>There are eight measured cells: terse sets of 1, 3, 6 and 12 tools, and verbose sets of 1, 3 and 6, plus a 0-tool baseline. Sets of N always use the first N of twelve fixed tools, so the cells are nested and the slope across 1/3/6/12 is attributable to the added definitions and nothing else. A terse definition is a name, a one-line description and flat string or number arguments; a verbose definition carries a 110-word description plus a nested <code>options</code> object with an enum, numeric bounds, defaults and <code>additionalProperties: false</code>. Both forms are published in <code>tool-fixture.json</code> so the comparison can be inspected rather than taken on trust.</p>
<p>Two deviations from the research plan are recorded here rather than smoothed over. First, three repeats per cell were collected instead of five. The first request of a cell is the one that pays model load, and its rate is quoted here as prompt tokens per wall-clock second: the records carry <code>total_duration_ns</code> and the driver&rsquo;s <code>wall_s</code>, not <code>prompt_eval_duration</code> (the API reports that field in its example response, but the fixture driver does not record it), so that denominator includes model load and the 8-token generation the fixture asked for, and these figures are completion rates rather than prompt-evaluation throughput. On this CPU host the first requests of the six tool-bearing cells other than the 6-tool verbose one ran at 5.8 to 13.0 prompt tokens per wall-clock second (144 tokens in 24.9 s for one terse tool, 764 in 58.9 s for twelve), the 6-tool verbose cell ran at 38.8, and the no-tools baseline falls outside that band at 0.9 prompt tokens per wall-clock second (18 tokens in 19.2 s). The stage budget did not allow five such cold requests for every cell. Second, the verbose 12-tool cell was not measured: its prompt — an extrapolated 3.6k tokens — exceeded the driver&rsquo;s 260-second client timeout (<code>REQUEST_TIMEOUT</code>, overridable with <code>--timeout</code>). The attempt is kept as a timeout row with no count (<code>TimeoutError: timed out</code>, <code>n_ok=0</code> in the published JSONL and CSV), and the run then stopped at the driver&rsquo;s own deadline guard before the cross-surface cell, so no completion for that request exists in the record. The driver published beside this article catches a timeout, prints <code>error=&lt;type&gt;</code>, moves to the next cell and still writes the summary, so a rerun on a slower host degrades instead of ending early. Everything else in the tables below is a measured value, and every repeat returned an identical <code>prompt_eval_count</code>, so each range is degenerate and each median is an exact repeat rather than an average.</p>
<h2 id="the-measured-numbers">The measured numbers</h2>
<p>Terse definitions, three repeats per cell, identical across repeats:</p>
<table>
	<thead>
			<tr>
					<th>Tools registered</th>
					<th><code>prompt_eval_count</code></th>
					<th>Added over previous cell</th>
					<th>Per added tool</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>0</td>
					<td>18</td>
					<td>—</td>
					<td>—</td>
			</tr>
			<tr>
					<td>1</td>
					<td>144</td>
					<td>+126</td>
					<td>126</td>
			</tr>
			<tr>
					<td>3</td>
					<td>250</td>
					<td>+106</td>
					<td>53</td>
			</tr>
			<tr>
					<td>6</td>
					<td>415</td>
					<td>+165</td>
					<td>55</td>
			</tr>
			<tr>
					<td>12</td>
					<td>764</td>
					<td>+349</td>
					<td>58</td>
			</tr>
	</tbody>
</table>
<p>The first tool is not like the others, and that is the first thing worth understanding. Going from no tools to one tool costs 126 tokens; each additional tool after that costs 53 to 58. The difference is a fixed block the template emits as soon as <code>tools</code> is non-empty — a preamble, an instruction line and the array scaffolding — before the first definition is serialised.</p>
<p>A least-squares line through the 3, 6 and 12-tool cells has a slope of 57.3 tokens per terse tool and an intercept of 75.5 at zero tools, which decomposes against the 18-token baseline into roughly 57.5 tokens of tools-block scaffolding plus 57.3 tokens per definition. That model predicts 133 tokens for a single tool against 144 measured: an 11-token residual at the smallest count, where the tokenizer&rsquo;s share of the array framing is largest. The straight line through the 3- and 12-tool cells alone gives 57.1 per tool and an intercept of 78.7, so the construction matters less than it looks — the scaffolding term lands between 57 and 61 tokens and the slope between 57.1 and 57.3.</p>
<h2 id="what-a-verbose-description-costs">What a verbose description costs</h2>
<p>The same twelve tools rewritten with a 110-word description and a nested options object:</p>
<table>
	<thead>
			<tr>
					<th>Tools registered</th>
					<th>Terse</th>
					<th>Verbose</th>
					<th>Difference</th>
					<th>Difference per tool</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>1</td>
					<td>144</td>
					<td>383</td>
					<td>239</td>
					<td>239.0</td>
			</tr>
			<tr>
					<td>3</td>
					<td>250</td>
					<td>965</td>
					<td>715</td>
					<td>238.3</td>
			</tr>
			<tr>
					<td>6</td>
					<td>415</td>
					<td>1844</td>
					<td>1429</td>
					<td>238.2</td>
			</tr>
			<tr>
					<td>12</td>
					<td>764</td>
					<td>not measured</td>
					<td>—</td>
					<td>—</td>
			</tr>
	</tbody>
</table>
<p>The description cost is stable at roughly 238 tokens per tool and the verbose slope is 292.3 tokens per definition with a 71.8-token preamble at zero tools, so the verbose form is about five times the terse form and does not drift with tool count. If you were assuming a long, careful tool description is nearly free because the model &ldquo;needs&rdquo; it, the fixture says otherwise: one paragraph of guidance per tool is roughly 240 tokens of every turn, which at twelve tools would be nearly 2900 tokens of standing cost — an extrapolation from the measured 6-tool cell, not a measurement.</p>
<p>A rougher sizing rule comes out of the same rows: at twelve terse tools the set runs about 4.8 bytes of JSON per token (3558 bytes for 746 tokens) and the verbose set about 6.3 bytes per token (11531 bytes for 1826 tokens). The ratio is only usable once the fixed scaffolding is amortised, because that part does not scale with bytes: applied below the cell where it was derived, it predicts 357 tokens where 397 were measured at six tools (10% low), 171 against 232 at three (26% low) and 50 against 126 at one (61% low). For a registry of roughly six tools or more it is good enough to decide whether the set fits; for one or two tools, use the two-term formula below.</p>
<h2 id="the-cross-surface-question-stays-open">The cross-surface question stays open</h2>
<p>Ollama&rsquo;s OpenAI-compatible endpoint also accepts <code>tools</code> and <code>stream_options.include_usage</code> (<a href="https://docs.ollama.com/api/openai-compatibility.md">OpenAI compatibility</a>), so <code>usage.prompt_tokens</code> is available as a second accounting surface. That cell was not collected in this run: the run ended at the 12-tool verbose timeout before the cross-surface step was reached, and no honest comparison exists without it. The compatibility table is a support matrix, not a statement that both surfaces count tokens identically, and nothing here should be read as one — the question of whether <code>/api/chat</code> and <code>/v1/chat/completions</code> agree on the same tools payload is listed as unmeasured.</p>
<h2 id="what-this-means-for-a-small-window">What this means for a small window</h2>
<p>Ollama&rsquo;s documented defaults choose the context length from total VRAM: 4k below 24 GiB, 32k at 24–48 GiB, 256k at 48 GiB and above (<a href="https://docs.ollama.com/context-length.md">context length</a>). This host sits in the first tier, so the denominator here is 4096 — the same default a laptop or a small CI runner gets.</p>
<p>At 4096 tokens, twelve terse tools are 18.2% of the window and the 12-tool verbose registry would be roughly 87% on the extrapolated slope, which is not a context budget but a context ceiling reached before the first user message. At 32k the same twelve terse tools are 2.3%. That is why &ldquo;just raise <code>num_ctx</code>&rdquo; is popular, and also why it is the wrong first lever on the small tier: the tier is chosen by VRAM, and raising <code>num_ctx</code> on a small host moves the cost from the prompt into the KV cache.</p>
<p>The arithmetic to run on your own registry is one line — <code>57.5 + 57.3 × tools</code> for terse definitions, <code>71.8 + 292.3 × tools</code> for verbose ones — added to whatever your template costs. Twelve terse tools is about 745 tokens; twelve verbose ones about 3579.</p>
<h2 id="decision-rules-that-follow-from-the-numbers">Decision rules that follow from the numbers</h2>
<p>Trim before you cut. Converting twelve verbose tools to terse saves about 2800 tokens (3579 against 745), more than deleting ten of twelve terse tools would save (about 570). Rewriting descriptions is the highest-leverage edit available and changes nothing about what the model can do.</p>
<p>Cut when the registry itself is the problem. On a 4k window with 2k of conversation, twelve terse tools leave about 1350 tokens of slack; a third registry is not affordable and no amount of description trimming fixes that. Splitting an agent into two six-tool registries costs a second prompt and a second model load but holds each turn&rsquo;s standing overhead near 415 tokens instead of 764.</p>
<p>Do not expect prompt caching to make the tools block free. <code>prompt_eval_count</code> reports what was evaluated, and a cached prefix is reported separately where the server supports it — this fixture&rsquo;s single-turn requests recorded no cached count at all. Even a cheap-to-evaluate tools block still occupies the window, because context capacity and prompt evaluation cost are two different budgets and tools spend the first one whether or not they spend the second.</p>
<h2 id="reproduce-it">Reproduce it</h2>
<p>The fixture is published: the driver script, the twelve tool definitions in both forms, the raw per-request JSONL with request hashes and <code>prompt_eval_count</code>, the aggregation script that turns that JSONL into the summary CSV, and the CSV itself.</p>
<ul>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-tool-definition-prompt-token-cost-2026/tool_schema_token_audit.py">tool_schema_token_audit.py</a></li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-tool-definition-prompt-token-cost-2026/tool-fixture.json">tool-fixture.json</a></li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-tool-definition-prompt-token-cost-2026/fixture-raw.jsonl">fixture-raw.jsonl</a></li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-tool-definition-prompt-token-cost-2026/aggregate_fixture.py">aggregate_fixture.py</a></li>
<li><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/ollama-tool-definition-prompt-token-cost-2026/fixture-summary.csv">fixture-summary.csv</a></li>
</ul>
<p>Run <code>python3 tool_schema_token_audit.py --out &lt;empty dir&gt; --repeats 3</code> against any Ollama server and you get the same table for your own model, because the script sends the definitions verbatim and records what the server reports rather than what a tokenizer is expected to do. Give it a fresh output directory each time: the driver appends to <code>fixture-raw.jsonl</code>, so a second run into the same directory mixes two runs into one summary. <code>python3 aggregate_fixture.py --raw fixture-raw.jsonl --csv fixture-summary.csv</code> rebuilds the summary from whatever records exist, one row per cell, with the columns <code>surface,cell,tool_count,n_ok,n_attempts,median,min,max,full_range,tools_json_bytes,errors</code> — the failed 12-tool verbose cell is kept as a row with <code>n_ok=0</code> and its error string, because a dropped cell would hide the most expensive result. On a slow host expect the same timeout this run hit: the first request after each tools-set change is the cold one, and the 12-tool verbose cell can exceed the 260-second client timeout before it produces a count.</p>
<p>What these numbers do not cover: other models, other quantisations, other templates. <code>prompt_eval_count</code> is tokenizer-specific, and a different chat template can move both the preamble and the per-tool slope. The values here are one model on one host on 2026-11-12 — a measurement of that configuration, not a constant of Ollama.</p>
<p>If you are sizing the wider agent cost equation rather than the tool term alone, the <a href="https://baeseokjae.github.io/posts/browser-mcp-snapshot-token-cost-2026/">browser MCP snapshot cost analysis</a> does the same accounting for page snapshots. If you are still deciding how to structure the calls themselves, the <a href="https://baeseokjae.github.io/posts/llm-function-calling-tool-use-guide-2026/">function calling guide</a> covers the mechanics, and the <a href="https://baeseokjae.github.io/posts/ollama-api-local-llm-integration-developer-guide-2026/">Ollama API integration guide</a> documents the surrounding request surface.</p>
]]></content:encoded></item><item><title>What an Ollama Vision Tag Really Ships: qwen3-vl Projector Packaging, Image-Token Cost and the 4096-Token Wall (2026)</title><link>https://baeseokjae.github.io/posts/local-vlm-tag-projector-image-token-cost-2026/</link><pubDate>Tue, 03 Nov 2026 17:20:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/local-vlm-tag-projector-image-token-cost-2026/</guid><description>Separate the three failures behind &amp;#39;the model ignored my image&amp;#39; — a tag that 404s, a tag that drops the pixels, and image tokens eating a 4096-token context — with registry layer tables, a measured prompt-token count and the KV arithmetic for one image.</description><content:encoded><![CDATA[<p>You pull a vision model, send a screenshot, and get an answer that reads as if no image was attached. Or the request that worked with one image dies as soon as a second one is in the prompt. The error message, when there is one, points at the request, not at the three very different things that can be wrong: the tag you typed does not exist, the tag exists but the runtime drops the image, or the model does see the image and the image tokens have already consumed the context window.</p>
<p>This page separates those three cases with registry metadata, one measured prompt-token count and arithmetic you can redo. Everything below was observed on 2026-11-03 on a CPU-only host with Ollama 0.20.3; the registry and config figures carry that date because tags change, and the 32B tag was never executed here — its row is metadata and arithmetic only.</p>
<h2 id="three-failures-that-all-look-like-the-model-ignored-my-image">Three failures that all look like &ldquo;the model ignored my image&rdquo;</h2>
<p>The first case fails at pull time. The family name is <code>qwen3-vl</code>, with the hyphen; typing it as <code>qwen3vl</code> produces a manifest request for <a href="https://registry.ollama.ai/v2/library/qwen3vl/manifests/32b">registry.ollama.ai/v2/library/qwen3vl/manifests/32b</a> that returns HTTP 404, while the same request against <a href="https://registry.ollama.ai/v2/library/qwen3-vl/manifests/32b">the correctly hyphenated repository</a> returns a manifest. That distinction matters because a 404 on pull is unambiguous while a successful pull of a tag that cannot see is not.</p>
<p>The second case fails silently. A text-only tag given an image payload returns HTTP 200, accepts the <code>images</code> array without comment — the <a href="https://github.com/ollama/ollama/blob/main/docs/api.md">Ollama API documentation</a> describes <code>images</code> as &ldquo;a list of base64-encoded images (for multimodal models such as <code>llava</code>)&rdquo; and documents no capability validation — and then answers as though nothing was attached. Measured on <code>qwen2.5:0.5b</code>: the same question with a 640x480 PNG cost 46 prompt tokens, without it 41. A five-token difference is an image marker, not a 300-token pixel grid, and the assistant replied &ldquo;I&rsquo;m sorry, but I cannot see the image you&rsquo;re referring to&rdquo; in both requests.</p>
<p>The third case is the one that produces confusing errors on large prompts. The image is tokenized correctly and the tokens are simply expensive: the same model with <code>qwen3-vl:2b</code> and one 640x480 image reported 330 prompt tokens against 28 for the same question with no image.</p>
<h2 id="what-the-tag-actually-contains">What the tag actually contains</h2>
<p>An Ollama manifest is a layer list with media types. Reading it tells you what the tag packs, and the answer differs between sizes of the same family. Observed on 2026-11-03 at 16:13 UTC:</p>
<ul>
<li><a href="https://registry.ollama.ai/v2/library/qwen3-vl/manifests/2b">qwen3-vl:2b</a> — model layer 1889496384 B, two licence layers, params; no projector layer.</li>
<li><a href="https://registry.ollama.ai/v2/library/qwen3-vl/manifests/4b">qwen3-vl:4b</a> — model layer 2497281600 B plus a projector layer of 839325984 B.</li>
<li><a href="https://registry.ollama.ai/v2/library/qwen3-vl/manifests/8b">qwen3-vl:8b</a> — model layer 5028391424 B plus a projector layer of 1162568672 B.</li>
<li><a href="https://registry.ollama.ai/v2/library/qwen3-vl/manifests/32b">qwen3-vl:32b</a> — model layer 20910274496 B, two licence layers, params; no projector layer.</li>
<li><a href="https://registry.ollama.ai/v2/library/llava/manifests/7b">llava:7b</a> — model layer 4108916992 B plus a 624434368 B projector.</li>
<li><a href="https://registry.ollama.ai/v2/library/gemma3/manifests/4b">gemma3:4b</a> — model layer 2499012288 B plus an 851251104 B projector.</li>
<li><a href="https://registry.ollama.ai/v2/library/qwen2.5/manifests/0.5b">qwen2.5:0.5b</a> — model layer 397807936 B, system, template, licence; no projector.</li>
</ul>
<p>So within one family the 4B and 8B tags carry a separate <code>projector</code> blob (~800 MB and ~1.1 GB) and the 2B and 32B tags do not. Read that as packaging, not capability: a single-GGUF tag can carry the vision tower and the merger inside the model layer, and the next section shows that this is exactly what happens with the 2B.</p>
<h2 id="why-the-layer-table-is-not-a-capability-check">Why the layer table is not a capability check</h2>
<p><code>ollama show qwen3-vl:2b</code> on the same host reported architecture <code>qwen3vl</code>, parameters 2.1B, context length 262144, quantization Q4_K_M and capabilities <code>completion, vision, tools, thinking</code>. The tag has no projector layer and still declares vision, and the trace below shows it tokenizing an image into the prompt.</p>
<p>That observation retires the stronger hypothesis this page was originally planned around — that a qwen3-vl size shipping without a projector layer is a text-only tag. It is not: for the 2B, the <a href="https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct">model card of the checkpoint behind the tag</a> describes a conditional-generation vision-language model, and the manifest layer list alone cannot prove runtime capability either way. The layer table is useful for explaining download size and for spotting a tag that ships a separate projector; it is not a suitability test.</p>
<h2 id="the-arithmetic-of-one-image">The arithmetic of one image</h2>
<p>Qwen3-VL splits an image into 16-pixel patches and merges 2x2 blocks, so the token count for a square input with no rescaling is <code>(H/16) * (W/16) / 4</code>. That is 300 tokens for 640x480, 1024 for 1024x1024 and 1600 for 1280x1280. The geometry comes from the published config of each checkpoint: <a href="https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct/raw/main/config.json">2B</a>, <a href="https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct/raw/main/config.json">4B</a>, <a href="https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct/raw/main/config.json">8B</a> and <a href="https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct/raw/main/config.json">32B</a> all declare <code>patch_size</code> 16, <code>spatial_merge_size</code> 2 and <code>head_dim</code> 128, and differ in <code>num_hidden_layers</code> (28, 36, 36, 64) with 8 key/value heads throughout.</p>
<p>Cache cost uses the same formula as the site&rsquo;s <a href="https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/">text KV budget guide</a>: twice (key and value) × layers × KV heads × head_dim × 2 bytes at f16. At 32B geometry that is 262144 bytes per token, at 8B and 4B 147456, at 2B 114688.</p>
<table>
	<thead>
			<tr>
					<th>Geometry</th>
					<th>KV per token (f16)</th>
					<th>1 image 640x480 (300 tok)</th>
					<th>1 image 1024x1024 (1024 tok)</th>
					<th>1 image 1280x1280 (1600 tok)</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Qwen3-VL-2B</td>
					<td>112 KiB</td>
					<td>32.8 MiB</td>
					<td>112 MiB</td>
					<td>175 MiB</td>
			</tr>
			<tr>
					<td>Qwen3-VL-4B</td>
					<td>144 KiB</td>
					<td>42.2 MiB</td>
					<td>144 MiB</td>
					<td>225 MiB</td>
			</tr>
			<tr>
					<td>Qwen3-VL-8B</td>
					<td>144 KiB</td>
					<td>42.2 MiB</td>
					<td>144 MiB</td>
					<td>225 MiB</td>
			</tr>
			<tr>
					<td>Qwen3-VL-32B</td>
					<td>256 KiB</td>
					<td>75 MiB</td>
					<td>256 MiB</td>
					<td>400 MiB</td>
			</tr>
	</tbody>
</table>
<p>Two of those numbers are worth holding on to. A single 1280x1280 image is 1600 prompt tokens, which is 39.1% of a 4096-token context, and at 32B geometry it costs 400 MiB of KV before the model has generated a single token. For scale, the 32B tag&rsquo;s model layer is 20910274496 B — 19.47 GiB at Q4 — so one full-resolution image is about 2% of the weights in cache, per image, per concurrent request.</p>
<p>One more config field deserves a flag. Every one of those configs declares <code>num_position_embeddings</code> 2304 for the vision tower, which after 2x2 merging implies a ceiling of 576 merged tokens for a single image — an inference from that field, not a limit the config states. A 1280x1280 image computed from raw geometry is 1600 tokens, 2.78x that ceiling, so a runtime that honours the cap and a runtime that does not will disagree badly at high resolutions. No request in this run got above 640x480, so which behaviour Ollama 0.20.3 has here is unverified, and the 1600-token figure above should be read as the uncapped arithmetic.</p>
<h2 id="measured-what-one-image-actually-adds-to-the-prompt">Measured: what one image actually adds to the prompt</h2>
<p>On a 2 vCPU Neoverse-N1 host with no GPU, <code>qwen3-vl:2b</code> Q4_K_M, <code>num_ctx</code> set to 4096 and <code>num_predict</code> 32, the same 85-character question was sent through <code>/api/chat</code> twice: once alone and once with one 1956-byte 640x480 PNG attached. The text-only request returned HTTP 200 after 44.4 s with <code>prompt_eval_count</code> 28; the with-image request returned HTTP 200 after 115.7 s with <code>prompt_eval_count</code> 330. The image therefore added 302 prompt tokens against 300 computed from the config geometry — a 0.67% gap, inside the 5% band this check was allowed to treat as reconcilable. An earlier baseline in the same session used a shorter prompt (<code>Reply with the single word: ready.</code>, 34 characters, 18 tokens); because the prompt text differed, that pair is not used for the delta and is disclosed here only so the two records are not confused.</p>
<p>The 2-token gap is consistent with chat-template image placeholders and any grid padding the runtime adds, but this run did not isolate which, so the honest statement is that the computed grid predicts the measured cost to within 1% on this input. The 115.7 s is prefill-bound and is wall-clock on two cores for one request; it is not a benchmark and says nothing about relative model quality. The assistant <code>response</code> field is empty because the 32-token budget ran out, but the <code>thinking</code> field of that same with-image request is populated: <code>So, let's look at the image. It's a flag divided into three horizontal sections. The top is red, then green, then blue.</code> with <code>done_reason</code> <code>length</code>. That text matches the fixture&rsquo;s three colour bands and shows the budget being consumed before any answer text, so the measurement establishes that the image reached the model and was tokenized; the truncated thinking is not a scored answer about the image.</p>
<p>A second request with one 1024x1024 PNG and the same prompt did not complete: the client timed out at 180.1 s with no HTTP status. That row is a timeout, not a measurement, and no token count is attributed to it anywhere on this page.</p>
<h2 id="the-4096-token-wall">The 4096-token wall</h2>
<p>Ollama&rsquo;s FAQ answers the context question directly: &ldquo;By default, Ollama uses a context window size of 4096 tokens&rdquo; (<a href="https://docs.ollama.com/faq">Ollama FAQ</a>). Combining that default with the arithmetic above is where vision requests start failing in ways that look like model bugs.</p>
<p>At 1600 tokens per 1280x1280 image, two images consume 3200 tokens — 78.1% of the default context — leaving 896 tokens for the system prompt, the chat template, your question and the answer. Three images do not fit at all. At 1024x1024 the budget is 1024 tokens per image, so four fit with nothing left for text. Nothing in the earlier sections is needed to see the shape of the problem: default context minus image tokens is the room your actual instructions get.</p>
<p>Raising <code>num_ctx</code> is the fix, and it is not free. At 32B geometry the KV cache costs 256 KiB per token, so 4096 tokens of context is 1024 MiB, 16384 tokens is 4096 MiB, and every additional 1000 tokens of context adds 250 MiB — on top of 19.47 GiB of weights. If the request fits after raising the context, the answer is memory, not a faster model.</p>
<h2 id="a-check-you-can-run-on-your-own-machine">A check you can run on your own machine</h2>
<p>Three commands separate the three cases, and none of them needs a GPU:</p>
<pre tabindex="0"><code># 1. what the tag packs, and what the runtime says it can do
curl -s https://registry.ollama.ai/v2/library/qwen3-vl/manifests/2b \
  | python3 -c &#39;import json,sys; [print(l[&#34;mediaType&#34;], l[&#34;size&#34;]) for l in json.load(sys.stdin)[&#34;layers&#34;]]&#39;
ollama show qwen3-vl:2b

# 2. the same question with and without an image, reading prompt_eval_count
#    (send via POST /api/chat with options.num_ctx set explicitly; the field
#     prompt_eval_count is what you compare, not the wall time)

# 3. compute the grid your image should produce
python3 -c &#39;print((480//16)*(640//16)//4)&#39;   # 640x480 -&gt; 300
</code></pre><p>Read the comparison in step 2 like this: a with-image delta of roughly zero means the runtime took the payload and dropped the pixels, so the tag has no vision path; a delta close to the grid from step 3 means the image was tokenized and the cost is real; a delta far above the grid means the runtime rescaled or padded the image and your context budget is worse than the arithmetic says. Run the same pair at <code>num_ctx</code> 4096 and 16384 to see whether your failure is the context wall rather than the image itself.</p>
<h2 id="what-this-does-not-prove">What this does not prove</h2>
<p>Only one tag was executed: <code>qwen3-vl:2b</code>. The 4B, 8B and 32B rows are manifest and config evidence, and no claim is made about how those tags behave at runtime — the 32B would not fit in this workspace&rsquo;s ~10 GiB of free disk at a 19.47 GiB model layer. Everything ran on one CPU-only host with one Ollama version, 0.20.3, and one image size that completed; the 1024x1024 attempt is a timeout, and the two-image and 1280x1280 requests were never sent.</p>
<p>The 0.67% gap between measured and computed image tokens is reported, not explained: placeholder tokens and grid padding were not isolated. The <code>num_position_embeddings</code> ceiling is unresolved in both directions and is presented as an inference, not as a documented limit. The <code>llama-server --mmproj</code> path was not exercised at all, so this page says nothing about which projector blob a llama.cpp server needs. And the <code>qwen3vl</code> 404 was verified at the registry manifest endpoint, not traced through a full <code>ollama pull</code>.</p>
<p>The raw capture files behind these numbers are a private local author verification, not a published dataset: registry manifests and model configs are third-party bytes and are not redistributed here. The registry layer sizes, config fields and all derived arithmetic are reproducible from the commands above and the linked endpoints. The prompt-token counts — 28 and 330 here, 46 and 41 in the text-only control — are one-host, one-version observations: they come from Ollama 0.20.3 with this chat template on the host described above, and a comparable runtime should land close but is not guaranteed to match to the token.</p>
<p>If you want the text-only memory model that this page builds on, the <a href="https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/">KV cache and context sizing guide</a> derives the per-token formula and measures it; for the endpoint mechanics of <code>/api/chat</code> see the <a href="https://baeseokjae.github.io/posts/ollama-api-local-llm-integration-developer-guide-2026/">Ollama API integration guide</a>; for choosing which text tag to run see <a href="https://baeseokjae.github.io/posts/best-ollama-models-coding-2026/">Ollama models for coding</a>; and for how to evaluate multimodal inputs by task rather than by capability label see <a href="https://baeseokjae.github.io/posts/multimodal-ai-2026/">multimodal AI evaluation</a>.</p>
]]></content:encoded></item></channel></rss>