<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Gqa Kv Heads on RockB</title><link>https://baeseokjae.github.io/tags/gqa-kv-heads/</link><description>Recent content in Gqa Kv Heads on RockB</description><image><title>RockB</title><url>https://baeseokjae.github.io/images/og-default.png</url><link>https://baeseokjae.github.io/images/og-default.png</link></image><generator>Hugo</generator><language>en-us</language><lastBuildDate>Tue, 03 Nov 2026 06:50:12 +0000</lastBuildDate><atom:link href="https://baeseokjae.github.io/tags/gqa-kv-heads/index.xml" rel="self" type="application/rss+xml"/><item><title>Local LLM VRAM Planning 2026: The KV Cache Math Behind the Context You Can Actually Serve</title><link>https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-sizing-2026/</link><pubDate>Tue, 03 Nov 2026 06:29:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-sizing-2026/</guid><description>Does this model, quant and context fit your card? Compute weights plus KV from the model&amp;#39;s own config at a pinned revision, with a per-model table, a hybrid-attention counterexample and a runnable planner.</description><content:encoded><![CDATA[<p>A model card says 40,960 tokens. The Q4_K_M file is 4.68 GiB. The card has 12 GB. You set <code>-c 40960</code>, and one of three things happens: the load fails, the runtime spills layers to CPU and tokens per second collapse, or the server starts and silently serves a shorter context than you asked for.</p>
<p>Which of the three you get depends on a number that appears in none of those three places — the key/value cache the runtime allocates at load time from the context you request. This page computes it for four specific models from their own <code>config.json</code> at a pinned revision, turns it into a &ldquo;largest context that fits this card&rdquo; table, and ships a planner you can run offline.</p>
<p>If you searched for benchmark charts, this page will disappoint you on purpose. There are no throughput measurements here: this environment has no GPU, and inventing a tokens-per-second table is worse than saying so. What is here is arithmetic you can check in a minute, plus the one failure mode that arithmetic gets wrong.</p>
<h2 id="the-budget-is-three-terms">The budget is three terms</h2>
<pre tabindex="0"><code>resident ≈ GGUF file size + KV cache at your context + runtime overhead
</code></pre><p>Only the first term is on the download page. The third term — compute buffers, CUDA context, the graph allocator — is machine- and backend-specific; the planner below takes it as a declared allowance and prints it separately, so you can see what you are assuming rather than have it hidden in a fudge factor.</p>
<p>The middle term is the one that changes with context:</p>
<pre tabindex="0"><code>bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
</code></pre><p>Two for key plus value. <code>n_layers</code>, <code>n_key_value_heads</code> and <code>head_dim</code> come from the model&rsquo;s <code>config.json</code> — not from the parameter count, and not from the size of the file. Two 8B-class models can differ by 8× here depending on how many heads share keys and values.</p>
<p>Qwen3-8B, read from <a href="https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json">its config</a> at revision <code>b968826d9c46dd6066d109eabc6255188de91218</code>: 36 layers, 8 KV heads, head_dim 128, and <code>max_position_embeddings</code> 40,960. That is 2 × 36 × 8 × 128 × 2 = 147,456 bytes per token — 144 KiB — which is 0.56 GiB at 4,096 tokens and 5.62 GiB at the native 40,960.</p>
<p>Same model, same file, a cache five times larger than the weights at the context the card advertises. That is the whole trap in one number.</p>
<h2 id="four-models-computed-from-their-own-configs">Four models, computed from their own configs</h2>
<p>Every row below is computed at batch size 1 with an f16 cache, from the config fields saved at the revisions read on 2026-11-03, and the GGUF sizes are the published file sizes in the same repositories. None of it is a measured load.</p>
<table>
	<thead>
			<tr>
					<th>Model</th>
					<th>Layers</th>
					<th>KV heads × head dim</th>
					<th>Native ctx</th>
					<th>f16 KV B/token</th>
					<th>KV at 8k</th>
					<th>KV at native</th>
					<th>Q4_K_M file</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Qwen3-8B</td>
					<td>36</td>
					<td>8 × 128</td>
					<td>40,960</td>
					<td>147,456</td>
					<td>1.12 GiB</td>
					<td>5.62 GiB</td>
					<td>4.68 GiB</td>
			</tr>
			<tr>
					<td>Qwen3-14B</td>
					<td>40</td>
					<td>8 × 128</td>
					<td>40,960</td>
					<td>163,840</td>
					<td>1.25 GiB</td>
					<td>6.25 GiB</td>
					<td>8.38 GiB</td>
			</tr>
			<tr>
					<td>Devstral-Small-2507</td>
					<td>40</td>
					<td>8 × 128</td>
					<td>131,072</td>
					<td>163,840</td>
					<td>1.25 GiB</td>
					<td>20.0 GiB</td>
					<td>13.35 GiB</td>
			</tr>
			<tr>
					<td>Gemma 4 31B (hybrid)</td>
					<td>60</td>
					<td>16 × 256</td>
					<td>262,144</td>
					<td>see below</td>
					<td>1.41 GiB</td>
					<td>20.8 GiB</td>
					<td>17.07 GiB</td>
			</tr>
	</tbody>
</table>
<p>Config sources: <a href="https://huggingface.co/Qwen/Qwen3-14B/raw/main/config.json">Qwen3-14B</a>, <a href="https://huggingface.co/mistralai/Devstral-Small-2507/raw/main/config.json">Devstral-Small-2507</a>, <a href="https://huggingface.co/google/gemma-4-31B-it/raw/main/config.json">Gemma 4 31B</a>. The file-size column comes from one mirror per model, each an <code>/api/models/&lt;repo&gt;/tree/main?blobs=true</code> listing read on 2026-11-03: the <a href="https://huggingface.co/api/models/unsloth/Qwen3-8B-GGUF/tree/main?blobs=true">unsloth Qwen3-8B GGUF tree</a> behind 4.68 GiB, the <a href="https://huggingface.co/api/models/unsloth/Qwen3-14B-GGUF/tree/main?blobs=true">unsloth Qwen3-14B GGUF tree</a> behind 8.38 GiB, the <a href="https://huggingface.co/api/models/unsloth/Devstral-Small-2507-GGUF/tree/main?blobs=true">unsloth Devstral GGUF tree</a> behind 13.35 GiB, and the <a href="https://huggingface.co/api/models/unsloth/gemma-4-31B-it-GGUF/tree/main?blobs=true">unsloth Gemma 4 31B GGUF tree</a> behind 17.07 GiB. Another quantiser&rsquo;s Q4_K_M is a different file with a different byte size, so treat the mirror as part of the number.</p>
<p>Read the Devstral row before you buy: a 13.35 GiB file whose native context needs 20 GiB of cache on top. The weights fit 24 GB comfortably; the context does not.</p>
<h2 id="quantisation-moves-the-left-hand-term-not-the-right-hand-one">Quantisation moves the left-hand term, not the right-hand one</h2>
<p>Going from BF16 to Q4_K_M shrinks the file by roughly 3.3× for Qwen3-8B (15.26 GiB to 4.68 GiB). It does not touch the cache unless you also quantise the cache. With q8_0 packing — 32 elements in 34 bytes, 1.0625 bytes per element — the same 40,960-token context costs 2.99 GiB instead of 5.62, and the whole plan lands at 8.67 GiB including a 1 GiB allowance.</p>
<p>Ollama documents that this only works with Flash Attention enabled, and that the setting is global rather than per model (<a href="https://docs.ollama.com/faq">Ollama FAQ</a>). A quantised cache type on a backend that cannot run Flash Attention is inert — the flag is accepted, the memory does not move. The CPU-only probe published with <a href="https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/">the budget article</a> shows exactly that: the same slope with <code>OLLAMA_KV_CACHE_TYPE=q8_0</code> until Flash Attention was turned on, then half.</p>
<h2 id="where-the-single-slope-formula-fails-gemma-4-31b">Where the single-slope formula fails: Gemma 4 31B</h2>
<p>Take the naive rule to Gemma 4 31B and you get 2 × 60 × 16 × 256 × 2 = 983,040 bytes per token, or 240 GiB at its 262,144-token maximum. That number is correct arithmetic and a wrong plan. The config declares a <code>layer_types</code> list in which only 10 of 60 layers are full attention, with <code>global_head_dim</code> 512 and 4 global KV heads; the other 50 slide over a 1,024-token window.</p>
<p>A sliding layer does not grow with context. It holds one window, ever. So the cost is a slope plus a fixed term:</p>
<ul>
<li>10 full layers: 2 × 10 × 4 × 512 × 2 = 81,920 bytes per token</li>
<li>50 sliding layers: 2 × 50 × 16 × 256 × 2 × 1,024 = 800 MiB fixed, per slot</li>
<li>at 32,768 tokens: 2.50 GiB + 0.78 GiB = 3.28 GiB, not 30 GiB</li>
</ul>
<p>This is the correction that matters most for 2026 models: if <code>layer_types</code> or <code>sliding_window</code> appears in a config, the per-token formula alone overestimates the cache by an order of magnitude, and a planner built on it will tell you a 24 GB card cannot run a 31B at 32k when the arithmetic says it can.</p>
<h2 id="largest-context-that-fits-at-a-declared-1-gib-overhead">Largest context that fits, at a declared 1 GiB overhead</h2>
<p>Computed with the same fixtures: file size plus f16 cache plus 1 GiB for compute buffers and CUDA context, capped at the model&rsquo;s native context. If your runtime&rsquo;s overhead is larger, subtract it and recompute — the planner does this for you.</p>
<table>
	<thead>
			<tr>
					<th>Q4_K_M</th>
					<th>8 GiB</th>
					<th>12 GiB</th>
					<th>16 GiB</th>
					<th>24 GiB</th>
					<th>48 GiB</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Qwen3-8B</td>
					<td>16,893</td>
					<td>40,960</td>
					<td>40,960</td>
					<td>40,960</td>
					<td>40,960</td>
			</tr>
			<tr>
					<td>Qwen3-14B</td>
					<td>does not fit</td>
					<td>17,170</td>
					<td>40,960</td>
					<td>40,960</td>
					<td>40,960</td>
			</tr>
			<tr>
					<td>Devstral-Small-2507</td>
					<td>does not fit</td>
					<td>does not fit</td>
					<td>10,813</td>
					<td>63,242</td>
					<td>131,072</td>
			</tr>
			<tr>
					<td>Gemma 4 31B (hybrid)</td>
					<td>does not fit</td>
					<td>does not fit</td>
					<td>does not fit</td>
					<td>67,485</td>
					<td>262,144</td>
			</tr>
	</tbody>
</table>
<p>Capped cells are written as the native context, not as the larger number the formula would allow. An 8 GiB card holds Qwen3-8B at 16,893 tokens and nothing else in this set; a 12 GB card holds it at the full 40,960 with about 0.7 GiB to spare, which is not a margin to plan a desktop session around once the display compositor takes its share.</p>
<h2 id="concurrency-multiplies-the-cache-not-the-weights">Concurrency multiplies the cache, not the weights</h2>
<p>Four parallel slots at 32,768 tokens on Qwen3-8B is 18 GiB of cache against 4.5 GiB for one slot — the weights are loaded once, the cache is allocated per slot. A server sized for one user&rsquo;s long context is not sized for four users&rsquo; long contexts, and the difference is not in the file size anyone quotes.</p>
<h2 id="the-planner">The planner</h2>
<p><code>plan.py</code> reads a fixture of the four configs above and prints the breakdown, exiting non-zero on overflow so it can gate a launch script:</p>
<pre tabindex="0"><code>$ python3 plan.py --model qwen3-8b --quant Q4_K_M --context 40960 --vram 12
model    : qwen3-8b (Qwen/Qwen3-8B, native 40960 tokens, dense)
weights  : 4.68 GiB  (Q4_K_M GGUF file)
kv       : 5.62 GiB  (f16, 147,456 B/token x 40960)
overhead : 1.00 GiB  (declared allowance, not measured)
total    : 11.30 GiB
device   : 12.00 GiB  margin +0.70 GiB
verdict  : fits at the declared allowance
</code></pre><p>Full output of the two-slot run, unedited (the verdict line goes to stderr, so it prints before the buffered stdout when the terminal interleaves them):</p>
<pre tabindex="0"><code>$ python3 plan.py --model gemma-4-31b-it --quant Q4_K_M --context 32768 --parallel 2 --vram 24
model    : gemma-4-31b-it (google/gemma-4-31B-it, native 262144 tokens, hybrid)
weights  : 17.07 GiB  (Q4_K_M GGUF file)
kv       : 6.56 GiB  (f16, 81,920 B/token x 32768 + 0.78 GiB fixed window per slot, x2 slots)
overhead : 1.00 GiB  (declared allowance, not measured)
total    : 24.63 GiB
device   : 24.00 GiB  margin -0.63 GiB
verdict  : DOES NOT FIT - lower context, quantise the KV cache, or reduce parallel slots
</code></pre><p>Two slots on the hybrid model is the difference between fitting 24 GB and not: one slot totals 21.35 GiB, two slots 24.63 GiB.</p>
<p>Files, first-party and published with the article:</p>
<table>
	<thead>
			<tr>
					<th>File</th>
					<th>What it is</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-sizing-2026/plan.py">plan.py</a></td>
					<td>The planner: weights + KV + declared overhead, hybrid-aware, non-zero exit on overflow</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-sizing-2026/plan-fixtures.json">plan-fixtures.json</a></td>
					<td>The four configs and their published GGUF sizes, with the Qwen3-8B revision pinned</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-sizing-2026/test_plan.py">test_plan.py</a></td>
					<td>Six tests (14 assertions) pinning 147,456 B/token, 5.62 GiB at 40,960, the 4× slot multiplier and the hybrid fixed term</td>
			</tr>
	</tbody>
</table>
<p>To add your own model, copy a config object into the fixtures file. The planner reads six fixture fields and nothing else; <code>num_hidden_layers</code>, <code>num_key_value_heads</code>, <code>head_dim</code> and <code>gguf_gib</code> drive the arithmetic, and the other two are for display:</p>
<ul>
<li><code>num_hidden_layers</code>, <code>num_key_value_heads</code>, <code>head_dim</code> — from the model&rsquo;s <code>config.json</code></li>
<li><code>max_position_embeddings</code> — from <code>config.json</code>; printed in the header and used only to cap the &ldquo;fits&rdquo; search</li>
<li><code>gguf_gib</code> — an object of <code>{&quot;Q4_K_M&quot;: 4.68, ...}</code>, one entry per quant you intend to plan for, each value being the published file size in bytes divided by 1024³</li>
<li><code>repo</code> — the model id, printed in the header; if you omit it the printer says <code>repo not recorded</code> instead of failing</li>
</ul>
<p><code>hybrid</code> is the only optional block, and it needs all five of <code>full_layers</code>, <code>sliding_layers</code>, <code>window</code>, <code>global_head_dim</code>, <code>global_kv_heads</code>. Add it only if the config carries <code>layer_types</code> or <code>sliding_window</code>: the mere presence of the key switches the model onto the hybrid branch, so a <code>hybrid</code> block invented for a dense model produces a wrong total rather than a denser one. <code>revision</code> is optional bookkeeping, never read by the arithmetic.</p>
<p><code>plan.py --list</code> shows what loaded, and <code>python3 plan.py --model &lt;name&gt; --quant &lt;quant&gt; --context &lt;n&gt; --vram &lt;GiB&gt;</code> prints the breakdown and exits 1 when the total exceeds the card.</p>
<h2 id="what-this-page-does-not-establish">What this page does not establish</h2>
<p>Every number above is arithmetic from published configs and file sizes, not an observed successful load: no GPU was available for this run. The formula itself was validated on a CPU-only host against a 0.5B model in the budget article — 11.98 KiB per token measured against 12.00 computed, and 6.37 against 6.38 for q8_0 with Flash Attention — which validates the arithmetic, not GPU behaviour, and not any large model.</p>
<p>Also out of scope: throughput, quality, tokens per second, vendor performance claims, whether a given runtime honours <code>layer_types</code> rather than allocating a uniform cache per layer, and vision projectors, which ship as separate <code>mmproj</code> files in the Gemma repositories and are not in any figure here.</p>
<p>Two gated repositories stayed gated: <a href="https://huggingface.co/api/models/meta-llama/Llama-3.3-70B-Instruct">meta-llama/Llama-3.3-70B-Instruct</a> reports <code>gated: manual</code> and both it and <code>google/gemma-3-27b-it</code> returned 401 for <code>config.json</code> from this host. Neither has a per-token figure here, and neither appears in any table above; the 70B is named only because it was the fifth repo the fixture plan asked for and could not be measured the same way.</p>
<p>If your runtime reports a served context shorter than the one you requested, that is a different failure from the one this page predicts — <a href="https://baeseokjae.github.io/posts/ollama-silent-context-truncation-2026/">the truncation diagnosis</a> and <a href="https://baeseokjae.github.io/posts/llama-server-context-overflow-diagnosis-2026/">the overflow diagnosis</a> cover it. Once the model fits, <a href="https://baeseokjae.github.io/posts/local-llm-rig-cost-payback-2026/">the payback model</a> turns a throughput number into a purchase decision, and <a href="https://baeseokjae.github.io/posts/best-local-llm-models-2026/">the model-list page</a> is the place to choose between these four.</p>
]]></content:encoded></item><item><title>Your 18GB GGUF Is Not an 18GB Budget: Sizing KV Cache and Context for a Local Coding Model</title><link>https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/</link><pubDate>Sun, 01 Nov 2026 13:00:00 +0000</pubDate><guid>https://baeseokjae.github.io/posts/local-llm-vram-kv-cache-budget-2026/</guid><description>A GGUF file is the weights, not the budget. Run the KV-cache formula before you download, with a measured 0.5B check against it on a 12 GB CPU host and the q8_0 setting that saves nothing without Flash Attention.</description><content:encoded><![CDATA[<p>You download a 4-bit quantised model, the file is roughly 18 GB, you have 24 GB of VRAM, and the arithmetic looks settled. Then the agent session gets long, the server either dies or quietly starts answering with a fraction of the context you asked for, and the model that &ldquo;fit yesterday&rdquo; no longer fits.</p>
<p>The missing term is the key/value cache. It is not in the GGUF file, it is not on the model card, and it is allocated by the runtime at load time from the context length you set — not from the number of tokens you happen to send. On the 12 GB CPU box used for the measurements below, the same 0.5B model occupied 417 MB of reported model memory at a 2,048-token context and 794 MB at 32,768, with an identical file on disk.</p>
<p>This article gives you one formula to run before downloading, one measured check of that formula that you can repeat, and one warning about a setting that looks like it halves memory and does not.</p>
<h2 id="what-the-cache-is-in-one-paragraph">What the cache is, in one paragraph</h2>
<p>Every transformer layer keeps, for every token in context, one key vector and one value vector per <em>key/value head</em>. The attention heads read them; they are not learned weights, they are state. The size is fixed by the model geometry and the cache element type:</p>
<pre tabindex="0"><code>bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
</code></pre><p>The 2 is key plus value. <code>n_layers</code> and <code>n_kv_heads</code> come from the model&rsquo;s own <code>config.json</code>. <code>head_dim</code> is either an explicit field or <code>hidden_size / num_attention_heads</code>. <code>bytes_per_element</code> is 2 for f16 and about 1.0625 for q8_0, because q8_0 stores 32 values in 34 bytes — 8 bits each plus a scale.</p>
<p>Grouped-query attention is why this number is not as frightening as it looks. A model can have 64 attention heads and only 8 key/value heads; the cache scales with the second number. That is also why the per-token cost cannot be guessed from parameter count alone: two models of the same size can differ by 8x in cache cost depending on how aggressively they share KV heads.</p>
<p>Ollama&rsquo;s own documentation fixes the default that catches people: &ldquo;By default, Ollama uses a context window size of 4096 tokens&rdquo;, overridable with <code>OLLAMA_CONTEXT_LENGTH</code> or per request with <code>num_ctx</code> (<a href="https://docs.ollama.com/faq">Ollama FAQ</a>). A coding agent session with file contents and tool output routinely exceeds that. Either the runtime truncates, or you raise the context and the cache grows with it.</p>
<h2 id="worked-example-one-the-small-model-measured">Worked example one: the small model, measured</h2>
<p>The tag used here is <code>qwen2.5:0.5b</code> at Q4_K_M. Its own config gives <code>num_hidden_layers</code> 24, <code>num_key_value_heads</code> 2, <code>num_attention_heads</code> 14 and <code>hidden_size</code> 896, so <code>head_dim</code> is 896 / 14 = 64 (<a href="https://huggingface.co/Qwen/Qwen2.5-0.5B/raw/main/config.json">Qwen2.5-0.5B config.json</a>).</p>
<p>That yields 2 × 24 × 2 × 64 × 2 = 12,288 bytes per token, or 12.00 KiB. At a 32,768-token context the cache alone is 384 MiB — comparable to the model itself, on a model whose file is about 400 MB.</p>
<p>This is the shape of the problem in miniature: at a context the model advertises, the cache can be the same order of magnitude as the weights. Nobody notices on a 0.5B model because 384 MiB is noise. Scale the geometry up and the same term decides whether the card is big enough.</p>
<h2 id="worked-example-two-a-32b-model-computed-not-measured">Worked example two: a 32B model, computed not measured</h2>
<p>Take Qwen3-32B, whose config was fetched and read for this article: 64 layers, 64 attention heads, 8 key/value heads, <code>head_dim</code> 128, 40,960 maximum positions (<a href="https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json">Qwen3-32B config.json</a>).</p>
<p>2 × 64 × 8 × 128 × 2 = 262,144 bytes per token = 256 KiB for f16. With q8_0 packing at 34 bytes per 32 elements the same geometry costs 136 KiB per token.</p>
<table>
	<thead>
			<tr>
					<th>Context</th>
					<th>f16 cache</th>
					<th>q8_0 cache (with Flash Attention)</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>8,192 tokens</td>
					<td>2 GiB</td>
					<td>1.06 GiB</td>
			</tr>
			<tr>
					<td>32,768 tokens</td>
					<td>8 GiB</td>
					<td>4.25 GiB</td>
			</tr>
			<tr>
					<td>40,960 tokens</td>
					<td>10 GiB</td>
					<td>5.31 GiB</td>
			</tr>
	</tbody>
</table>
<p>Every number in that table is <strong>computed</strong> from the config and the formula. None of it was measured on a 32B model — this host has no GPU and 12 GB of RAM. It is arithmetic you can redo in a spreadsheet in a minute, and that is the point: the term is predictable before you download anything.</p>
<p>Note what changed the number: not the parameter count, but 8 KV heads instead of 64, and a quantised cache. A dense 32B model without grouped-query attention would have cost eight times more per token and would not fit at that context on any consumer card.</p>
<h2 id="the-measured-check-on-this-machine">The measured check on this machine</h2>
<p>The probe starts a private Ollama server, forces one load per cell at a requested <code>num_ctx</code>, reads the runner&rsquo;s <code>VmRSS</code> from <code>/proc/&lt;pid&gt;/status</code>, and confirms what the server actually honoured with <code>ollama ps</code>. Nothing else runs, so the only moving part is the cache allocation.</p>
<table>
	<thead>
			<tr>
					<th>num_ctx</th>
					<th>KV type</th>
					<th>Flash Attention</th>
					<th>runner VmRSS</th>
					<th><code>ollama ps</code> SIZE</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>2,048</td>
					<td>f16</td>
					<td>off</td>
					<td>494,748 kB</td>
					<td>417 MB</td>
			</tr>
			<tr>
					<td>8,192</td>
					<td>f16</td>
					<td>off</td>
					<td>567,752 kB</td>
					<td>492 MB</td>
			</tr>
			<tr>
					<td>32,768</td>
					<td>f16</td>
					<td>off</td>
					<td>862,668 kB</td>
					<td>794 MB</td>
			</tr>
			<tr>
					<td>2,048</td>
					<td>q8_0</td>
					<td>off</td>
					<td>494,220 kB</td>
					<td>417 MB</td>
			</tr>
			<tr>
					<td>8,192</td>
					<td>q8_0</td>
					<td>off</td>
					<td>568,064 kB</td>
					<td>492 MB</td>
			</tr>
			<tr>
					<td>32,768</td>
					<td>q8_0</td>
					<td>off</td>
					<td>862,896 kB</td>
					<td>794 MB</td>
			</tr>
			<tr>
					<td>8,192</td>
					<td>q8_0</td>
					<td>on</td>
					<td>521,648 kB</td>
					<td>442 MB</td>
			</tr>
			<tr>
					<td>32,768</td>
					<td>q8_0</td>
					<td>on</td>
					<td>678,308 kB</td>
					<td>593 MB</td>
			</tr>
	</tbody>
</table>
<p>The reported context matched the requested context in all eight rows, so no row is secretly a measurement of a different context.</p>
<p>From 2,048 to 32,768 tokens, f16 grew by 367,920 kB over 30,720 tokens: 11.98 KiB per token against 12.00 computed, an error of minus 0.20 percent. Subtracting the slope leaves a base of about 470 MB that barely moves across contexts (720 kB spread on eight rows), which is the Q4_K_M weights plus graph and allocator overhead. The slope is the portable part; the base is not.</p>
<h2 id="the-trap-a-setting-that-looks-like-half-the-memory">The trap: a setting that looks like half the memory</h2>
<p><code>OLLAMA_KV_CACHE_TYPE=q8_0</code> with default settings produced the same slope as f16: 12.00 KiB per token, 228 kB apart at 32,768 tokens, which is noise. The knob did nothing.</p>
<p>Ollama&rsquo;s documentation says why, in the sentence most people skim: the K/V cache &ldquo;can be quantized to significantly reduce memory usage <strong>when Flash Attention is enabled</strong>&rdquo;. Re-running the same two cells with <code>OLLAMA_FLASH_ATTENTION=1</code> halved the slope: 156,660 kB over 24,576 tokens is 6.37 KiB per token against 6.38 computed, and 0.532x the f16 slope. The saving is real — but only after the precondition is met.</p>
<p>Two practical consequences. First, verify a memory setting by its effect on resident memory, not by the name of the flag; a quantised cache type on a backend that cannot run Flash Attention is inert, and you will only find out when the box swaps. Second, compare slopes, not single readings: the base also moved with Flash Attention (567,752 kB to 521,648 kB at 8,192 tokens), so a one-point comparison would have been misleading in the opposite direction.</p>
<h2 id="turning-it-into-a-budget">Turning it into a budget</h2>
<p>The budget is three terms, and only one of them is on the download page:</p>
<pre tabindex="0"><code>resident ≈ weights (GGUF bytes) + KV cache at your context + runtime overhead
</code></pre><p>Work the numbers in the same unit before you buy anything. For a 32B-class model at a 32,768-token context, the cache term alone is 8 GiB at f16 or 4.25 GiB at q8_0 with Flash Attention. If your file is the 18 GB the title refers to — a figure this article takes from the reader&rsquo;s rule of thumb and does <strong>not</strong> verify for any specific 31B release — then 18 + 8 + 2 GB of overhead is 28 GB, which does not fit a 24 GB card at full context. The same model with a quantised cache lands near 24 GB: still on the edge, and still not something to plan around without measuring your own box.</p>
<p>Three corrections that change the answer more often than a bigger GPU:</p>
<ul>
<li><strong>Parallel slots multiply the cache.</strong> <code>OLLAMA_NUM_PARALLEL</code> defaults to 1, and the docs state that required memory scales with it. A shared server sized for one user&rsquo;s context is not sized for four.</li>
<li><strong>The context default is 4,096, not the model&rsquo;s maximum.</strong> A setup that &ldquo;fit yesterday&rdquo; may have fit because the runtime was quietly using a small context. Raising <code>num_ctx</code> to the model&rsquo;s <a href="https://baeseokjae.github.io/posts/llm-context-window-comparison-2026/">advertised maximum</a> is where the memory goes.</li>
<li><strong>Quantising the cache is not free of preconditions.</strong> It needs Flash Attention, which needs a backend and device that support it. Check the effect, not the flag.</li>
</ul>
<p>When the total does not fit, the levers in order of cost are: lower the context to what the agent actually uses, quantise the cache if Flash Attention is available, reduce parallel slots, and only then move to a smaller model.</p>
<h2 id="doing-it-on-your-own-machine-in-ten-minutes">Doing it on your own machine in ten minutes</h2>
<p>The probe is small enough to read before you run it. It starts its own server on a free loopback port so it does not disturb a running instance, samples the runner&rsquo;s resident set after each load, records what <code>ollama ps</code> reports as the honoured context, and writes one CSV row per cell.</p>
<p>Files, all first-party and published with the article:</p>
<table>
	<thead>
			<tr>
					<th>File</th>
					<th>What it is</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-budget-2026/scripts/kv_budget_probe.py">kv_budget_probe.py</a></td>
					<td>The probe: context and cache-type matrix, RSS sampling, <code>ollama ps</code> cross-check</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-budget-2026/kv_budget_prompt.txt">kv_budget_prompt.txt</a></td>
					<td>The fixed 30-token fixture with a head and a tail token, kept for replay and hashing; the probe sends a 1-token generation because the allocation happens at load</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-budget-2026/kv-budget-2026-11-01.csv">kv-budget-2026-11-01.csv</a></td>
					<td>The eight measured rows behind the table above</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-budget-2026/kv-budget-formula-check.md">kv-budget-formula-check.md</a></td>
					<td>Measured slope against the config-derived slope, with the residual explained</td>
			</tr>
			<tr>
					<td><a href="https://raw.githubusercontent.com/baeseokjae/baeseokjae.github.io/main/research/evidence/local-llm-vram-kv-cache-budget-2026/method.md">method.md</a></td>
					<td>Server flags, sampling points, host, and what was deliberately not measured</td>
			</tr>
	</tbody>
</table>
<p>Point it at a model that fits your box. Two cells are enough to get your own slope: the same cache type at two contexts, resident memory at each, difference divided by the context difference. If your slope is within a quarter of the formula, you can trust the arithmetic for the model you actually want; if it is not, the runtime is allocating something the formula does not describe and the formula&rsquo;s answer for a bigger model is not safe to use.</p>
<h2 id="what-this-does-and-does-not-establish">What this does and does not establish</h2>
<p>Measured, on one CPU-only host with one model tag: the cache is allocated at load from <code>num_ctx</code>, it grows linearly at the formula&rsquo;s rate within 0.20 percent for f16 and 0.01 percent for q8_0 with Flash Attention, and the q8_0 setting is inert without Flash Attention.</p>
<p>Not measured, and not claimed: any throughput or tokens-per-second figure, any quality comparison, any GPU-backend behaviour, any multi-slot configuration, and any resident-memory number for a 32B or 27B model. The 32B figures are computed from its published config, and the 27B family is absent on purpose: the <code>google/gemma-3-27b-it</code> config returned HTTP 401 on both attempts, so its geometry is unverified here and no number for it appears in this article. One sample per cell was taken; repeat-run spread between identical configurations was 228 kB against a 368 MB slope, which is enough for a slope and not enough for sub-percent claims about the constant overhead.</p>
<p>The one-sentence version: the file you download is the smallest term in the budget, and the term that grows is the one nobody puts on the model card. Compute it before you download, then confirm it on your own machine with two loads.</p>
<p>If you have not chosen the model yet, <a href="https://baeseokjae.github.io/posts/best-local-llm-models-2026/">the model-selection comparison</a> and <a href="https://baeseokjae.github.io/posts/best-ollama-models-coding-2026/">the coding-model shortlist</a> are the better starting points; <a href="https://baeseokjae.github.io/posts/llm-inference-engineering-guide/">the inference-engineering guide</a> explains PagedAttention and serving-side batching, which this article deliberately does not re-explain; and <a href="https://baeseokjae.github.io/posts/gemma-4-local-setup-guide-2026/">the 31B local setup guide</a> covers the download half of the question.</p>
]]></content:encoded></item></channel></rss>