Local LLM VRAM Planning 2026: The KV Cache Math Behind the Context You Can Actually Serve

A model card says 40,960 tokens. The Q4_K_M file is 4.68 GiB. The card has 12 GB. You set -c 40960, and one of three things happens: the load fails, the runtime spills layers to CPU and tokens per second collapse, or the server starts and silently serves a shorter context than you asked for. Which of the three you get depends on a number that appears in none of those three places — the key/value cache the runtime allocates at load time from the context you request. This page computes it for four specific models from their own config.json at a pinned revision, turns it into a “largest context that fits this card” table, and ships a planner you can run offline. ...

November 3, 2026 · 10 min · baeseokjae

Your 18GB GGUF Is Not an 18GB Budget: Sizing KV Cache and Context for a Local Coding Model

You download a 4-bit quantised model, the file is roughly 18 GB, you have 24 GB of VRAM, and the arithmetic looks settled. Then the agent session gets long, the server either dies or quietly starts answering with a fraction of the context you asked for, and the model that “fit yesterday” no longer fits. The missing term is the key/value cache. It is not in the GGUF file, it is not on the model card, and it is allocated by the runtime at load time from the context length you set — not from the number of tokens you happen to send. On the 12 GB CPU box used for the measurements below, the same 0.5B model occupied 417 MB of reported model memory at a 2,048-token context and 794 MB at 32,768, with an identical file on disk. ...

November 1, 2026 · 9 min · baeseokjae