LLM Inference Engineering: From KV Cache to PagedAttention
LLM inference engineering is the discipline of making large language models run fast, cheap, and reliably in production. It centers on two phases — prefill and decode — and on the KV cache, the memory structure that stores attention keys and values. The single biggest win in the field is PagedAttention, which manages the KV cache like an operating system manages virtual memory, cutting memory waste from up to 60–80% down to near zero and boosting serving throughput by up to 24x over naive implementations. ...