Prompt-Prefix Cache in a Local Agent Loop: One Mutable Line at the Top Defeats Prefix Reuse (2026)
Two consecutive turns of a local agent loop can differ by more than an order of magnitude in prefill time while the model, the output length and - within 31 tokens - the prompt size stay nearly the same. On the host used for this article, an append-only second turn prefilled 962 tokens in 2.4711 s. The same second turn, with one line regenerated at the top of the system block, prefilled 993 tokens in 36.8326 s. Nothing errored, nothing was truncated, and no tokens-per-second figure on either turn would have told you which one you were looking at. ...