prefix caching (inference)

Many requests to a deployed model share an identical opening: the same long system prompt, the same few-shot examples, the same retrieved document, or the same conversation history on every follow-up turn. Recomputing the KV cache for that shared prefix on every request is pure waste, because attention over a fixed prefix produces identical keys and values no matter what follows. Prefix caching keeps the computed KV for common prefixes around and reuses it, so a new request that begins with a seen prefix skips straight to the novel suffix.

Implementations hash prompt content into blocks and key a shared cache by those hashes; thanks to paged KV memory, multiple sequences simply point their block tables at the same physical pages, with copy-on-write the moment their continuations diverge. The payoff is concentrated in the prefill phase: time-to-first-token drops sharply for long shared contexts, and the GPU is spared redundant compute. In multi-turn chat the entire prior conversation becomes a free prefix on each new turn.

The design questions are eviction policy under memory pressure and exactness — cache hits must be byte-identical prefixes, or the reused KV would be silently wrong.