KV cache

Attention lets each new token look back at every earlier token by comparing a query against the keys and values of the history. Without caching, every decode step would recompute the keys and values for the entire past — pure waste, since those vectors never change once a token is fixed. The KV cache simply stores them.

After prefill, the model keeps the key and value vectors for every layer and every past token in GPU memory. To generate the next token it computes only the new token's query, key, and value, then attends against the cached K and V. This turns the per-step cost from quadratic in sequence length into linear, and is the single biggest reason interactive generation is affordable at all — paid for with memory that grows with every token you keep in context.