Inference & Serving

KV cache memory cost

People assume the model weights are what fill the GPU, but for long conversations the KV cache can dwarf them. Every token you keep in context carries a key and a value vector in every layer, and that pile grows linearly with the context length and with the number of concurrent users — two multipliers that get large fast.

Roughly, the cache size is two (key and value) times the number of layers, times the number of key-value heads, times the head dimension, times the number of tokens, times the bytes per number — summed across all active requests. A long context or a high batch size multiplies this quickly, and when it overflows VRAM the server must evict tokens or reject requests. This is why long context windows are genuinely expensive to serve, and why grouped-query attention, KV-cache quantization, and paging all exist: they each attack this one growing cost.

\text{KV bytes} \approx 2 \cdot n_{\text{layers}} \cdot n_{\text{kv-heads}} \cdot d_{\text{head}} \cdot T \cdot b

Per request: T tokens in context, b bytes per number; multiply by batch size for the whole server.