KV cache quantization

At long context lengths the KV cache, not the model weights, becomes the dominant consumer of GPU memory: it grows linearly with sequence length, batch size, layers, and heads, and at tens of thousands of tokens it can dwarf the parameters themselves. KV cache quantization stores those cached key and value tensors in a low-precision format — often eight or even four bits instead of sixteen — so the same hardware holds far longer contexts or far larger batches. Because the cache is read on every single decode step, shrinking it also reduces the memory traffic that bottlenecks generation.

The catch is that keys and values have very different statistics, and a few channels carry large-magnitude outliers that dominate error if quantized naively. Effective methods therefore quantize keys per-channel and values per-token, with small group sizes and per-group scales, and sometimes keep the most recent tokens in full precision since errors there propagate most. Unlike weight quantization, which is done once offline, KV quantization happens on the fly during generation, so the encode and decode must be cheap enough not to erase the bandwidth they save.

Done well, four-bit KV caches roughly double the servable context with negligible quality loss; pushed too far, they degrade long-range recall and reasoning.