Why the cache is the bottleneck on concurrency
After weights, the KV cache is the largest claim on GPU memory, and unlike the fixed weights it grows with every token and every concurrent request. Its size is roughly layers × 2 × heads × head-dim × sequence-length × batch × bytes-per-element. For a long context window and many users this can dwarf the model itself, and it is the real reason you cannot just batch more requests to fill the GPU: you run out of cache memory long before you run out of compute.
KV-cache memory grows linearly in layers L, KV heads n_kv, head dim d_head, sequence length s and batch b — the factor 2 counts both keys and values, and p is bytes per element.
PagedAttention: virtual memory for the cache
Classic serving allocated one contiguous cache buffer per request, sized for the maximum possible length. That wastes enormous memory to internal fragmentation: a request that finishes early, or never reaches max length, leaves a hole nobody else can use. PagedAttention borrows the operating-system idea of virtual memory. It chops the cache into fixed-size blocks (pages) and lets a request's logical sequence map to non-contiguous physical blocks through a block table.
The payoff is near-zero fragmentation: blocks are allocated on demand and freed the instant a request ends, so memory utilization climbs from roughly half to well over ninety percent. Higher utilization means more requests fit, which means bigger batches, which — recall the wall from guide one — means higher throughput. PagedAttention is the foundation almost every modern engine is built on.
Prefix caching: never recompute a shared prefix
Many requests share a leading prefix — a long system prompt, a few-shot template, a retrieved document, a multi-turn chat history. Recomputing that prefill for every request is pure waste. Prefix caching keeps the KV blocks for already-seen prefixes and lets new requests attach to them instead of re-prefilling. Because PagedAttention already stores the cache in shareable blocks, two requests with the same prefix can simply point their block tables at the same physical pages.
Diagram of attention as a soft query–key–value lookup whose keys and values are what the cache reuses.
Chunked prefill: keeping latency steady
A long prompt creates a problem: prefilling it is one big compute-heavy burst that monopolizes the GPU and stalls every other request's decode, causing TPOT spikes. Chunked prefill slices a long prefill into smaller pieces and interleaves those pieces with ongoing decode steps in the same batch. No single iteration is dominated by one giant prefill, so decode latency stays smooth even when a huge prompt arrives.
Chunked prefill also gives the scheduler a continuous knob to balance prefill and decode work — a theme we develop fully in the next guide. The lesson here: prefill and decode compete for the same hardware, and good serving is largely about scheduling that competition gracefully.
KV-cache quantization: more context per byte
If memory bandwidth and capacity are the bottleneck, store the cache in fewer bits. KV-cache quantization keeps keys and values in 8-bit or even 4-bit instead of 16-bit, immediately halving or quartering the cache footprint and the bytes moved per decode step. That lets you fit longer contexts and larger batches on the same GPU.
Symmetric b-bit quantization stores each key/value as an integer sharing one scale s; dropping from 16 to 8 or 4 bits is what buys more context per byte.