paged KV cache

The KV cache grows as text is generated, but you cannot know in advance how long an answer will be. Early systems reserved one big contiguous block per request, sized for the maximum possible length, and then wasted most of it. Paging borrows the operating system's trick of virtual memory to stop that waste.

Instead of one contiguous region, the cache is split into fixed-size blocks, or pages, that can live anywhere in GPU memory; a per-request block table maps logical token positions to physical pages. Memory is handed out a page at a time as the sequence grows, which nearly eliminates fragmentation and lets many more requests share the same GPU. As a bonus, requests with a shared prefix can point at the very same physical pages, saving both memory and the prefill compute that would have recreated them.