chunked prefill

LLM serving has two very different kinds of work. Prefill ingests the whole prompt at once and is compute-bound, easily saturating the GPU's math units. Decode produces one token at a time and is memory-bound, leaving those math units largely idle. When a long prompt arrives, its single giant prefill monopolizes the GPU for a while, and every other request that is mid-generation stalls — a sudden spike in their time-per-output-token that users feel as stutter. Chunked prefill removes the spike by never running a whole prefill at once.

It splits a long prompt into fixed-size chunks and processes one chunk per scheduler iteration, interleaving those chunks with the ongoing decode steps of other requests in the same batch. Each iteration therefore carries a controlled amount of prefill compute piggybacked onto memory-bound decodes, which conveniently fills the idle math units and keeps the GPU near full utilization. The result is smoother, more predictable inter-token latency for everyone, at the price of slightly delaying any single prompt's first token.

Chunk size is the central knob: larger chunks finish prefill faster but reintroduce decode jitter, smaller chunks smooth latency but add scheduling overhead.