continuous batching

Classic static batching groups a fixed set of requests, runs them together, and cannot return any result until the slowest one in the batch finishes generating. Because LLM outputs vary wildly in length, short replies are held hostage by long ones, and finished slots sit idle while the rest grind on. Throughput collapses and tail latency balloons. Continuous batching breaks this by operating at the granularity of a single decode step rather than a whole request.

After every token-generation iteration the scheduler revisits the running batch: any sequence that hit its stop condition is retired and its KV-cache slot freed, and any waiting request is admitted into the now-open slot for the next iteration. So the batch composition changes continuously, every step, and the GPU is kept saturated with useful work instead of padding. Combined with paged KV memory, which makes admitting and evicting sequences cheap, this is what lets a server sustain high throughput under a churning, mixed-length workload.

The remaining subtlety is balancing newly arriving prompts, which need a heavy prefill pass, against ongoing decodes that each need only one step — the motivation for chunked prefill and prefill-decode scheduling.