continuous batching

Requests arrive at different moments and finish at different moments. Classic static batching gathers a fixed group, runs them together, and refuses to admit anyone new until the slowest member finishes — so fast requests are held hostage by slow ones and the GPU sits half-idle whenever sequences complete early.

Continuous batching schedules at the granularity of a single decode step instead. After every step, finished sequences leave the batch and freshly arrived ones join immediately, so the GPU stays as full as the workload allows. Paired with a paged KV cache, this lifts throughput dramatically on real, bursty traffic without forcing any one request to wait for a batch to form. It is the core scheduling idea behind essentially every modern serving engine.