JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The scheduler's view: continuous batching, disaggregation, and parallelism

A serving engine is really a token-level scheduler. See how continuous batching, prefill/decode disaggregation, tensor-parallel sharding, and streaming combine to keep an expensive GPU saturated without breaking anyone's latency SLO.

Static batching wastes the GPU

The simplest form of request batching is static: collect N requests, run them together until all are done, then start the next batch. This breaks badly for generation because requests finish at wildly different lengths. A batch of N is only as fast as its longest sequence; once the short ones are done, their slots sit idle producing padding while the GPU waits for the straggler. Utilization collapses precisely when you wanted efficiency.

U=\frac{\sum_{i=1}^{N} L_i}{N\,\max_i L_i}\le 1

Static batching wastes the GPU: the batch is held until the longest sequence finishes, so utilization U falls far below 1 when generation lengths vary.

Continuous (in-flight) batching

Continuous batching, also called in-flight batching, makes batch membership dynamic at the granularity of a single decode step. After each iteration the scheduler retires any sequences that just emitted an end-of-sequence token, frees their KV blocks, and admits waiting requests into the freed slots — immediately, without waiting for the whole batch to drain.

  1. Run one forward step over every sequence currently in the batch.
  2. Sample the next token for each; append it to that request's output stream.
  3. Retire any sequence that hit its stop condition and free its KV blocks.
  4. Admit queued requests (their prefill, possibly chunked) into the freed slots.
  5. Repeat. The batch is never artificially held back by its longest member.

Combined with chunked prefill, this gives the scheduler a unified queue of work where prefill chunks and decode steps coexist in the same iteration. That single design choice is responsible for most of the throughput gap between a research-grade generate loop and a production engine.

Disaggregating prefill from decode

Even with chunking, prefill and decode have opposite hardware profiles — prefill is compute-bound, decode is memory-bound — so forcing them onto the same GPUs means neither runs at its best, and a burst of prefill can still jitter decode latency. Disaggregated prefill/decode runs them on separate pools of GPUs: prefill workers compute the prompt's KV cache, ship it across the interconnect to decode workers, which then stream the output.

\underbrace{t_{\text{decode}}\approx\frac{2\,N_{\text{params}}}{\mathrm{BW}_{\text{HBM}}}}_{\text{memory-bound}}\qquad\underbrace{t_{\text{prefill}}\approx\frac{2\,N_{\text{params}}\,L_{\text{prompt}}}{\mathrm{FLOPs}_{\text{peak}}}}_{\text{compute-bound}}

Why disaggregate: a decode step is memory-bound (read every weight per token) while prefill is compute-bound — opposite hardware profiles that want different GPUs.

Sharding the model: tensor-parallel serving

When a model is too large for one GPU, or when you need lower latency than one GPU can give, you shard it. Tensor-parallel inference splits each weight matrix across GPUs so every device holds a slice of every layer and they cooperate on each token, exchanging partial results through an all-reduce. This is the inference-time use of the tensor parallelism you met in the training track, but the trade-offs differ: at serving time the all-reduce sits on the critical path of every decode step, so the interconnect bandwidth directly bounds your TPOT.

Tensor-parallel serving shards attention by head across GPUs — each device runs parallel heads, then an all-reduce concatenates and mixes them.

Diagram of multi-head attention with parallel heads concatenated and mixed.

The practical guidance: tensor parallelism cuts latency and unlocks big models, but past a point the communication overhead eats the gains. Within a single fast-interconnect node it is the default lever; across nodes you reach for other forms of parallelism or disaggregation instead.

t(p)\approx\frac{T_{\text{compute}}}{p}+\underbrace{2\,\frac{p-1}{p}\cdot\frac{M_{\text{act}}}{\mathrm{BW}_{\text{link}}}}_{\text{all-reduce per layer}}

Tensor parallelism over p GPUs cuts compute by p but adds a per-layer all-reduce; past a point that communication term dominates and latency stops improving.

Streaming tokens back

Finally, the user experience. Because decode produces tokens one at a time anyway, token streaming sends each token to the client the moment it is sampled rather than waiting for the full response. This decouples perceived latency from total latency: a user reading along feels the system is fast even if the full answer takes many seconds, because the relevant metric becomes TTFT plus a comfortable TPOT, exactly the metrics from guide one.