disaggregated prefill/decode

Prefill and decode have opposite hardware appetites — prefill is compute-bound and bursty, decode is memory-bandwidth-bound and steady — yet when they share the same GPUs they interfere: a big prefill spikes the latency of in-flight decodes, and decode underuses the math units. Even chunked prefill only smooths this interference rather than removing it. Disaggregation removes it by physically separating the two phases onto distinct pools of machines, each tuned and scaled for its own bottleneck.

A request first lands on a prefill worker that ingests the prompt and computes its KV cache; that cache is then transferred — typically over a fast interconnect such as NVLink or RDMA — to a decode worker that streams out the tokens. The two pools scale independently, so an operator can add prefill capacity for prompt-heavy traffic or decode capacity for long-generation traffic without overprovisioning the other. Each phase can also run with its own parallelism strategy and batch policy, and decode latency stops being hostage to prefill spikes.

The cost is the KV-transfer overhead and added system complexity, so disaggregation pays off mainly at large scale where strict, separate latency targets justify it.