prefill vs. decode

Inference has two phases with very different personalities. Prefill reads the prompt — all of its tokens processed in parallel as one big matrix multiply, so it is compute-heavy. Decode then writes the answer token by token; each step is tiny in arithmetic but spends its time waiting on memory.

During prefill the GPU is genuinely busy (compute-bound): it crunches hundreds or thousands of prompt tokens together and fills the KV cache in one go. During decode it handles just one token per step, so the math is trivial, but it must re-read the whole cache and all the weights from memory every single step — making decode memory-bandwidth-bound. This split is why a long prompt with a short answer feels fast, while a short prompt with a long answer is slow per token, and why the two phases are often scheduled and even hardware-placed differently.