speculative decoding

Generating one token at a time is slow because each step has to re-read the whole big model from memory. Speculative decoding uses a small, cheap draft model to guess several next tokens quickly, then lets the big model check all the guesses in a single pass — like a fast typist drafting while a careful editor approves in bulk.

The draft model proposes k candidate tokens. The target model then runs once over the prompt plus those k tokens — a parallel forward pass, cheap relative to k separate steps — and computes its own probabilities. A verification rule accepts the longest prefix of guesses that the target would itself have produced and corrects the first mismatch. Because a forward pass is memory-bound, checking k tokens costs almost the same as generating one, so when the draft is often right you get a two-to-three-times speedup with a provably identical output distribution.