speculative decoding

Autoregressive generation is slow because each token needs a full forward pass through a huge model, and those passes are memory-bound, not compute-bound — the GPU spends most of its time waiting on the weights, not multiplying. Speculative decoding exploits that idle compute. A cheap draft model guesses a short run of future tokens, and then the large target model checks all of them in a single batched forward pass. Easy tokens, which a small model already predicts correctly, get accepted for almost free; only the hard tokens force the big model to truly decide.

The trick is that verification is exact, not approximate. Given a draft of γ tokens, the target runs one parallel pass, and a modified rejection-sampling rule accepts each drafted token with a probability that guarantees the output distribution is identical to sampling from the target alone. On the first rejection the algorithm resamples that position from an adjusted residual distribution and discards the rest of the draft. So you pay one big forward pass per accepted block instead of per token, and the quality is provably unchanged.

Speedup is governed by the acceptance rate α and the draft length γ: a well-matched draft yields two-to-three times fewer target calls. The cost is a second model to host and a tuning knob — too long a draft wastes verification when acceptance is low. It is now the default acceleration in most production serving stacks.