Why one token per pass is the enemy
Recall the wall: each decode step reads the whole model just to emit one token, so at small batch you are paying a full memory sweep per token. But that memory sweep can verify several candidate tokens almost as cheaply as one — the extra arithmetic is nearly free because you were memory-bound, not compute-bound. The entire idea of speculation is to exploit this slack: propose multiple future tokens cheaply, then confirm them in a single expensive pass.
Diagram of an autoregressive loop feeding each generated token back in as input for the next pass.
Draft-then-verify: the core algorithm
Speculative decoding pairs the large target model with a small fast draft model. The draft proposes a short run of, say, four tokens autoregressively; the target then runs one forward pass that scores all four positions in parallel. A clever acceptance rule keeps the longest prefix of drafted tokens that the target would have produced anyway and resamples at the first disagreement.
Speculative sampling accepts each drafted token with probability min(1, p/q) and resamples from the residual on rejection — making the output distribution exactly the target's.
The catch is sourcing a good draft model: it must be cheap to run yet predict the target well. Distilling a small model from the target (recall knowledge distillation) is one route, but a separate draft adds memory and operational complexity. That tension motivates everything that follows.
Self-drafting: Medusa, EAGLE, and skipping layers
What if the target drafts for itself? Medusa bolts several extra decoding heads onto the target so that, from one hidden state, it predicts the next token plus a few subsequent ones in parallel; tree-structured candidates from these heads are then verified together. EAGLE goes a level deeper, drafting at the feature (hidden-state) level rather than the token level, which makes its guesses far more accurate and yields some of the highest acceptance rates in the family.
A third variant skips the extra parameters entirely. Self-speculative decoding uses a subset of the target's own layers — an early-exit shallow pass — as the draft, then verifies with the full depth. One set of weights, no separate model, no extra heads to train; the draft is just the model running cheaper.
Draft-free: parallelizing with n-grams
Lookahead decoding removes the draft model altogether. It runs a parallel Jacobi-style iteration that maintains a window of guessed future tokens and simultaneously collects verified n-grams from the sequence so far; each step both refines the guesses and checks them, breaking the strict sequential dependency without any auxiliary network. It is purely algorithmic — attractive when you cannot or do not want to train or host extra parameters.
Multi-token prediction, and when speculation pays
The cleanest fix is to train the model to predict ahead from the start. Multi-token prediction adds auxiliary objectives during pretraining so each position predicts the next several tokens, not just one. This improves the model itself and provides ready-made, high-quality draft heads at inference time — self-speculation that came free with training. Several frontier models now ship with such heads built in.
Expected tokens generated per target pass for acceptance rate α over γ drafted tokens — speedup grows with the acceptance rate, explaining why speculation shines on predictable text.