lookahead decoding

Lookahead decoding accelerates a model without any draft model or extra trained heads at all. It reframes autoregressive generation as solving a system of equations: instead of producing tokens strictly one after another, it makes parallel guesses at several future positions at once and iteratively refines them, a fixed-point iteration in the spirit of the Jacobi method. Each step nudges a window of tokens toward the values the model would eventually settle on, breaking the strict left-to-right dependency that normally serializes decoding.

Concretely it runs two interleaved branches in each forward pass. A lookahead branch generates and collects short n-grams from the parallel guesses, building up a pool of plausible continuations as a side effect of the iterations. A verification branch then checks candidate n-grams from that pool against the model's true next-token distribution, accepting any that match exactly. As generation proceeds the n-gram pool grows richer, acceptance rises, and more tokens per step come for free — all while provably emitting the same sequence greedy decoding would.

The appeal is zero training and zero extra models; the cost is more compute per step, so it pays off mainly when the GPU has spare FLOPs relative to memory bandwidth.