EAGLE speculative decoding

EAGLE sharpens speculative decoding by drafting in the right space. Earlier draft models guess raw tokens, which are discrete and noisy, so acceptance rates suffer. EAGLE instead predicts the target model's own second-to-last-layer feature vectors — the smooth, continuous hidden states just before the output projection — and only then maps them to tokens. Forecasting features is an easier regression problem than forecasting tokens, because the feature trajectory is more autoregressively predictable, which yields markedly higher acceptance.

The draft is a single small autoregressive head that takes both the previous feature and the previously sampled token as input, resolving the uncertainty that pure feature prediction would otherwise leave when the next token is genuinely ambiguous. It expands several candidate continuations into a tree and the target verifies the whole tree in one pass, exactly as in tree-based speculation, keeping the output distribution identical to the target. Later versions add dynamic draft trees whose shape adapts to per-step confidence.

Among lossless accelerators EAGLE reaches some of the highest reported speedups, often three times or more, which is why it has become a common default draft head in open serving frameworks.