Medusa decoding

Medusa is a form of self-speculation that avoids hosting a separate draft model. Instead it bolts several small extra prediction heads onto the frozen backbone of the target model — head one predicts the token two steps ahead, head two the token three steps ahead, and so on. Each head is a lightweight feed-forward layer trained on top of the model's last hidden state, so a single forward pass now emits not just the next token but a fan of candidate continuations several positions deep.

Those candidates are assembled into a small tree of possible token sequences, and a tree-structured attention mask lets the target verify every path in the tree within one batched forward pass, accepting the longest prefix that matches what the model itself would have sampled. Because the heads and backbone share the same weights and features, training is cheap — you freeze the base model and fit only the heads — and there is no second model to deploy, schedule, or keep memory-resident. The acceptance is still exact, preserving the original output distribution.

Medusa typically delivers a roughly two-times speedup with minimal engineering overhead, trading a little extra compute per step and head-training for the convenience of a single self-contained model.