self-speculative decoding

Self-speculative decoding gets the benefits of speculative decoding from a single model, with no separate draft network and no extra trained heads. The idea is to draft using a deliberately weakened version of the model itself — typically by running only its first few layers and skipping the rest — producing fast, approximate tokens. The same full model then verifies those drafted tokens with one complete forward pass, accepting the prefix it agrees with, exactly as in standard speculation, so the output distribution is unchanged.

Because the draft and the target share all weights, the draft costs only the compute of the early layers and reuses the same KV cache and memory, which sidesteps the deployment burden of hosting two models. Approaches such as LayerSkip train the network to be robust to dropping later layers and to exit early, so its shallow sub-network is already a competent drafter; the verification then runs the remaining layers only on the candidate tokens. The whole scheme fits inside one model's memory footprint.

It trades a small drop in acceptance rate, since a partial model drafts less accurately than a dedicated one, for the simplicity of having nothing extra to deploy.