Inference & Serving

draft-token verification

Speculative decoding only helps if the guesses are checked in a way that never changes what the model would have said. Verification is that careful accept-or-reject step: it keeps the big model's output statistically exactly what it would have been had it generated every token on its own, so there is no quality loss to trade for the speed.

For each drafted token the target model has already produced a probability. The verifier accepts a draft token with probability equal to the smaller of one and the ratio of the target probability to the draft probability; on the first rejection it resamples from an adjusted residual distribution and stops the round. This rejection-sampling scheme guarantees the final sequence is distributed identically to plain sampling from the target. How many tokens get accepted per round — the acceptance rate — sets the real speedup, so a well-aligned draft model matters far more than a merely small one.

p_{\text{accept}}(x) = \min\!\left(1, \frac{p_{\text{target}}(x)}{p_{\text{draft}}(x)}\right)

Accept probability for a drafted token x; rejection triggers a corrected resample.