Two levers on the memory wall
Since decode is memory-bound, the two ways to go faster are: move less memory per token, or produce more tokens per memory sweep. Quantization is the first lever; speculative decoding is the second. They attack the same bottleneck from opposite ends, and they stack — you can use both at once.
Decode is memory-bound: each token's time tracks the bytes read per unit of memory bandwidth — so reading fewer bytes (quantization) or producing more tokens per sweep (speculation) is the whole game.
Quantization: store the weights in fewer bits
A model's weights are numbers. By default each is a 16-bit float; quantization for inference re-stores them in fewer bits — commonly 8-bit integers, increasingly 4-bit. Eight-bit roughly halves the model's memory footprint; four-bit roughly quarters it. Because decode speed tracks how many bytes you read per token, cutting the bytes cuts the time almost proportionally.
The most common flavor is weight-only quantization: compress just the static weights, and de-quantize each tile back to full precision on the fly for the actual multiply. Activations stay high-precision, so accuracy barely moves while the big memory saving stays. This is why a model that needed two cards in fp16 can often fit on one when quantized.
Weight-only quantization stores each weight as a low-bit integer q and de-quantizes on the fly: w ≈ s(q − z), with a per-tile scale s and zero-point z.
Watch the accuracy you trade
Fewer bits means coarser numbers, and at some point quality slips. Eight-bit is usually indistinguishable from full precision; four-bit needs careful, calibrated methods (rounding each weight to minimize the error it actually causes) to stay safe; below four bits, degradation is real and task-dependent. The discipline is simple: quantize, then re-run your evaluations — never trust the speedup until the quality numbers confirm it held.
Speculative decoding: guess ahead, verify in bulk
Decode is sequential because each big-model pass yields one token. Speculative decoding breaks that by adding a small, cheap draft model. The draft quickly guesses the next, say, 4 tokens. Then the big model checks all 4 in a single forward pass — and verifying several tokens at once costs it almost the same as generating one, because it was memory-bound and had spare math capacity anyway.
Diagram of an autoregressive next-token generation loop that feeds each new token back in as input.
The magic is in the verification step. The big model accepts the longest prefix of the guess it agrees with and rejects the rest, so the output is provably identical to what the big model would have produced alone — same distribution, zero quality loss. You are not trading accuracy for speed here; you are getting speed for free whenever the draft guesses well.
- Draft model proposes a short run of tokens (e.g. 4) cheaply and quickly.
- Big model verifies all of them in one pass, finding the longest prefix it would have produced itself.
- Accept that prefix in one go; on the first disagreement, keep the big model's own token and discard the rest.
- Repeat. On easy, predictable text many tokens land per pass; on hard text it gracefully falls back to roughly normal speed.
Choosing your levers
Quantization is nearly always worth trying first: it shrinks memory, raises tokens per second, and enlarges your batch, with only an evaluation to babysit. Speculative decoding shines for interactive, single-stream latency — one user wanting their answer fast — but it adds little to an already-saturated batch where the math units are full. Read the room: an offline batch job wants quantization and big batches; a live coding assistant wants quantization and speculative decoding.