autoregressive decoding
/ aw-toh-ree-GRES-iv dee-KOH-ding /
Autoregressive decoding is how most language models actually write: one token at a time, each new token chosen based on everything written so far, including the model's own previous outputs. "Auto-regressive" simply means the model regresses on itself — it feeds its own output back in as input for the next step. It is exactly like composing a sentence word by word, where each word you pick narrows and shapes what can sensibly come next.
The loop is mechanical. The model looks at the prompt plus whatever it has generated, produces a probability distribution over the next possible token, picks one (greedily, or by sampling with some randomness), appends it, and repeats — until it emits a special stop token or hits a length limit. This is why chatbots stream their replies left to right, and why the KV cache exists: each step needs the keys and values of all prior tokens, so caching them is what keeps the loop fast.
Two honest consequences follow. First, generation is inherently sequential: token one hundred cannot be produced until token ninety-nine exists, so this step-by-step nature is a fundamental reason large models feel slow and why so much effort goes into speeding it up. Second, and more importantly, the model only ever predicts a plausible next token — it has no plan for the whole answer and no check on truth. A single wrong early token can send the rest of the response confidently down a false path, which is one root of why fluent models can state falsehoods so smoothly. Sounding right and being right are produced by the very same mechanism.
Given "The capital of France is," the model assigns high probability to "Paris" and picks it, then with "The capital of France is Paris" it likely picks a period and stops. But if an early token goes astray, the model will gamely keep extending that wrong start into a fluent, confident falsehood.
One token feeds the next — and an early wrong step rarely self-corrects.
Autoregressive models predict the next plausible token, not the truth, and have no plan for the whole answer before they start. That single fact explains both why they can be wrong while sounding perfectly fluent, and why they are slow — each token must wait for the one before it.