JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

From Probabilities to Text: What Decoding Means

A language model never 'writes a word' — it scores every token, and decoding picks one. Meet the loop behind every reply.

The step nobody shows you

When an assistant answers you, it feels like it is composing sentences. Under the hood it is doing something far more mechanical: predicting the next token. At every position the model looks at everything so far and produces a score for every single token in its vocabulary — tens of thousands of candidates at once. It does not decide anything yet. Turning that giant table of scores into one actual token is a separate, swappable step called decoding, and the family of rules you can use is the set of decoding strategies.

Scores, softmax, probabilities

The raw scores are called logits — unbounded numbers, one per token. To make them comparable, the model passes them through softmax, which squashes the whole row into positive numbers that sum to 1: a genuine probability distribution. So at each step you really have something like “ ‘ the ’ = 41%, ‘ a ’ = 18%, ‘ cat ’ = 0.003% … ” across the whole vocabulary. The (natural) log of those probabilities, the log-probabilities (logprobs), is what most APIs report, because tiny probabilities are easier to read as negative log numbers.

p_i = \dfrac{e^{z_i}}{\sum_{j} e^{z_j}}

Softmax turns the raw logits into a probability distribution over the whole vocabulary.

Greedy: just take the biggest

The simplest possible decoder is greedy decoding: at every step, pick the single highest-probability token and move on. It is fast, fully deterministic, and a great mental starting point. Its weakness is that always grabbing the local favourite can paint you into a corner — and because there is zero randomness, you get the exact same answer every time for the same prompt.

x_t = \arg\max_{i}\; p_i

Greedy decoding simply takes the arg-max — the single highest-probability token at each step.

The autoregressive loop

Decoding is not a one-shot act; it is a loop. Once a token is chosen it gets appended to the input, and the model runs again to score the next position. This feedback — generate one, feed it back, generate the next — is what makes the model autoregressive. Because the text is built one token at a time, interfaces can show it appearing word-by-word; that live drip is streaming generation.

  1. Run the model on everything so far; get logits for the next position.
  2. Apply softmax to turn logits into a probability distribution.
  3. Use your decoding strategy to choose one token from that distribution.
  4. Append the chosen token and repeat — until a stop condition fires.
Autoregressive decoding in one loop: score, choose a token, append it, and repeat.

Diagram of the autoregressive generation loop feeding each chosen token back into the model.

Knowing when to stop

The loop needs an exit. Models are trained to emit a special end-of-sequence token when they are done, and generation halts there. You can also set a `max_tokens` ceiling, or define stop sequences — strings like `"\n\n"` or `"User:"` that, once produced, cut the generation off so it does not run on into the next turn.

That is the whole skeleton: score → choose → append → stop. Everything in this track is about that one choose step. Greedy is the boring baseline; the moment you let chance in, you reach sampling versus greedy — the fork in the road we walk down next.