decoding strategies

At every step, an LLM does not hand you a word — it hands you a probability for every token in its vocabulary, often tens of thousands of them. A decoding strategy is the rule that turns that whole distribution into one actual choice, then repeats the process for the next token. Think of it as the policy a writer follows: always pick the safest word, roll dice weighted by likelihood, or keep a few drafts going at once. The same model can sound cautious, creative, or repetitive depending purely on this rule.

Because the model only ever predicts one token ahead, the strategy is applied again and again, each pick feeding back in as context for the next. The main families are deterministic methods (greedy, beam search) that aim for the most probable text, and stochastic methods (temperature, top-k, top-p, min-p sampling) that deliberately inject randomness. Most production systems combine several knobs at once. Choosing well matters as much as the model itself: the wrong settings can make a capable model babble or loop.