sampling vs. greedy

This is the fork in the road for every decoder: be deterministic or be random. Greedy (and beam search) always chase the most probable text, so they are repeatable and well suited to tasks with a right answer — arithmetic, code, structured extraction, anything you want to test. Sampling — temperature, top-k, top-p, min-p — rolls weighted dice, so the same prompt yields different outputs and the writing feels more human, varied, and alive. The price is unpredictability and the occasional wrong turn.

The deep reason sampling wins for open-ended text is subtle: the single most probable continuation of human language is usually bland and repetitive, while real human writing lives in the broad, slightly-less-likely middle of the distribution. Picking only the top token starves the output of that natural variety. The practical rule of thumb: deterministic when there is one correct answer, sampling when you want creativity — and remember that even at temperature zero, hardware and batching can make outputs not perfectly reproducible.