top-k sampling

Top-k sampling puts a guard rail on randomness. Instead of sampling from the model's full vocabulary — where tens of thousands of tokens each carry a tiny probability that can add up to real nonsense — it keeps only the k most likely tokens (k might be 40 or 50), renormalizes their probabilities to sum to one, and samples from just that shortlist. The model still gets to be creative, but only among the candidates it actually thought were plausible.

The weakness is that k is a fixed count, blind to context. When the model is very sure — only two or three tokens are sensible — a large k drags in long-shot options that should not be there. When the model is genuinely torn among many good continuations, a small k cuts off perfectly fine choices. Top-p (nucleus) sampling was invented to fix exactly this, by sizing the shortlist to the model's confidence instead of fixing its length.