degeneration and repetition

Neural text degeneration is the failure where a model's output goes flat, generic, or stuck — most visibly when it falls into a loop, repeating a phrase or sentence over and over until you cut it off. You see it most with deterministic decoders: greedy and beam search keep chasing the highest-probability continuation, and once a repeated pattern becomes locally the most likely thing to say, the model digs the rut deeper with every step instead of climbing out.

The surprising lesson from research is that the cure is more randomness, not less. Letting the model sample from the natural spread of likely tokens — via top-p or min-p, with a sensible temperature — breaks the self-reinforcing loop and restores variety, because human language genuinely lives in that broad middle rather than at the single peak. Repetition penalties patch the symptom from the other side. When you see an LLM babbling or looping, the first suspect is almost always the decoding settings, not the model's intelligence.