perplexity

Perplexity measures how surprised a model is by real text it has never seen. Imagine the model reading a sentence one word at a time, each time guessing what comes next and placing a bet. If it expected the true next word, it is barely surprised; if the true word shocks it, the surprise is large. Perplexity averages that surprise over a held-out passage. Lower is better: a perplexity of 10 roughly means that, at each step, the model was as unsure as if choosing fairly among about 10 equally likely words.

Formally it is the exponential of the average negative log-likelihood the model assigns to the held-out tokens. Because it only needs the text's true continuations, no labels or human grading are required, which makes it the cheapest signal we have and the standard yardstick during pretraining. A model with lower perplexity has, by definition, learned the statistics of language more tightly.

But low perplexity is not the same as being useful, truthful, or safe. It rewards predicting the next token of fluent text, which is necessary but far from sufficient — and you cannot compare perplexity across models that chop text into different tokens. Treat it as a training thermometer, not a report card.