perplexity
/ per-PLEK-suh-tee /
Perplexity measures how surprised a language model is by real text. Picture the model reading a sentence one word at a time, each time guessing what comes next. If the next word is usually one it expected, it's barely surprised; if it's constantly blindsided, it's very surprised. Perplexity packages that average surprise into a single number — and lower means less surprised, which means a better fit to the language.
There's an intuitive reading: a perplexity of N is roughly like the model being as confused as if it had to choose uniformly among N equally-likely words at every step. A perplexity of 10 means it's about as unsure as picking blindly from 10 options each time; a perplexity of 100 means picking from 100. So a model with perplexity 20 is, loosely, twice as good at anticipating the next word as one with perplexity 40. Mathematically it's just the exponential of the average log loss on the text.
Perplexity is the workhorse for tracking whether a language model is learning during training, and lower numbers genuinely reflect better next-word prediction. But beware the overclaim: low perplexity does not mean the text is true, helpful, or sensible. A model can be wonderfully unsurprised by fluent nonsense. And perplexity numbers are only comparable when computed on the same text with the same tokenization — different vocabularies make the figures incomparable, so a perplexity quoted with no context is close to meaningless.
Reading "the cat sat on the ___," a good model expects "mat," "floor," "sofa" — only a few likely words, so low perplexity. A weak model finds nearly any word plausible there, as if choosing among hundreds — high perplexity. The text is the same; the models' surprise differs.
Perplexity ≈ how many words the model effectively wavers between at each step.
Perplexity scores are only comparable under identical tokenization and test text — a model with a bigger vocabulary or different chunking can post a different number for no real improvement. And crucially, perplexity measures fluency-fit, not factual correctness; a confidently fluent hallucination can have excellent perplexity.