MMLU (Massive Multitask Language Understanding)
MMLU is a multiple-choice exam spanning 57 subjects — from elementary mathematics and US history to professional law, medicine, and college physics. Each question has four options and the score is plain accuracy, usually reported in a few-shot setting (classically 5-shot). It became the default headline number for how much a model knows precisely because its breadth resists being gamed by narrow optimization on one skill.
The scoring has subtleties that break naive comparisons. You can read off the answer letter the model generates, or compare the log-probabilities of the four option tokens and take the argmax — these give different numbers, so cross-model results require an identical harness. Random guessing scores 25%; strong frontier models now exceed 88%, compressing the headroom and amplifying the effect of the handful of mislabeled or ambiguous items that MMLU is known to contain.
Because MMLU has circulated publicly since 2020, contamination is a live worry, and the ceiling is close enough that successors like MMLU-Pro (ten options, harder distractors, reasoning-heavy items) were built to restore discrimination. Read a single MMLU number as a coarse proxy for stored knowledge, not a verdict on reasoning depth.
Plain accuracy over N four-way questions; the chance baseline is 0.25.
A high MMLU score reflects stored knowledge plus test-taking, not reasoning depth; saturation near the human-expert ceiling makes small gaps statistically fragile.