Why standardized suites exist
If every lab tested its model on its own private questions, no two numbers would be comparable. Benchmark suites solve this with a fixed set of questions, a fixed answer key, and a fixed scoring rule. Everyone runs the same exam, so a score becomes a shared currency you can put on a leaderboard.
Most public benchmarks are static datasets: a list of inputs paired with correct outputs. The model answers each item, an automatic checker compares against the key, and you report accuracy. The art is in choosing items that are hard enough to separate strong models, unambiguous enough to grade automatically, and broad enough to mean something.
Benchmark scoring in one line: accuracy is just the fraction of items whose answer matches the key.
Knowledge and multiple choice: MMLU-style
The workhorse format is multiple choice. MMLU-style benchmarks — named after MMLU, Massive Multitask Language Understanding — ask thousands of exam-style questions across dozens of subjects, from history to medicine to law, each with four options. Grading is trivial: did the model pick the right letter? A single accuracy percentage summarizes broad book-knowledge.
Multiple choice is convenient but narrow. It tests recognition of the right answer among given options, not generation of an answer from scratch — and it is vulnerable to lucky guessing and to subtle formatting tricks. Strong models long ago pushed past 85% on MMLU, which is why newer 'MMLU-Pro' style sets add harder questions and more options.
Coding benchmarks
Code has a wonderful property for evaluation: you can run it. Coding benchmarks give the model a function description and hidden unit tests. Classic HumanEval asks for small standalone functions and checks them with tests; the score is the fraction that pass — often reported as pass@1, the chance the first sample is fully correct.
HumanEval's pass@k: the chance that at least one of k sampled solutions passes the hidden tests.
Toy functions are not real engineering, so harder suites moved to whole repositories. SWE-bench hands the model a real GitHub issue and the full codebase, and asks it to produce a patch that makes the project's actual test suite pass. This measures something much closer to a working agent than HumanEval ever did — and scores there climbed from near-zero to the majority in just a couple of years.
Reasoning and long-context probes
Reasoning benchmarks target multi-step thinking rather than recall: grade-school and competition math, logic grids, science word-problems. Because they usually have a single checkable final answer (a number or short string), they grade cleanly while still demanding a chain of correct intermediate steps — which is exactly what separates strong reasoners from pattern-matchers.
As context windows grew to hundreds of thousands of tokens, we needed long-context benchmarks. The classic probe is 'needle in a haystack': bury one fact in a huge document and ask for it. Models often retrieve fine at the edges but fail in the middle — the lost-in-the-middle effect — so good long-context tests scatter many needles, require combining several, and vary their position.
Interactive self-attention: clicking a token highlights the other tokens it attends to across the sequence.
Holistic suites: many axes at once
No single benchmark covers everything, so holistic evaluation frameworks report a matrix: many scenarios crossed with many metrics — accuracy, calibration, robustness, fairness, toxicity, efficiency. Stanford's HELM popularized this. Instead of one headline number, you get a profile that shows trade-offs: model A is more accurate but slower and more toxic than model B.