From one number to a moving target
In Volume I you scored a classifier by holding out a test set and computing accuracy, precision and recall: there was one correct label per example, so grading was mechanical. A large language model (LLM) breaks every assumption behind that recipe. Its output is open-ended text, there is rarely a single gold answer, the same prompt can be answered well in a thousand different wordings, and a small change in phrasing can swing the score. Evaluation stops being a measurement you read off and becomes a research instrument you must design.
A two-by-two confusion matrix with true positive, false positive, false negative and true negative cells.
The pretraining proxy you already know — perplexity, how surprised the model is by held-out text — correlates with capability but does not tell you whether the model writes correct code, refuses a harmful request, or stops hallucinating citations. We need task-level evaluation, and the moment we leave perplexity behind, the hard problems begin.
Perplexity is the exponential of the model's average per-token negative log-likelihood on held-out text — a measure of surprise that is blind to whether the answer is correct.
The anatomy of a benchmark
A benchmark is four decisions bundled together: a dataset of inputs, a task format (multiple choice, free generation, code execution), a scoring function, and an aggregation into a single number that lands on a leaderboard. Each decision leaks assumptions. Multiple choice is cheap to grade but tests recognition, not generation. Free-form grading is realistic but needs a fallible grader. The scoring function defines what 'good' means, and the leaderboard rank you finally publish hides all of it.
Three things that go wrong
Contamination. Models train on most of the public web, so test questions may already be in the training data; the score then measures memorisation, not skill. This motivates contamination-free evaluation and the detection methods in Volume II's data-contamination detection.
Data split into train, validation, and test partitions, with test items leaking back into the training set.
Saturation and Goodhart's law. Once a benchmark is famous, the whole field optimises against it; the number climbs toward the ceiling and stops discriminating between models — and 'when a measure becomes a target, it ceases to be a good measure'. Construct validity. A benchmark is a proxy for an ability you care about; a high score on a multiple-choice medical exam does not prove a model gives safe medical advice. Holding these three failure modes in mind — contamination, saturation, validity — is the discipline this track teaches.
A map of evaluation methods
The rest of this track walks four families. Static benchmarks with automatic scoring: knowledge tests like MMLU, truthfulness tests, and executable code tests. Holistic frameworks like HELM that report many metrics across many scenarios instead of one headline number. Preference and judge-based evaluation, where a strong model or a crowd of humans grades open-ended answers — including LLM-as-a-judge. Adversarial evaluation, or red-teaming, which asks not 'how good on average' but 'how does it fail when someone is trying to break it'.
- Guide 2 — static benchmarks: knowledge, truth, and executable code.
- Guide 3 — model judges, win-rates, and the Elo arena.
- Guide 4 — red-teaming, jailbreaks, and prompt injection.
- Guide 5 — assembling a research-grade evaluation harness you can trust.