Design around a claim, not a number
Every good evaluation starts from a sentence you want to defend: 'model X is better at multi-step financial reasoning than Y for our users'. That claim dictates the scenarios, the metrics, and the baseline. Borrow the HELM discipline of many axes — accuracy and calibration and robustness and cost — and resist collapsing them prematurely into one score. An evaluation built backward from a claim is interpretable; one assembled from whatever benchmarks are lying around is not.
- Write the claim and its decision: what action changes if the evaluation comes out one way vs. the other?
- Choose scenarios that represent the real input distribution, not just convenient public sets.
- Pick metrics per scenario and a fixed baseline; decide grading (exact-match, execution, or judge) up front.
- Pre-register the protocol so you cannot tune it until the result looks good.
Get the statistics right
Benchmark numbers are estimates with error bars, yet papers routinely report bare point estimates and crown winners on differences smaller than the noise. Two sources of variance matter: sampling variance over the finite test set (report a confidence interval, e.g. via bootstrap) and decoding variance from stochastic generation (average over seeds, and report the temperature). Before claiming model A beats B, check statistical significance with a paired test on per-item scores — and remember that significant is not the same as large enough to matter.
A benchmark score is an estimate: report it with a confidence interval, never a bare point estimate.
Control contamination and leakage
The threat from Guide 1 returns operationally. Public benchmarks may be in the training corpus, so a high score can be memorisation. Defences: prefer freshly created or time-gated test sets (data created after the model's training cutoff), apply contamination detection (n-gram overlap with the corpus, or canary strings), and design a contamination-free private hold-out you never publish. Beware subtler leakage too: few-shot exemplars that hint the answer, or a judge that has seen the reference.
Data divided into separate train, validation, and test partitions.
Trust the judge only as far as you have calibrated it
If your harness uses LLM-as-a-judge or rubrics, the judge is now part of your measuring instrument and must itself be evaluated. Hold out a few hundred human-labelled items, measure judge–human agreement, and report it next to every judged result. Apply the position-bias and length controls from Guide 3, and probe whether the judge's verdict survives paraphrasing the rubric. A judged metric without a reported agreement number is a measurement without units — and treat the judge's own calibration as a property to verify, not assume.
A confusion matrix of true positives, false positives, false negatives, and true negatives.
Report honestly
The final deliverable is a document a skeptical peer can check. State the exact prompts, decoding settings, and grader; show per-scenario results with intervals, not just an average; include the failure cases and the regressions, not only the wins; and write down what the evaluation does not cover. The most trustworthy evaluations are the ones that most clearly mark their own blind spots.
The open frontier
Evaluation is the bottleneck on AI progress: capabilities now outrun our ability to measure them. Open problems are everywhere — evaluating long-horizon agents whose trajectories no human can fully grade, building benchmarks that resist saturation, judging open-ended creative and reasoning tasks, and scaling oversight to models stronger than their evaluators. If Volume I taught you to build models, this track's lesson is that knowing what you built is its own deep, unfinished science — and one where careful work still changes the field.
An agent loop cycling through reason, act, and observe steps.