Static Benchmarks: Knowledge, Truth, and Code

Multiple-choice knowledge: MMLU

MMLU (Massive Multitask Language Understanding) asks ~16,000 multiple-choice questions across 57 subjects, from elementary maths to professional law. Scoring is a single mechanical operation: did the model pick the right letter? Its strength is breadth and reproducibility. Its weaknesses are subtle: the score is sensitive to prompt format and answer-ordering, four-way multiple choice gives 25% accuracy for free, and a few questions are mislabeled. MMLU measures recognition of stored knowledge under a clean format — useful, but a narrow slice of 'intelligence'.

Holistic, multi-metric evaluation: HELM

A single leaderboard number hides trade-offs. HELM (Holistic Evaluation of Language Models) answers this by evaluating a model across many scenarios and reporting many metrics per scenario — not just accuracy but calibration, robustness to perturbations, fairness, bias, toxicity, efficiency, and more — in a transparent matrix. The point is not to crown a winner but to make the trade-offs legible: a model that is more accurate may be less calibrated or slower, and HELM shows you that on one screen.

Holistic evaluation is expensive — many scenarios times many metrics times many models — but it is the honest counterweight to the single-number culture. When you design your own suite in Guide 5, HELM is the template: pick the axes that matter for your deployment and report them all.

Truthfulness versus fluency: TruthfulQA

Fluent text can be confidently wrong. TruthfulQA is built from questions where humans hold common misconceptions ('What happens if you crack your knuckles?'), so a model that imitates the internet will reproduce the popular falsehood. It directly probes the gap between sounding right and being right, and connects to the broader hallucination problem. Note the scoring subtlety: judging truthfulness itself requires a grader, which is why TruthfulQA shipped with both a fine-tuned classifier and human labels — a foreshadowing of the judge problem in Guide 3.

\text{score}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\text{truthful}_i]\cdot\mathbf{1}[\text{informative}_i]

TruthfulQA rewards answers that are both truthful and informative, so fluent-but-false answers score zero.

Executable evaluation: HumanEval and pass@k

Code is the one domain where correctness can be checked by running it. HumanEval gives 164 Python problems with hidden unit tests; a solution is correct only if it passes every test. Because models sample stochastically, we report pass@k: the probability that at least one of k sampled completions passes. Crucially, pass@k is computed with an unbiased estimator from n > k samples rather than naively, to avoid high-variance estimates.

\mathrm{pass@}k=\mathbb{E}_{\text{problems}}\!\left[\,1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\,\right]

The unbiased pass@k: draw n samples, count c correct, and estimate the chance a random k-subset holds at least one passing solution.

# Unbiased pass@k from n samples, c of which pass (Chen et al., 2021)
def pass_at_k(n, c, k):
    if n - c < k:        # fewer than k failures: success is certain
        return 1.0
    # 1 - P(all k drawn are failures)
    prod = 1.0
    for i in range(n - c + 1, n + 1):
        prod *= 1.0 - k / i
    return 1.0 - prod

Estimating pass@k without the high variance of taking exactly k samples.

From toy functions to real repositories: SWE-bench

HumanEval problems are self-contained snippets. SWE-bench raises the bar to real software engineering: each task is an actual GitHub issue from a popular Python project, and the model (usually an agent that can browse files and run tests) must produce a patch that makes the repository's hidden test suite pass. This rewards multi-file navigation, understanding existing code, and end-to-end problem solving — and scores are far lower than on HumanEval, which is the honest signal that real engineering is hard. It is the benchmark that best tracks agentic coding progress, complementing the broader agent evaluation benchmarks.

Reading long: long-context evaluation

Vendors advertise enormous context windows, but a long window is not the same as a usable one. Long-context evaluation stresses this: the classic 'needle in a haystack' test plants a fact at a random depth in a huge document and asks the model to retrieve it, exposing positions the model effectively ignores. Harder variants demand multi-hop reasoning across several needles, or aggregation over the whole context, where many models that ace single-needle retrieval collapse.