JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

When Benchmarks Lie: Contamination, Saturation, and Reproducibility

A high score is not the same as a capable model. The failure modes that quietly inflate or scramble the numbers.

Goodhart's law comes for benchmarks

'When a measure becomes a target, it ceases to be a good measure.' Every public benchmark is also a target labs optimize toward — directly or by osmosis. So a rising score can mean two very different things: the model genuinely got better at the underlying skill, or the model got better at the test. Evaluating LLMs honestly means constantly distinguishing those two.

Contamination: the test in the training set

The sharpest failure is benchmark contamination: the benchmark's questions and answers leak into the model's training data. Pretraining scrapes much of the web, and benchmarks live on the web, so a model may have memorized the exact answers it is being tested on. The score then measures recall of leaked data, not the skill the benchmark was meant to probe.

Benchmarks assume a clean wall between training and test data; contamination is that wall breaking when test items leak into pretraining.

A dataset split into separate train, validation, and test portions.

Defenses fall into detection and prevention. Contamination detection looks for tell-tale signs — abnormally low perplexity on test items, or near-perfect scores that collapse when a question is slightly reworded. Prevention favors contamination-free evaluation: hold-out private test sets, benchmarks built after a model's training cutoff, or freshly generated problems the model could not have seen.

\mathrm{PPL}(x)=\exp\!\left(-\frac{1}{N}\sum_{i=1}^{N}\log p(x_i\mid x_{<i})\right)

One contamination tell-tale: abnormally low perplexity on test items, because the model has effectively memorized them.

Saturation: the ceiling we hit

Even a clean benchmark dies of success. Benchmark saturation happens when top models cluster near the maximum score — 92%, 93%, 94% — and the benchmark can no longer tell them apart. Worse, much of the remaining gap is often mislabeled items and ambiguous questions, so chasing the last few points means fitting the benchmark's noise, not real skill.

Saturation is why the field churns through benchmarks: MMLU gave way to MMLU-Pro and GPQA; easy math sets gave way to competition-level ones. A useful evaluation needs headroom — questions hard enough that today's best models clearly fail some of them.

Reproducibility: same model, different score

Run the 'same' benchmark twice and you can get different numbers — a reproducibility failure. The culprits are mundane but huge: the exact prompt template, whether few-shot examples were used, how answers are parsed from the model's text, the sampling temperature, stop sequences, and even library versions. Two papers reporting MMLU can differ by ten points purely from these choices.

\widehat{\mathrm{acc}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i=y_i]\;\pm\;z\,\sqrt{\frac{\widehat{\mathrm{acc}}\,(1-\widehat{\mathrm{acc}})}{N}}

A leaderboard number is an estimate with error bars, not a fixed truth — which is why a score is meaningless without its protocol.

Reading a leaderboard with healthy suspicion

Put it together and a benchmark number deserves three questions before you believe it. Could the test have leaked into training (contamination)? Is the benchmark still discriminating, or has everyone maxed it out (saturation)? Was it run under a protocol you could reproduce (reproducibility)? A score that survives all three is informative; one that fails any of them is a marketing figure.