contamination-free evaluation

A benchmark is only meaningful if the model has not already seen its answers. Contamination is the leakage of test items — questions, solutions, or close paraphrases of them — into the pretraining corpus, which inflates scores with no real capability gain. As web-scraped corpora swallow more of the internet, popular public benchmarks are increasingly likely to be memorized rather than solved.

Contamination-free evaluation combines detection and prevention. Detection probes for memorization: n-gram or embedding overlap between test items and training data, suspiciously high per-example confidence, or canary-string and exam-swap tests where a model that has seen a benchmark scores much higher on the original ordering than on a perturbed copy. Prevention uses truly held-out or freshly authored data — rolling benchmarks dated after the training cutoff (LiveCodeBench, recent-exam sets), private test splits, or dynamically generated items.

Detection is hard because you usually cannot inspect a closed model's training set, and paraphrase contamination slips past exact-match scans. The honest stance is to assume some leakage on any old public benchmark, weight recent and private evaluations more heavily, and always report the training cutoff date alongside the score so readers can judge for themselves.