evaluation reproducibility

Reproducibility means that if you run the same evaluation again — or someone else does — you get essentially the same score. It sounds obvious, yet LLM benchmarks are notoriously fragile: the same model on the same test can shift by several points depending on tiny choices that have nothing to do with its real ability. That fragility makes leaderboard comparisons across papers genuinely hard to trust.

The culprits are many. Prompt wording and formatting, the number of in-context examples, how the final answer is parsed out of free text, the sampling temperature, the random seed, the exact model version behind an API, and even floating-point and hardware differences all move the number. Two labs reporting on the same benchmark can disagree simply because their harnesses differ.

The fixes are discipline: publish the exact prompts and harness, pin model and library versions, fix or report the decoding settings, run multiple seeds and report variance, and prefer shared open evaluation frameworks so everyone scores the same way. A score without its protocol is not a result you can reproduce or compare.