reasoning benchmarks

Reasoning benchmarks test whether a model can work through a problem in steps rather than just recall a fact. The classic examples are grade-school word problems (GSM8K) and competition mathematics (MATH), where the answer is a specific number and you cannot fake it — either the multi-step chain lands on the right value or it does not. Newer sets push into logic puzzles, science questions, and graduate-level problems built to resist memorisation.

Because the final answer is checkable, grading stays mostly automatic, and these benchmarks exposed something important: prompting a model to show its working, step by step, often raises accuracy sharply. That observation made reasoning benchmarks the proving ground for chain-of-thought and for the recent wave of models trained to think longer before answering.

Two honest cautions. First, a correct final number can hide a flawed chain that got lucky, so the headline accuracy overstates genuine reasoning. Second, once a math set is widely used, its problems and solutions flood the internet and get memorised, which is why the hardest modern sets are kept fresh, private, or contamination-checked.