benchmark suites

A benchmark suite is a fixed, public collection of tasks that everyone runs the same way, so different models can be compared on equal footing. Think of it as a standardised exam: the same questions, the same scoring rules, the same conditions for every test-taker. Each item usually comes with a known correct answer, so grading can be automatic — count the fraction right and you have a score. Bundling many tasks into one suite gives a fuller picture than any single test.

Suites differ by what they probe: broad knowledge, coding, math, reading comprehension, safety, multilingual ability, long inputs. They also fix the protocol — how prompts are phrased, how many examples are shown, how answers are extracted from free text. Those details matter enormously; the same model can swing several points just from a different prompt template, which is why reputable suites publish their exact harness.

Their value is comparability and reproducibility, but they are static snapshots. Once a suite is popular, its answers leak online and models start memorising it, so a suite slowly stops measuring reasoning and starts measuring familiarity.