the SPEC benchmarks
/ SPEC: speck /
If every car maker invented its own private test and declared itself the winner, no buyer could compare them. So the industry agreed on shared, refereed tests. SPEC — the Standard Performance Evaluation Corporation — is that referee for computers: a nonprofit consortium that publishes standardized benchmark suites of real programs and strict rules for running and reporting them, so results from different vendors can be lined up side by side.
The best-known suite, SPEC CPU, bundles a couple dozen real applications drawn from compilers, simulations, compression, AI, and scientific computing, split into integer (SPECint) and floating-point (SPECfp) groups. Each machine runs all of them; each program's run time is compared against a fixed reference machine to get a per-program ratio, and the ratios are combined with a geometric mean into a single headline score. Crucially, SPEC defines rules: which compiler flags are allowed, that runs must be reproducible, and that results are published with full disclosure so they can be audited. There are other SPEC suites too — for power efficiency, web servers, virtualization, and more.
SPEC is the most respected general benchmark, but it is not the last word, and pretending otherwise is the mistake. Vendors still optimize their compilers specifically for SPEC programs, so a SPEC score can overstate everyday performance; SPEC CPU stresses the processor and memory but barely touches disk, network, or interactivity; and your particular application may look nothing like SPEC's mix. Use SPEC as a credible, comparable starting point — precisely because its rules resist the worst gaming — but always finish by measuring the workload you actually care about.
SPEC CPU reports a per-program ratio versus a reference machine: if program X takes 100 s on the reference and 25 s on your machine, its ratio is 4. The suite's headline score is the geometric mean of all such ratios — so one program cannot dominate the result by sheer size of its number.
SPEC combines per-program ratios with a geometric mean, under audited rules — credible, but still tuned for and not equal to your own workload.
A high SPEC score is a credible signal, not a promise about your application. Compilers are tuned specifically for SPEC programs, and SPEC CPU largely ignores I/O and interactivity, so it can diverge from what you actually experience.