reproducible benchmarking (controlling frequency scaling)

You time your optimized function: 8.2 ms. You run it again: 9.7 ms. Again: 7.9 ms. Did your change help, or is the machine just noisy? If two measurements of the SAME code disagree by 20%, you cannot trust a 5% improvement you think you see. Reproducible benchmarking is the discipline of pinning down the environment so that the same code gives the same number, and a real change stands out from the noise.

The single biggest source of this drift is dynamic frequency scaling. A modern CPU does not run at a fixed clock: to save power it slows down when idle (frequency scaling / power-saving governors), and to go fast briefly it boosts above its base clock (turbo / boost) — but turbo is limited by temperature and by how many cores are busy, so the same code runs at different clock speeds depending on how warm the chip is and what else is running. Measure during a cool turbo burst and it looks fast; measure after the chip heats and throttles and it looks slow. The cures are concrete: fix the CPU frequency (set the governor to 'performance' or disable turbo so the clock is constant), and warm up before timing. Beyond frequency, control the other noise sources: pin the benchmark to specific cores (CPU affinity) and ideally isolate those cores from the scheduler and from interrupts so other work cannot preempt it; disable hyper-threading so a sibling thread does not steal resources; quiet the machine (close background processes); account for address-space layout randomization, which can shift performance run-to-run, by running many trials; and report a robust statistic (median plus a high percentile) over many repetitions rather than a single best-or-average number. The aim is that re-running the benchmark unchanged reproduces the result.

It matters because an unstable benchmark produces confident nonsense: you 'confirm' improvements that are noise and discard real ones, and nobody can reproduce your numbers on another machine. The honest tension: a maximally controlled benchmark (fixed clock, isolated cores, no other load) is the right way to compare two code versions fairly, but it is NOT the environment your software runs in production, where turbo, noisy neighbors, and contention are real. So use a controlled setup to attribute a change to your code, and a separate, production-like measurement to learn how it actually behaves in the wild — and always report the conditions, because a benchmark number without its environment is unreproducible by definition.