The naive benchmark, and why it lies
The last guide drilled in measure, don't guess: profile the whole program, find where the time actually goes, and aim your effort there. Suppose the flame graph points at one small function — say a hash, a parser inner loop, a math kernel. The natural next move is to isolate it: write a tiny program that calls just that function in a loop a million times, time the loop, divide, and announce "3.2 nanoseconds per call." This is a micro-benchmark, and the instinct is exactly right. The trouble is that the smaller and tighter the thing you measure, the more ways the measurement can quietly betray you.
Here is the single most common way it betrays you, and it catches almost everyone the first time. You write the loop, compile with optimizations on — `gcc -O2`, as you should for any honest performance measurement — and the benchmark reports a time so absurdly fast it would shame the laws of physics. You did not just write the world's fastest hash. The compiler noticed something: nobody uses the result of the function, so by the dead-code elimination pass the entire call is useless work, and the as-if rule lets the optimizer delete it outright. Your million-iteration loop got compiled down to nothing. You timed an empty loop — or no loop at all.
The compiler is not being malicious; it is doing precisely its job. It proved your computation has no observable effect and removed it, which is the same brilliance that makes your real program fast. But in a benchmark it is catastrophic, because you wanted that work done so you could time it. And dead-code elimination is only the first trick: constant propagation is just as deadly. If you feed the function a literal — `hash("hello")` — the compiler may fold the whole thing to a constant at compile time, so the loop measures the speed of returning a number it already knew. You must defeat both, or your number is fiction.
Keeping the work alive: sinks and unknown inputs
The cure for both traps is to make the compiler unable to prove the work is useless. You attack from two sides. First, make the inputs unknown at compile time, so constant propagation has nothing to fold: read the value from argv, from a file, from a random buffer filled at runtime — anything the optimizer cannot see through. Second, make the outputs observable, so dead-code elimination cannot delete the call: accumulate every result into a running total, then do something with that total the compiler must respect, like printing it or storing it through a pointer it cannot trace.
// WRONG: result unused -> the whole loop is deleted at -O2
for (size_t i = 0; i < N; i++)
hash(i); // dead code
// RIGHT: input unknown (from argv), output consumed via a sink
unsigned seed = (unsigned)atoi(argv[1]); // optimizer can't fold
unsigned long acc = 0;
for (size_t i = 0; i < N; i++)
acc += hash(seed + i); // every result feeds acc
if (printf("%lu\n", acc) < 0) // sink: forces the work to exist
return 1;A serious benchmark framework gives you a cleaner tool than printf: a do-not-optimize sink, sometimes a function like `benchmark::DoNotOptimize(x)` or a tiny inline-assembly barrier, that tells the compiler "pretend this value escaped to code you cannot see." That stops dead-code elimination at exactly the right point without you having to invent a fake printout. The principle underneath is the same one the whole rung leans on: the optimizer reasons about observable behavior, so to measure work you must make that work observable — no more, no less.
Warmup, the steady state, and the noise floor
Even with the work kept alive, your first timed iteration is a trap of a different kind. The very first call runs cold: the function's code is not yet in the instruction cache, its data is not in the data cache, the branch predictor has never seen these branches and will mispredict them, and on a JIT-compiled or paged-in binary the code may not even be fully resident. The first run can be ten or a hundred times slower than the hundredth. If you time one shot, or average the cold runs in, you measure startup, not steady-state speed.
So you warm up: run the benchmark untimed for a while until performance stops improving, then start measuring once it has settled into its steady state. But which state you actually care about depends on the question. If you are optimizing a long-running server loop, steady-state is exactly right. If you are timing something that runs once per program — startup parsing, first-request latency — then the cold path is the thing, and warming it away measures a fantasy. Decide which regime your real workload lives in before you decide what to discard.
Now run the steady-state benchmark several times and you will see the numbers wobble — 3.1 ns, 3.4 ns, 3.0 ns, 3.6 ns. That spread is the noise floor, and it comes from everything you do not control: the OS scheduler moving your thread to another core (cold caches again), frequency scaling spinning the clock up and down, other processes stealing cache and memory bandwidth, interrupts and even thermal throttling. A measured difference smaller than the noise floor is not a result — it is the weather. The cardinal sin is to run the old code once, the new code once, see 3.3 vs 3.1, and declare a 6% win that is pure scheduler jitter.
Reading the distribution, not the average
Because the noise floor is real, a single number is never an answer; a distribution is. Run many timed iterations and look at the whole shape. The most robust summary is usually not the mean — one stray interrupt produces a huge outlier that drags the average up and tells you nothing about the typical call. Prefer the minimum when you want the cleanest estimate of the code's intrinsic cost (the fastest run is the one least disturbed by noise), and report the median and the spread to show how stable it is. If the minimum is 3.0 ns but the median is 9 ns, the code is not actually fast — it is fast only when the stars align.
When you compare two versions, the question is not "is A's number smaller than B's number?" but "do A's and B's distributions actually separate?" If old code clusters around 3.2 ns with a spread of ±0.4, and new code clusters around 3.0 ns with the same spread, those clouds overlap heavily and your "win" is inside the noise. A real improvement is one where the two distributions barely touch. This is why reproducible benchmarking insists on many runs and a stated variance, not a triumphant single figure — it is the difference between a measurement and a guess wearing a number's clothes.
There is a deeper reason to keep the distribution: the average can hide the part that actually hurts. A function averaging 5 ns but spiking to 500 ns once in a thousand calls — because that one call triggered an allocation or a cache miss storm — looks fine on a mean and is a disaster in a latency-sensitive system. The tail of the distribution is often the real story, and it is the entire subject of the next guide on latency, throughput, and the tail. For micro-benchmarks, the habit to build now is simple: never collapse your measurements to one number until you have looked at their shape.
The micro you measured is not the macro you ship
Suppose you have done everything right — kept the work alive, warmed up, run a distribution, beaten the noise — and your function really is 30% faster in isolation. The last and most humbling trap remains: a faster micro-benchmark does not guarantee a faster program. In your tight loop the function ran with its data hot in cache, its branches perfectly trained on one repeated input, and the CPU's instruction-level parallelism feeding it from the same memory over and over. In the real program that function is called once between a thousand other things, with cold caches, an untrained branch predictor, and contended memory. The very conditions that made the micro-benchmark clean are the ones that do not hold in production.
This connects straight back to the cost model idea from the last guide. A micro-benchmark measures the function's cost in isolation; what you ship is its cost in context. Sometimes a clever optimization that wins the micro-benchmark loses in the macro because it bloated the code and pushed a hotter neighbor out of the instruction cache, or because it traded a few instructions for a memory access pattern that thrashes a shared cache. The micro-benchmark cannot see those second-order effects, because by design it removed the context that produces them.
Pinning it down: a checklist you can run
All of this distills into a short, repeatable procedure. None of it is exotic; it is just the set of habits that, applied every time, keep a micro-benchmark honest. Walk through it deliberately the first few times and it soon becomes reflex — the same way checking the return value of malloc() and read() became reflex earlier on the ladder.
- Compile at the optimization level you deploy (usually -O2 or -O3), never -O0 — measure the program you actually ship.
- Defeat dead-code elimination: consume every result into an accumulator and feed it to a do-not-optimize sink or a real side effect.
- Defeat constant propagation: take inputs from a source the compiler cannot see through — argv, a file, a runtime-filled buffer — never a literal.
- Warm up to the steady state if your real workload runs warm; keep the cold path if what you ship runs once.
- Run many iterations, collect a distribution, and report the minimum plus the median and spread — never a lone average.
- Quiet the machine: pin the thread to a core, disable frequency scaling and turbo if you can, and close noisy neighbors to lower the noise floor.
- Confirm any win end-to-end by re-profiling the whole program — a micro-benchmark is a hypothesis, the macro measurement is the verdict.
Run that list and your numbers start telling the truth. The thread tying it together is the same one from the first principle of benchmarking: the compiler and the CPU are both relentlessly trying to do less work than your benchmark naively asks for, and a measurement is only as good as your account of what they actually did. With honest micro-numbers in hand, you are ready for the next guide, where we stop asking "how fast on average?" and start asking the harder, more truthful question — how slow is it when it is slow, and why the tail, not the mean, is what your users feel.