micro-benchmarking pitfalls

A micro-benchmark is a tiny program that times one small thing in isolation — say, how long a single call to your hash function takes. It sounds foolproof: put the call in a loop, time the loop, divide. In practice micro-benchmarks are one of the easiest things in all of programming to get wrong, and a wrong benchmark is worse than none because it lies with a confident-looking number.

Here are the classic traps, each with its cure. First, dead-code elimination: if you compute a result and never use it, the optimizer is allowed to delete the entire computation, and you end up timing an empty loop that reports an impossibly fast time. The cure is to consume the result — store it to a volatile, return it, or use a 'do not optimize away' helper your benchmark library provides. Second, constant folding and hoisting: if your inputs are compile-time constants, the compiler may compute the answer once at build time, or hoist an invariant computation out of the loop, so you measure nothing. Use inputs the compiler cannot predict. Third, warm-up effects: the first iterations pay cold-cache and (in JIT languages) compilation costs, so timing from the very first run mixes startup with steady state — run a warm-up phase, then measure. Fourth, measurement overhead: if the thing you time is only a few nanoseconds, the clock-reading call itself can cost as much, so time a batch of many iterations and divide, rather than timing each one. Fifth, lack of statistical rigor: one run is noise; report a distribution (median and a high percentile) over many runs, not a single number.

It matters because these traps routinely produce results that are off by 10x, 100x, or infinity (a deleted loop), and they fail silently. Two deeper honest cautions: a micro-benchmark measures a function in an artificial environment — perfectly warm caches, perfect branch prediction, no surrounding work — so a routine that wins a micro-benchmark can lose in the real program where caches are cold and other code competes; always validate a micro-benchmark win against a whole-program measurement before believing it.