a sampling profiler versus an instrumentation profiler
/ perf: PURF /
There are two honest ways to find out where a program spends its time, and they are like two ways of finding out what a busy office does all day. The first: every so often you walk in, snap a photo of who is doing what, and after a thousand photos you have a good statistical picture — that is a sampling profiler. The second: you give every worker a stopwatch and have them log entry and exit from every task — that is an instrumentation profiler. Each tells you something true; each disturbs the office differently.
A sampling profiler interrupts the program at a regular rate (say 1000 times a second, often using a hardware timer or the PMU) and records the current call stack at each interrupt — which function was running, and who called it. It never touches your code; it just observes from outside. Because it only peeks periodically, its overhead is small and roughly fixed regardless of program size, but it sees only a statistical sample, so very short or rarely-hit functions may be undercounted, and you read its results as percentages, not exact counts. An instrumentation profiler, by contrast, inserts extra recording code at the start and end of every function (the compiler can do this, e.g. gcc -pg, or a tool can rewrite the binary). It counts exact call counts and measures every call, which is precise — but the inserted code itself takes time, and for tiny hot functions that overhead can dwarf and distort the very thing you are measuring.
Which to reach for: use a sampling profiler first for a realistic, low-distortion picture of where wall-clock time goes in a whole program, especially in production; use instrumentation when you need exact call counts or want to time a specific bounded region. The classic gotcha for instrumentation is that it changes the program's timing (and can defeat inlining), so a function that is cheap in reality can look expensive purely because of measurement overhead — which is exactly why sampling is the default for whole-program work.
Sampling: timer fires 1000x/s -> record stack -> {parse: 612 hits, sort: 200, io: 100} ~ 61% parse Instrumentation: enter/exit every call -> {parse called 1.2M times, 480 ms total, sort 5k times, 1 ms}
Sampling gives proportions of where time went; instrumentation gives exact call counts and per-function totals at the price of overhead.
A subtlety: a pure sampling profiler can miss time spent BLOCKED (waiting on I/O or a lock) because a sleeping thread is not running when the timer fires — for that you want off-CPU profiling, not on-CPU sampling.