Performance Engineering

the flame graph

A profiler can collect tens of thousands of call-stack samples, but a giant table of stacks is unreadable. A flame graph is a picture that makes that pile of stacks instantly legible: it is the single most popular way to SEE where a program's time goes.

Here is how to read one. Each sampled call stack is a vertical column: main at the bottom, the function it called above it, and so on up to whatever was running at the moment of the sample. The flame graph merges all those columns side by side and sorts them, so the width of any box is the fraction of samples in which that function (and everything it called) was on the stack — wider means more time. The y-axis is call depth, NOT time, and the x-axis is NOT time order; it is just sorted alphabetically so identical stacks pile up into one wide box. So you scan along the top edge: any wide plateau at the top is a function that was itself running (not just waiting on a callee) for a lot of samples — that is your hot spot. Two important variants: a differential flame graph colors boxes by how much they grew or shrank between two profiles (red wider, blue narrower), which is how you see what a change made faster or slower; and an off-CPU flame graph is built from samples of BLOCKED time instead of running time, so it shows where the program waited (on locks, on I/O) rather than where it computed.

It matters because it turns a profiling result into something you can read in seconds and point a teammate at. The honest caveats: width is proportion of SAMPLES, not exact time, so it inherits the statistical nature of sampling; recursion makes a function appear stacked on itself; and the picture only shows what was sampled, so an on-CPU flame graph that looks empty during a slow phase usually means the time was spent off-CPU, blocked, where you need the off-CPU variant instead.

$ perf record -F 999 -g ./myprog $ perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg # wide plateau at top edge = parse_line(); that is where the CPU actually sat

A standard pipeline: record stacks, collapse identical ones, render to an SVG whose box widths are sample proportions.

Read width as 'fraction of stacks containing this frame', not 'time inside this function' — a wide box low down can be wide only because of an expensive callee above it; the function's own cost is the part NOT covered by its children.

Also called
differential flame graphoff-CPU flame graphicicle graph差分火焰圖離 CPU 火焰圖