JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Continuous Profiling and Crash Analysis

The first three guides taught you how to watch a running program with tracing. This one answers two harder questions: where does the time actually go when it is fine but slow, and what exactly happened in the half-second before it died? You will see how a sampling profiler turns a stack of frames into a flame graph, why running it always-on across the fleet is now affordable, and how a core dump lets you walk a dead process's memory as if it were paused under a debugger.

Sampling vs instrumentation: two ways to ask 'where does the time go?'

A program that crashes tells you loudly that something is wrong. A program that merely runs slow tells you nothing — it just quietly burns time, and your job is to find out where. There are two fundamentally different ways to find out, and choosing wrong is the first mistake. The instrumenting way is to add a tiny timer around every function: record the clock on entry, record it on exit, add up the difference. This is exact — you learn the real cost of every call — but it has a brutal flaw you already half-know from the observer effect in guide 1: the timing code itself runs on every single call, so a function called a million times now does a million extra clock reads. The measurement changes what it measures, and a tiny hot function can be slowed several-fold just by watching it.

The sampling way is the opposite, and it is the one continuous profilers use. Instead of timing every call, you let the program run untouched and just peek at it many times per second — say 99 times a second — and at each peek you record one thing: the current call stack, the chain of return addresses from the innermost function all the way out to main(). You do nothing else. Each peek costs almost nothing, so the program runs at nearly full speed. The trade is that you no longer have exact times; you have a statistical picture. But that picture is exactly the one you want, and the reasoning is simple: if a function is where the CPU spends 40% of its time, then roughly 40% of your random peeks will catch the CPU inside that function. Frequency of appearance in the samples is proportional to time spent. That is the whole idea of a sampling profiler.

Reading a flame graph: width is time, depth is the call chain

Once you have tens of thousands of stack samples, you face a new problem: how do you look at them? A raw list is useless. The answer is one of the great visualizations in systems work, the flame graph, and once you can read it you can diagnose a slow program at a glance. The trick is to collapse all the samples that share the same stack and lay them out as nested rectangles. Each box is a function. A box sits on top of the box that called it, so the vertical axis is the call stack — main() along the bottom, the deepest callee at the top. And here is the part that matters: the width of a box is how many samples caught that function on the stack, which is to say how much time was spent in it and everything it called.

|<--------------------- all samples (100%) --------------------->|
[ main                                                            ]
[ handle_request                          ][ log_line   ][ idle  ]
[ parse_json        ][ db_query           ][ format     ]
[ strlen ][ memcpy  ][ recv  ][ parse_row ]
                              ^^^^^^^^^^^^^
      width  = time spent (sample count)   ->  WIDE box = hot
      height = depth of the call stack
      a wide PLATEAU at the top = a leaf burning CPU on its own
A flame graph: x-axis is time (box width = fraction of samples), y-axis is call depth. You hunt for the widest boxes, and especially a wide flat top — a leaf function that is itself doing the work, not just calling down further.

Reading it becomes a reflex. A tall, thin spike is a deep call chain that barely costs anything — ignore it. A wide box is where your time lives. Scan along the top edge: a wide plateau at the top means a leaf function that is itself burning the CPU rather than calling further down, and that leaf is your number-one suspect. Crucially, the x-axis is not time order — boxes are sorted to merge identical stacks, so left-to-right means nothing; only width and stacking carry meaning. One honest caveat: this 'on-CPU' flame graph shows where the CPU was busy, so it is blind to time spent waiting — blocked on a lock, asleep in a blocking read, parked off-CPU. If your program is slow but the flame graph looks idle, the time is being spent waiting, and you want an off-CPU profile instead, which the next idea makes cheap to collect.

Why profiling can now run always-on, everywhere

Traditionally you profiled by reproducing a slow case: run the program under a profiler on your laptop, hit it with a load test, read the result. But the bug that matters is often the one that only appears at 3 a.m. on one machine out of a thousand, under a traffic mix you cannot reproduce. The modern answer is continuous profiling: collect a low-frequency stack-sample profile from every process on every machine, all the time, and ship it to a central store you can query weeks later. When something was slow last Tuesday, you do not re-run anything — you just go look at the profile that was already being recorded. It turns profiling from a special expedition into ambient, always-available data, exactly the philosophy of observability from guide 1 applied to CPU time.

What makes this newly affordable is the machinery from guide 2. A continuous profiler is typically a small eBPF program attached to the CPU's sampling timer: on each tick the kernel-side eBPF code walks the current stack and increments a counter in a BPF map keyed by that stack, entirely in the kernel, with no copy of raw samples crossing into user space until you aggregate. The overhead lands around a fraction of one percent of CPU — cheap enough to leave on in production forever. And because eBPF can walk the kernel stack too, one profiler sees a request flow from your function, down through a system call, into the kernel, in a single unified flame graph — something no user-space-only profiler could ever show you.

Be honest about the limits, because a profiler that lies is worse than none. Sampling is statistical, so a function that runs for a microsecond but never happens to be on the stack at a tick can be invisible — sampling is great for the fat parts and weak on the thin, infrequent ones. The default per-CPU profile also misses waiting (off-CPU) time, as we said. And walking a stack in the kernel needs reliable frame information; a binary compiled to omit the frame pointer can give the walker a truncated or wrong stack unless it has DWARF unwinding or the compiler kept the frame pointer. Treat a profile as a strong statistical hint about the hot paths, confirmed before you optimize — the measure, don't guess discipline applies to the profiler's own claims too.

The core dump: a snapshot of memory at the instant of death

Profiling is for the program that is too slow. The other half of this guide is for the program that is dead. When a process commits a fatal error — dereferences a null pointer and takes a segmentation fault, or trips an assertion — the kernel can, on its way to killing it, write out a core dump: a file containing a frozen snapshot of that process's memory and registers at the exact instant it died. Think of it as a photograph of the corpse. Every byte of the heap, the full call stack of every thread, the value in rip pointing at the very instruction that faulted, the contents of rax and rsp and rbp — all of it, captured. The crash already happened and the process is gone, but the evidence is on disk, and you can examine it at leisure.

You open that photograph with the same debugger you used for live debugging two rungs ago, only now you point it at the corpse: gdb ./myprog core. The debugger loads the dump as if the process were paused right at the crash, and the first thing you ask is the one that solves most crashes outright — for a backtrace. The command bt prints the backtrace: the chain of stack frames from the function that faulted, out through each caller, to main(). In one screen you see which line of which function crashed and the exact path of calls that led there. You can then print any local variable or follow any pointer, reading the dead program's memory as plain data — was that pointer 0x0, was that index past the end of the array, was that length absurdly large?

When the kernel itself dies: kdump and production discipline

A core dump captures one dead process. But sometimes the thing that dies is the kernel itself — a kernel panic, where the operating system hits an error it cannot recover from and halts the whole machine. You cannot write a normal file at that moment, because the very subsystem that writes files is part of what just died; you cannot trust the running kernel to dump itself. The clever solution is kdump: a second, tiny crash kernel is loaded into a reserved slice of memory at boot and kept dormant. When the main kernel panics, instead of just freezing, it hands control to that pre-loaded crash kernel, which boots in the reserved memory, treats the dead kernel's memory as a giant core dump, and writes it out to disk for analysis.

Whether you are reading a process core or a kdump image, the discipline that makes production crash analysis actually work is the same, and it is worth stating plainly as a routine. A crash is just a symptom; the dump is your evidence; the goal is the root cause, not the line that happened to fault. The disciplined walk is short and you will run it many times in your career.

  1. Confirm a dump exists and matches: load the core with the exact build that produced it (gdb ./prog core), checking the binary and the dump are the same version — a mismatched binary gives a garbage, lying backtrace.
  2. Get the backtrace: run bt to see which line faulted and the full chain of callers, then move up and down the frames to read locals — this alone explains most crashes.
  3. Inspect the faulting state: look at the registers and the offending pointer or index — was it 0x0 (a null deref), a wild address, or just past an array end? Match the value to a known bug class.
  4. Cross-check with the trail: line the dump up against the logs, metrics, and traces around that timestamp from guide 1, so you understand the conditions that led in, not just the final instruction.
  5. Name the root cause and fix that, not the symptom: a null deref is usually a missing check upstream, not a bug at the line that crashed — fix the upstream cause and add the check that would have caught it.

Tying the two halves together

Step back and these two tools are the same instinct pointed at two states of a program. Continuous profiling watches the living system — always sampling, statistical, answering 'where does the time go?' at fleet scale and near-zero cost thanks to eBPF. Crash analysis examines the dead system — one frozen snapshot, exact, answering 'what state was it in when it died?' through a core dump and a backtrace. One is a continuous film of the healthy program; the other is a single sharp photograph of the moment of failure, and a serious production engineer reaches for whichever the situation calls for.

Both also share one honest theme worth carrying out of this rung: the measurement is only as good as the symbols and the matching build behind it. A profile of stripped binaries is a hex wall; a core dump opened against the wrong build is a confident lie. The cheap, boring habit that makes everything in this guide work is to keep, somewhere safe, the unstripped binary with full debug info for every version you ship — so that whether the fleet is slow or one machine is dead, the raw addresses can always be resolved back into your function names and line numbers. The last guide in this rung takes the final step: following a single request not down into one machine's stack, but across many machines, with distributed tracing.