JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Core Dumps and Post-Mortem Debugging

Your program already crashed — on a server at 3am, with no debugger attached and no way to reproduce it. A core dump is the frozen photograph of that exact moment. Here is how the kernel takes the picture, how to make sure it actually gets saved, and how to load it into gdb and walk back to the line that killed you.

Debugging a corpse, not a patient

In the previous guide you drove gdb while a program was alive: you set a breakpoint, you ran stepping line by line, and when something looked wrong you printed a backtrace to see who called whom. That is the happy case — you have a running process you can poke at. But the bugs that hurt most never give you that luxury. They strike once, on a machine you were not watching, and by the time you arrive the process is gone. You cannot attach a debugger to a process that has already died. So instead of debugging a living patient, we learn to debug a corpse.

A core dump is the autopsy material. When the kernel kills a process for a fatal fault — a segmentation fault, say — it can, before reclaiming the memory, write out a single file that captures the process's entire state at the instant of death: the contents of memory (the heap, the call stack, global variables), and the CPU registers (rax, rsp, rip, and the rest). The name is an antique: in the 1960s main memory was woven from tiny magnetic "cores," so dumping memory to disk was literally a core dump. The metaphor that helps today is a photograph: not a video of the run, but one perfectly still frame snapped at the microsecond the program fell over.

Why the file is usually missing

The cruel surprise for newcomers is that you crash, you go looking for the core file, and there is nothing there. This is not a bug; it is policy. Writing a full memory image to disk can mean hundreds of megabytes, and a crashing process may be one of thousands, so most systems ship with core dumps throttled or switched off by default. The first gate is a per-process resource limit. The shell command `ulimit -c` shows the maximum core-file size, and on a fresh login it is very often 0 — meaning "write zero bytes," i.e. never produce a core at all. Raising it with `ulimit -c unlimited` lifts that gate for the current shell and its children.

Pass that gate and a second one appears: even when a core is allowed, where it lands and what it is called are decided by the kernel, not your shell. On Linux this is governed by the pseudo-file /proc/sys/kernel/core_pattern. The simplest setting is a literal name like `core`, which drops a file named core (or core.PID) in the process's working directory. But on many modern desktops core_pattern instead begins with a pipe character, handing the dump to a system service — systemd's coredumpd, or apport on Ubuntu — which tucks it away in a system directory and compresses it. That is why the directory looks empty: the core was taken, just not where you looked. You retrieve it with the service's own tool, such as `coredumpctl list` and `coredumpctl gdb`.

The signal that pulls the trigger

What actually decides whether a death produces a core is the signal that ended the process. When your code dereferences a bad pointer, the hardware traps into the kernel, and the kernel delivers a signal to your process — typically SIGSEGV for an invalid memory access, or SIGABRT when something like a failed assert() or a detected double free calls abort(). Some signals have a default action of "terminate and dump core"; others merely terminate. SIGSEGV, SIGABRT, SIGFPE (a bad arithmetic op like divide-by-zero), SIGBUS, and SIGILL all dump by default. SIGTERM and a plain Ctrl-C (SIGINT), by contrast, just end the process with no photograph taken — which is exactly why a clean shutdown leaves no core behind.

You can also take the photograph deliberately. The library call abort() raises SIGABRT on yourself, which is the principled way to halt a program the instant it detects an impossible state — a failed invariant, a corrupted data structure — so that the core captures the scene while it is still fresh, before the damage spreads. This is the same mechanism behind a failed assertion: assert(p != NULL) that fires will abort() and, if cores are enabled, leave you a dump pointing straight at the offending line. Crashing early and loudly with a core in hand beats limping onward and corrupting state you will debug an hour later from a useless symptom.

Loading the photograph into gdb

Now the payoff. You hand gdb two things: the executable that crashed, and the core file it produced — `gdb ./a.out core`. gdb does not run anything; the program is dead. Instead it reconstructs the world from the dump, dropping you at a prompt as if you had been standing at the breakpoint of the crash all along. The very first command to type is `bt` (backtrace). It prints the call stack frozen at the moment of death — the chain of function calls from main() down to the exact line that triggered the fault — letting you read upward to see how the program arrived at its doom. This is the same backtrace you learned last guide, but now reconstructed from a corpse rather than a paused live process.

But the backtrace is only readable if you compiled with debug information. This is the single most important habit in this entire guide. Without it, gdb can still show you raw addresses and register values, but the frames read as a wall of hex — `0x000055e3a1b4 in ?? ()` — with no function names, no line numbers, no variables. The fix is to compile with debug info turned on by adding the `-g` flag: `gcc -g -O0 main.c`. The -g flag tells the compiler to embed a map from machine addresses back to your source lines and variable names, so gdb can translate the corpse's raw bytes into the words you actually wrote.

  1. Compile with debug info and no optimization: `gcc -g -O0 main.c -o a.out`. Keep -O0 here so the code in the dump matches your source one-to-one.
  2. Enable cores in the launching shell: `ulimit -c unlimited`, and confirm where they go with `cat /proc/sys/kernel/core_pattern`.
  3. Run the program so it crashes: `./a.out`. You should see a message like "Segmentation fault (core dumped)."
  4. Open the corpse in the debugger: `gdb ./a.out core` (or `coredumpctl gdb` if a service captured it).
  5. Type `bt` for the backtrace to find the crashing frame, then `frame N` to step into the frame you care about.
  6. Inspect the dead variables with `print p` or `info locals` to see the exact value that killed you — often a null or wild pointer.

Reading the scene, and its honest limits

Once you are at the crashing frame, the corpse is fully inspectable. `print p` shows the value of a pointer; `print *p` would have been the very thing that crashed, but `print p` is safe because you are only reading the dump's stored bytes, not running code. `info locals` dumps every local variable in the current frame, and `info registers` shows rax, rsp, rip and friends exactly as they stood at death — rip in particular points at the very instruction that faulted. A classic read goes: bt shows the crash is in a function called process(); `frame 1` moves there; `print buf` reveals buf is 0x0, a null pointer; and you now know process() was called with a buffer that some caller never allocated. The autopsy is complete.

Three honest caveats keep you from over-trusting the photograph. First, the executable and the core must match — if you recompiled even slightly after the crash, the address-to-line map no longer lines up and gdb will mislead you; debug a core only with the exact binary that produced it. Second, optimization muddies the scene: at -O2 the compiler reorders and inlines code, so a frame may be "optimized out" and a variable may read `<optimized out>` even with -g present. That is why -O0 is kindest for debugging, though release crashes you must read as-is. Third, the crash site is not always the root cause: a pointer that is null here may have been corrupted by a buffer overflow in entirely different code a million instructions ago.

That last caveat is the bridge to the rest of this rung. A core dump tells you where the body fell, which is sometimes far from where the wound was inflicted. When the symptom and the cause live far apart — a stale pointer here, a stray write there — the static photograph runs out of road, and you reach for tools that watch memory as it is used rather than after the fact. That is precisely what Valgrind and the sanitizers do, and they are the subjects of the next two guides. The core dump remains your indispensable first responder: cheap to capture, priceless when a bug strikes once on a machine you will never see, and the fastest way to turn "it crashed" into "it crashed on line 42 with buf equal to 0x0."