production core analysis

When a process crashes in production, it usually dies in the dead of night, on a machine you cannot interrupt, after some input you cannot reproduce. You cannot attach a debugger to a process that has already vanished. The answer is to capture a snapshot of the process at the instant it died and examine it afterwards, at leisure — this is core analysis: generating, capturing, and post-mortem-analysing a core dump from a production crash. (The offline single-process debugger workflow is covered in Volume I; here the focus is doing this in a live production environment.)

Here is the workflow in plain steps. A core dump is a file the operating system writes when a process crashes, containing the contents of that process's memory and registers at the moment of death — effectively a frozen photograph of its state. You must first arrange for cores to be generated and captured: the OS only writes them if a resource limit allows it (the core file size limit must not be zero), and on a modern Linux system the kernel hands the crashing process's memory to a configured handler (commonly systemd-coredump or a tool like the ABRT collector) that stores it somewhere durable rather than dropping a huge file in the working directory. Then, after the fact and on a different machine, you load the core into a debugger (gdb core ./prog, or specialised post-mortem tools) together with the exact binary and its debug symbols, and you inspect the backtrace, the local variables, and the heap as they were at the crash. To make sense of a stripped production binary, you keep the matching debug-info separately and pair it with the core.

It matters because for a class of failures (a crash that fires once in a million requests) a core is the only honest evidence you will ever get — you analyse the actual moment of death rather than guessing. The honest caveats: a core dump contains the process's full memory, which can include secrets (passwords, keys, customer data), so capturing and storing cores in production is a real privacy and security responsibility, not a casual setting to flip on; cores can be very large for a big process; and a core only captures the final state — it tells you WHERE it died and WHAT the memory looked like, but the root cause (often earlier undefined behaviour or a stale pointer) may have happened long before, so the core is a powerful clue, not always the whole story.