A different kind of bug, a different kind of tool
Across this rung you built a kit for looking inward. gdb froze your program and let you read its registers and backtrace; core dumps let you do that on a corpse, after the crash; Valgrind and the sanitizers watched every load and store to catch a use-after-free or a buffer overflow. All of those answer the question *what is my code doing to memory?* But a whole class of bugs lives outside that question entirely. Your program opens the wrong file, hangs waiting on a socket that never answers, exits with a mysterious status, or works perfectly on your machine and dies on the server. The memory is fine. The bug is in the conversation your program is having with the outside world.
Recall the boundary you learned two rungs ago. A system call is not an ordinary function call: it crosses from your user-space program into the kernel, the only code allowed to touch hardware, the filesystem, or the network. Every file your program reads, every byte it sends over a socket, every process it spawns — all of it happens by your program asking the kernel, through calls like open(), read(), write(), and connect(). That request stream is the conversation. And because it is a sharp, well-defined boundary, we can stand at it and write down every single thing that crosses. The tool that does this is strace.
strace: every system call, written down
Run your program under strace and it prints one line for every system call the program makes: the call's name, its arguments (decoded into readable form — flags spelled out, strings shown in quotes), and its return value. You do not recompile, you do not need source code, you do not even need debug info. strace works by using the kernel's own tracing machinery (the ptrace() facility) to intercept your process at each kernel boundary. The cost is real — every traced call is stopped and reported, so a program can run many times slower under strace — but for seeing what happened, nothing is more direct.
$ strace -f -e trace=openat,read,write ./myprog openat(AT_FDCWD, "config.txt", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/config.txt", O_RDONLY) = 3 read(3, "port=8080\nhost=localhost\n", 4096) = 26 read(3, "", 4096) = 0 write(1, "listening on 8080\n", 18) = 18 +++ exited with 0 +++
Read that transcript again, because it shows strace's superpower. The bug here is invisible from inside the program — the code did open a config file and did read it, so it never crashes. But strace reveals that the first openat() failed with ENOENT and the program quietly fell back to a different file in /etc. If your program is reading the wrong settings, this is the line that tells you why. strace is the tool for *'it works on my machine'* — the file it cannot find, the permission it is denied, the library it loads from an unexpected path, the errno it gets back and ignores. You are not guessing about the environment any more; you are watching the program touch it.
ltrace: the other side of the boundary
strace sees system calls — your program talking to the kernel. But much of what your program does never reaches the kernel at all. When you call malloc(), strlen(), or printf(), you are calling a function inside a shared library (the C library, libc), and most of that work happens entirely in user space. malloc() usually just carves a block out of memory the library already owns; it only descends to a real system call (like mmap() or brk()) once in a while, when it needs more memory from the kernel. strace cannot see the malloc() — only the rare syscall underneath it. To watch the library calls, you need its sibling, ltrace.
ltrace prints a line for each call your program makes into a shared library, again with arguments and return values decoded. This makes the distinction from the previous rung concrete and visible: it is exactly the difference between a library call and a system call. strace shows you the kernel boundary; ltrace shows you the library boundary. Tracing the same program both ways and laying the outputs side by side is one of the clearest ways to feel where user space ends and the kernel begins — ltrace shows ten malloc() calls, strace shows the single mmap() underneath them all.
printf-debugging, done with respect
There is a debugging tool older than all the others, one that needs no special install and runs anywhere: printing a line to say where you are and what you saw. printf-debugging has a bad reputation among people who think gdb makes it obsolete, and that reputation is wrong. A debugger is unbeatable for inspecting a single frozen moment, but a log is unbeatable for seeing a sequence over time — especially for a bug that vanishes under a debugger, or one that only appears once in ten thousand runs on a server you cannot attach to. The two tools answer different questions, and a serious programmer keeps both sharp.
But do it with respect, because naive printing lies to you in two famous ways. First, buffering: printf() writes to stdout, which is line-buffered to a terminal but fully buffered when redirected to a file or a pipe. So if your program crashes, the last few printf() lines — often the most important ones, right before the crash — may still be sitting in the buffer and never reach the file. The output you see ends before the real failure, sending you to hunt in the wrong place. Print to stderr (which is unbuffered) for debug lines, or call fflush(stdout) after each one, or set the stream to unbuffered. Second, interleaving: two threads printing at once can splice their output mid-line into nonsense, so always tag each line with its thread or process id when concurrency is in play.
Good logging: from scattered prints to a record you can trust
Scattered printf() calls you add and delete are fine for a five-minute hunt. But the same instinct, made disciplined, becomes logging — a permanent, structured record your program emits as it runs, the thing you read when a failure happened yesterday and you cannot reproduce it. The difference between a stray print and a good log line is what you put in it. A good line carries a timestamp (so you can order events and measure gaps), a severity level (so you can later show only the warnings and errors), a stable location or component name (so you can grep for one subsystem), and — this is the part people skip — the actual values that matter, not just "got here". "failed to open config" is a lament; "open('/etc/app.conf') failed: errno=2 (ENOENT)" is a diagnosis.
Severity levels are the dial that keeps logging useful instead of overwhelming. The common ladder is DEBUG (firehose detail, off in production), INFO (normal milestones), WARNING (something odd but handled), ERROR (an operation failed), and FATAL (the program cannot continue). The discipline is to log at the right level and let a runtime setting choose how much to show: chatty DEBUG while you hunt, quiet WARNING-and-above when things are healthy. This ties straight back to the rung on errors — every place you check a return value of open(), read(), or malloc() is a natural place to log the failure with its errno before you handle it, turning your error-checking into a paper trail.
Step back and see how this whole rung fits together. gdb and core dumps freeze a moment and let you inspect it; Valgrind and the sanitizers watch memory for the bugs you cannot see; strace and ltrace stand at the kernel and library boundaries and transcribe the conversation; and logging records the story over time that the others cannot. No single tool is the answer — the skill is knowing which question you have. Is it *what is the state right now?* (a debugger), *what touched this memory?* (a sanitizer), *what did it ask the system to do?* (strace), or *what was the sequence that led here?* (a log). Master that choice and you stop guessing — you start, as this rung promised, to see.