tracing and profiling
Tracing and profiling are the two main ways to look inside a running system and answer the question every performance engineer asks: where is the time going? Picture a doctor with two instruments — a stethoscope that samples your heartbeat now and then to estimate your overall state, and an EKG that records every single beat as it happens. Profiling is the periodic-sample instrument; tracing is the record-every-event instrument. Both turn an opaque, fast-moving system into something you can actually see.
Profiling works by sampling: many times a second it interrupts the program and notes which function is currently running, then builds a statistical picture of where most time is spent — perfect for finding the hot spot without recording everything. Tracing instead records specific events as they occur — every system call, every context switch, every function entry — giving an exact timeline you can replay to understand a sequence or a rare glitch. The classic toolkit reflects this split: perf samples CPU activity and reads hardware performance counters; ftrace traces events and function calls inside the Linux kernel; DTrace (from Solaris, now on macOS and BSD) pioneered safe, programmable, system-wide tracing of both kernel and applications. Each lets you watch what the OS is really doing instead of guessing.
The honest caveat is the observer effect: measuring costs something, and the heavier the instrumentation, the more it perturbs the very thing you are measuring — tracing every event can itself slow the system and flood you with data. Profiling is cheap but only statistical (it can miss rare events between samples); full tracing is exact but expensive. Good practice is to start with light, low-overhead sampling to find the rough area, then add focused tracing only where you need the detail.
A web service is slow under load. You run a profiler for ten seconds and it reports that 70% of CPU time is inside one JSON-parsing function — a hot spot you would never have guessed. You then trace that function's calls to confirm it is being invoked far more often than expected, and fix the real cause: a loop reparsing the same data.
Profiling finds the hot spot; tracing confirms the exact sequence behind it.
Beware the observer effect: instrumentation is not free. Heavy tracing can slow the system enough to change its behavior, so the numbers you collect may not reflect the system as it runs untouched.