From debugging one machine to debugging a system you cannot pause
Everything you learned in the debugging rung assumed a luxury you will not have in production: you could stop the program. You set a breakpoint, read a backtrace, single-stepped, re-ran with a sanitizer. That works at your desk because you control the process. A live service is the opposite world. It is serving real users right now, the fault fired ten minutes ago and is gone, it may be spread across forty machines, and freezing it to poke around is itself an outage. The question changes from "let me step through it" to "what did the system already record about itself before I arrived?"
Observability is the property of a system that lets you answer questions about its internal state purely from the signals it emits to the outside — without shipping new code, without reproducing the bug, without attaching anything. The word is borrowed honestly from control theory, where a system is observable if its internal state can be reconstructed from its outputs. Applied to software, observability asks: when something weird happens that nobody anticipated, do you have enough recorded output to figure out why, or do you have to redeploy a build with one more print statement and wait for it to happen again? A system you can interrogate after the fact is observable; one you can only re-run is not.
Why monitoring is not the same word
People use "monitoring" and "observability" interchangeably, and the conflation hides a real distinction worth holding crisply. Monitoring watches a fixed, predefined set of things you already decided matter — CPU above 90%, error rate above 1%, disk filling up — and pages you when one crosses a threshold. It answers questions you wrote down in advance. That is genuinely valuable: monitoring is how you learn that something is wrong. But it is structurally blind to the failure you did not foresee, because there is no dashboard for a question nobody thought to ask.
Observability is the broader property that lets you ask new questions after the fact — questions you did not anticipate when you wrote the system. Monitoring tells you the checkout latency spiked; observability lets you then discover it was only requests from one region, only on the new database replica, only for carts over a certain size. The honest framing is not "observability replaces monitoring." Monitoring is the always-on alarm; observability is the rich record that lets you investigate once the alarm rings. You want both, and they share the same raw material — the next section names it.
The three pillars: metrics, logs, traces
The conventional vocabulary calls the raw signals the three pillars — metrics, logs, and traces — and the cleanest way to understand them is by what each one trades away. A metric is a number measured over time: requests per second, the 99th-percentile latency, bytes free. It is tiny and cheap because it is aggregated — you keep the count, not the individual events — so you can store it forever and graph a year of it, but you have thrown away the detail of any single request. Metrics are how you see trends and thresholds cheaply, which is exactly why monitoring is built on them.
A log is a record of a discrete event, classically a line of text: "2026-06-24T03:14:07 ERROR payment failed for order 4192, code 0x2." A log keeps the detail a metric throws away, but the cost flips — logs are voluminous and, written as free text, hard to query. The grown-up form is structured logging: instead of a sentence, you emit a record with named fields (event, order_id=4192, code=0x2, latency_ms=812) so a machine can filter and aggregate them. This guide's successors lean hard on that idea; the takeaway now is that a log answers what exactly happened in this one event, at the price of volume.
A trace follows a single request as it flows through the system, stitching together every stage it touched into one causal story. In a service made of many pieces, one user click might hit the web tier, two internal services, a cache, and a database; a distributed trace records each of those as a span with a start, a duration, and a parent, and ties them all under one trace id. That is the pillar that answers where did the time go and which hop failed — the question neither a metric (too aggregated) nor a single log line (too local) can answer on its own. The last guide of this rung is devoted to it.
one request -> one trace, several spans: [trace 0x7af...] |-- span web-frontend 12 ms | |-- span auth-service 3 ms | |-- span cart-service 45 ms <-- the time went here | |-- span db-query 41 ms <-- and really, here metric: cart_p99_latency_ms = 812 (a number, no story) log: ERROR db-query slow order=4192 ms=41 (one event) trace: the tree above (the whole causal path)
Correlation is the whole game
Here is the part that separates real observability from three disconnected data lakes: the pillars are only powerful when you can pivot between them. A metric pages you that p99 latency jumped. From that spike you want to jump straight to the traces of the slow requests in that window; from a slow trace's failing span you want to jump straight to the logs that span emitted. If your metrics, logs, and traces share no common keys — no request id, no trace id, no consistent service name — each one is a separate dead end and you are back to redeploying with a print statement. Correlation, not collection, is what makes the jump from symptom to root cause possible.
This is why the field converged on shared conventions: a trace id propagated through every hop, structured fields with agreed names, one clock. The open standard for this is OpenTelemetry — a vendor-neutral way to generate, tag, and ship all three pillars with consistent metadata so they can be joined. You do not need it to understand the idea, but it is the reason a modern stack can show you a latency graph, let you click the spike, see the offending trace, and open its logs, all carrying the same trace id. That single id threaded through everything is the spine that turns three piles of data into one explanation.
Honest costs: the observer effect and where this rung goes
Observability is not free, and pretending otherwise produces bad engineers. Every signal you emit costs CPU to produce, network to ship, and disk to store — and worse, the act of measuring can perturb the thing you measure. This is the observer effect: a log line on a hot path adds a syscall and a lock; a trace span allocates and serializes; turning on verbose instrumentation can itself slow a service enough to change its behavior, so the bug you were chasing shifts or vanishes. You already met its cousin in the debugging rung as the heisenbug. The discipline is to make the common case cheap: aggregate into metrics, sample traces rather than recording every one, and keep the hottest paths quiet.
The tension running through the rest of this rung is exactly this: how do you see deep inside a running production system without perturbing it badly or stopping it at all? The answers are the chapters ahead. eBPF lets the kernel run tiny safe programs at chosen events with very low overhead. ftrace, kprobes, and uprobes hook the kernel and your binaries to watch them live. Hardware performance counters read cycle and cache statistics the CPU is already counting, essentially for free. Continuous profilers sample stacks across the fleet to find where CPU and memory actually go. Each is a different way to extract signal from a live system on a strict perturbation budget — and they all serve the goal you now hold clearly: turning a system that cannot be paused into one that explains itself.