Structured and Distributed Tracing in Production

First, make a log a record instead of a sentence

Before we cross machines, fix the humble log line, because everything else is built on it. In the debugging rung you printed sentences for a human to read: "payment failed for order 4192." That is fine when one person reads one screen, and useless when a million such lines land in a search system and you need every failure for orders over a certain size, in one region, in a five-minute window. The problem is that a sentence has no fields — to a machine it is one opaque blob of characters. Structured logging is the simple, load-bearing fix: emit each event as a record of named key-value pairs, not prose.

unstructured (a sentence, one opaque blob):
  2026-06-24T03:14:07 ERROR payment failed for order 4192, code 0x2

structured (named fields a machine can filter and join):
  {
    "ts":       "2026-06-24T03:14:07Z",
    "level":    "error",
    "event":    "payment_failed",
    "order_id": 4192,
    "amount":   18999,
    "region":   "eu-west",
    "code":     "0x2",
    "trace_id": "0x7af3c1...",
    "span_id":  "0x91b0"
  }

The format matters less than the discipline. JSON is common, but so are key=value lines and binary encodings; what counts is that the fields are named and consistent across services, so amount means the same thing everywhere and a query engine can pivot on it. Recall from the first guide that a log keeps the per-event detail a metric throws away — structuring it is what makes that detail queryable at scale instead of a wall of text you grep through and pray. This is also the natural seam to attach the two identifiers that turn a lone log into part of a trace, which is where we are headed.

The anatomy of a span

The first guide introduced a distributed trace as a tree of spans following one request. Now let us open up the span itself, because it is a precise little object and most of tracing is just creating and linking these correctly. A span represents one unit of work with a duration — a database query, an outgoing HTTP call, a chunk of computation. Concretely it carries: a span_id (unique to this span), a trace_id (shared by every span in the same request), a parent_span_id (the work that caused this work), a name, a start time and end time, and a bag of attributes — your structured fields again: db.statement, http.status_code, region.

Those three ids are the entire trick. The span_id names this node; the parent_span_id is the edge to its parent; the trace_id labels which tree it belongs to. Give every span the same trace_id and a correct parent_span_id and you have not stored a tree at all — you have stored a flat list of spans, each pointing at its parent, and the tree reconstructs itself when a viewer groups by trace_id and links child to parent. That is why a span can be emitted independently from forty machines, buffered, and shipped separately, yet still reassemble into one coherent picture: the structure lives in the ids, not in any central coordinator. It is the same id-threading idea the first guide called the spine of correlation, now made concrete.

Make it concrete with four spans sharing trace_id 0x7af3c1. Span A is web-frontend with no parent and a 12 ms duration; span B is auth with parent A and 3 ms; span C is cart with parent A and 45 ms; span D is db-query with parent C, 41 ms, carrying the attribute db.rows=0. Stored on disk that is just four flat rows, each naming its parent and possibly written by a different machine. A viewer groups them by trace_id and follows the parent links to draw A branching to B and to C, and C down to D — and the shape immediately shows that 41 of cart's 45 ms was the database query. No central coordinator drew that tree; the parent_span_id edges did.

Propagation: carrying the context across a process boundary

Here is the question that makes distributed tracing genuinely hard, and it is a systems question you are now equipped to feel. Inside one process, passing the current trace_id and span_id from function to function is easy — they sit in thread-local storage or get passed along the call chain. But the moment the web service makes an outgoing call to the cart service, that is a different process, very likely on a different machine, reached over a socket. None of your in-memory context crosses that wire by itself. The trace_id and the calling span_id have to be serialized into the request and read back out on the other side, or the child span on the cart service will have no idea which trace it belongs to.

This step is called context propagation, and for HTTP the convention is dead simple: the caller writes the ids into request headers and the callee reads them. The W3C standard header is traceparent, carrying a version, the trace_id, the caller's span_id (which becomes the child's parent_span_id), and flags. The receiving service parses that header, starts its own span with the inherited trace_id and the inherited parent, and the chain continues — across as many hops as the request takes. The mechanism is unglamorous on purpose; it is just a string riding along in the metadata of whatever protocol you already speak. But this is exactly the boundary the rung's overarching warning targets: a trace is only as connected as its weakest hop, and any service that drops or forgets to forward the header silently breaks the tree below it.

You cannot afford to trace everything: sampling

Now the cost reasserts itself, exactly as the first guide promised with the observer effect. A busy service handles tens of thousands of requests per second; recording a full span tree for every single one would generate more trace data than the actual work, saturate the network shipping it, and cost a fortune to store. So in production you almost never trace every request — you sample. The honest tension is real: a trace you did not record is one you cannot inspect later, so sampling is deliberately throwing away evidence to keep the system affordable, and the whole craft is throwing away the boring evidence while keeping the interesting kind.

There are two broad strategies, and the difference is when you decide. Head-based sampling decides at the very first span, before the request has even finished: roll a die at the front door, say "keep one in a hundred," and propagate that keep-or-drop decision in the trace flags so every downstream hop honors it and the trace stays whole. It is cheap and simple, but it is blind — it commits before knowing whether this request will be the slow one or the failing one. Tail-based sampling buffers all the spans of a trace and decides after it completes, so it can keep every trace that erred or exceeded a latency threshold and drop the boring fast successes. That is far more useful for chasing tail latency and errors, but it costs real memory and infrastructure to hold every trace in flight long enough to judge it.

Sampling is precisely why traces and metrics are partners rather than rivals, and why the three pillars earn their separate keep. Because you sampled traces, you do not have a trustworthy count of total requests from your traces alone — a one-in-a-hundred sample undercounts everything by design. So you still keep an unsampled metric for the rate and the error percentage: the metric, cheap and aggregated, tells you reliably that error rate doubled across all requests, and the kept sample of failing traces tells you why on the examples you did retain. Reach for the metric for the trustworthy aggregate, the trace for the causal detail; they answer different questions on purpose.

Putting it together, and the honest limits

Walk one real investigation end to end to see every piece engage. A monitoring alert (a metric crossing a threshold) pages you: checkout p99 latency doubled. You open the latency graph, click the spike, and pivot to the sampled, kept slow traces in that window — tail-based sampling made sure the slow ones survived. You pick one trace, see its span tree, and the time is plainly not in the web tier or auth but in a single db-query span that took 41 of the request's 45 ms. You click that span and jump to the structured logs it emitted, joined by its trace_id and span_id, and read db.rows=0 with a retry loop — the query was scanning a table whose index had just been dropped. Metric to trace to log, each hop a pivot on a shared id: that is the symptom-to-root-cause path the whole rung has been building toward.

The open standard that makes this work across a polyglot fleet is OpenTelemetry — a vendor-neutral set of APIs, libraries, and a wire format for emitting all three pillars with consistent metadata, so a span produced by a service written in one language and a log produced by another can still be joined on the same trace_id. Its value is not that it invents tracing; it is that it standardizes the field names and the propagation headers so the correlation actually happens across teams that never coordinated. You do not need it to grasp any idea here, but it is why a modern stack can offer the click-through-the-spike experience instead of forty incompatible tracing dialects that never join up.