a USDT probe
/ you-ESS-dee-TEE /
A tracepoint gives the kernel stable, developer-placed observation points. USDT is the same good idea, but for user-space programs: it lets the AUTHOR of an application or library bake named, stable tracing points directly into their own code, at events that matter to that program — 'a query started', 'a garbage collection began', 'a request was retried'. USDT stands for User Statically-Defined Tracing.
Here is how it works in practice. A developer marks a spot in their source with a probe macro, giving it a name and the values to expose (for example, a probe named query__start carrying the SQL text and a connection id). When compiled, this leaves a special marker recorded in the binary's notes section that says 'there is a probe named query__start at this address, with these arguments'. Crucially, when no tracer is attached the probe compiles down to essentially a single no-op instruction, so it costs almost nothing in normal operation. When a tracing tool (bpftrace, BCC, or perf) is asked to attach to that probe, it reads the marker, finds the address, and places a uprobe-style breakpoint there to capture the named arguments each time the spot is reached. Databases and language runtimes such as PostgreSQL, MySQL, Python, Node.js, and the JVM ship with USDT probes precisely so operators can trace meaningful application events without reading the source.
It matters because it bridges a gap: dynamic uprobes can reach any function but only see raw, unstable internals, whereas USDT gives you stable, MEANINGFUL, author-blessed events with friendly arguments — the application's own observability contract. The honest caveats: USDT probes only exist where the author chose to add them and only if the build included them (some distributions compile them out); the inactive probe is nearly free but not perfectly free; and because the names and arguments are an interface the author maintains, they can still change across major versions, though that is the explicit thing a good author tries to avoid.
// in the application's C source, the author plants a named probe: DTRACE_PROBE2(myapp, query__start, sql_text, conn_id); // later, an operator attaches with no recompile: $ bpftrace -e 'usdt:/usr/bin/myapp:myapp:query__start { printf("%s\n", str(arg0)); }'
The author defines a stable, named application event with friendly arguments; an operator traces it live without touching or rebuilding the program.
Despite the macro name DTRACE_PROBE, USDT is not DTrace — the probe-definition convention was inherited from DTrace, but on Linux the probe is consumed by eBPF/bpftrace/perf, not by DTrace itself.