The hook is the hard part
Guide 2 left you with a powerful little machine: eBPF lets you load a small, verified program into the kernel and run it safely. But a program that runs is useless until it runs at the right moment — the instant a file is opened, a packet arrives, or a particular function is called. That "at the right moment" is the missing half, and it is genuinely the harder half. This guide is about the four attachment points Linux gives you to say where and when your observation fires: ftrace, kprobes, uprobes, and tracepoints. eBPF is the engine; these are the places you can bolt it on.
It helps to see the family split along two lines. The first line is kernel versus user: a kprobe hooks code inside the kernel, while a uprobe hooks code inside an ordinary user-space program like your own a.out or a copy of nginx. The second line is dynamic versus static: a kprobe and a uprobe are dynamic — they can attach to almost any instruction address with no cooperation from the target, found and patched at runtime — whereas a tracepoint is static, a named hook the kernel's authors deliberately placed in the source and promised to keep stable. Each cell of that two-by-two grid trades safety against reach, and knowing which you are using tells you exactly how much to trust what you see.
ftrace: the tracer already living in your kernel
Of the four, ftrace is the gentlest place to start, because it needs no program from you at all — it is a tracing framework built into the kernel and steered entirely through files. It lives under a special directory, /sys/kernel/tracing, and you drive it by writing plain text into those files and reading the results back out. Its headline trick is the function tracer: with one compiler feature, the kernel was built so that every function begins with a tiny placeholder the tracer can switch on, turning the whole kernel into something you can watch function-by-function with no rebuild.
The mental model is a light switch you flip from outside. To trace, say, every call into the function that handles opening files, you write that function's name into a filter file, write the word "function" into the current tracer file, flip tracing on, and then read a trace file to see the live stream of calls with timestamps and CPU numbers. The point is not the exact filenames — they shift between kernel versions — but the shape: ftrace is configured, not programmed. That makes it the fastest way to answer "is this function even being called, and how often, and from where," which is often the only question you actually have.
# turn on ftrace's function tracer for one kernel function, # read a few lines, then turn it back off. (run as root) cd /sys/kernel/tracing echo do_sys_openat2 > set_ftrace_filter # watch just this function echo function > current_tracer # pick the function tracer echo 1 > tracing_on # flip the switch ON cat trace | head # see live calls: # TASK-PID CPU# TIMESTAMP FUNCTION # cat-3812 [002] 91.123456: do_sys_openat2 <-__x64_sys_openat # cat-3812 [002] 91.123512: do_sys_openat2 <-__x64_sys_openat echo 0 > tracing_on # flip it OFF (always clean up)
kprobes and uprobes: a breakpoint that does not stop the world
ftrace shows you the functions the kernel was built to expose. A kprobe goes further: it lets you attach to almost any instruction address in the kernel, even deep inside a function, named or not. The trick will feel familiar if you remember debuggers from the debugging rung. A kprobe works like a breakpoint: the kernel saves the original instruction at your chosen address and overwrites its first byte with a trap instruction. When the CPU reaches that address, it faults into the kprobe machinery, runs your handler, then quietly restores and executes the original instruction and continues. The code you patched never knew it was watched.
But hold the analogy honestly, because the difference from a debugger breakpoint is the whole point. A debugger breakpoint stops the program and waits for a human; that is fatal in production, where halting the kernel halts the machine. A kprobe never stops anything — its handler runs to completion in a few instructions and execution flows on. That is why you can run one on a live server serving real traffic. The cost is real but small: each hit is a trap and a handler, so a kprobe on a function called millions of times a second is measurable. You are buying x-ray vision into a running kernel, and you pay for it in nanoseconds per call.
A uprobe is the same idea aimed at user space. Instead of patching a kernel function, you point it at an address inside an ordinary program or shared library on disk — say the malloc() entry in libc, or a function in your own binary. The kernel sets the equivalent of a breakpoint there, and from then on, every process that executes that code trips your handler. This is how you trace a program you cannot or will not recompile: a uprobe on a function needs nothing from the program's author, no special build, no source. Note the asymmetry, though — a uprobe involves a switch into the kernel on every hit, so it is heavier than a kprobe and far heavier than a hook the program built in for itself, which is exactly the gap the next section fills.
Tracepoints and USDT: the stable hooks the author left for you
Dynamic probes are wonderfully powerful and quietly dangerous, and you should feel both. A kprobe attaches to a function by name and offset — but those names are internal kernel details with no promise of stability. Next kernel release the function might be renamed, inlined away, or split in two, and your kprobe silently attaches to the wrong thing or fails to attach at all. A tracepoint is the cure. It is a hook the kernel's own authors placed at a meaningful spot in the source, gave a stable name, and committed to keeping working across versions, along with a documented set of fields it hands you. Tracing sched_switch via its tracepoint will keep meaning "a context switch happened" for years; a kprobe on whatever function happens to implement it this month will not.
The same idea exists for user-space programs, and there it has a name: USDT, for Userland Statically Defined Tracing. A USDT probe is a tracepoint a programmer deliberately wrote into their own application — a named marker like "a request started" or "a query finished," placed where it is semantically meaningful rather than wherever a function boundary happens to fall. Databases like PostgreSQL and runtimes like the JVM and Node.js ship with USDT probes built in. When the marker is not firing, it compiles down to a single no-op instruction, so it costs essentially nothing; a tracer can flip it live when someone is actually watching.
Putting it together, and the heritage behind it
In practice you rarely poke these files by hand. The four hooks are the plumbing; the tool you actually type is usually bpftrace, a one-liner language that attaches an eBPF program to any of them. A single line says where to hook and what to do: "on every kprobe of this function, count by some key" or "on this tracepoint, record a latency." The same little script can target a tracepoint, a kprobe, a uprobe, or a USDT probe just by changing the prefix that names the hook — which is the whole reason it was worth learning the four as one family. The hooks are the verbs; bpftrace and eBPF are the grammar that strings them into a sentence.
None of this sprang from nowhere. The whole approach — safe, programmable, production-grade tracing of a live system — was pioneered by DTrace, built at Sun Microsystems around 2003 for Solaris. DTrace introduced the now-standard ideas you have been reading: dynamic probes that patch a running kernel, a safe little language to run at each probe, static USDT markers an application author defines, and the discipline of designing all of it to be safe enough to leave on in production. Linux's ftrace, kprobes, uprobes, and tracepoints grew up partly as the Linux world's answer to DTrace, and eBPF eventually became the safe in-kernel engine that ties them together. Crediting the lineage is not trivia — it tells you these are mature, deliberate tools, not clever hacks.
Step back and the map is clean. ftrace is the configure-don't-program tracer already in your kernel; kprobes and uprobes are dynamic breakpoints that attach almost anywhere — in the kernel and in user programs respectively — at the cost of coupling to internal detail; tracepoints and USDT are the stable, author-blessed hooks you should prefer. They live on the kernel-versus-user and dynamic-versus-static axes, and choosing the right cell is most of the skill. With these as your attachment points and eBPF as the engine, guide 4 turns from watching a running system to sampling it continuously and to picking apart a system that has already crashed.