kprobes and uprobes
/ KAY-probes, YOO-probes /
Most tracing can only see at points someone planned for in advance. But sometimes you need to look at one specific function — a particular kernel routine, or a function inside a running program — that nobody added a tracing point to. kprobes and uprobes are the mechanism that lets you place a probe on ALMOST ANY function at run time, dynamically, without recompiling anything: a kprobe attaches to an arbitrary kernel function, and a uprobe attaches to an arbitrary function in a user-space program.
Here is the trick, in plain steps. To probe a function, the kernel takes the first machine instruction at the target address and replaces it with a breakpoint instruction (on x86 the single-byte int3, 0xCC), saving the original instruction aside. When execution reaches that address, the breakpoint fires a trap into the kernel; the kernel runs your attached handler (which can read the function's arguments, registers, or memory), then single-steps the saved original instruction and lets execution continue as if nothing happened. A return-probe variant (kretprobe / uretprobe) additionally traps when the function returns, so you can read the return value and measure how long the call took. A uprobe works the same way but the breakpoint is patched into a user-space binary's code, which the kernel intercepts. These probes are typically driven by ftrace or, far more commonly today, attached to eBPF programs.
It matters because it is what makes tracing truly ad-hoc and powerful: you are not limited to the events someone foresaw — you can ask about any named function the moment you realise you need to. This is the dynamic counterpart to the static, stable tracepoint. The honest caveats: because a probe patches a breakpoint into live code, it adds a trap (a few hundred nanoseconds) on every hit, so probing a hot function can hurt; and because it targets an internal function by name or address rather than a stable interface, the probe can break or move when the kernel or the program is updated — a kprobe is precise but inherently fragile across versions, which is exactly why tracepoints exist for the things meant to be watched long-term.
# kprobe: trace every call to the kernel's do_sys_openat2 and print the filename arg $ bpftrace -e 'kprobe:do_sys_openat2 { printf("%s opens %s\n", comm, str(arg1)); }' # uprobe: trace readline() inside /bin/bash, capturing what the user typed $ bpftrace -e 'uretprobe:/bin/bash:readline { printf("typed: %s\n", str(retval)); }'
A kprobe instruments an arbitrary kernel function by name; a uretprobe reads a user function's return value — both placed live, with no recompile.
kprobes target internal implementation details by name, so unlike a tracepoint they offer NO stability promise across kernel versions: a script that works today can silently stop matching after an upgrade if the function is renamed, inlined, or removed.