The problem eBPF solves
From the previous guide you carry a sharp distinction: observability is the power to ask new questions of a running system, not just watch a fixed set of dials. But the most interesting questions live inside the kernel — why is this disk write slow, which process is flooding the network, how long does each system call really take. Your ordinary program lives in user space and is walled off from all of that. The classic way to reach across the wall was to write a loadable kernel module: full privilege, no safety net, one stray pointer and the whole machine panics. That is a heavy, dangerous tool to reach for just to measure something.
eBPF — the extended Berkeley Packet Filter — is the way out of that dilemma. It lets you load a tiny program of your own into the running kernel and attach it to a precise event: a packet arriving, a function being entered, a tracepoint firing, a system call returning. When that event happens, your code runs, right there in kernel mode, with direct access to the kernel's data — and then the kernel keeps going. No reboot, no module compiled against a specific kernel version, and crucially no way for your program to crash the machine. The name is a historical accident; the original BPF (1992) was a small filter language for tcpdump, and eBPF extended that idea into a general-purpose in-kernel execution engine. The packet-filtering ancestry is just where it was born.
A virtual machine living inside the kernel
How can the kernel run your code without trusting your code? The answer is that it does not run your raw machine instructions at all. eBPF defines its own small virtual machine — an abstract CPU with eleven 64-bit registers (called r0 through r10), a fixed-size stack of 512 bytes, and a compact instruction set of its own. You write your logic in a restricted dialect of C; a compiler like clang turns it not into x86 or ARM machine code, but into eBPF bytecode — instructions for this imaginary CPU. Because the kernel defines that machine, it controls exactly what each instruction is allowed to do. This is the same idea as the WebAssembly runtime you may have met: a portable, sandboxed instruction set that a host gets to interpret on its own terms.
Running an interpreter for every packet would be far too slow, though — defeating the point of measuring without disturbing. So the kernel takes a second step: once a program is accepted, an in-kernel JIT compiler (just-in-time) translates the verified eBPF bytecode into the real native machine code of the host CPU, register for register. From then on your trace handler runs at essentially native speed, as a tight sequence of x86 or ARM instructions, not as an interpreted loop. So the bytecode is a transport and verification format — small, checkable, portable — and the JIT is what makes it fast. Verify slowly and carefully once; run quickly forever after.
restricted C source
| clang -target bpf
v
eBPF bytecode (instructions for the in-kernel VM)
| bpf() syscall: load into kernel
v
+-------------------- VERIFIER --------------------+
| walks every path; proves: no out-of-bounds |
| memory, no infinite loop, no leaked kernel ptr, |
| every map access bounds-checked. REJECT or PASS. |
+--------------------------------------------------+
| pass
v
JIT -> native x86 / ARM machine code
| attach to event (tracepoint, kprobe, packet hook)
v
event fires --> your code runs in-kernel at native speed
--> writes results into a BPF map
| read out from user space
v
your tool prints the histogram / count / latencyThe verifier: how the kernel proves your code is safe
Everything hinges on the verifier, and it is worth understanding what it actually does rather than treating it as magic. When you load a program through the bpf() system call, the verifier performs a static analysis before a single instruction runs: it walks every possible path through your bytecode, simulating the range of values each register could hold at each step. It is proving theorems about your program — and it rejects anything it cannot prove safe. The big things it insists on: the program must terminate (historically loops were banned outright; modern kernels allow bounded loops the verifier can prove will end), so an eBPF program cannot hang the kernel. Every memory access must be provably in bounds — you cannot read off the end of a packet or past your 512-byte stack. And you cannot leak a raw kernel pointer back to user space, which would defeat kernel address-space protections.
This is the honest heart of eBPF, and it cuts both ways. The good side: a verified program genuinely cannot corrupt kernel memory or loop forever, which is exactly why operators trust it in production where a kernel module would be too risky. The painful side, which every eBPF author meets: the verifier is conservative. It rejects many programs that are in fact correct, simply because it cannot prove they are safe within its analysis budget. You will write a loop you know terminates and watch the verifier reject it; you will add a bounds check that looks redundant to a human but that the verifier demands before it will let you touch a byte. The error messages are notoriously cryptic. This is not a bug — it is the price of the guarantee. A verifier that accepted your correct-but-unprovable program would also accept someone's subtly-broken one.
Maps: how the in-kernel program talks to you
Your eBPF program runs in the kernel, fires thousands of times a second, and has only its tiny stack — so where do its results go, and how do you ever read them? The answer is BPF maps. A map is a key-value data structure that lives in the kernel and is shared between your eBPF program and your user-space tool. There are several flavors — a hash map, an array, a per-CPU array, a ring buffer for streaming events — but the core idea is one: the in-kernel program writes into the map as events happen, and your user-space program reads from the same map to display what it found. The map is the bridge across the user/kernel boundary that the trace itself cannot cross.
This design is what makes eBPF cheap enough to leave on in production — and that matters, because of the observer effect you met earlier: any measurement disturbs the thing measured. Consider counting system calls per process. The naive approach is to log every call and have user space tally them — but that copies millions of records out of the kernel and crushes performance. The eBPF way is to keep a hash map keyed by process id, and have the in-kernel program do count = count + 1 on every call. The expensive work — counting — happens in-kernel at native speed, touching only one map entry; user space wakes up once a second and reads the already-aggregated totals. You moved the aggregation to where the data is, instead of moving the data to the aggregation. That is the whole reason eBPF can profile a busy production server without bringing it to its knees.
From bytecode to a one-line tool: bpftrace
Writing restricted C, compiling to bytecode, loading through bpf(), managing maps, fighting the verifier — that is a lot of machinery just to count something. Most of the time you do not touch it directly, because higher-level tools generate all of it for you. The friendliest is bpftrace, a tiny high-level language for ad-hoc tracing. You describe what to trace and what to record, and bpftrace compiles your one-liner down into eBPF bytecode, loads it, sets up the maps, and prints the results — the whole pipeline from the diagram, hidden behind a single line. It is the eBPF equivalent of how you would reach for a quick awk script instead of writing a C program for a throwaway question.
A concrete example makes it land. Suppose you want a histogram of how long the read() system call takes, across the whole machine, live. In bpftrace that is roughly one line: attach to the entry and exit of the read tracepoint, record a timestamp on entry, and on exit subtract to get the duration and feed it into a histogram aggregation. Behind that line, bpftrace built two eBPF programs, a map to stash the entry timestamps keyed by thread, and a histogram map for the durations — then on exit it printed a neat power-of-two histogram. You learned a real, hard-to-get number about a live production system, asked as a fresh question, with the machine barely noticing. That is observability delivered.
Where this sits, and what comes next
Step back and hold the whole shape. eBPF is a small in-kernel virtual machine; you compile restricted C to its bytecode; a verifier proves the bytecode is safe before it runs; a JIT turns it into native code so it runs fast; it attaches to kernel events; and maps carry results out to user space. That one paragraph is the entire mechanism. The deep reason it is special is the trade it strikes — in-kernel reach without in-kernel trust — purchased by accepting a conservative verifier that will sometimes reject correct code. Knowing this, you can reach for eBPF-based tools with real understanding of what they cost and what they guarantee, instead of as a black box.
But we glossed over one question this whole guide leaned on: what exactly are the events you attach to? We spoke loosely of 'a function being entered' and 'a tracepoint firing,' and those are the real attach points the next guide unpacks. It steps below eBPF to the kernel's own tracing plumbing — ftrace as the built-in function tracer, kprobes and uprobes that can dynamically instrument almost any kernel or user function, and static tracepoints and USDT probes that developers place deliberately. eBPF is frequently the program you attach at those points; the next guide is the places you can attach it. Together they are the foundation that the rung's later guides — on continuous profiling and crash analysis, and on distributed tracing — quietly build on.