First, an honest rule: measure before you touch
You now understand the machinery of an operating system from the inside: processes and threads, the scheduler, paging and the TLB, file systems, interrupts and I/O. This guide is about a different question — not 'how does it work?' but 'why is it slow, and how do I make it fast?' The single most important lesson of performance tuning is also the most often ignored: you must measure first. Intuition about where a program spends its time is wrong far more often than it is right, and changing code based on a guess usually just moves the bottleneck somewhere you cannot see.
Think of it like a plumber called to a house where the water is slow. The amateur replaces the tap he can see; the professional first puts a pressure gauge on the line and finds the clog is three rooms away. Performance work is the same discipline. Before you rewrite a loop, add a cache, or twiddle a kernel knob, you attach instruments and let them tell you where the time actually goes. Only then do you change one thing, measure again, and keep the change only if the number moved. Tune in the dark and you will optimize the wrong 95% of the program.
Where the time really goes
To know what to measure, you need a feel for the costs that dominate a modern machine. The deepest one is the memory wall: a single access that misses every cache and goes to main RAM can cost on the order of a hundred CPU cycles, during which the processor would otherwise have done a hundred bits of arithmetic. This is exactly why locality of reference — keeping the data you touch next physically close to the data you just touched — quietly governs real-world speed. A loop that strides nicely through an array can be ten times faster than one that hops randomly through memory of the same size, not because it does less work but because it waits less.
The second great cost is crossing the boundary you met long ago: the trip from user mode into the kernel and back. Every system call — every read, write, or network send — is a controlled fall through the door into kernel mode, and that crossing is not free: registers must be saved, the mode switched, caches and the TLB partly disturbed, and the whole journey reversed on return. One system call is cheap; a million of them, one byte at a time, can dominate your program. A huge fraction of real tuning is simply doing fewer, bigger crossings: read 64 KB at once instead of one byte sixty-five thousand times.
The third cost only appears when many cores work together: contention. When two threads on two cores both want the same lock or the same cache line, the hardware must shuttle ownership back and forth between them, and they spend their time fighting rather than working. This is the heart of multicore scalability: a program that runs beautifully on one core can run no faster — or even slower — on sixteen, if those sixteen are all queued behind a single shared lock. The cure is to share less: give each core its own data, or use lock-free structures, so cores stop tripping over each other.
The toolkit: perf, ftrace, and eBPF
So how do you actually see these costs? With the tracing and profiling toolkit. The two ideas behind it are worth keeping straight. Profiling is sampling: many times a second, an interrupt fires and the tool snapshots which function is running right now. Collect a hundred thousand such snapshots and the functions that appear most often are where the time goes — a statistical X-ray of the CPU. Tracing is recording specific events as they happen: 'a system call just started', 'this lock was just taken', 'a packet just arrived'. Profiling answers 'where is the CPU spending time?'; tracing answers 'what exactly happened, in order, and how long did each step take?'
On Linux, perf is the classic profiler: it leans on the CPU's hardware performance counters and on periodic sampling to tell you the hottest functions, the cache-miss rate, the branch-mispredict rate — the raw vital signs. ftrace is the kernel's built-in tracer: a framework baked into the Linux kernel that can log function entries and exits and a vast menu of predefined trace points as the kernel runs. Both are mature and powerful. But both share a limitation that is easy to miss: you largely get the data the tool authors decided in advance to expose, in roughly the shape they chose. If your question is unusual, the classic tools may simply not have an answer ready.
This is exactly the gap eBPF fills, and why it is the most important systems idea of the last decade. eBPF lets you write a tiny, safe program and attach it to almost any event inside the running kernel — a system call, a network packet, a function entry, a context switch — and run your own logic there, live, without rebooting or recompiling the kernel. Instead of being limited to the data the kernel already exports, you ask your own question, in your own words, at the exact spot where the answer lives. It is the difference between reading the instruments someone bolted to the engine and being able to solder on a new sensor wherever you like, while the engine keeps running.
How eBPF can be safe inside the kernel
That last claim should make you nervous, and it should. Running your own code inside the kernel is normally the most dangerous thing imaginable: a single bad pointer or infinite loop in kernel space crashes the whole machine, because down there nothing protects the kernel from itself. So how can eBPF possibly let ordinary programs inject code into that sacred space and still be safe? The answer is a beautiful piece of engineering: the kernel refuses to simply trust your program, and instead proves it harmless before it ever runs.
- You write a small program (often in a restricted subset of C) describing what to do at some kernel event — for example, 'when any process calls read, add one to a counter kept per process'.
- It is compiled into eBPF bytecode — a tiny, restricted instruction set, deliberately not a full programming language with arbitrary power.
- The bytecode is handed to the kernel's verifier, which examines every possible path through the program before loading it.
- The verifier rejects anything unsafe: no unbounded loops (so it must end), no reads of arbitrary memory, no out-of-bounds access. If it cannot prove safety, the program is refused — full stop.
- Only a program that passes is attached to the event and run, often through a just-in-time compiler to near-native speed, safely inside the kernel.
Read those steps again and notice the trade. eBPF buys its safety by giving up generality: the verifier can only approve programs simple enough to prove safe, so eBPF deliberately is not a place to run arbitrary software. That restriction is the whole point — it is what makes 'run untrusted code in the kernel' a sane sentence instead of an oxymoron. The result is that observability, networking, and security tools can now live exactly where the action is, with the kernel's blessing, rather than copying mountains of data out to a user-space program that watches from a distance.
Going faster still: bypassing the slow path
Sometimes measurement reveals that the kernel itself is the bottleneck. For a network card receiving ten million packets a second, even the well-tuned cost of one trip through the kernel per packet — the interrupt, the system call, the copying — becomes unaffordable. The radical response is kernel bypass: hand a device directly to a user-space program and let it talk to the hardware itself, skipping the kernel's general-purpose path entirely. Frameworks like DPDK do this for networking; the result can be dramatically faster, because you have stopped paying for crossings you no longer need.
But be honest about the cost of bypass — there is no free speed. By stepping around the kernel, that user-space program also steps around everything the kernel was doing for everyone: the fair sharing, the safety checks, the clean abstraction that let other programs use the same card. Bypass trades generality and protection for raw throughput, which is a fine deal for one dedicated high-speed appliance and a terrible deal for a shared machine. A gentler middle road on the storage and I/O side is io_uring: a newer interface that keeps the kernel in charge but slashes the number of crossings by letting a program queue many asynchronous requests and collect their results in batches, rather than making one blocking system call at a time.
One last frontier worth naming, because it runs opposite to raw speed: power management. On a phone or a laptop, the fastest setting is rarely what you want — burning full clock speed flat-out empties a battery and cooks a lap. So a modern OS constantly negotiates speed against energy: dropping cores to sleep when idle, lowering the clock when the work is light, racing to finish a burst quickly so the chip can sleep sooner. 'Fast' on a battery means 'finishes the job using the least energy', not 'always runs at maximum', and a good power policy is its own kind of performance tuning — measured in joules instead of seconds.
Putting it together
Step back and the whole guide collapses into a loop you can carry anywhere. Measure to find which of the four costs — CPU, memory, I/O, or contention — is actually hurting. Reach for the right instrument: perf to profile the hot functions, ftrace or eBPF to trace the exact events, hardware counters for cache and branch behaviour. Change one thing — improve locality, batch the system calls, shrink the lock, bypass the kernel only where it truly pays. Then measure again, and keep the change only if the number moved. That loop, not any single trick, is what 'making systems fast' actually means.