Why one client is easy and a thousand is hard
In the last few guides you built a working TCP server: you called accept to take a new connection, then read and wrote on it like a file. That server is correct, and for a single user it is delightful. The trouble starts the moment two people show up at once. To understand why, we have to look closely at one small word that has been hiding in every call you made: blocking.
A blocking call is one that puts your program to sleep until it can finish. When you call read on a socket and no data has arrived yet, the kernel does not return an error; it simply parks your thread, frozen, until a byte shows up. The same is true of accept when no client is waiting, and of write when the send buffer is full. From your code's point of view, the line of execution just stops dead, sometimes for milliseconds, sometimes forever. For one client that is fine, because there is nothing else you wanted to be doing anyway.
Now picture two clients, Mia and Theo. Your single thread blocks inside read waiting for Mia to send something. Mia steps away for coffee. Theo, meanwhile, is hammering the server with urgent requests, and every one of them sits unanswered, because your thread is asleep on Mia. One slow or idle client has frozen everyone. This is the central problem of network programming at scale: a server must serve many connections that are each ready at unpredictable, independent moments, but a blocking call can only ever wait for one thing at a time.
The obvious fix, and why it eventually breaks
The first idea everyone reaches for is to give each client its own helper. When accept hands you a new connection, you spin up a fresh thread (or process) whose only job is to read and write that one socket. Now Mia's thread can sleep on her coffee break without touching Theo's thread at all. This works, it is easy to reason about, and for a few dozen or even a few hundred connections it is a perfectly sensible design. Many real servers do exactly this.
But scale it up and the seams show. Each thread needs its own stack, often around a megabyte of memory, so ten thousand idle connections can cost you gigabytes of RAM doing nothing. Worse, the operating system must constantly switch the CPU between all those threads, and each switch costs time saving and restoring state. When most of your ten thousand connections are idle most of the time, which is exactly the shape of a real chat server or web server, you are spending enormous resources to keep thousands of sleeping threads ready just in case. The waste is in the waiting, and there are far too many waiters.
Non-blocking sockets: never wait, just ask
The first half of the breakthrough is to change how the socket itself behaves. You can flip a socket into non-blocking mode (with a small system call, fcntl with O_NONBLOCK on Unix). After that, a call that would have put you to sleep instead returns immediately. If you call read and no data is waiting, it does not park your thread; it returns right away with a special signal (the error code EAGAIN, meaning try again later). If write cannot accept everything because the send buffer is full, it takes what it can and tells you how much, rather than freezing until there is room.
On its own this seems useless, even worse. If read just keeps saying try again, what stops you from spinning in a tight loop, asking ten million times a second and burning a whole CPU core to discover nothing has changed? That busy-waiting would be a disaster. Non-blocking mode by itself does not solve the problem; it only removes the freezing. We still need something to answer a different, far more useful question: out of all my connections, which ones are ready right now, so I can ask only those?
Notice this maps cleanly onto something you already know. TCP gives your application a byte stream, not neat messages, so a single read may return half a request or two requests glued together. Non-blocking mode makes this even more visible: you read whatever bytes happen to be ready this instant, stash them in a buffer, and only act once you have a whole message according to your own framing rule. The non-blocking world forces you to be honest about the fact that data arrives in dribs and drabs, on no schedule of yours.
I/O multiplexing: one watcher for every connection
The missing piece is I/O multiplexing: a way to hand the kernel a whole list of sockets and say, put me to sleep, but wake me the instant any one of these becomes ready, and tell me which. Now your single thread blocks in exactly one place, on the watcher, instead of on any individual socket. The thread sleeps cheaply while every connection is quiet, and the moment Theo's data arrives, it wakes, learns Theo is ready, and serves only him, leaving Mia's idle socket untouched. One thread, ten thousand connections, and effort spent strictly in proportion to real activity.
The classic tools for this are select and poll. Both let you pass a set of sockets and block until some are ready. They work, and they are portable, but they share a fatal flaw at large scale: every single call, you hand over the entire list of (say) ten thousand sockets, and the kernel scans all ten thousand to find the handful that are ready. You pay a cost proportional to the total number of connections on every wakeup, even when only one connection had news. Watching grows more expensive the more you watch, which is exactly backwards from what you want.
This is where epoll comes in (its cousins are kqueue on BSD and macOS, and IOCP on Windows; the idea is the same). Instead of re-handing the whole list every time, you register each socket with the kernel once, and the kernel maintains the interest list for you. When you ask which sockets are ready, it returns only the ready ones, with no scan of the idle majority. The cost of each wakeup now scales with the number of active connections, not the total. Ten thousand mostly-idle connections become genuinely cheap to watch, and that single change is what made serving thousands on one thread routine.
The event loop, step by step
Put the two halves together, non-blocking sockets plus an epoll watcher, and you get the heart of nearly every high-performance server, from Nginx to Redis to Node.js: the event loop. It is a single thread that spends its life in one tight rhythm, sleep until something is ready, then do exactly the ready work, then sleep again. Here is the shape of it.
set listening socket to non-blocking
epoll_create -> make the watcher
epoll_ctl ADD -> register the listening socket
loop forever:
ready = epoll_wait(...) # sleep here until something happens
for each socket in ready:
if it is the listening socket:
accept all pending clients (loop until EAGAIN)
set each new socket non-blocking, epoll_ctl ADD it
else if readable:
read whatever bytes are there (loop until EAGAIN)
append to that connection's buffer
while buffer holds a full message (your framing rule):
process it, queue a reply
else if writable:
send as much of the queued reply as fits, keep the rest- The thread blocks in epoll_wait. It is asleep, using no CPU, while all ten thousand connections sit quiet. This is the only place it ever waits.
- Theo's data arrives at the network card; the kernel marks his socket ready and wakes the loop, handing back just the ready sockets, his alone, not the idle 9,999.
- The loop reads from Theo's socket in non-blocking mode, draining whatever bytes are there until it gets EAGAIN, then appends them to Theo's own buffer.
- It applies the framing rule. If a whole message is present, it handles it and prepares a reply; if only a fragment arrived, it leaves the bytes in the buffer and moves on. Half a message is normal, not an error.
- When the reply cannot all be sent at once (the send buffer fills), it sends what fits, remembers the rest, and asks epoll to wake it when the socket becomes writable again, then loops back to sleep.
Honest limits and what comes next
Be clear about what this design buys and what it costs. The event loop wins because it spends almost no resources on idle connections, which is the dominant reality for chat, web, and streaming servers. But its single thread must never run a long, slow piece of work in the middle of the loop, because while it is busy on one request, every other connection waits. A heavy computation, or worse, an accidental blocking call (like a synchronous disk read or a DNS lookup that parks the thread), stalls the whole server. The discipline of the event loop is simple to state and hard to keep: never block, never dawdle.
One last honest note: non-blocking I/O and epoll are about handling many connections efficiently on one machine. They do nothing to make any single connection faster, and they change nothing about TCP itself, the connection is still a reliable byte stream, still unencrypted, still subject to the same round-trip delays. They are a concurrency tool, not a speed tool. In the final guide of this rung we will turn to the small, sharp gotchas that bite even a perfectly written server: byte order across machines, the lingering TIME-WAIT state after you close, and Nagle's algorithm quietly buffering your tiny writes.