JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

RDMA and the Race to Microseconds

Inside the datacenter, software that once measured delay in milliseconds now counts microseconds. This guide shows where those microseconds hide, why the operating system kernel is the bottleneck, and how RDMA lets one machine read another's memory almost as if it were local.

Where did the milliseconds go?

By now in this rung you have seen what makes a datacenter network special: enormous scale, mostly east-west traffic between servers, fat-tree and leaf-spine fabrics for high bisection bandwidth, and congestion control like DCTCP tuned for tiny round trips. Now we ask a sharper question. When two servers sit a few metres apart on the same fabric, how fast can one actually hand data to the other? On the public Internet a round trip is tens of milliseconds, so saving a microsecond is pointless. Inside the datacenter the physical round trip can be a handful of microseconds, and suddenly every microsecond of software overhead is the whole game.

It helps to remember a rule from the very first rung: bandwidth, throughput, and latency are three different things, and more bandwidth does not cut latency. A 100 Gbps link moves more bits per second, but it does not make the first bit arrive sooner. The race to microseconds is a latency race, and you cannot beat the speed of light: light in fibre crosses a 100-metre datacenter aisle in well under a microsecond, so the wire is hardly ever the problem. The problem is everything the bits must pass through at each end before and after the wire.

So before we reach for a fancy solution, let us be honest about where the time goes. Picture sending one small message with ordinary TCP: your application calls into the operating system, the data is copied into a kernel buffer, the kernel builds the segment and IP headers, hands it to the driver, the network card sends it; on the far side a hardware interrupt wakes the kernel, the data is copied again into a kernel buffer, the receiving application is woken from sleep and copies the data into its own memory. Each step is cheap, but they add up to several microseconds per message, dwarfing the sub-microsecond wire time.

The kernel is the toll booth

Think of the operating system kernel as a toll booth that every byte must stop at on the way in and out. It is there for good reasons: it protects one program from another, multiplexes one network card among many applications, and enforces the reliable, ordered stream that makes TCP so easy to use. But each pass through the booth costs a context switch (the CPU stops running your code and runs kernel code), one or more memory copies, and the bookkeeping of TCP. For a big file transfer this overhead is a rounding error. For a key-value store answering a million tiny requests a second, the toll booth is the traffic jam.

ordinary TCP send/receive (each '|' is time at a CPU)

  sender                              receiver
  app: write() ---> kernel copy
       build headers
       NIC sends  =====[ ~0.5 us wire ]=====> NIC, interrupt
                                              kernel copy
                                              wake app, copy
                                              app: read()

  several us of CPU work  >>  the ~0.5 us on the wire
With ordinary TCP, the CPU work at each end dominates; the wire is the fast part.

There are two complementary ways to shrink the booth. The first is kernel bypass: let the application talk to the network card almost directly, mapping the card's queues into the program's own memory so sending a packet means writing to a ring buffer the card is already watching, with no system call and no kernel copy. The second is to push work into hardware: the network card itself handles the headers, the checksums, even reliability, so the CPU is barely involved. RDMA, our destination, combines both, but the same instinct already powers user-space packet frameworks and the offload engines on a modern SmartNIC.

RDMA: reading another machine's memory

Remote Direct Memory Access, or RDMA, takes the bypass idea to its logical end. Inside one computer, a device can already use direct memory access to read and write main memory without bothering the CPU. RDMA stretches that across the network: a special network card lets one server place data straight into a region of another server's memory, or read it out, with the remote CPU never touching the message and often never even noticing. The kernel set up the permissions once, in advance; after that the data path is pure hardware. The toll booth is gone from the fast path entirely.

  1. Register memory: each side asks its RDMA card to pin a buffer in physical memory and hand back a key naming that region. Pinning matters because the hardware will reach into that memory later without asking the operating system again.
  2. Exchange keys: the two sides swap memory addresses and keys once, over an ordinary connection. This is the only slow, kernel-involved step, and it happens before the hot loop.
  3. Post a request: to send, the application writes a small work request into a queue mapped into its own memory, naming the local buffer and the remote address and key. No system call is needed.
  4. The card does the rest: the local card reads the buffer, ships it across the fabric, and the remote card writes it straight into the named memory. For a one-sided write the remote CPU is never interrupted; the data simply appears.
  5. Completion: the card drops a note into a completion queue so the sender knows the transfer finished. The application polls that queue instead of sleeping, which is why latencies can fall to a few microseconds.

There are two flavours of operation worth distinguishing. A two-sided operation (RDMA send and receive) still works like a message: the receiver must have posted a buffer to catch it, much like a normal socket. A one-sided operation (RDMA read and write) is the magic trick: the initiator names a remote address directly and the remote application is not involved at all. One-sided writes are how a distributed database can update a peer's memory, or a storage system can fetch a remote page, in a couple of microseconds, leaving the remote CPU free for real work.

RoCE: RDMA over the Ethernet you already have

RDMA was born on InfiniBand, a separate, purpose-built fabric with its own cabling and switches. That is wonderful if you can afford a second network, but most clouds want RDMA on the same Ethernet they run everything else on. The answer is RoCE (RDMA over Converged Ethernet), which wraps RDMA messages inside ordinary Ethernet frames and IP packets so they ride the leaf-spine fabric you already built. The promise is real, but it comes with a sharp condition that connects straight back to the previous guide.

Here is the catch. The original RoCE hardware assumed a lossless network: if the wire never drops a packet, the card's simple reliability logic stays fast and cheap. But ordinary Ethernet does drop packets when buffers overflow, exactly the TCP incast and microburst situations you met earlier. So RoCE leans on two tools you already know. Priority Flow Control lets a switch tell the upstream port to pause briefly rather than drop, and ECN marks packets when queues start filling so senders slow down before anything is lost. It is the same congestion-signalling instinct behind DCTCP, now keeping an RDMA fabric from ever having to retransmit.

When microseconds are worth it (and when they are not)

It is tempting to think faster is always better, but RDMA earns its keep only for specific shapes of work. Disaggregated storage, distributed databases, large machine-learning training where thousands of GPUs must exchange gradients every step, and in-memory key-value stores all share a trait: a flood of small east-west transfers where the kernel overhead, not the data volume, sets the pace. Cutting a message from ten microseconds to two there can mean millions more operations per second across a cluster.

For most other traffic, plain TCP is the right and honest choice. A web server streaming a video, a backup copying terabytes overnight, a request crossing the public Internet where the round trip is already milliseconds: in all of these the kernel toll is invisible, and TCP's universal reach, congestion fairness, and decades of hardening matter far more than shaving a microsecond you will never feel. Remember too that, like TCP, RDMA on its own encrypts nothing; confidentiality is a separate layer, just as TLS sits above TCP. The skill is not always reaching for the fastest tool, but knowing which bottleneck you are actually fighting.