RDMA
/ AR-dee-em-AY /
Normally, sending data from one computer to another is like passing a note through a long chain of office workers: your application hands it to the operating system, which copies it, wraps it in headers, hands it to the network card, and on the far side the whole chain runs in reverse — each handoff costs CPU time and adds delay. RDMA (Remote Direct Memory Access) is like installing a pneumatic tube that runs straight from your desk drawer into your colleague's desk drawer across the building: data goes directly from one machine's memory into another's, without bothering either computer's CPU or operating system along the way.
Concretely, RDMA lets a network card on one server read from or write to a region of memory on another server directly, bypassing the remote CPU and the usual kernel networking stack and avoiding the data copies that stack normally makes. The application sets up the transfer once (registering memory regions and posting work requests), and the network cards do the rest in hardware. The two main ways to run RDMA in a datacenter are InfiniBand (a separate, purpose-built network technology) and RoCE (RDMA over Converged Ethernet, pronounced 'rocky'), which carries RDMA over ordinary Ethernet so operators can reuse their IP fabric. The payoff is dramatic: latencies of a few microseconds and very high throughput with almost no CPU involvement, because the host processor is freed from shuffling every byte.
Why it matters: RDMA is the backbone of the most latency-sensitive datacenter workloads — distributed storage, high-performance computing, and especially large-scale machine-learning training, where thousands of GPUs must exchange data constantly and any CPU overhead or extra microsecond is multiplied across the cluster. The honest caveats are real: RDMA was born on InfiniBand's tightly controlled, essentially lossless fabric, so running it over Ethernet (RoCE) traditionally demanded a near-lossless network using Priority Flow Control, which is fragile and can cause deadlocks and congestion spreading if misconfigured. Making RDMA behave well at scale on lossy Ethernet is an active engineering problem, and it is one of the main reasons datacenters invest in SmartNICs, careful congestion control, and programmable fabrics.
During distributed training, GPU server A needs a tensor sitting in GPU server B's memory. With RDMA, B's network card writes the bytes straight into A's pre-registered memory region in a few microseconds, and neither server's CPU is interrupted to copy or process the data — leaving those cores free for actual computation.
Memory-to-memory transfer that bypasses both CPUs.
RDMA's huge speedup comes from bypassing the CPU and kernel, but that assumes a near-lossless network. Classic RoCE leans on Priority Flow Control to avoid loss, which is brittle: misconfiguration can cause congestion spreading or even deadlock, so RDMA at scale is far from plug-and-play.