The cloud is a building
Up to now this ladder has treated the network as the public Internet: a sprawling mesh of independent networks, glued together by IP and routing protocols, crossed by TCP connections that fight for room on shared links. That picture is real, but most of the bytes you touch never travel it. When you open an app, the heavy lifting happens inside a datacenter — a warehouse packed with tens or hundreds of thousands of servers, all owned by one operator, all wired together by a network they designed top to bottom. The cloud is not a place in the sky. It is a building, and a very particular kind of network lives inside it.
That single owner changes everything. On the Internet, no one controls the whole path, so designs must be cautious, decentralized, and tolerant of strangers misbehaving. Inside a datacenter, one operator owns every switch, every cable, every server, and can tune the whole system as one machine. They can pick the topology, choose the addressing, run their own protocols, and rebuild it all next year. So the rules you learned for the wild Internet still apply, but the operator gets to bend them. This rung is about what they do with that freedom.
Traffic goes sideways now
The biggest surprise is the direction of the traffic. The old web picture is north-south: a request comes in from a user outside, a single server answers, the reply goes back out. But a modern cloud service is not one server — it is a swarm. To build one web page the front-end server fans out to dozens of back-ends: a search index here, a recommendation model there, three database shards, a cache, an auth service. Each of those may call others. The reply to the user is tiny; the conversation among servers to produce it is enormous. This server-to-server traffic that never leaves the building is called east-west traffic, and in a real datacenter it dwarfs the north-south traffic, often by a factor of ten or more.
Picture the difference physically. North-south is customers walking through one front door to a counter. East-west is the whole staff behind the scenes constantly running between desks, passing notes, the kitchen yelling to the bar to the storeroom and back, hundreds of times for every single customer order. If you only built a grand front entrance and forgot the corridors between desks, the place would seize up. Datacenter networks are designed corridors-first: the priority is letting any server talk to any other server fast, because that is what almost all the traffic actually is.
Two server-to-server qualities matter most. One is bisection bandwidth: imagine slicing the datacenter into two equal halves and asking how much traffic could cross the cut at once. High bisection bandwidth means any-to-any communication does not bottleneck no matter who is talking. The other is low latency — and here, honesty: more bandwidth does not buy you low latency. They are different things. Bandwidth is how wide the pipe is; latency is how long one message takes to arrive; throughput is how much actually gets through. You cannot beat the speed of light, but inside a building distances are tiny, so the latency that remains is mostly queuing and switching — which is exactly the part the operator can engineer away.
Wiring it: from oversubscribed trees to Clos fabrics
The classic way to wire a building was a three-tier tree. Servers plug into access switches at the bottom; access switches connect up to a few aggregation switches; aggregation connects up to one or two beefy core routers at the top. Traffic between two servers under different access switches must climb up the tree and back down. The trouble is the climb. Each level up has fewer, busier links than the level below, so the tree is oversubscribed: it is built assuming not everyone talks at once. A 4-to-1 oversubscription means four units of demand at the bottom share one unit of capacity higher up. That bet is fine for north-south web traffic, but east-west traffic breaks it — when every server really is talking, the upper links jam.
The modern answer flips the design: instead of a few fat links at the top, use a mesh of many cheap, identical switches so the bandwidth up the tree equals the bandwidth at the bottom. This is the Clos / fat-tree idea, and its practical form is the leaf-spine fabric. Every server connects to a leaf switch; every leaf connects to every spine switch; spines connect to nothing but leaves. So any server reaches any other in exactly two hops — up to a spine, down to the destination leaf — and there is no congested top, because there are many spines sharing the load equally. Add capacity by adding spines. The full mechanics, and the math of why fat-trees give full bisection bandwidth, are the whole of the next guide.
A mesh creates a happy new problem: between any two servers there are now many equally good paths, one through each spine. How do you use them all? The trick is ECMP: when a switch sees several next-hops of equal cost, it hashes each flow's header fields (source and destination IP, ports) to pick one path and pins that flow to it. Different flows hash to different spines, so load spreads across the fabric, while every packet of a single flow follows one path and stays in order. It is a receptionist who, faced with five identical lifts, sends each visitor to a lift by a rule — keeping any one visitor's bags together while keeping all five lifts busy.
Two networks in one: overlays and the virtual switch
There is a second twist. The servers you rent in the cloud are usually not whole machines but virtual machines or containers sharing physical hosts, and thousands of separate customers (tenants) live on the same hardware. Each tenant wants their own private network — their own IP ranges, their own broadcast domains — and must never see another tenant's traffic. But the physical fabric underneath is one shared thing the operator wired once. So the cloud runs two networks at once: the real physical network, the underlay, and a software-defined logical network for each tenant painted on top, the overlay.
The overlay works by tunneling, and the common tool is VXLAN. When a tenant's virtual machine sends a packet, a piece of software on the host — the virtual switch, the receptionist for that physical machine's many guests — wraps the whole tenant packet inside a fresh outer packet addressed between physical hosts, with a tag saying which tenant it belongs to. The underlay fabric only ever sees and routes the outer packet, host to host; it knows nothing of tenants. At the far host the virtual switch unwraps it and hands the original packet to the right virtual machine. This is just encapsulation — nested envelopes — used to make one physical network pretend to be thousands of private ones. The overlay, the virtual switch, and the controller that programs them are the third guide.
Sitting at the building's front door is one more cloud workhorse: the load balancer. A service is not one server but a pool of identical ones, so something must spread incoming requests across them, hide failed members, and present one stable address to the world. A simple load balancer works at the transport layer, steering whole TCP connections; a smarter layer-7 load balancer reads the HTTP request and routes by URL or cookie. Either way it is the head waiter at the door, seating each arriving party at whichever identical table is free — which is also how the cloud scales a service by simply adding more tables.
Transport rewritten for microseconds
Now the part where the transport rung you climbed gets bent the hardest. TCP was tuned for the Internet, where round trips are tens of milliseconds and buffers are deep. Inside a datacenter, round trips are tens of microseconds — a thousand times shorter — and switch buffers are tiny. Two things that were harmless on the Internet become vicious here. The first is incast: remember that fan-out where one front-end queries many back-ends? When all those back-ends answer at almost the same instant, their replies converge on one switch port at once, overflow its tiny buffer, and a burst of packets is dropped together. Classic TCP reacts to the loss by backing off and waiting for a timeout — but a timeout is milliseconds, an eternity here. One synchronized answer-storm can stall a whole request.
The second is the microburst. Datacenter traffic is spiky: a link can sit nearly idle yet receive a burst of packets in a few microseconds that briefly overwhelms a buffer. Averaged over a second the link looks empty, so capacity charts swear nothing is wrong, yet packets are being dropped in bursts too short to see. The fix for both troubles is a smarter congestion signal. DCTCP leans on ECN: switches mark packets the moment their queue starts to build, well before it overflows, and the sender reacts in proportion to how many marks it sees — a gentle, early, graduated brake instead of waiting blind for a catastrophic drop. The careful driver now has an early-warning light on the dashboard. All of incast, microbursts, and DCTCP are the fourth guide.
Some workloads need to go faster still — distributed training, storage, in-memory databases want one machine to read another's memory in a few microseconds, and even a lean TCP stack spends too long in the operating system copying data and handling interrupts. The escape is RDMA, remote direct memory access: a network card writes straight into a remote server's memory with the host CPU barely involved, like one office reaching directly into another's filing cabinet instead of mailing a request and waiting for a reply. Running RDMA over an ordinary Ethernet fabric is called RoCE, and it demands a near-lossless network, which is why it leans on the same congestion machinery DCTCP introduced. The race to microseconds is the fifth and final guide.
The map ahead
Pull it together with one picture. A datacenter is a building where the staff talk to each other far more than they talk to customers, so it is wired as a flat mesh with many equal paths, painted over with thousands of private overlay networks, and run on a transport stack rebuilt to brake within microseconds. Everything special about it follows from two facts: one owner controls it all, and the traffic mostly goes sideways. Here is the rest of the rung.
- Fat-Trees and Leaf-Spine: how the Clos idea builds full bisection bandwidth from cheap identical switches, why any-to-any takes a fixed number of hops, and the arithmetic of how big a fabric you can grow.
- Overlays and Virtual Switches: VXLAN tunnels over the underlay, the virtual switch on each host, and the controller that programs thousands of private tenant networks onto one shared fabric.
- TCP Incast and Datacenter Congestion Control: why fan-out replies and microbursts wreck tiny buffers, and how DCTCP uses early ECN marks for a gentle, proportional brake.
- RDMA and the Race to Microseconds: bypassing the operating system to write straight into remote memory, RoCE over Ethernet, and the near-lossless network it needs.