Two problems the bare fabric cannot solve
The previous guide gave us a beautiful machine: a leaf-spine fabric that moves packets between any two servers at full speed, spreading flows across many equal paths with ECMP. But that fabric is deliberately simple. It knows about physical servers, their addresses, and the shortest way to reach each one — and almost nothing else. In a real cloud, the customers aren't physical servers at all. They are virtual machines and containers, dozens packed onto each physical box, springing into life and vanishing thousands of times an hour.
This creates two headaches the bare datacenter network cannot solve on its own. First, isolation: customer A and customer B might both pick the address 10.0.0.5 for one of their machines, and they must never see each other's traffic, even when their VMs land on the very same physical server. Second, mobility: when a VM migrates from one rack to another for maintenance, its address has to follow it — but in the underlay, an address belongs to a location, so moving it would normally break every connection. The fabric is fast precisely because it refuses to track any of this churn.
The overlay idea: a private network drawn in software
A network overlay is a network that exists only as an agreement between the machines at the edges. Nothing in the middle of the fabric needs to know it's there. Think of it like the postal system: the highways and sorting centres (the underlay) only ever read the outer address on an envelope. If you seal a second, fully-addressed letter inside that envelope, the postal system happily carries the whole thing to the building on the outer address — and only the person who opens it there reads the inner letter and forwards it on. The outer envelope is the underlay's job; the inner letter is the overlay's private business.
That nested-envelope trick is exactly encapsulation — the same idea you met long ago when a TCP segment went inside an IP packet inside an Ethernet frame. The overlay just adds one more layer: it takes a complete packet from a customer VM and wraps it inside a brand-new outer packet addressed between two physical servers. The fabric forwards the outer packet by its outer address and never inspects what's inside. When it arrives, the receiving server peels off the wrapper and hands the original packet to the right VM, untouched. The customer's network never knew it took a ride through someone else's wires.
This single move solves both headaches at once. Isolation comes free: each overlay carries a tag identifying which customer's network it belongs to, so two VMs that both use 10.0.0.5 sit on different overlays and can never be confused. Mobility comes free too: when a VM migrates to a new rack, only the outer address changes — the inner packet, with the VM's own unchanged address, looks identical to everyone in the overlay. The connection survives the move because the part the customer sees never moved at all.
VXLAN: how the wrapping actually works
The most common way to build these overlays is VXLAN — Virtual eXtensible LAN. The name is a clue to what it does: it lets the customer's virtual machines behave as if they were all plugged into one big switch on the same local network, even though they're scattered across racks and connected only by routed IP underneath. VXLAN takes the customer's whole Ethernet frame and stuffs it inside a UDP packet that travels across the underlay. The endpoints that do this wrapping and unwrapping are called VTEPs (VXLAN Tunnel Endpoints), and they almost always live in software on each physical server.
What goes on the wire when VM-A (on server S1) sends a frame to VM-B (on server S2):
+----------------------------------------------------------------+
| Outer Ethernet | Outer IP | Outer UDP | VXLAN | INNER |
| S1-nic -> S2-nic| 192.0.2.1->.2 | dst 4789 | VNI | FRAME |
+----------------------------------------------------------------+
\____ added by the VTEP on S1 (the wrapper) _________/ ^
|
INNER FRAME = the customer's original, untouched Ethernet frame:
[ VM-A mac -> VM-B mac | 10.0.0.5 -> 10.0.0.6 | TCP ... data ]
VNI = the 24-bit tenant tag: which customer network this belongs to.
The leaf-spine fabric only ever reads the OUTER IP (192.0.2.1->.2).Two details in that sketch carry the real magic. The first is the VNI, the VXLAN Network Identifier: a 24-bit tag stamped into every wrapped packet that says which tenant's network it belongs to. Twenty-four bits gives about 2^24 networks — over sixteen million — which is why VXLAN replaced the old VLAN tag's mere 4096. The second is that outer UDP. Choosing UDP wasn't about reliability — the inner TCP already handles that. It was a trick: the underlay's ECMP spreads flows by hashing port numbers, so the VTEP varies the outer UDP source port per inner flow. That way thousands of tunnelled flows scatter evenly across all the spine paths instead of jamming one.
The virtual switch: where the thinking moved
So who does this wrapping, decides which physical server to send to, and enforces the rules about who may talk to whom? Not the fabric — it stayed dumb on purpose. The work moved into a virtual switch: a piece of software running on each physical server, sitting between the VMs and the real network card. Recall the very first networking analogy of a switch as a receptionist who learns where everyone sits. A virtual switch is that same receptionist, but now living in software inside the host, handling a building full of virtual tenants who keep moving desks.
Because it's software, the virtual switch can be far smarter than a hardware switch ever was. It learns where every VM lives, it does the VXLAN wrapping and the VNI tagging, it enforces per-VM firewall rules (called security groups) before a packet even reaches the wire, it meters bandwidth, and it can mirror traffic for monitoring. This is the quiet foundation of software-defined networking: the network's intelligence has been pulled out of fixed boxes and into programmable software at the edge, where it can be updated by an API in seconds. The fabric forwards; the virtual switch thinks.
- VM-A on server S1 sends an ordinary frame to VM-B (inner address 10.0.0.6). It has no idea it's on a cloud — it thinks it's on a plain LAN.
- S1's virtual switch catches the frame, looks up which physical server hosts VM-B in this tenant's network, and finds it's on server S2 at underlay address 192.0.2.2.
- The virtual switch wraps the whole frame in VXLAN: it stamps the tenant's VNI, picks an outer UDP source port (so ECMP spreads the load), and addresses the outer IP from 192.0.2.1 to 192.0.2.2.
- The leaf-spine fabric forwards the outer packet by its outer IP alone, hashing it onto one of many equal paths. It never looks inside.
- S2's virtual switch receives it, peels off the VXLAN wrapper, checks the VNI to confirm the tenant, and delivers the original untouched frame to VM-B — which never knew any of this happened.
Cloud load balancers: one door, many rooms
There's one more piece the cloud needs, and it builds straight on overlays. A popular service isn't one server — it's a fleet of identical VMs that grow and shrink with demand. But the outside world needs a single, stable address to connect to. The cloud load balancer is that single front door: clients all connect to one virtual IP, and the load balancer spreads their connections across the healthy VMs behind it, the way a good host seats arriving guests across many open rooms instead of cramming them into one.
You've met load balancing before in the CDN rung, but the cloud version leans hard on everything in this guide. The load balancer keeps health checks running so it never sends traffic to a crashed VM. It can balance at layer 4 (just spreading TCP connections by speed) or, like a layer-7 balancer, peek inside HTTP and route by URL path — /images to one pool, /checkout to another. And because the back-end VMs live on overlays, the load balancer can pick a new VM in a different rack, or a freshly-launched one, and the encapsulation quietly delivers the connection there. The single front door stays put while the rooms behind it are rebuilt at will.
It's worth being honest about what a load balancer does and doesn't fix. It spreads load and routes around dead VMs beautifully, and the virtual IP lets you scale invisibly. But it cannot beat physics: if every back-end VM is genuinely overloaded, the balancer just spreads the misery evenly. And it adds a hop, so it costs a little latency — more bandwidth or more VMs behind it will never shave the round-trip time below what the speed of light and the extra hop allow. The load balancer manages where work goes; it does not make the network faster.
Putting the layers back together
Step back and the whole cloud network is a clean division of labour. At the bottom, a fast, dumb leaf-spine fabric moves outer packets between physical servers and balances them with ECMP — and that's all it does. On top, a software virtual switch on every host builds overlays with VXLAN, giving every customer a private, mobile, isolated network drawn entirely in software. And in front, cloud load balancers hand each service a single stable address over a fleet that scales freely behind it. Each layer hides its mess from the one above, exactly like the protocol stack you learned in the very first rung.
But there's a tension hiding underneath, and it sets up the rest of this rung. All this software wrapping and unwrapping happens on the host CPU, on a network where deadlines are measured in microseconds and a single user request can fan out to thousands of back-end VMs at once. When all those replies come back together, they slam the receiver and the fabric in ways the public Internet never sees. Why that synchronized flood is so dangerous — and the special congestion control built to tame it — is exactly where the next guide takes us.