TCP Incast and Datacenter Congestion Control

A network so fast that TCP gets confused

By now you have toured the inside of the cloud: the datacenter network with its rivers of east-west traffic between servers, the leaf-spine fabric that gives any two machines a fat, low-latency path, and the ECMP spreading of flows across many equal links. The whole point of all that engineering was to make the network blisteringly fast — round trips inside a single building are measured in tens of microseconds, not the tens of milliseconds you see across the Internet. You might expect that a faster network can only make TCP happier. The surprising truth of this guide is the opposite: the very speed and shape of the datacenter break assumptions that ordinary TCP was built on, and a transport that behaves beautifully on the open Internet can fall apart here.

Recall the mental model of classic congestion control from the transport rung: TCP is a careful driver who speeds up gently and brakes hard. It has no speedometer for the network, so it learns the road by feel — it keeps enlarging its congestion window (sending a little more each round) until a packet is lost, treats that loss as the signal i pushed too hard, and slams the window down. On the wide Internet this works because a lost packet really does mean a router queue overflowed somewhere, and a round trip is long enough that the driver has time to react. Inside a datacenter both of those comforts vanish: the round trip is so short there is almost no time to react, and the switch buffers are so small that loss arrives in violent, all-at-once bursts rather than as a gentle warning.

Incast: when everyone answers at once

Here is the pattern that breaks things, and it falls straight out of the east-west traffic you already met. Many datacenter jobs are scatter-gather: one machine asks a question that is split across dozens or hundreds of servers — read this row from forty shards, fetch these search results from a hundred index servers — and every one of those servers answers at almost the same moment, because the request reached them all together. All those replies converge on the single link leading back to the one requester. That synchronized flood onto a shared bottleneck is TCP incast: many senders, one receiver, all arriving in the same instant.

Scatter-gather -> incast at the last-hop switch

  worker 1 --\
  worker 2 ---\
  worker 3 ----\   (all reply in the SAME millisecond)
   ...          >===[ switch port buffer ]===> aggregator
  worker 39 ---/        ^ tiny: fills in microseconds
  worker 40 --/         -> overflows -> packets DROPPED

  Dropped TCP segments wait for a retransmission timeout (RTO).
  Datacenter RTT ~ 100 us, but a default minimum RTO ~ 200 ms.
  One unlucky flow stalls 200,000 us to recover a 100 us trip.

Now the small numbers turn cruel. The buffer on that last-hop switch port overflows in microseconds, and several of the synchronized replies are simply dropped. A dropped TCP segment is normally rescued by a fast retransmit — but fast retransmit needs a few later packets to arrive and trigger duplicate acknowledgments, and in a tight incast burst the sender may have nothing else in flight to trigger them. So the flow falls back on its safety net, the retransmission timeout. The catch is brutal: that timeout has a floor, historically around 200 milliseconds, while the datacenter round trip is around 100 microseconds. The connection stalls for two hundred thousand microseconds to recover from losing a one-hundred-microsecond trip. A handful of unlucky flows sitting idle on a timeout can drag the whole scatter-gather request to a crawl.

Be precise about what failed, because it is not really TCP being wrong. TCP's loss-equals-congestion rule and its timeout are perfectly sensible on the Internet; they simply assume losses are occasional and round trips are long. Incast violates both assumptions at once. The first fixes people reached for were narrow: shrink that minimum timeout into the microsecond range so a stalled flow recovers fast, or keep replies from synchronizing so perfectly. Useful, but they treat the symptom — the flood still slams a buffer that overflows. The deeper fix is to stop the buffer from overflowing in the first place, which means changing the congestion signal itself.

Microbursts: the trouble you cannot see on a graph

Incast is one cause of a more general datacenter affliction: the microburst. A microburst is a flood of traffic so brief that it lasts only microseconds, yet during those microseconds it arrives faster than a switch port can drain it, so the buffer briefly fills and may overflow. The maddening part is that it is invisible to ordinary monitoring. A switch counter that reports utilization averaged over one second can read a comfortable 30 percent while, inside that second, a 50-microsecond burst momentarily demanded 300 percent of the port and dropped packets. The average is calm; the millisecond is on fire.

This is the datacenter's version of a misconception worth naming. On the open Internet you met bufferbloat — the problem of buffers that are too big, where excess data sits in a fat queue and inflates latency for everyone. Inside the datacenter the danger leans the other way: buffers are deliberately small to keep latency low, so the failure mode is not a bloated queue but a sudden overflow that drops packets. The shared lesson is that a queue is a delay you are choosing to take on, and the real art is keeping the queue short enough to be quick yet not so short that a normal burst spills over. You want the queue almost empty almost always.

DCTCP: braking on a hint, not a crash

The clever idea that ties this all together is DCTCP, Data Center TCP. Its insight is simple to state: do not wait for a packet to be lost before braking — react to the queue starting to build, while there is still room. To get an early warning, DCTCP leans on a feature you met in the transport rung: Explicit Congestion Notification (ECN). With ECN, a switch that sees its queue growing past a small threshold does not drop the packet; it sets a single mark bit in the packet's header that means i am getting congested. The packet still arrives, and the receiver echoes that mark back to the sender. The signal is now a gentle warning ahead of the cliff, not the crash at the bottom of it.

But DCTCP's real cleverness is in how it reacts, and it differs sharply from the classic brake. Ordinary TCP, on any congestion signal, roughly halves its window — a hard, one-size stomp. DCTCP instead measures how much congestion there is by counting the fraction of its recent packets that came back marked. If almost none are marked, the queue is barely forming and DCTCP shaves its window only a sliver. If nearly all are marked, the queue is genuinely full and it cuts back hard. The reaction is proportional: a light tap on the brake for a light touch of congestion, a firm press for a real jam. That gentleness is exactly what keeps queues short and steady instead of swinging wildly between empty and overflowing.

How DCTCP keeps the queue empty

1. A switch watches its queue. As soon as it climbs past a small threshold (not when it is full), the switch sets the ECN mark on passing packets instead of dropping them.
2. The receiver sees the marks arrive and reflects that information back to the sender in its acknowledgments.
3. The sender keeps a running estimate of the marked fraction — what share of its recent packets met a building queue.
4. It shrinks its congestion window in proportion to that fraction: a sliver if marks are rare, a large cut if marks are common.
5. Because everyone reacts early and gently, the shared queue settles to a short, steady length — leaving headroom for the next incast burst instead of overflowing into it.

Why this helps incast directly: if the bottleneck queue is kept short and almost empty, then when forty replies arrive together there is real buffer space to absorb the spike, and far fewer packets are dropped. Fewer drops means fewer flows stalling on a timeout, which is what crushed throughput in the first place. DCTCP does not abolish incast — a large enough fan-in can still overrun any finite buffer — but by keeping the buffer empty in the calm moments, it gives the burst somewhere to land. Notice the shift in philosophy: classic TCP uses the buffer as its congestion signal and only learns it overdid things once the buffer is full; DCTCP treats a full buffer as a failure to be avoided and signals long before then.

Pulling it together

Step back and the through-line is clear. The datacenter's gifts — huge bandwidth, microsecond round trips, lots of east-west fan-in — are exactly what makes ordinary congestion control stumble. Incast turns the natural scatter-gather pattern into a synchronized flood; tiny buffers turn that flood into dropped packets; the mismatch between a microsecond round trip and a millisecond timeout turns a few dropped packets into a stalled request. Active queue management with ECN marking, and a transport like DCTCP that reads those marks and reacts in proportion, attacks the chain at its root by keeping the queue short so the burst has somewhere to go.

Keep two honest boundaries in view as you climb on. First, none of this is about security or correctness — DCTCP changes when a sender slows down, not whether the data is reliable or encrypted; TCP is still reliable but not secret, and TLS still does the encrypting. Second, congestion control is only half the latency story. Even a perfectly empty queue still costs you the bare round trip and the work of the TCP and operating-system machinery on each end, and for some datacenter jobs even tens of microseconds is too much. That is where the final guide picks up: bypassing the kernel and reshaping the transport entirely with RDMA, in the race to microseconds.