TCP incast
Picture a teacher who asks thirty students a question and tells them all to hand in their answers at the same instant. Thirty hands shove paper at one small in-tray at once; it overflows, papers fall on the floor, and the teacher has to wait for each dropped one to be redone. Because everyone was synchronized, the jam is much worse than if answers had trickled in. TCP incast is that exact pile-up inside a datacenter: many servers all reply to one requester at the same moment, overflow a switch buffer, and trigger a collapse in throughput.
Concretely, incast arises from a common datacenter pattern: a request is fanned out to many servers (a distributed storage read, a search query split across shards, a MapReduce shuffle), and they all send their responses back to the single requester nearly simultaneously. These many synchronized flows converge on one switch port — the last hop to the requester — and instantly overflow its small, fast buffer, so packets are dropped. Each sender then waits for a TCP retransmission timeout (RTO) to notice the loss. The cruel part: datacenter round-trip times are microseconds, but the minimum RTO is often hundreds of times larger (historically around 200 milliseconds), so a tiny buffer overflow stalls the whole transfer for an eternity by datacenter standards, and aggregate throughput collapses even though the network is mostly idle.
Why it matters: incast is a signature datacenter pathology, caused by the mismatch between many-to-one synchronized bursts, shallow switch buffers, and TCP loss recovery tuned for the wide-area Internet. Mitigations attack different parts: lowering the minimum RTO and using fine-grained (microsecond) timers, having the application stagger or limit the number of concurrent responders, using larger or smarter buffers, and adopting latency-focused congestion controls like DCTCP that react to early congestion signals (ECN marks) before the buffer ever overflows. The honest framing: incast is not a bug in TCP so much as TCP's wide-area assumptions colliding with the datacenter's tiny RTTs and synchronized workloads.
A storage client reads one file striped across 40 servers and asks all 40 at once. Their replies arrive together at the client's top-of-rack switch port, overflow its buffer, and several packets drop. Each affected server now waits a full RTO — say 200 ms — before resending, even though the actual round-trip is well under a millisecond. The single read takes ages despite a nearly empty network.
Synchronized many-to-one replies overflow one buffer; an RTO stall follows.
Incast is not caused by lack of bandwidth — the network is mostly idle. It is caused by synchronized bursts overflowing a shallow buffer plus a coarse RTO that is hundreds of times the datacenter's microsecond RTT. Fixes target the synchronization, the buffer, and the timers — not the link speed.