ring all-reduce

Averaging gradients across N devices is an all-reduce, and the naive way — every device sends its full gradient to one node that sums and broadcasts back — makes that node a bottleneck whose traffic grows with N. Ring all-reduce removes the bottleneck by arranging the devices in a logical ring where each only ever talks to its two neighbors. The gradient is chopped into N chunks, and the chunks circulate around the ring so the work and the bandwidth are shared evenly by everyone.

It runs in two phases. In the reduce-scatter phase, over N−1 steps each device sends one chunk to its successor and adds the chunk it receives, so that after the phase each device holds one fully-summed chunk. In the all-gather phase, those summed chunks circulate another N−1 steps until every device has the complete reduced gradient. The total data each device sends is about twice the gradient size regardless of N — this is the bandwidth-optimal property: the per-device communication volume does not grow with the number of devices, only the latency (step count) does.

Ring all-reduce, popularized for deep learning by Baidu's implementation and embodied in libraries like NCCL, is the workhorse behind data-parallel gradient synchronization. Its weakness is latency on large rings — N−1 sequential hops — so production collectives use hierarchical or tree-based variants that combine ring's bandwidth optimality within a node with shallower topologies across nodes.