Why every large model is a distributed program
No single accelerator holds a frontier model's weights, gradients, and optimizer state, so distributed training shards the work across many. But sharding is the easy part; the hard part is that the shards must constantly agree. Data-parallel replicas must average their gradients each step; tensor-parallel shards must exchange partial activations within a layer; FSDP must gather each layer's parameters just-in-time and re-scatter them after. All of this agreement is expressed in a tiny vocabulary of collective communication operations.
The collective vocabulary
NCCL (NVIDIA Collective Communications Library) is the de-facto implementation on NVIDIA hardware, and it provides a handful of primitives that every parallelism strategy is built from. All-reduce sums a tensor across all ranks and gives everyone the result — this is gradient averaging. All-gather collects each rank's shard into the full tensor on everyone — this rebuilds FSDP's parameters. Reduce-scatter sums and leaves each rank holding one slice — this produces sharded gradients. Broadcast and all-to-all round out the set; the last is the workhorse of mixture-of-experts routing.
Ring all-reduce: bandwidth-optimal by design
A naive all-reduce that sends every gradient to one master node makes that link a bottleneck and scales terribly. Ring all-reduce instead arranges the ranks in a logical ring and passes chunks around it: a reduce-scatter phase circulates and sums chunks, then an all-gather phase circulates the finished chunks back. The beauty is that the data each rank sends becomes independent of the number of ranks — roughly 2× the tensor size, no matter whether you have 8 GPUs or 8,000. That bandwidth-optimality is why ring (and its tree and hierarchical cousins) underpins large-scale training.
Ring all-reduce moves 2(N−1)/N × |g| bytes per rank, approaching a constant 2|g|/B independent of N — the sense in which it is bandwidth-optimal.
The fabric: NVLink, InfiniBand, and topology
Collectives are only as fast as the wires under them, and those wires are not uniform. NVLink connects the GPUs inside a node at terabytes per second — close to HBM speed — while nodes are joined across the data center by InfiniBand or Ethernet at perhaps an order of magnitude less. This hierarchy is the single most important fact about a cluster. NCCL is topology-aware: it discovers the fabric and builds rings and trees that keep the heaviest traffic on the fastest links. Your job is to place parallelism to match: put the most chatty dimension (tensor parallel) inside a node on NVLink, and the most latency-tolerant (data/pipeline parallel) across the slower inter-node fabric.
Hiding communication behind compute
Even bandwidth-optimal collectives cost time, and the cleanest way to pay for that time is to spend none — by overlapping communication with computation. While the GPU computes layer L's math, the network can be all-gathering layer L+1's parameters or all-reducing layer L-1's gradients in the background. Done well, the comms disappear entirely behind the compute and the cluster scales almost linearly; done poorly, ranks stall at every collective and you pay full price. Achieving overlap is the central engineering problem of every serious training stack.
Perfect overlap hides communication entirely: a step costs the max of compute and comm, not their sum.
# FSDP-style overlap, schematically
for layer in layers:
prefetch_all_gather(layer.next) # launch comm now, async
y = layer.forward(x) # compute hides the gather
x = y
# backward mirrors it: reduce-scatter grads of layer L while computing L-1