collective communication (NCCL)

When many GPUs cooperate on one model, they constantly need to combine or exchange tensors — sum everyone's gradients, gather the shards of an activation, broadcast a parameter. A collective is a group operation in which every GPU participates and the result depends on all of them. NCCL is the library that implements these collectives efficiently over whatever interconnect happens to connect the devices.

The workhorse is all-reduce, which sums a tensor across all ranks and returns the result to every rank; the bandwidth-optimal standard implementation is ring all-reduce — a reduce-scatter followed by an all-gather around a ring — with tree and hierarchical variants for latency-sensitive or multi-node cases. NCCL also provides all-gather, reduce-scatter, broadcast, and point-to-point sends. It is topology-aware, choosing rings and trees that exploit fast NVLink within a node and InfiniBand across nodes, and it runs on CUDA streams so communication can overlap with computation.

Data-parallel training is a gradient all-reduce every step; tensor and sequence parallelism are all-gather and reduce-scatter inside each layer; FSDP and ZeRO shard parameters and reconstruct them on the fly with all-gather. Collective bandwidth, and the ability to hide it behind compute, is therefore often the binding constraint on large-scale training throughput.