interconnect topology (NVLink)

At scale, the wires between accelerators matter as much as the accelerators themselves. Collective operations move enormous tensors between GPUs on every step, so the interconnect topology — which devices are wired to which, at what bandwidth, in what shape — sets a hard ceiling on training throughput. The design question is the layout of the links, not just their speed.

Inside a node, NVLink provides direct GPU-to-GPU links far faster than PCIe, often arranged through an NVSwitch crossbar so any GPU can reach any other at full bandwidth; this is the fast intra-node domain. Across nodes, InfiniBand (or RoCE) with RDMA connects servers in fat-tree or dragonfly topologies, an order of magnitude slower per link than NVLink. Collective libraries are topology-aware: they run bandwidth-optimal rings and trees inside the NVLink domain and hierarchical algorithms across the slower fabric, overlapping this traffic with computation to hide it.

This bandwidth hierarchy directly shapes how a model is sharded. Tensor parallelism, the most communication-intensive, is kept inside the high-bandwidth NVLink domain, while pipeline and data parallelism, which exchange less per step, span the slower inter-node network. Mapping a parallelization strategy onto the physical topology is a central part of large-scale training design.