3D parallelism
No single parallelism strategy scales to the largest models, because each has a different ceiling: tensor parallelism is bandwidth-bound to one node, pipeline parallelism is limited by the bubble and by stage count, and data parallelism is capped by batch size and gradient-sync cost. 3D parallelism composes the three into orthogonal axes. Devices are arranged on a conceptual grid; one axis shards each layer (tensor), one axis splits the layer stack into stages (pipeline), and one axis replicates the whole pipeline over different data shards (data).
The art is mapping these axes onto the physical topology so each kind of traffic rides the fabric that suits it. Tensor parallelism's frequent, heavy all-reduces are placed within a node's high-bandwidth NVLink domain; pipeline parallelism's light point-to-point hand-offs cross the slower inter-node network; data parallelism's once-per-step gradient reduce-scatter spans the outermost, often slowest links and can be overlapped with the backward pass. ZeRO or FSDP usually rides the data axis to also shard optimizer state. Choosing the three degrees is a constrained optimization over memory limits, bubble fraction, and communication volume for a given cluster.
3D parallelism (often with expert parallelism added as a fourth axis) is the standard blueprint behind trillion-parameter training runs. Its difficulty is not any single technique but their interaction — a change to one degree ripples into the memory and communication budgets of the others, so tuning the configuration is a large part of the engineering effort.
Device count factorizes into data (d), tensor (t), and pipeline (p) degrees, with expert degree e as an optional fourth axis.
The degrees multiply: total devices equal data-degree times tensor-degree times pipeline-degree, so the grid factorization is itself a design choice with no single right answer for a given cluster.