distributed LLM training

A frontier LLM is far too large and trained on far too much data to fit on one chip, so training is spread across hundreds or thousands of accelerators working as a single machine. Distributed training is the craft of slicing the model, the data, and the optimizer state across all those devices and keeping them in lock-step so the result is identical to training on one impossibly huge GPU.

Several axes of parallelism are combined. Data parallelism replicates the model and splits the batch; tensor parallelism splits individual matrix multiplies within a layer; pipeline parallelism assigns different layers to different devices; and sharding spreads the optimizer state. Real runs stack these into three-dimensional parallelism, choosing the mix to fit memory limits and the cluster's network. Between steps the devices exchange gradients through collective operations like all-reduce.

The central battle is communication. Every device must share results constantly, and if computation cannot hide that traffic the expensive accelerators sit idle. So distributed training is as much a networking and systems problem as a machine-learning one, and a poorly mapped run can waste most of its hardware.