Training at Scale: Distributed Pretraining

Why one GPU is never enough

Two hard walls force pretraining onto many machines. First, memory: a model with tens of billions of parameters, plus its gradients and the bulky AdamW optimizer state, needs far more memory than any single GPU has. Second, time: the compute budget of a serious run — think 10²³ FLOPs — would take a single chip many decades. The answer is distributed training: spread the work across hundreds or thousands of GPUs that cooperate as one.

In mixed-precision Adam training each parameter costs about 16 bytes — weights, gradients and optimizer state together — which is why memory, not compute, forces a model onto many GPUs.

The challenge is that the GPUs must stay in sync — they are all training one model — so they constantly exchange data over fast interconnects. The art of distributed training is splitting the work so that chips spend their time computing, not waiting on each other. There are three complementary ways to split, and large runs use all three at once.

Data parallelism: clone the model, split the batch

The simplest and most common split is data parallelism. Put a full copy of the model on each GPU, hand each copy a different slice of the training batch, and let them all compute gradients in parallel. Then comes the key step: every GPU shares its gradient and they are averaged together (an operation called all-reduce), so all copies apply the identical update and stay perfectly in sync.

Data parallelism is how you make a run faster: double the GPUs, double the batch you can process per second. But by itself it does not help with memory, because every GPU still holds a whole model. The modern fix is sharded data parallelism (the FSDP / ZeRO family), which also splits the weights, gradients, and optimizer state across the data-parallel group and gathers each piece only when needed — keeping the simple programming model while slashing per-GPU memory.

Tensor and pipeline parallelism: split the model itself

When a model is too big to fit on one GPU even after sharding, you must cut the model itself into pieces — that is tensor and pipeline parallelism. They slice along two different axes:

Tensor parallelism splits within a layer. A single huge matrix multiply is divided column-wise across several GPUs; each does part of the math and they combine results. It is very fast but chatty — the GPUs must talk on every layer — so it is used within a tightly-connected group of chips (typically inside one server, linked by the fastest interconnect).

Pipeline parallelism splits across layers. Layers 1–8 live on GPU A, 9–16 on GPU B, and so on, like stations on an assembly line. A batch flows through the stages in turn. The danger is idle "bubbles" while later stages wait for earlier ones, so the batch is chopped into micro-batches that keep every stage busy at once.

a frontier run, 3D parallelism on 1024 GPUs:
  tensor   parallel = 8    (within a server, splits each layer)
  pipeline parallel = 8    (across servers, splits the layer stack)
  data     parallel = 16   (replicas, splits the batch)
  8 * 8 * 16 = 1024 GPUs cooperating on one model

The bottleneck is communication, not math

Here is the counterintuitive truth of scale: with thousands of GPUs, the limiting factor is rarely raw arithmetic — it is moving data between chips. Every all-reduce of gradients and every hand-off between pipeline stages is communication, and the network is slower than the math units it feeds. A run that uses 90% of its theoretical compute is excellent; many waste a third or more just waiting on the network.

So the parallelism strategy is chosen to fit the hardware topology: put the chatty tensor-parallel group on the fastest links inside a server, run pipeline stages across servers, and overlap communication with computation wherever possible — send one layer's gradients while the next layer is still computing. Mixed precision helps here too: 16-bit gradients are half the bytes to ship, so mixed precision buys not just compute speed but communication speed.

Keeping a months-long run alive

Across thousands of GPUs running for weeks, something will break: a chip fails, a network link flaps, a node reboots. At that scale hardware failure is not an accident but a near-certainty over the run. Add the loss spikes from the last guide, and a real pretraining run is as much an operations problem as a machine-learning one. The survival kit:

1. Distributed checkpointing: every node saves its shard of the state regularly, so the full model can be restored after any crash.
2. Automatic restart: on failure, swap in a spare node and resume from the last checkpoint with no human awake.
3. Health and metric monitoring: watch loss, gradient norm, throughput, and per-GPU temperature for the first sign of trouble.
4. Determinism and logging: record data order and seeds so a run can be rewound, a bad batch skipped, and the whole thing reproduced.

Put it all together — a clean trillion-token corpus, a Chinchilla-aware budget, AdamW with a warmup-decay schedule in mixed precision, and 3D parallelism wrapped in fault-tolerant operations — and you have what it takes to turn a blank network into a base model. From here the journey continues into fine-tuning and alignment, where that raw next-token predictor is shaped into the helpful assistant you actually talk to.