distributed training
/ dis-TRIB-yoo-ted TRAY-ning /
Distributed training is the practice of spreading one training job across many chips, or many machines, so it finishes in reasonable time. Imagine a giant ledger that one accountant would take a year to total. Hire a hundred accountants, hand each a stack of pages, and have them periodically compare running totals — the same work now finishes in days. Distributed training is that, with GPUs or TPUs instead of accountants, and a neural network's parameters instead of a ledger.
The catch is coordination. Each chip computes on its own slice and then they must agree on a shared result, which means constantly shipping numbers back and forth over the network between them. That communication is pure overhead: time spent talking instead of computing. So doubling the number of chips never quite doubles the speed, and past a certain point adding more can even slow things down, because the chips spend more time synchronizing than working. Engineers fight hard to overlap communication with computation and to shrink how much must be exchanged.
Why it matters: the largest models simply cannot be trained any other way — they are too big for one chip's memory and would take lifetimes on one device. Distributed training is what makes today's frontier models possible. But it is genuinely difficult to do well: a single slow or failed machine can stall the whole cluster, costs scale with every chip-hour, and debugging a job sprawled across hundreds of nodes is far harder than fixing a program on one computer.
Training on 256 GPUs ideally runs 256× faster than on one. In practice you might see 200×: the missing 56× vanished into time spent synchronizing parameters between the chips after every step.
Near-linear, never perfectly linear — communication overhead is the tax you always pay.
More machines does not always mean faster training. There is a point of diminishing returns where the cost of keeping everyone in sync overtakes the benefit of extra hands. The art is finding that sweet spot, not just throwing hardware at the problem.