From a job to a fleet
The previous guides made one training run fast. This one keeps thousands of them alive on shared hardware. At scale the statistics turn against you: if a single GPU has a tiny failure rate, a job spanning thousands of them will hit a hardware fault every few hours — guaranteed. A frontier run that takes weeks will be interrupted dozens of times. So infrastructure stops being about peak speed and becomes about goodput: the fraction of wall-clock time spent making real progress rather than waiting, restarting, or recomputing lost work. The systems scaling laws only pay off if goodput stays high.
Why the statistics turn against you: with N nodes, a job's mean time between failures shrinks to a single node's MTBF divided by N — so at thousand-GPU scale failures arrive every few hours.
Scheduling: placing jobs on a contended cluster
A GPU cluster scheduler (Slurm, Kubernetes, or a bespoke system) decides which job runs on which accelerators and when. Distributed training imposes two hard constraints ordinary schedulers ignore. First, gang scheduling: all N workers of a job must start together or none can make progress — a half-allocated all-reduce job just deadlocks. Second, topology awareness: the scheduler should bin-pack a job's GPUs onto the same NVLink island and adjacent network spines, because, as the last guide showed, placement decides communication cost. A scheduler that hands you 256 GPUs scattered across the data center has quietly halved your throughput.
Distributed checkpointing: saving state without stalling
A checkpoint is your only insurance against failure, but for a model whose weights and optimizer state run into terabytes, saving naively is its own disaster: funneling everything to one rank to write a single file stalls the whole cluster for minutes. Distributed checkpointing writes in parallel — each rank persists the shard it already holds, so checkpoint bandwidth scales with cluster size. This is the natural partner of checkpoint sharding under 3D parallelism: the on-disk layout mirrors the in-memory sharding, and a separate resharding step lets you reload onto a different parallelism configuration when you resume.
The cost-benefit knob made precise: Young's optimal checkpoint interval balances the write cost C against the work lost on a crash, setting how often to save.
Fault-tolerant and elastic training
Checkpoint-and-restart is the baseline, but restarting the whole job for one dead node wastes every other healthy GPU's time. Fault-tolerant training does better: it detects the failure, evicts or replaces the bad node, rebuilds the communication groups, and resumes from the last checkpoint — elastically, often without a full job teardown. The frontier here is reducing lost work toward zero: redundant in-memory replicas of state so a survivor can re-seed a replacement without touching storage, and reconfiguration that lets a run continue at reduced scale until spare capacity returns. Every minute shaved off recovery is a minute added to goodput.
- Detect: a health check or a hung collective flags a failed rank within seconds.
- Isolate: drain the bad node, ask the scheduler for a replacement from the spare pool.
- Reconfigure: rebuild NCCL communication groups for the new membership.
- Restore: every rank reloads its shard from the distributed checkpoint in parallel.
- Resume: continue training; ideally only the steps since the last checkpoint are lost.
Heterogeneous accelerators and the road ahead
Real fleets are not uniform. A datacenter accumulates several GPU generations, plus TPUs and custom ASICs, and increasingly we want to use them together. Heterogeneous-accelerator training and serving must cope with devices of different speed, memory capacity, and even numerical behavior. That breaks the comfortable assumption of balanced data parallelism: a slower device becomes a straggler that gates every collective, so work must be partitioned unevenly to match each device's throughput, and abstractions like XLA earn their keep by targeting wildly different backends from one program.