fault-tolerant training

When a job runs on thousands of GPUs for weeks, hardware failure stops being a rare accident and becomes routine — at that scale something breaks every few hours. Fault-tolerant training is the machinery that lets a run survive these failures without restarting from scratch, treating a dead node as an expected event rather than a catastrophe.

The baseline mechanism is checkpoint-and-restart: periodically save model and optimizer state, and on failure reload the last checkpoint and resume, so the lost work is bounded by the checkpoint interval. Elastic training goes further — a coordinator detects the failure, the surviving workers re-form their process group, and the job continues on the remaining or replacement nodes, re-sharding state to the new world size. Supporting pieces include health monitoring and failure detection, hot spares to swap in, and increasingly in-memory or redundant checkpoints and partial recovery that avoid rereading from slow storage. The tension is overhead versus lost work: frequent checkpoints and redundancy cost throughput but make each failure cheaper.

At frontier scale, fault tolerance is what makes a run finish at all, and its quality sets the effective utilization — the fraction of GPU-hours spent making progress rather than recomputing after crashes. It leans entirely on fast distributed checkpointing and an elastic scheduler underneath.