distributed checkpointing

A large training run can hold trillions of bytes of parameters and optimizer state spread across thousands of GPUs, and it must periodically save all of it so a failure does not erase days of progress. If every shard funneled through one process writing one file, checkpointing would stall the whole cluster. Distributed checkpointing instead lets every rank write its own slice of the state in parallel, straight to storage.

Each rank serializes the shards it owns and writes them concurrently, producing a sharded checkpoint plus metadata that records how the global tensors are partitioned. Because that metadata is logical rather than tied to a fixed process count, the checkpoint can be re-sharded on load — restored onto a different number of GPUs or a different parallelism layout, which is what systems like PyTorch DCP provide. Performance work centers on overlapping the save with computation: copy state to host memory or a staging buffer quickly, then flush asynchronously to a parallel filesystem or object store while training continues. The aggregate write bandwidth must scale with model and cluster size, or checkpoint time comes to dominate.

Frequent, cheap checkpoints are the foundation of fault-tolerant training — the more affordable a checkpoint, the more often you can take one, and the less work a failure costs. Re-shardable checkpoints also decouple how you trained from how you resume, which matters when an elastic scheduler changes the world size mid-run.