checkpoint sharding

A trillion-parameter training run must save its state periodically — to survive hardware failures, which on a cluster of thousands of accelerators are routine, and to resume later. But the full state (parameters plus optimizer moments) can be terabytes, and the model is already sharded across hundreds of devices. Checkpoint sharding writes that state as many parallel shards, each device persisting the slice it already holds, instead of gathering everything onto one rank and serializing a single monolithic file that would be slow and memory-prohibitive.

The hard part is decoupling the on-disk layout from the parallelism layout. A naive sharded checkpoint is tied to the exact configuration that wrote it, so you could not resume on a different number of devices or a different tensor/pipeline split. Distributed-checkpoint formats solve this by storing each parameter with metadata describing its global shape and how the shards tile it, so a loader can re-shard on the fly to whatever topology the resuming job uses. Asynchronous checkpointing further hides the cost by snapshotting tensors to host memory quickly and flushing to storage in the background while training continues.

Checkpoint sharding is what makes long runs robust and elastic: failures are recovered in minutes rather than restarting from scratch, and a job can be reshaped onto a different cluster topology between checkpoints. Its engineering burden is correctness under concurrency and topology change — a checkpoint that silently loses or misplaces a shard is worse than none, so these systems are validated carefully.