You cannot tune hyperparameters at full scale
A frontier run is a single shot: you get one learning rate, one initialization scheme, and if they are wrong you may not find out until you have burned a fortune in compute. Ordinary hyperparameter tuning by sweeping is impossible because each trial costs as much as the final model. This is the problem maximal update parameterization (muP) solves: by scaling the initialization variance and per-layer learning rates with the model's width in a principled way, it makes the optimal hyperparameters invariant as the model grows.
- Reparameterize the model under muP so activations, gradients, and updates all stay order-1 as width grows.
- Sweep learning rate, warmup, and init on a small but muP-parameterized proxy model that is cheap to train.
- Transfer the winning hyperparameters directly to the full-width model — no re-tuning at scale.
Under muP, the optimal learning rate is width-invariant — tune it on a small model and transfer it to the full-scale run.
Extending a model without starting over
Once a base model exists, you rarely retrain from scratch to add a capability. continued pretraining resumes training from an existing checkpoint on new data — a fresh domain (code, medicine, a new language), a longer context window, or simply more of the same to push further down the loss curve. The art is avoiding catastrophic forgetting: a naive high learning rate on the new corpus erases what the model already knew. The usual recipe is a gentle re-warmed learning rate, a data mixture that retains a slice of the original distribution, and a token budget large enough to learn the new domain but small relative to the original pretraining.
Continued pretraining is also where the FP8 and parallelism choices from earlier guides get reused wholesale: you resume into the same 3D layout, often with FP8 training still on, and the only new wrinkle is making sure the optimizer state and learning-rate schedule resume coherently rather than restarting cold.
At ten thousand GPUs, something is always broken
Hardware reliability is brutal arithmetic. If a single GPU has a mean time-to-failure of years, a cluster of ten thousand of them sees a failure every few hours. A run that cannot survive a dead node is a run that never finishes. fault-tolerant training is the discipline of expecting failure: detect the dead rank, evict it, and restart from the last good checkpoint with minimal lost work. The whole strategy rests on checkpointing being fast — because the more expensive a checkpoint is, the less often you can afford to take one, and the more progress you lose when you roll back.
Cluster reliability is brutal arithmetic: ten thousand GPUs turn a multi-year per-GPU lifetime into a failure every few hours.
Checkpointing that keeps up with the cluster
When state is sharded across thousands of ranks, you cannot funnel it through one writer. distributed checkpointing has every rank write its own slice in parallel — this is checkpoint sharding — so wall-clock save time stays roughly flat as the cluster grows. The subtle requirement is that the saved format be resharding-aware: you should be able to reload a checkpoint written under one parallel layout (say TP=8, PP=16) into a different layout when you resume on a differently shaped cluster, which means the checkpoint must store the logical, layout-independent tensors rather than each rank's raw physical shard.
- Save asynchronously: snapshot tensors to host memory fast, then flush to storage in the background while training continues.
- Store layout-independent logical tensors so a checkpoint can be resharded onto a different parallel configuration.
- Set the checkpoint interval from the expected failure rate: frequent enough to bound lost work, rare enough not to dominate the run.
- Verify on reload — a corrupt checkpoint discovered only at restart is its own kind of disaster.
The Young–Daly optimum sets the checkpoint interval from the checkpoint cost δ and the cluster's mean time between failures M.