training instability and loss spikes

A healthy pretraining run shows the loss gliding smoothly downward for weeks. Instability is when that calm breaks: the loss suddenly leaps upward, sometimes by a lot, occasionally never recovering and ruining the run. These loss spikes are one of the most feared events in large-scale training, because a single bad one can waste days of compute on thousands of GPUs.

They arise when an update is too violent for the current state of the model — often from a rare batch with extreme values, an exploding gradient in a particular layer, or numerical overflow in low precision. The standard defences are a toolkit rather than one cure: gradient clipping caps the size of any update, careful learning-rate warmup eases early volatility, normalisation choices and weight initialisation keep activations in range, and skipping or rolling back over a poisonous batch saves a run mid-flight.

Larger models tend to be twitchier, so teams checkpoint often and watch the loss curve like a hawk. Recovering from a spike usually means rewinding to the last good checkpoint, possibly lowering the learning rate or stepping over the offending data, and pressing on.