learning-rate warmup and decay

The learning rate controls how big a step the optimizer takes on each update. Rather than holding it fixed, LLM training shapes it over time. Warmup starts the rate near zero and ramps it up over the first few thousand steps; decay then slowly lowers it back down across the rest of training. The picture is a gentle climb to a peak followed by a long descent.

Each half earns its keep. A fresh model has chaotic, ill-conditioned gradients, and a large step early on can send the loss diverging — warmup eases in until the optimizer's running statistics settle. Late in training, near a good solution, big steps overshoot and bounce; decaying the rate (often along a cosine curve down to a small floor) lets the model fine-tune into a sharper, lower-loss region.

The schedule's shape is a real hyperparameter: peak rate, warmup length, and final value all matter, and they interact with batch size and model width. A poorly tuned schedule is one of the most common causes of stalled or unstable pretraining.