learning-rate schedule and warmup
/ LUR-ning rayt SKE-jool and WORM-up /
A learning-rate schedule is a plan for changing the step size over the course of training, rather than holding it fixed. The intuition matches how you'd search for something: stride boldly while you're far away and still getting oriented, then take smaller, careful steps as you close in, so you settle precisely instead of stomping past the target. Almost every well-trained modern model uses some schedule, not a constant rate.
The most common move is decay: start with a relatively large learning rate to make fast early progress, then shrink it as training goes on — by steps, smoothly, or following a cosine curve down toward zero. Warmup is the opposite trick applied at the very beginning: start with a tiny rate and ramp it up over the first few hundred or few thousand steps. Early in training the weights are random and the gradients can be wild, so a too-big step right away can throw the model off a cliff; warmup eases it in. Large models and transformers in particular tend to need warmup to train stably.
Combined, the typical recipe is warm up, then decay: ramp the rate up, hold or peak, then taper down. This isn't superstition — the early phase explores the loss landscape broadly while big steps are useful, and the late phase fine-tunes within a good basin where small steps avoid bouncing around the minimum. The schedule itself becomes a set of hyperparameters (peak rate, warmup length, decay shape) that you tune like any other.
A common transformer recipe: over the first 4,000 steps, ramp the learning rate linearly from near zero up to 0.0005 (warmup); after that, let it decay smoothly back toward zero for the rest of training. Skip the warmup and the same model often diverges in its first few hundred steps.
Ramp up gently, then taper down — the standard modern shape.
Warmup and adaptive optimizers like Adam both stabilize early training, but they're not interchangeable. Many large models use both — adaptive steps and a warmup ramp — because the first few hundred updates are the most fragile.