learning rate schedule

A fixed learning rate forces one compromise to serve the entire run, but training has distinct phases with different needs. Early on, the weights are random and the loss landscape is treacherous, so you want to move cautiously; in the middle you can stride confidently; near the end you want to take ever-smaller steps to settle precisely into a minimum instead of bouncing around it. A learning-rate schedule simply varies the learning rate over time to match these phases — typically warming up, then decaying — and it reliably improves both final accuracy and training stability.

The decay shapes you will meet most often are step decay (cut the LR by a factor, e.g. ÷10, at fixed milestones — the classic recipe for training ResNets on ImageNet), exponential decay (multiply by a constant <1 each step), and cosine annealing (smoothly follow half a cosine curve from the initial LR down to near zero over the run, now the default for ViT, ConvNeXt, and most modern vision training). Cosine annealing with warm restarts (SGDR) periodically jumps the LR back up to escape the current basin and explore others, which can also produce an ensemble of snapshots taken at each low point.

Warmup is the front half of the modern recipe: start the LR at near zero and linearly (or otherwise) ramp it up to the target over the first few hundred to few thousand steps before any decay begins. The reason is sharpest for adaptive optimizers like Adam — early in training the second-moment estimate v̂ is computed from very few gradients and is unreliable, so a full-size step can be wildly miscalibrated and destabilize the model. Warmup also tames the large effective steps that occur with large batch sizes and with the high-variance gradients of a freshly-initialized deep network. The transformer recipe of linear warmup followed by decay is now ubiquitous in vision transformers and detection models.

Schedules interact with everything else. The peak LR after warmup is what the linear scaling rule applies to; the total number of steps sets how fast cosine decay falls; and an overly aggressive decay can freeze learning before convergence while too gentle a decay leaves the model jittering. Because of this coupling, the schedule is best chosen together with the batch size, optimizer, and total training budget rather than in isolation.