cosine annealing schedule
You generally want a large learning rate early, to traverse the landscape quickly, then a smooth decline so the optimizer can settle gently into a minimum rather than rattling around it. A cosine annealing schedule shapes the learning rate exactly this way: it starts high and eases down to near zero following the smooth curve of the first half of a cosine wave, with no abrupt drops.
The rate at step t is eta_t = eta_min + half (eta_max - eta_min)(1 + cos(pi t / T)), where T is the length of the cycle. The 'warm restarts' variant, SGDR, periodically resets t to zero, often with a growing period, so the learning rate jumps back up and re-anneals; this lets the optimizer escape one basin and explore others over the course of training. A short linear warmup is usually prepended so the very first steps do not start at full rate.
Cosine schedules are ubiquitous in modern vision and language-model training because they are smoother than step decay and have fewer knobs to set. Warm restarts can additionally be exploited to collect snapshot ensembles, one model per cycle, almost for free. The two choices that matter most are the final learning rate and the total horizon T over which the cosine descends.
The learning rate follows half a cosine wave from eta_max down to eta_min.
For long single-cycle LLM runs the schedule is typically annealed to a small fraction of the peak rather than literally zero; decaying fully to zero too early can stall learning before the data is exhausted.