learning rate
/ LUR-ning rayt /
The learning rate is the size of each step the model takes while learning. Picture walking downhill toward a valley: the slope tells you which direction to go, but the learning rate decides how far you stride each time. Tiny steps and you'll get there eventually but painfully slowly; giant steps and you might leap clean over the valley and end up higher than you started, bouncing wildly.
Technically, after the gradient says "adjust this weight in this direction," the learning rate is the multiplier that controls how big the adjustment actually is. A learning rate of 0.1 moves ten times as far per step as 0.01. It is almost always the single most important hyperparameter to get right: get it wrong by a factor of ten and training can fail completely — either diverging into nonsense (too high) or crawling so slowly it never finishes (too low).
There's no magic value, because the right step size depends on the loss landscape's shape, which you can't see in advance. So people search: try a few values spread across orders of magnitude (0.001, 0.01, 0.1), watch the loss curve, and pick the largest rate that still trains stably. Modern practice rarely keeps it fixed — it's usually warmed up at the start and decayed over time, which is the job of a learning-rate schedule.
Same model, three learning rates. At 0.5 the loss explodes to infinity within a few steps (overshooting). At 0.00001 the loss barely moves after thousands of steps (crawling). At 0.01 it slides smoothly down to a good value. Only experimentation reveals which is which.
Too big overshoots; too small crawls; just right slides down.
If your loss suddenly shoots up to a huge number or becomes "NaN," the learning rate is almost always too high. It's the first dial to turn down when training blows up.