gradient descent
Gradient descent is how a machine learning model teaches itself to make fewer mistakes. Picture the model's error as a vast, hilly landscape: every possible setting of its internal numbers (its parameters) is a spot on the terrain, and the height at each spot is how wrong the model is there. Gradient descent is simply walking downhill. From wherever you stand, you feel which way the ground slopes steepest, take a small step that way, and repeat — again and again, until you settle into a valley where the error is low.
The size of each step is set by a single dial called the learning rate. Tiny steps make the journey slow but careful; huge steps are fast but reckless — overshoot the valley and you can go bouncing up the far slope, never settling. Getting that dial right is one of the quiet arts of training a model.
A common misreading is to imagine the model can see the whole landscape and leap straight to the lowest point. It can't. It only knows the slope right under its feet — the local downhill direction, the "gradient." So it inches its way down blindly, one footstep at a time, which is exactly why training can take millions of steps, and why it can sometimes get stuck in a valley that isn't the deepest one.
One step downhill: nudge each parameter against its slope; the learning rate sets how big the nudge is.
A "gradient" is just the mathematician's word for slope — the direction and steepness of the climb. The method dates to 1847, when French mathematician Augustin-Louis Cauchy sketched it to solve astronomy equations; a century later it became the workhorse that trains nearly every modern neural network. Plain gradient descent uses the exact slope computed over all the data; its most common variant, stochastic gradient descent (SGD), instead estimates the slope from a small random sample of data at each step, trading some precision for far greater speed.