gradient descent
Gradient descent is the practical, workhorse version of following the slope downhill, and it is the algorithm that trains nearly every modern neural network. The idea is disarmingly simple: to minimize a function, repeatedly nudge your current guess a little bit in the direction opposite to the gradient, because that is the direction in which the function decreases fastest.
Each update reads x_new = x_old - eta times grad f(x_old), where eta is the learning rate or step size — a small positive number controlling how big a step you take. Pick eta too small and progress is painfully slow; pick it too large and you overshoot the valley and may diverge. Unlike textbook steepest descent, practical gradient descent usually fixes eta (or schedules it) rather than doing an exact line search at every step, because computing the gradient even once is already expensive when there are millions of variables. In machine learning a further twist, stochastic gradient descent, estimates the gradient from a small random batch of data each step, trading exactness for speed.
This is the honest mechanism behind training: a neural network's weights are the variables, the loss is the function f, and the gradient is computed efficiently by backpropagation (the chain rule applied through the network). The method does not magically find the best possible weights — it descends to whatever local minimum the path leads to, and its success depends on the learning rate, initialization, and the shape of the loss landscape. That gradient descent works as well as it does on enormous, non-convex problems is partly empirical good fortune and partly the result of decades of engineering refinement.
To fit a line by minimizing squared error, gradient descent repeatedly adjusts the slope and intercept by small steps proportional to the negative gradient of the error. Over many iterations the parameters settle near the least-squares solution — the same answer the normal equations give in one shot.
Gradient descent reaches iteratively what the normal equations solve directly — useful when a direct solve is too big.
Gradient descent finds a local minimum, not necessarily the global one, and only on a non-convex landscape that distinction bites. On a convex function every local minimum is global, which is why convexity makes the method provably reliable.