Nesterov accelerated gradient

Plain momentum accumulates velocity from past gradients, which speeds things up but also tends to overshoot, because it reacts to the slope where it currently stands. Nesterov's accelerated gradient adds a small foresight: first take the momentum step to a look-ahead point, then measure the gradient there, so the method can sense an upcoming wall and brake before it overshoots. That single change of where the gradient is evaluated turns out to give a provably faster convergence rate.

The update is v <- mu v - eta grad L(theta + mu v) followed by theta <- theta + v, so the gradient is taken at the look-ahead position theta plus mu v rather than at theta itself. For smooth convex objectives this method attains the order one-over-t-squared convergence rate, against order one-over-t for ordinary gradient descent, and this rate matches the known lower bound for first-order methods, which is why it is called optimal acceleration.

Nesterov's scheme is the canonical notion of acceleration in optimization and underlies many accelerated and proximal algorithms, including FISTA. In deep learning the look-ahead form is a small, usually beneficial tweak to ordinary momentum, though it becomes sensitive when the momentum coefficient mu is pushed close to one, where it can amplify noise instead of damping it.