RMSprop

RMSprop is an optimizer that gives each weight its own step size based on how turbulent that weight's gradients have recently been. The intuition: if a particular direction has been swinging wildly, take small careful steps there; if it's been calm and consistent, you can afford to move faster. It's like driving — you slow down on the bumpy, twisty stretch and speed up on the smooth straightaway.

Technically, RMSprop keeps a running average of the squared gradient for each weight — a measure of recent magnitude — and divides each weight's step by the square root of that average (hence "root mean square"). Directions with consistently large gradients get damped; directions with small gradients get amplified. Crucially, the average is a decaying one, weighting recent steps more than old ones, so the step sizes keep adapting as training moves into new regions of the landscape.

RMSprop was proposed (famously, in a lecture slide rather than a paper) to fix a flaw in its predecessor AdaGrad, whose step sizes shrank monotonically toward zero and could stall training. By using a decaying average instead of an ever-growing sum, RMSprop keeps step sizes alive throughout long training runs. It works well on noisy, non-stationary problems and on recurrent networks, and its core idea — the squared-gradient denominator — lives on inside Adam, which simply adds momentum on top.