AdaGrad
/ AD-uh-grad /
AdaGrad was one of the first optimizers to give each weight its own learning rate, automatically. The guiding idea: weights that have already been updated a lot probably don't need big steps anymore, while weights that have rarely moved deserve bigger ones. So it gradually slows down the frequently-updated directions and lets the neglected ones catch up — handy when some features appear constantly and others only rarely.
Mechanically, AdaGrad keeps, for each weight, a running sum of all the squared gradients it has ever seen, and divides that weight's step by the square root of this sum. Because the sum only ever grows, the effective step size for every weight keeps shrinking over time. This made AdaGrad especially good at sparse problems — like text, where most words are rare — because rare features, having accumulated little, still get meaningful updates.
But that same ever-growing sum is AdaGrad's Achilles' heel. The denominator never stops increasing, so step sizes eventually shrink toward zero and learning grinds to a halt before the model has finished — a real problem for deep networks that train for a long time. This flaw is precisely what RMSprop and Adam were designed to fix, by replacing the cumulative sum with a decaying average. AdaGrad is rarely the final choice today, but it's the historical root of the adaptive optimizers everyone now uses, so it's worth understanding.
Training a text classifier, the word "the" appears in nearly every example while "perihelion" appears once. AdaGrad lets the rare word's weight take a big, meaningful step on its single appearance, while "the" — already adjusted thousands of times — gets a tiny step. The downside: after enough passes, every step has shrunk so far that learning nearly stops.
Great for rare features early; stalls late as steps shrink to nothing.
AdaGrad's monotonically shrinking step size is a feature for sparse, short training and a bug for long deep-learning runs. That single limitation is the reason RMSprop and Adam exist.