Lookahead optimizer

Standard optimizers can oscillate across a ravine or overshoot a good region, and tuning them out of that is fiddly. Lookahead is a wrapper around any base optimizer that adds a second, slower set of weights for stability. The idea is to let fast weights scout ahead by running several ordinary update steps, then let the slow weights take a cautious partial step toward wherever the scouts ended up, smoothing out the jitter.

Mechanically, Lookahead keeps slow weights phi. From phi it runs the inner base optimizer, such as Adam or SGD, for k steps to produce fast weights theta. It then updates phi <- phi + alpha (theta - phi) and resets the fast weights back to the new phi before the next inner loop. The slow update is an interpolation, like an exponential moving average taken along the fast trajectory rather than over time, with alpha controlling how boldly it follows.

The appeal is that Lookahead reduces the variance of the iterates and improves stability across many tasks with very little tuning, just the two hyperparameters k and alpha. It is orthogonal to the base optimizer, so you can wrap it around your favorite one, and the extra cost is small: a single additional copy of the weights and a cheap interpolation every k steps.