Lion optimizer

Almost every optimizer in use was hand-designed by a researcher; Lion, whose name stands for EvoLved Sign Momentum, was instead discovered automatically by a symbolic program search over candidate optimizer programs. What the search found is striking for how simple and memory-light it is: Lion stores only a momentum buffer, and its actual weight update is just the sign of an interpolated momentum, so every coordinate moves by the same magnitude in whatever direction the sign points.

Each step computes the update as the elementwise sign of beta1 times the old momentum plus (1 minus beta1) times the current gradient, then refreshes the momentum buffer with a second decay beta2. Because the sign discards magnitude information, the effective per-parameter step is uniform, much like signed SGD; in exchange the learning rate must be set roughly ten times smaller and the weight decay several times larger than what you would use for AdamW, since decay now competes with a fixed-size update.

Lion keeps only half the optimizer state of Adam because it has no second-moment estimate, which matters at large scale, and it is competitive with or better than AdamW on image classification and language-model pretraining. The trade-offs are real: the sign update can be unstable at small batch sizes, and its performance is sensitive to getting the rescaled learning rate and weight decay right.