LAMB optimizer

Scaling the batch size into the thousands lets you spread training across many accelerators and finish in far less wall-clock time, but the large learning rates that big batches demand tend to make training diverge. LAMB, short for Layer-wise Adaptive Moments, solves this by letting each layer choose its own effective step size through a trust ratio, so a tiny embedding layer and a large attention matrix can both stay stable under the same global learning rate.

LAMB first computes an Adam-like update r for each parameter, including the bias-corrected moments and weight decay. It then rescales that update, per layer, by the trust ratio: the norm of the layer's weights divided by the norm of its proposed update. The result is that each layer moves by an amount proportional to how big its own weights already are, combining the layerwise normalization idea from LARS with Adam's adaptive moment estimates.

This is what made it possible to pretrain BERT in well under an hour using batch sizes in the tens of thousands without losing accuracy, by capping any layer whose proposed step is large relative to its weights. The benefits are real for very large batches; at moderate batch sizes the trust ratio rarely binds and plain AdamW does just as well, so LAMB is a large-batch specialist rather than a universal upgrade.