maximal update parameterization (μP)

Tuning hyperparameters directly on a giant model is prohibitively expensive — a single learning-rate sweep at full scale can cost more than the final run. μP attacks this by changing how a network is parameterized so that its optimal hyperparameters stay essentially the same as the model gets wider. The promise is concrete: tune the learning rate and initialization on a small proxy model, then transfer those settings to a model orders of magnitude larger and have them remain near-optimal, a procedure called μTransfer.

The theory comes from analyzing the infinite-width limit. In the standard parameterization, as width grows the scale of activations, gradients, and weight updates drifts, so the learning rate that works at one width is wrong at another. μP prescribes width-dependent scalings — for the initialization variance, the learning rate, and a per-layer multiplier on certain matrices — chosen precisely so that every layer receives feature updates of the same characteristic size regardless of width. Under this parameterization the loss landscape's relevant geometry is width-invariant, so the hyperparameter optimum is too.

μP turns hyperparameter search for frontier models from a guess into a transfer: you pay the tuning cost once on a cheap model and inherit it at scale. Its caveats are that the cleanest guarantees concern width (depth and other axes need additional care), and that it requires actually rewriting the model's scalings — a real but one-time engineering investment that has nonetheless become standard practice for large pretraining.