Muon optimizer

When you train a weight matrix, its accumulated momentum is often close to low rank: a few dominant directions carry most of the update, so the step is lopsided and under-uses the other directions. Muon counters this by orthogonalizing the momentum matrix before it is applied, so every direction in that layer's update gets a balanced, roughly unit-scale push. The effect is a bit like whitening the update so no single direction dominates.

Mechanically Muon forms the usual momentum matrix M and then replaces it by the orthogonal factor of its singular value decomposition, the polar factor U V-transpose, where M equals U Sigma V-transpose. Rather than running an actual SVD, it approximates this orthogonalization cheaply with a few Newton-Schulz iterations applied to M divided by its norm, which run efficiently in low precision. This orthogonalized update is used only for the two-dimensional weight matrices of hidden layers; embeddings, the output head, and one-dimensional bias and normalization parameters fall back to AdamW.

In recent large-language-model training speedruns Muon has delivered faster, more compute-efficient pretraining than AdamW for the same budget, which is why it has drawn attention. It is still a maturing method: only the matrix-shaped hidden parameters use it, the Newton-Schulz steps must be tuned for stability, and questions about scaling behavior and learning-rate transfer across model sizes are areas of active research.