The Frontier: Discovering, Orthogonalizing, and Choosing Optimizers

Optimizers can be discovered, not just designed

For decades optimizers were hand-derived from convergence proofs. Lion arrived by a different door: a program-search over the space of possible update rules, where candidate optimizers — expressed as short programs over gradients and momentum buffers — were evolved and evaluated on proxy training tasks. The search rediscovered momentum, simplified it, and converged on a compact sign-based rule that no one would have written by hand but that holds up across many real workloads.

\theta_{t} = \theta_{t-1} - \eta\left(\operatorname{sign}\!\big(\beta_1\, m_{t-1} + (1-\beta_1)\, g_t\big) + \lambda\,\theta_{t-1}\right)

The Lion update rule that program search surfaced: a sign-of-interpolated-momentum step with decoupled weight decay.

The lesson is not that humans are obsolete; it is that the design space is larger and stranger than our proofs cover. Search is most valuable as a generator of hypotheses — it surfaces candidate rules that we then analyze, simplify, and stress-test. Lion's sign update, once found, turned out to have a clean interpretation as a constrained, normalized step, which is exactly how a machine-found rule becomes durable knowledge: by being explained after the fact.

Muon: orthogonalize the momentum

Muon is the sharpest recent idea for the hidden weight matrices of a network. Its insight: a momentum matrix often has a few dominant singular directions that crowd out the rest, so a raw step over-updates a handful of directions and under-updates the others. Muon orthogonalizes the momentum — replacing it with the nearest matrix whose singular values are all one — so every direction receives a balanced update. It computes this orthogonalization cheaply with a few Newton-Schulz iterations instead of a full singular value decomposition.

M = U\,\Sigma\,V^{\top} \;\Longrightarrow\; O = U\,V^{\top}, \qquad W \leftarrow W - \eta\,O

Muon's core step: orthogonalize the momentum matrix to U Vᵀ, equalizing its singular directions before the update.

M = beta*M + grad                 # momentum (a 2D weight matrix)
X = M / M.norm()
for _ in range(5):                # Newton-Schulz -> approx orthogonal factor
    X = 1.5*X - 0.5 * X @ (X.T @ X)
W = W - lr * X                    # all singular directions get a unit-scale step

Muon updates hidden matrices with an orthogonalized step; 1-D params (biases, norms) still use AdamW.

Conceptually Muon is a close cousin of Shampoo — both impose matrix structure on the update rather than treating parameters as a flat vector — but Muon is far cheaper because Newton-Schulz needs only matrix multiplies. On large language-model training it has matched or beaten well-tuned AdamW at lower step counts, which is why it has become one of the most watched optimizers at the frontier. Note it is applied to 2-D hidden weights; embeddings, biases, and norm scales are still handled by a standard optimizer.

Proximal methods: when the objective is not smooth

Everything so far assumed a differentiable loss. But many useful objectives add a nonsmooth regularizer — an L1 penalty for sparsity, a nuclear-norm penalty for low rank, a hard constraint set. Proximal gradient methods handle these by splitting the step in two: a normal gradient step on the smooth part, followed by a proximal operator that solves a small, structured subproblem for the nonsmooth part. For an L1 penalty the proximal operator is soft-thresholding — shrink each coordinate toward zero and clip — which is exactly how lasso produces truly sparse solutions where a plain gradient penalty only produces small ones.

\operatorname{prox}_{\lambda g}(v) = \arg\min_{x}\Big(g(x) + \tfrac{1}{2\lambda}\lVert x - v\rVert_2^{2}\Big), \qquad x_{k+1} = \operatorname{prox}_{\lambda g}\!\big(x_k - \lambda\,\nabla f(x_k)\big)

The proximal operator and the proximal-gradient step: take a gradient move on the smooth part, then resolve the nonsmooth penalty.

Proximal methods complete the geometric picture from Guide 1. Just as mirror descent generalizes the metric of a step, the proximal operator generalizes the constraint or penalty a step must respect. Together they let you optimize objectives that mix a smooth, easily-differentiated term with a structured term you would rather enforce exactly than approximate. In deep learning this shows up in structured pruning, constrained fine-tuning, and any setting where you want hard sparsity rather than a soft nudge.

Benchmarking without fooling yourself

Optimizer comparisons are a minefield of self-deception. The most common error is unequal tuning budget: a new optimizer is swept over a fine grid while the baseline keeps its paper defaults, so the new method 'wins' on tuning effort, not on merit. A fair comparison gives every contender the same hyperparameter-tuning budget — including learning rate, weight decay, warmup, and schedule — and reports the result after each has been independently optimized.

Fix the budget: same number of trials, same search space shape, for every optimizer including the baseline.
Compare at matched compute, not matched steps — a method that needs two passes per step must show twice the per-step gain to break even.
Report generalization (held-out / downstream), not just training loss — faster optimization can mean worse models.
Repeat across seeds and at least one larger scale; many optimizer wins evaporate when the model or batch grows.

A decision guide and the open questions

Bring the whole track to a practical point. Start with AdamW plus warmup-then-cosine; it is the right default for almost everything. Reach for curvature (natural gradient, K-FAC, Shampoo) when the problem is ill-conditioned and steps are scarce. Reach for layerwise scaling at very large batch. Add SWA or SAM when generalization is the bottleneck. Try Muon on transformer hidden matrices when you want frontier step-efficiency and can tolerate a newer recipe. Use proximal methods when the objective is genuinely nonsmooth.

The open questions are humbling. We still lack a predictive theory of which optimizer suits which architecture; the interaction between optimizer, schedule, and the implicit bias toward flat minima is only partly understood; and we cannot yet reliably forecast an optimizer's behavior at a scale ten times larger than where it was tuned. Advanced optimization is, for now, a discipline of strong principles and honest empiricism — which is exactly why reading every result through the `P⁻¹ g` lens of Guide 1, and the benchmarking discipline of this section, is the most durable skill you can carry out of this track.