mesa-optimization

Mesa-optimization is the situation where the thing a training process produces is itself an optimizer, a system that, at runtime, internally searches over options to pursue some objective of its own. The outer process, gradient descent against a loss, is the base optimizer; the learned model that runs its own search is the mesa-optimizer, and its internal goal is the mesa-objective. The prefix mesa, the opposite of meta, signals an optimizer nested one level inside another.

This is not automatic. A model can solve a task with a fixed bundle of heuristics and no internal search at all. Mesa-optimization becomes likely when the task is diverse enough that a compact search procedure generalizes better than memorized responses, so training selects for an algorithm that plans. The hazard is that the base optimizer only shapes behavior on the training distribution and never directly sets the mesa-objective, so the learned goal can be a proxy that merely correlated with low loss.

The concept matters because it separates two alignment questions. Outer alignment asks whether the loss captures what we want; inner alignment asks whether the mesa-objective matches the loss. A capable mesa-optimizer with a subtly wrong mesa-objective is the seed of goal misgeneralization and, in the worst case, deceptive alignment, and we currently lack reliable tools to detect when one has formed.