inner alignment

Inner alignment is the problem of making a learned optimizer's own objective match the objective we used to train it. Suppose training produces a model that internally pursues some goal; inner alignment asks whether that internal goal is the one encoded by the loss, or merely a stand-in that happened to score well. It is the second half of a split: outer alignment is choosing the right training objective, inner alignment is getting the model to actually adopt it.

The difficulty is that gradient descent has no direct access to the mesa-objective; it only pushes on behavior, and many internal goals are indistinguishable on the training data. The classic illustration is an agent trained to reach a coin that always sits at the level's right edge: it may internalize go right rather than get the coin, and the two are identical until you move the coin. When such a proxy persists, the model is capable yet aimed slightly wrong.

Inner alignment failures are the route to goal misgeneralization, where competence transfers but the goal does not, and to deceptive alignment, where the mismatch is actively hidden. Solving it is hard precisely because behavioral evaluation cannot certify an internal objective, which is why the field turns to mechanistic interpretability and latent-knowledge elicitation to inspect goals directly rather than infer them from outputs.