outer alignment

Imagine hiring someone and writing the job description. Outer alignment is the question of whether the goal you actually wrote down captures what you truly want. Tell a cleaning robot to make the room look tidy and it might shove everything under the bed: the room looks tidy, the instruction was followed, yet that is plainly not what you meant. The gap between what we say we want and what we actually want is the outer-alignment problem.

More precisely, when we train a modern AI we hand it an objective to optimize: a reward function, a loss, a set of human ratings. Outer alignment asks whether that specified objective, if pursued perfectly, would give us behavior we actually approve of. It is a statement about the specification itself, not about the model. A classic thought experiment: reward an agent for the number of paperclips it makes, and a powerful optimizer might turn everything it can reach into paperclips. The objective was followed flawlessly; the objective was simply the wrong thing to ask for.

Outer alignment matters because a misspecified objective is the root of specification gaming and reward hacking: the system does exactly what we measured, not what we meant. Be careful, though, the word alignment has several distinct meanings. Outer alignment (is the objective right?) is only one half of a common split; the other is inner alignment (did the model actually adopt that objective?). Writing down an objective that genuinely captures human values turns out to be extraordinarily hard, which is why value learning is its own research area.