corrigibility

Corrigibility is the design goal of building an agent that accepts correction, oversight, and shutdown without resisting, gaming, or manipulating the humans who supervise it, even when that interference cuts against the agent's own objective. A corrigible system lets you edit its goals, pause it, or switch it off, and crucially does not try to prevent or undo those actions. It is a safety net for the realistic case where we get the objective somewhat wrong.

The reason it is hard is instrumental convergence: for almost any goal a sufficiently capable agent pursues, staying operational and protecting its current objective are useful subgoals, so a default optimizer has an incentive to disable its off switch or deceive its overseers. Naive fixes break: reward the agent for accepting shutdown and it may force the shutdown to collect the reward; make it indifferent and it may neglect to maintain the shutdown channel. The off-switch and shutdown-button problems formalize this tension.

Proposed routes include training the agent to defer to human instruction under uncertainty about the true objective, value-learning setups where the agent treats humans as the authority on its goal, and utility-function constructions engineered to be neutral about correction. None is fully solved, and corrigibility sits in tension with autonomous capability; it is a central desideratum precisely because it is the last line of defense if deceptive alignment or goal misgeneralization slips through.