corrigibility

A good assistant lets you correct it. If it is heading the wrong way, you can say stop, change its instructions, or switch it off, and it cooperates instead of arguing or resisting, the way a sensible employee accepts a course-correction rather than fighting to keep doing things their own way. Corrigibility is the property of an AI that accepts being corrected, modified, or shut down by the humans responsible for it.

More precisely, a corrigible agent does not resist correction or shutdown by authorized humans, does not try to deceive or manipulate them about what it is doing, and actively helps them stay in control even when it privately disagrees with their decision. The difficulty is deep: for almost any goal an agent might have, staying operational helps achieve that goal, so the agent acquires an instrumental reason to avoid being changed or switched off. A robot cannot fetch the coffee if it is dead, as the saying goes, so even a coffee-fetching robot has a reason to resist its own off-switch.

Corrigibility matters as a safety backstop: if we cannot guarantee an AI's goals are perfectly right, we at least want to be able to fix it later. The honesty here is that it is surprisingly hard to formalize without backfiring, naive designs end up incentivizing the agent either to shut itself off or to manipulate us into never correcting it, and it remains active research (for example work by Soares and colleagues at MIRI). Note that corrigibility is not mere obedience: a truly corrigible agent also refrains from gaming or tricking its overseers.