AI safety via debate

A non-expert can often pick the better side of an argument by watching two experts debate and rebut each other, because a lie usually has a weak spot an opponent can expose, even if the judge could never have found the truth alone. AI safety via debate turns this into a training method: have two AIs argue opposite answers to a question, and let a human decide who argued better.

More precisely, in the debate proposal (Irving and colleagues, OpenAI, 2018) two copies of a model are given a question and assigned to defend opposing answers. They take turns making statements and, importantly, pointing out flaws and gaps in each other's case. A human judge, who may be much weaker than the debaters, declares a winner, and the models are trained to win debates. The hope is that defending a false answer is harder than defending a true one, because a truthful opponent can always expose the lie, so the equilibrium of this game rewards honest, helpful, easy-to-verify arguments, giving us a training signal even for questions a human could not answer directly.

Debate matters as another candidate method for scalable oversight, a sibling of iterated amplification. The honesty here is essential: this is a hope, not a theorem. It depends on truth genuinely being easier to argue than falsehood and on judges not being fooled by rhetoric. Known failure modes, such as the obfuscated-argument problem (where a debater hides a flaw too deep to expose in the available steps) and persuasive manipulation of the judge, are real, and empirical results so far are mixed.