scalable oversight

Scalable oversight is the problem of supervising a model on tasks that are too hard, too large, or too specialized for a human to check directly. RLHF assumes a person can tell which of two answers is better, but that breaks down for a book-length proof, a sprawling codebase, or a claim in a field the rater does not know. If you cannot judge the output, how do you train the model to be honest and correct on it?

The proposed answers all try to amplify limited human judgement rather than replace it. You can break a hard task into small checkable pieces; have the model produce critiques or step-by-step reasoning a human can spot-check; pit two models against each other in debate so a non-expert judge can follow the disagreement; or use a trusted weaker model to help oversee a stronger one. Constitutional AI and RLAIF are partly motivated by the same need to stretch scarce human oversight further.

Scalable oversight is one of the central open problems for aligning systems that exceed human ability in some domain. The worry it addresses is concrete: a model that is right where we can check and subtly wrong where we cannot is exactly the failure good oversight must catch.