Scalable Oversight: Supervising What You Cannot Check

The problem: a judge weaker than the judged

Scalable oversight is the study of how to give a correct training signal on tasks where humans cannot directly evaluate the answer: a million-line codebase, a frontier physics derivation, a subtle medical recommendation. The naive failure is brutal — if you train against a judge who cannot tell good from bad, you do not get a neutral result, you get a model optimized to fool the judge. The supervision signal becomes adversarial against you.

The shared idea behind every approach below is decomposition of trust: find some sub-task the limited judge can reliably evaluate, then arrange the protocol so that getting the easy parts right forces the hard part to be right too. The approaches differ in how they manufacture those checkable sub-tasks.

Debate: let two experts cross-examine

Debate pits two strong models against each other on a question, arguing for opposing answers, while a weaker judge reads the transcript and declares a winner. The bet is asymmetric in our favor: it should be harder to defend a lie against a capable adversary than to defend the truth. A dishonest debater's claims can be challenged, decomposed, and checked at a leaf the judge understands, where the lie finally becomes visible.

V^{\star} \;=\; \max_{\pi_{\text{honest}}}\;\min_{\pi_{\text{deceptive}}}\; \mathbb{E}\!\left[\,J\!\big(\tau(\pi_{\text{honest}},\pi_{\text{deceptive}})\big)\,\right]

Debate as a two-player zero-sum game: the honest debater maximizes, the deceptive one minimizes, and the weaker judge J scores the resulting transcript.

Debate is beautiful in theory and fragile in practice. It assumes truth is genuinely easier to defend (not always true — some falsehoods are simpler than their messy true rebuttals), that the judge is not exploitable by rhetoric, and that the debaters actually find the strongest counterarguments rather than colluding. Empirically, stronger debaters do help weaker judges reach correct verdicts more often, which is the encouraging signal; but persuasion-vs-truth confounds are a live concern, and 'obfuscated arguments' that are too long to fully check remain an open hole.

Weak-to-strong generalization

Weak-to-strong generalization reframes the whole problem as an empirical analogy. We cannot yet study 'humans supervising superhuman models', but we can study the structurally similar case of a weak model supervising a strong one: fine-tune GPT-4-scale on labels generated by a much smaller, error-prone model, and ask how much of the strong model's latent capability survives.

The hopeful finding: a strong model trained on a weak teacher's flawed labels often outperforms its teacher, recovering much (not all) of the gap to its own ceiling. The interpretation is that the strong model already 'knows' the right concept internally, and weak labels mainly elicit that latent knowledge rather than teach it from scratch. The sobering finding: naive fine-tuning also makes the strong model imitate the teacher's mistakes, and the recovered fraction shrinks as the capability gap grows — exactly the regime we care about most.

\mathrm{PGR} \;=\; \frac{\operatorname{acc}(\text{weak}\!\to\!\text{strong}) \;-\; \operatorname{acc}(\text{weak})}{\operatorname{acc}(\text{strong ceiling}) \;-\; \operatorname{acc}(\text{weak})}

Performance-gap recovered (PGR): the fraction of the weak-to-strong gap the student closes — 1 means it reaches its own ceiling, 0 means it only matches its weak teacher.

# Weak-to-strong, the core experiment
weak_labels = weak_model.predict(unlabeled)      # noisy, ceiling-limited
strong_ft   = finetune(strong_base, on=weak_labels)
# Measure: PGR = (acc(strong_ft) - acc(weak)) / (acc(strong_ceiling) - acc(weak))
# PGR -> 1 means weak supervision fully elicited the strong model.

Performance-gap-recovered (PGR) asks: how much of the strong model's potential did weak labels unlock?

Eliciting latent knowledge

All three methods above lean on a deeper assumption: that the model contains the truth we want, and the task is to extract it. Eliciting latent knowledge (ELK) names this directly. Suppose a model has a rich internal world-model and knows the true answer to a question, but its reported answer is whatever its training rewarded — possibly 'what a human would believe' rather than 'what is actually true'. ELK asks: can we read out what the model actually believes, even when that diverges from what it says?

The canonical worry is the 'human simulator' failure: a probe trained to predict the human-endorsed answer might learn 'output what a human would think', which is correct on every training example a human could check and yet systematically wrong exactly where the model knows better than the human. Distinguishing the model's direct report of reality from its model-of-the-human is, in the general case, unsolved — ELK is more a sharply-posed open problem than a finished method, and it is the conceptual bridge to interpretability in guide 5.

Ask whether your task is genuinely beyond your judge, or just tedious — many 'oversight' problems are really annotation-cost problems with cheaper fixes.
Prefer protocols that produce a checkable leaf: a step the judge verifies, a decomposition, a cross-examination point.
Validate any oversight scheme on a sandwich: a weak judge, the real model, and a trusted strong expert as ground truth you withhold during training.
Treat persuasiveness and correctness as separable; measure whether your protocol rewards the second without being captured by the first.