iterated amplification
One person cannot personally answer a giant research question, but a person who can break it into smaller pieces, hand each piece to an assistant, and then combine the answers can tackle something far bigger than they could alone. Now imagine training a fast model to imitate that whole team, then giving the team copies of the new, better model as assistants, and repeating. That bootstrapping loop is the idea behind iterated amplification.
More precisely, IDA (proposed by Paul Christiano) alternates two steps. Amplification: take a human together with several calls to the current model, and use them to decompose a hard question into subquestions the model can already handle, then assemble the pieces into an answer to the hard question, a slow but more capable human-plus-AI team. Distillation: train a new model to imitate that amplified team, recovering speed. Repeat. Crucially, the training signal always comes from these amplified teams, so no human is ever asked to judge the full hard task alone.
Iterated amplification matters as a concrete proposal for scalable oversight: a way to grow aligned capability past the point of direct human judgement while keeping a human in the loop at every step. The honest caveats: it assumes that breaking a problem into pieces preserves alignment (that safe parts compose into a safe whole), and that the distillation step does not quietly introduce new misalignment. It remains largely a research proposal, with limited validation at large scale.
To answer is this 10,000-line program safe?, a human plus model copies split it into modules, each subquestion is answered by the same process, the verdicts are combined, and a faster model is then trained to imitate the whole thing.
Decompose, delegate, recombine, distill, then repeat.
IDA rests on a big assumption: that hard tasks decompose into pieces whose safe, correct answers compose back into a safe, correct whole. If decomposition loses or distorts the intent, amplification can amplify errors too.