recursive reward modeling

Suppose you want to reward an AI for doing a job you cannot fully judge, say, writing a good summary of a thick technical book. One move is to first build AI helpers that make the job judgeable: an assistant that pulls out the key claims and flags anything the summary got wrong, so that you only have to rate something you can now actually check. Recursive reward modeling builds those helpers using the very same technique, layer upon layer.

Precisely, ordinary reward modeling (Leike and colleagues, DeepMind, 2018) means learning a reward model from human feedback and then training an agent to maximize it, rather than hand-coding the reward. Recursive reward modeling adds bootstrapping: use agents you have already trained and aligned on easier tasks to assist humans in evaluating harder tasks; that assisted human feedback trains a reward model for the harder tasks; and those newly trained agents then help evaluate tasks harder still. You climb a hierarchy, each rung standing on the aligned agents built at the rung below.

It matters as the reward-model route to scalable oversight, closely related to iterated amplification and debate. The honest caveats are inherited from reward modeling itself: a reward model is a learned proxy for human preferences, not the preferences themselves, so it can be gamed (Goodhart's law and reward over-optimization). On top of that, each recursive layer can compound the errors of the layers beneath it, so it remains a research-stage approach rather than a deployed solution.