reward model ensembles

A single reward model has blind spots, and an optimiser will hunt them down. One defence is to ask several reward models instead of one. Train a handful on the same preferences but with different random seeds or data orderings, and they will mostly agree where the data is solid and disagree where it is shaky. Their disagreement is a built-in alarm for exactly the regions where any single one of them might be fooled.

You train an ensemble and combine its members — averaging their scores, or more usefully, penalising the policy by their disagreement (their variance). Where the models agree, the reward is trusted and the policy optimises normally; where they diverge, the effective reward is discounted, so the optimiser is discouraged from venturing into the uncertain, easily-hacked regions. This turns the ensemble's epistemic uncertainty into a conservative pressure that resists overoptimisation.

Ensembles help but they are not a cure. The members share training data and architecture, so they share blind spots and can be confidently wrong together; an ensemble also multiplies the compute and memory cost. It is a partial, practical hedge against reward hacking, best stacked with a KL penalty and fresh comparisons rather than leaned on alone as if disagreement could catch every failure.