preference-based reward learning

Preference-based reward learning infers what to reward from comparisons rather than from demonstrations. Instead of asking a person to perform the task — which they may not be able to do — you show them two of the agent's own behaviors and ask the far easier question: which of these is better? From many such yes-or-no judgments, a reward model is fitted, and ordinary RL then optimizes it.

The standard model treats each comparison as noisy evidence: a segment with higher true reward should be chosen more often, formalized by a Bradley-Terry likelihood where the probability of preferring one trajectory grows with the gap in their summed rewards. Fitting the reward model is maximum-likelihood over the collected preference labels, and the loop alternates between collecting comparisons on fresh behavior and improving the policy.

This is the backbone of RLHF for language models and of reward learning where goals are hard to script but easy to judge. Its risks are that the learned reward can be gamed by the policy (reward hacking) and that it only reflects the comparisons and judges you actually sampled.