preference-based reinforcement learning

Classic RL assumes the environment hands the agent a numeric reward at every step. But for many real tasks no such number exists — what is the reward for a 'natural-looking' backflip? Preference-based RL drops the assumption of a given reward and learns purely from comparative feedback: a human (or other oracle) is occasionally shown two behaviours and says which is better, and the agent must find a good policy from those judgments alone.

The standard approach interleaves two loops. The agent acts and collects experience; a teacher compares pairs of behaviours, often whole trajectories or short clips; a reward model is fit to those comparisons; and the agent does ordinary RL against the current reward model — then the cycle repeats, with new comparisons queried for the situations the agent now visits. RLHF for language models is the famous instance, but the framework is general and predates large models, with roots in robotics and control.

The central research question is feedback efficiency: human comparisons are expensive, so which pairs should you ask about? Good preference-based RL spends its query budget wisely — active learning, uncertainty sampling, asking only where the answer would most change the policy — and stays honest that the learned reward is only a model of the teacher, carrying every overoptimisation risk that entails.