trajectory preference feedback

It is hard for a person to say whether a single joint torque or a single token was good — those choices only make sense in context. It is much easier to watch two whole behaviours play out and say which one you liked better. Trajectory preference feedback is exactly that: humans compare entire sequences of behaviour — a full robot motion, a complete dialogue, a clip of gameplay — rather than judging step by painstaking step.

The human compares trajectories, but the reward model usually scores individual states or steps; the bridge is the assumption that a trajectory's total value is the sum of its per-step rewards. So a preference between two clips becomes a constraint on summed rewards, and fitting the reward model — via Bradley-Terry over trajectory return sums — distributes a single coarse judgment across all the steps inside. This is how a handful of clip comparisons can teach a dense, step-level reward.

The convenience hides a hard credit-assignment problem: if you preferred clip A, which of its many moments actually deserved the credit? Long trajectories make this worse, and humans may judge on one salient moment or on superficial features like smoothness. Shorter clips and well-chosen comparisons help, but spreading a trajectory-level preference down to a step-level reward is inherently ambiguous, and the reward model has to guess.