reward ambiguity in IRL

Reward ambiguity is the fundamental headache of inverse RL: the same demonstrations are explained equally well by many different reward functions. Most embarrassingly, the all-zero reward makes every policy optimal, so it trivially fits any behavior. Watching someone drive carefully, you cannot tell whether they value safety, dislike tickets, or simply enjoy a smooth ride — the trajectory alone underdetermines the goal.

More precisely, several transformations leave the optimal policy untouched: scaling the reward by a positive constant, adding a constant, and potential-based shaping that adds the difference of any function across a transition. Any reward and all of its shaped cousins induce identical behavior, so demonstrations can never single one out without further assumptions.

This is why every practical IRL method imposes a tie-breaker — maximum entropy, feature-expectation matching, a margin, or a prior. Recognizing ambiguity also tempers expectations: a recovered reward is one plausible explanation, not the expert's true mind, and should be validated by how well it transfers rather than taken as ground truth.