apprenticeship learning

Apprenticeship learning is an early, influential way to use inverse RL for control. Rather than insisting on the one true reward, it assumes the reward is a weighted sum of known features of a state — distance to obstacles, speed, lane position — and aims for a more modest goal: produce a policy that performs as well as the expert under any such reward, even without ever pinning down the exact weights.

The key quantity is feature expectations: the discounted average of each feature accumulated along a trajectory. The crucial observation is that two policies with the same feature expectations earn the same return for every linear reward. So the algorithm alternates between guessing reward weights, solving the resulting RL problem, and adjusting until the learner's feature expectations match the expert's — at which point its performance is provably close to the expert's.

Its strength is the guarantee on behavior despite reward ambiguity; its limits are the reliance on hand-chosen features and on rewards being linear in them.