From copying actions to inferring intent
Behavioral cloning answers 'what action did the expert take here?'. Inverse reinforcement learning (IRL) asks a deeper question: 'what goal makes this behavior look optimal?'. Instead of learning a policy directly, IRL learns the reward function the expert seems to be maximizing. Once you have the reward, you can run ordinary RL on it to recover — or even surpass — the expert.
Diagram of an agent taking actions in an environment and receiving state and reward in return.
The payoff is generalization. A cloned policy fails on any state the expert never visited. A recovered reward function, by contrast, is a compact statement of intent that transfers to new dynamics, new start states, even new bodies: a reward like 'reach the goal while staying upright' still makes sense if you swap the robot.
The IRL problem, stated
Set it inside a Markov decision process whose dynamics you (mostly) know, but whose reward is missing. You are handed expert trajectories assumed to be near-optimal. The task: find a reward function under which the expert's policy scores at least as high as any alternative. Early formulations write this as a set of linear constraints — 'the expert must beat every other policy' — and search for a reward satisfying them.
A standard simplification assumes the reward is linear in hand-designed features of a state: r(s) = wᵀφ(s), where φ(s) might encode 'distance to goal', 'speed', 'collision risk'. Then learning the reward reduces to learning the weight vector w — a much smaller, tractable target.
A linear reward model: each state's reward is a weighted sum of its features, so learning the reward reduces to learning the weight vector w.
Apprenticeship learning and feature matching
Apprenticeship learning reframes IRL around a clean goal: don't insist on recovering the true reward, just produce a policy that performs as well as the expert under any reward in the chosen family. The key device is the feature expectation — the discounted average of features φ a policy accumulates over time.
If the reward is linear in features, two policies with equal feature expectations earn equal value under every weight vector. So feature-expectation matching turns 'imitate the expert' into 'make my feature expectations equal the expert's'. The algorithm alternates: guess a reward that maximally separates the current policy from the expert, run RL to optimize it, measure the new policy's feature expectations, and repeat until they match the expert's.
Interactive gridworld where Q-learning finds a policy from a reward function.
Reward ambiguity: the central headache
Here is the inverse problem biting back. Many different rewards explain the same demonstrations equally well — including the degenerate reward that is zero everywhere, under which every policy (including the expert's) is trivially optimal. This is reward ambiguity, and without extra assumptions IRL cannot pick the 'right' reward out of this large equivalence class.
Practical IRL methods resolve ambiguity with a preference principle — a tie-breaker that selects one reward from the many. Margin-based methods prefer rewards under which the expert beats alternatives by the widest margin. The most influential tie-breaker, maximum entropy, picks the reward that explains the data while committing to the fewest extra assumptions — the subject of the next guide.
IRL vs imitation: when to use which
The distinction comes down to what you learn. Imitation (behavioral cloning, DAgger) learns a policy directly: cheap, fast, but brittle to distribution shift and unable to exceed the expert. IRL learns a reward, then a policy: expensive (an RL solve inside every iteration) but yielding an intent that transfers across environments and can support an agent that improves on a suboptimal expert.
Pipeline where human preferences train a reward model that then guides policy tuning.