Resolving Ambiguity: MaxEnt IRL and Adversarial Imitation

MaxEnt IRL: the least-committed explanation

Maximum-entropy IRL resolves reward ambiguity with a principle from statistics: among all explanations consistent with the data, prefer the one that assumes the least. Concretely, it models the expert as choosing trajectories with probability proportional to the exponential of their return — high-reward trajectories are exponentially more likely, but every trajectory keeps some probability.

p(\tau)\;\propto\;\exp\!\left(\sum_{t} r_\theta(s_t,a_t)\right)

MaxEnt IRL makes a trajectory exponentially more likely the higher its total reward — the least-committed distribution consistent with the demonstrations.

This single modeling choice does three things at once. It makes the model probabilistic (so you can fit it by maximum likelihood), it gracefully tolerates imperfect demonstrations (a slightly bad expert action is unlikely, not impossible), and it uniquely selects a reward — the flat zero reward, which would make all trajectories equally likely, no longer fits demonstrations that clearly favor some paths over others.

Why entropy is the right tie-breaker

Fitting the MaxEnt model by maximum likelihood turns out to reproduce feature-expectation matching automatically: at the optimum, the model's expected features equal the expert's observed features. But among the infinitely many distributions that match those features, the maximum-entropy one is the flattest — it spreads probability as evenly as the constraints allow, encoding no preference the data didn't force on it.

\mathbb{E}_{\pi_\theta}\!\big[\phi\big]\;=\;\mathbb{E}_{\text{expert}}\!\big[\phi\big]

At the maximum-likelihood optimum the model's expected features equal the expert's — MaxEnt reproduces feature-expectation matching automatically.

That is exactly the property we wanted from a tie-breaker: when the demonstrations underdetermine the reward, default to the explanation that adds the fewest hidden assumptions. The practical cost is computing a normalizing term over trajectories (the partition function), which in known small MDPs uses dynamic programming, and in large or unknown ones is approximated — a need that pushes the field toward sampling-based, adversarial methods.

GAIL: imitation as a two-player game

Generative adversarial imitation learning (GAIL) borrows the machinery of the generative adversarial network. Two networks compete: a discriminator tries to tell expert state–action pairs apart from the learner's, while the policy (the generator) tries to fool it by producing expert-like behavior. The discriminator's verdict serves as a learned reward signal that the policy maximizes with ordinary RL.

GAIL's breakthrough is that it skips the expensive inner RL solve that classic IRL repeats for every candidate reward. By matching the distribution of state–action pairs directly through the adversarial game, it scales to high-dimensional continuous control where running full RL inside an outer loop would be hopeless. The price is GAN-style training instability — the two players can oscillate instead of converging.

for iteration in range(N):
    learner_data = rollout(policy)              # generator samples
    # 1) Train discriminator: expert = real, learner = fake
    D = update_discriminator(expert_data, learner_data)
    # 2) Use -log D(s,a) as the reward, improve the policy with RL
    rewards = [-log(D(s, a)) for (s, a) in learner_data]
    policy = rl_update(policy, learner_data, rewards)

GAIL alternates a discriminator update with an RL policy update; the discriminator's score is the reward.

Adversarial IRL: recovering a reusable reward

GAIL imitates beautifully but throws the reward away — its discriminator entangles reward with the current policy, so you cannot lift out a clean reward to reuse. Adversarial inverse RL (AIRL) fixes this by structuring the discriminator so that part of it provably converges to a reward function disentangled from the dynamics. You get GAIL's scalability and IRL's transferable reward.

That disentangled reward is the prize. Train AIRL on flat ground, recover 'move forward efficiently', then drop the reward onto a robot facing hills or stronger gravity and re-run RL — the policy adapts because the intent transferred, even though the optimal actions changed. This robustness to shifting dynamics is exactly what pure imitation cannot offer.