Frontiers: Reward, Transfer, Hierarchy, Curiosity, and Many Agents

Where does the reward come from? — Inverse RL

Every algorithm so far assumed a given reward function. In the real world that assumption is the hardest part: hand-designing a reward is brittle and invites reward hacking, where the agent satisfies the letter of the reward while violating its intent. Inverse reinforcement learning (IRL) flips the problem: given expert demonstrations, infer the reward function that best explains them, then optimize it. The reward is the most transferable, compact description of a task, so recovering it can generalize far better than copying actions directly.

IRL is fundamentally ill-posed — many rewards explain the same behavior, and a constant reward explains everything. Maximum-entropy IRL resolves the ambiguity by preferring the reward under which the expert's trajectories are most probable while assuming the expert is only approximately optimal. Its modern adversarial form (the GAIL/AIRL family) trains a discriminator to tell expert from policy trajectories and uses it as the reward signal, drawing imitation learning and GANs into the same picture. This contrasts sharply with naive reward shaping, where you guess at a denser reward by hand.

p(\tau)=\frac{1}{Z}\exp\!\Big(\textstyle\sum_{t} r_\theta(s_t,a_t)\Big)

Maximum-entropy IRL makes a trajectory's probability grow exponentially with its reward, picking out which reward best explains the demonstrations.

Transfer via successor features

Suppose the dynamics of your world stay fixed but the reward changes — same maze, new exit. Re-solving from scratch wastes everything you learned about how the world moves. Successor features cleanly separate the two: factor the reward as a dot product of state features and a reward weight vector, then learn the expected discounted sum of future features under a policy. That successor-feature representation captures the dynamics once; when the reward weight changes, the new value function is just a dot product away — no re-learning of dynamics required.

Q^\pi(s,a)=\boldsymbol{\psi}^\pi(s,a)^{\top}\mathbf{w},\quad \boldsymbol{\psi}^\pi(s,a)=\mathbb{E}_\pi\!\Big[\textstyle\sum_{k\ge 0}\gamma^{k}\,\boldsymbol{\phi}(s_{t+k},a_{t+k})\,\Big|\,s_t=s,a_t=a\Big]

Successor features factor value into expected discounted features ψ times a reward vector w, so a new reward changes only w.

Paired with generalized policy improvement — evaluate a library of old policies on the new reward and act greedily over the best — successor features give zero-shot or few-shot transfer across a whole family of tasks that share dynamics. The honest limit is the feature basis: transfer is only as expressive as the chosen features, and learning features that are both reward-relevant and dynamics-faithful is itself an open problem.

Temporal abstraction — options and hierarchy

Flat RL chooses one primitive action per timestep, which is hopeless over long horizons — a thousand-step task is a thousand-deep credit-assignment problem. Options and hierarchical RL introduce temporally extended actions: an option bundles an internal sub-policy, an initiation set, and a termination condition, so a high-level policy can choose "go to the door" and let the option run for many steps. Formally this turns the problem into a semi-Markov decision process, where decisions occur at variable time intervals.

Exploration beyond ε-greedy — intrinsic motivation

When rewards are sparse, random exploration is hopeless — an agent could flail for the lifetime of the universe and never see a reward in a hard exploration game. Intrinsic motivation manufactures an internal reward to drive exploration when the external one is silent. The two dominant recipes are curiosity (reward the agent for transitions its own forward model predicts poorly — surprise as a signal of learnable novelty) and count/novelty bonuses (reward visiting rarely seen states, with pseudo-counts standing in for exact counts in large state spaces). Random Network Distillation is a robust modern instance of the novelty idea.

r_t = r^{\text{ext}}_t + \beta\,r^{\text{int}}_t,\qquad r^{\text{int}}_t=\big\lVert \hat f(s_t,a_t)-\phi(s_{t+1})\big\rVert_2^{2}

Curiosity adds an intrinsic bonus — here the prediction error of a learned forward model — on top of the sparse extrinsic reward.

Intrinsic motivation is powerful but treacherous. The noisy-TV problem is the canonical failure: an agent rewarded for surprise will fixate on a stochastic, unpredictable but useless source of novelty — a television showing static — because it is endlessly surprising. Good intrinsic rewards must chase epistemic novelty (things you could learn to predict) rather than mere aleatoric noise, echoing the uncertainty distinction we met in distributional RL.

When the world contains other learners — multi-agent RL

Every method so far assumed a stationary environment. Multi-agent RL (MARL) breaks that assumption: when other agents are simultaneously learning, the environment each one faces is non-stationary by construction — the ground shifts under your value estimates because everyone else is changing too. The right solution concept is no longer a single optimum but a game-theoretic equilibrium, and convergence guarantees from single-agent theory mostly evaporate.

The workhorse design pattern is centralized training with decentralized execution: during training a critic may see the joint state and all agents' actions to get a stable learning signal, but each agent executes using only its own local observation. Cooperative settings wrestle with credit assignment across the team; competitive settings lean on self-play to bootstrap ever-stronger opponents, the engine behind superhuman play in Go, poker, and StarCraft.