Reward the agent for being surprised
Children explore with no external reward at all — they are curious. Intrinsic motivation gives an RL agent the same drive: an internal reward, generated by the agent itself, that encourages it to seek out experiences it can learn from. Add this intrinsic reward to the (often near-zero) external reward, and even a goalless agent will roam, experiment, and master its environment.
The dominant way to formalize curiosity is prediction-error exploration: make the agent learn to predict something about the next state, and reward it in proportion to how wrong its prediction is. High error means the situation is unfamiliar — there is something to learn — so surprise itself becomes the bonus. As the agent practices a region, its predictions improve, the error falls, and the curiosity reward naturally fades, pushing it onward to fresh ground.
The Intrinsic Curiosity Module
A naive predictor of raw next-frame pixels would fixate on unpredictable but irrelevant visual noise — swaying leaves, flickering shadows. The Intrinsic Curiosity Module (ICM) solves this with a learned feature space that keeps only what the agent can affect. It trains two coupled models on those features: a forward model predicting the next state's features from the current features and action, and an inverse model predicting which action was taken between two states.
The inverse model is the clever part. By forcing the features to be good enough to recover the action, it teaches the encoder to ignore anything the agent cannot control — pixels that no action ever changes carry no signal for predicting the action, so they get filtered out. The forward model's prediction error in this purified feature space becomes the curiosity reward.
- Encode the current and next states into the learned feature space.
- Inverse model: predict the action from the two feature vectors — this trains the encoder to keep only controllable features.
- Forward model: predict the next state's features from current features + action.
- Intrinsic reward = the forward model's prediction error; add it to the external reward and learn as usual.
The ICM's intrinsic reward is the forward model's prediction error in the learned feature space φ.
The noisy-TV problem
Prediction-error curiosity has one notorious failure: the noisy-TV problem. Put a screen of random static in the environment, give the agent a remote, and it will sit forever flipping channels. The static is inherently unpredictable, so the prediction error never drops, so the curiosity reward never fades — the agent is hypnotized by pure noise. Any genuinely stochastic element (a dice roll, sensor noise) is a potential noisy TV.
Interactive Markov chain showing unpredictable random transitions between states.
Random Network Distillation
Random Network Distillation (RND) is a strikingly simple cure. Create a fixed, randomly initialized target network that maps each state to a random feature vector — it is never trained, just frozen. Then train a second predictor network to copy the target's output on the states the agent visits. The intrinsic reward is the predictor's error against the target.
RND's reward is the error between a trained predictor f-hat and a fixed, randomly initialized target network f.
Why does this dodge the noisy TV? Because the target is a deterministic function of the state — there is nothing random to predict. On states seen often, the predictor learns to match the target well, so error (and reward) is low. On novel states it has never trained against, error is high. Error is therefore a clean measure of familiarity, not of environmental randomness. A noisy TV is just as predictable to RND as a blank wall once both are visited, so it holds no special allure.