intrinsic curiosity module (ICM)

Children explore not for external reward but because surprise itself feels rewarding. The intrinsic curiosity module gives an agent the same drive: it learns a model that predicts the consequence of each action, and pays the agent a bonus whenever that prediction turns out wrong. States the agent cannot yet predict are exciting; states it has mastered are boring, so it keeps pushing toward the edge of what it understands.

The clever part is where the prediction happens. To avoid being mesmerised by unpredictable but irrelevant noise — a flickering television, swaying leaves — ICM first learns a feature space that keeps only what the agent's actions can influence, using an inverse model that predicts the action from the before-and-after features. Curiosity is then the error of a forward model in that space. This filters out distractions, though truly stochastic-but-controllable dynamics can still snag it, the so-called noisy-TV problem.