information-gain exploration

The most principled answer to where one should explore is: wherever you expect to learn the most. Information-gain exploration values an action not by the reward it brings but by how much it is expected to reduce the agent's uncertainty about the world — formally, the expected drop in the entropy of its beliefs, or equivalently the mutual information between the next observation and the unknown dynamics. The agent seeks experiments, not just rewards.

This casts exploration as optimal experimental design and gives a clean, unified account of curiosity — methods like variational information maximisation (VIME) add an intrinsic reward equal to the information a transition provides about the agent's model parameters. The appeal is that it targets useful surprise, ignoring noise that carries no information. The price is that estimating expected information gain requires a model of the agent's own uncertainty, which is approximate and expensive in deep networks.