Diversity Is All You Need

DIAYN is the best-known concrete recipe for reward-free skill discovery, and it turns a slogan into an objective: skills are valuable to the extent that you can tell them apart by what they do. Concretely, the agent samples a skill code z at the start of an episode, acts according to a skill-conditioned policy, and is rewarded whenever its visited states make it easy for a separate classifier to guess which z it was running. Skills compete to occupy distinct, recognisable corners of the state space.

The formal objective is to maximise the mutual information between the skill z and the states s the agent visits, while keeping the policy's actions varied via an entropy bonus. In practice you train a discriminator to predict z from s, and use the log-probability it assigns to the true skill as the intrinsic reward. The mutual-information goal pushes different skills toward different state distributions, and the entropy keeps each skill from collapsing to a single rigid behaviour.

What makes DIAYN striking is that with zero environment reward it discovers genuinely useful behaviours — agents learn to walk, hop, or move in different directions just from the drive to be distinguishable — which can then seed hierarchical control or jump-start a downstream task. Its known limitation is coverage: because the reward only asks for distinguishability, skills tend to spread over easily reachable states and may miss the rare, task-critical regions a real objective cares about.