JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Discovering Skills Without Rewards — and the Open Frontier

The hardest part of hierarchy is finding the pieces. See how agents discover diverse skills with no task reward at all, how those skills feed a hierarchy, and what remains genuinely unsolved.

Skills as reusable behaviours

We have now seen options (named skills) and goals (proposed destinations). Skill discovery asks the most ambitious version of the question: can an agent invent a useful repertoire of behaviours before it is ever told the task — purely by interacting with the world? If so, a later high-level controller can compose those pre-built skills into a hierarchy almost for free.

Skill discovery is the reinforcement-learning cousin of unsupervised learning: the agent learns how to act with no task reward, just as unsupervised methods find structure with no labels.

Diagram contrasting the three learning paradigms: supervised, unsupervised, and reinforcement learning.

Diversity is all you need (DIAYN)

The signature method is Diversity Is All You Need (DIAYN). Its insight is delightfully simple: a good skill is one you can tell apart from the others by looking at where it goes. So instead of a task reward, give the agent an intrinsic reward for making its skills distinguishable.

\mathcal{I}(S;Z) = \mathcal{H}(Z) - \mathcal{H}(Z\mid S)

DIAYN's objective: maximize the mutual information between the skill code z and the states it visits, so different skills become easy to tell apart.

  1. Sample a latent skill code z at the start of an episode and condition the policy on it (much like a goal in goal-conditioned RL).
  2. Train a discriminator that tries to guess which z produced each visited state.
  3. Reward the policy when the discriminator guesses correctly — i.e. reward each skill for visiting states that uniquely identify it. Maximising this mutual information spreads the skills out to cover the state space.
r_z(s) = \log q_\phi(z \mid s) - \log p(z)

DIAYN's pseudo-reward: the policy is paid when the discriminator q_φ confidently recovers the skill code z from the visited state s.

Astonishingly, with no task reward at all, DIAYN-style agents learn to walk, hop, run, and flip — distinct locomotion gaits emerge purely from the pressure to be different. Those gaits are exactly the kind of reusable primitives a hierarchy wants.

From discovered skills to a working hierarchy

Discovery and control then snap together. Freeze the learned skill set, and train a high-level policy that selects skills (or skill codes z) to maximise the real task reward. The hierarchy is now two clean layers: a discovered low level that knows how to move, and a learned high level that decides which behaviour to deploy when — the same manager-worker split from the feudal world, but with the worker's skills bootstrapped reward-free.

Because the skills were learned once and are task-agnostic, the same repertoire transfers across many downstream tasks — pay the exploration cost up front, reuse forever. This is why skill discovery is often pitched as a direct cure for the subgoal-discovery bottleneck and for the option collapse we met with Option-Critic.

The frontier: what's still genuinely hard

  1. Useful vs. merely diverse. Diversity objectives reward being different, not being useful. Skills may scatter to cover trivial state dimensions and miss the behaviours a real task needs — aligning discovery with downstream value is open.
  2. Deeper than two levels. Almost everything in practice is two layers. Stable learning of three or more levels of temporal abstraction remains largely unsolved.
  3. Non-stationarity between levels. While the worker is still changing, the manager is aiming at a moving target; co-adapting both layers stably is delicate.
  4. When does hierarchy even help? On many benchmarks a well-tuned flat agent matches a hierarchical one. Characterising which problems truly need hierarchy is an active debate.

Where to go next

You now hold the two great families of hierarchy — options (guides 2–3) and goals (guide 4) — plus the discovery machinery (this guide) that feeds both. The natural next steps are exploration (skill discovery is really directed exploration in disguise), transfer and meta-RL (where reusable skills shine), and multi-agent settings where managers coordinate whole teams.