subgoal discovery

If a hierarchy is only as good as its decomposition, the obvious next question is: where do the sub-tasks come from? Subgoal discovery is the problem of finding useful intermediate targets automatically, instead of a human writing them by hand. The dream is for the agent to notice on its own that 'reach the doorway' or 'pick up the key' are natural waypoints, and to spin up options or goals around them.

Approaches cluster into a few families. Graph-based methods build a model of which states lead to which and look for structurally special states — doorways, junctions, bottlenecks — that many paths must pass through. Frequency methods flag states that appear often on successful trajectories but rarely on failures. Reward- and novelty-based methods propose subgoals that are surprising, hard to reach, or that maximise some intrinsic signal. In each case the discovered states become the targets a goal-conditioned worker is trained to hit.

Good subgoals can transform a hopeless sparse-reward task into a sequence of short, learnable hops; bad ones add overhead for nothing or send the agent chasing dead ends. The honest state of the art is that fully automatic discovery is still brittle and an active research area — getting it reliably right across diverse tasks remains one of hierarchical RL's central open problems.