hierarchical reinforcement learning

Think about how you make coffee. You do not plan it as a thousand tiny muscle twitches; you think 'grind beans, then boil water, then pour', and each of those expands into smaller steps only when you reach it. Hierarchical reinforcement learning gives an agent the same gift: instead of choosing a low-level action every single timestep, a high level picks a short-term goal or sub-behaviour, and a low level carries it out. The same problem is solved at two or more time scales at once.

Mechanically, you split one flat policy into a layered controller. A top-level policy chooses among a small set of temporally extended behaviours — call them options, skills, or subgoals — and a bottom-level policy executes the chosen behaviour for many steps until it finishes. Each level is still an RL problem with its own states, actions, and rewards, but the top level now reasons over a much shorter, coarser sequence of decisions, which is far easier to learn and to plan with.

The payoff is sharpest on long, sparse-reward tasks where a flat agent would wander for thousands of steps before stumbling on anything good: structure shortens the effective horizon and lets useful behaviours be reused across tasks. The price is that you must now learn or design the hierarchy itself, and a badly chosen decomposition can hurt more than it helps. Most of this field is about getting that decomposition right.