The trouble with one step at a time
Standard reinforcement learning frames every decision as a single primitive action: nudge the joint, press the key, step left. That is fine for short tasks, but imagine an agent that must navigate a building, open a door, and fetch a cup. As a problem in sequential decision making, the chain of primitive actions is hundreds of steps long, and the reward only arrives at the very end. The agent then faces a brutal credit assignment problem: which of those hundreds of tiny moves actually mattered?
A loop diagram: the agent sends an action to the environment, which returns a reward and the next state.
The deeper issue is that delayed reward gets diluted over long horizons. Each tiny action takes a sliver of the blame or credit, so the learning signal per step is faint and noisy. Exploration suffers too: random one-step jitter almost never stumbles onto a hundred-step solution by chance.
Discounted return: distant rewards are scaled by gamma^k, so over a long horizon each step's slice of credit gets diluted.
Temporal abstraction: the core idea
Humans do not plan a trip to the kitchen as a thousand muscle twitches. We think walk to the door, open it, go to the counter — each a reusable behaviour that runs for many steps and then stops. Temporal abstraction gives an agent the same power: the ability to commit to an extended course of action and reason at the level of those chunks instead of raw primitives.
This is the whole premise of hierarchical reinforcement learning (HRL): build a hierarchy of policies where a high level picks coarse, long-lived behaviours and a low level executes the fine-grained primitives. The payoff is dramatic — credit travels over a handful of high-level decisions instead of hundreds of micro-steps, and exploration jumps across the state space in meaningful leaps.
Macro-actions: your first taste of abstraction
The simplest form of temporal abstraction is the macro-action: a fixed, open-loop sequence of primitives bundled into one chooseable unit — like a chess opening or a `go-forward-3-then-turn-left` routine. The high level treats the whole macro as a single decision; the environment just runs the canned sequence.
Macro-actions show the upside immediately — fewer decisions, longer reach — but also the limit: they are open-loop. A canned sequence cannot react if the world shifts mid-stride (someone closes the door you were walking toward). The next guide upgrades macros into options, which carry their own closed-loop policy and decide for themselves when to stop.
Where does the hierarchy come from?
A natural question: who decides what the useful sub-behaviours are? One classic answer is to look for bottleneck states — the narrow passages every successful path must funnel through. In a multi-room maze, doorways are bottlenecks: learn to reach a doorway, and you have a reusable skill that unlocks whole regions of the task.
An interactive gridworld where an agent learns action values; a single doorway connects two rooms.
More generally, subgoal discovery is the art of automatically proposing intermediate targets — landmarks the agent should learn to hit. Bottlenecks, frequently-visited states, and states that surprise the agent are all popular signals. We will return to discovery in depth in guides 3 and 5; for now, just hold the picture: a hierarchy needs pieces (the sub-behaviours) and a boss that strings them together.
What you'll build in this track
- Guide 2 — the options framework: the precise, closed-loop definition of a temporally extended action.
- Guide 3 — learning options end-to-end: intra-option learning and the Option-Critic architecture.
- Guide 4 — goal-conditioned and feudal RL: hierarchy by setting goals instead of choosing options.
- Guide 5 — skill discovery without rewards and the open frontier of HRL.