JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Learning Options End-to-End: Intra-Option Learning & Option-Critic

Stop hand-coding skills. Learn the option policies and their termination conditions directly from reward with intra-option learning and the Option-Critic architecture — and meet the failure mode that haunts it.

The bootstrapping problem: where do options come from?

Hand-designed options need a human who already knows the task structure — exactly the knowledge we hoped the agent would discover. The grand goal of hierarchical RL is to learn the whole stack from reward alone: the high-level policy and the option policies and their termination conditions, all at once. Two ideas make this practical.

Intra-option learning: squeeze every transition

A naive approach updates an option's value only when the option finishes — wasting all the rich experience collected mid-execution. Intra-option learning fixes this: every single primitive transition is used to update every option whose policy would have taken that same action, not just the one currently running.

This is an off-policy bonus: while running one skill, the agent quietly learns about all the others that agree with it, so data is shared instead of siloed. The result is far better sample efficiency — options improve even when they are not the one in control.

The Option-Critic architecture

The Option-Critic architecture is the landmark result that learns options end-to-end with gradients, requiring no subgoals, no pseudo-rewards, and no hand-specified termination. It extends the policy gradient theorem to the options setting, deriving gradients for two new objects at once.

Option-Critic learns its option policies and termination conditions end-to-end by following gradients down the loss surface — no subgoals or pseudo-rewards required.

Interactive gradient descent rolling down a loss surface toward a minimum.

  1. The intra-option policy gradient improves how each option behaves, pushing its actions toward higher long-run value.
  2. The termination gradient tunes when each option stops: an option should keep running while it is still advantageous, and terminate when a different option would do better.
  3. A critic estimates option-value functions to supply the advantage signals that drive both gradients — exactly the role a critic plays in an ordinary actor-critic setup.
\nabla_{\theta} J \;=\; \mathbb{E}\!\left[\,\nabla_{\theta}\log \pi_{\omega,\theta}(a\mid s)\; Q_{U}(s,\omega,a)\right]

The intra-option policy gradient: each option's policy is nudged toward actions whose option-value Q_U is high.

Discovery as a side effect — bottlenecks and subgoals

Option-Critic discovers structure implicitly: the termination gradient naturally learns to end options near useful decision points, which often turn out to be bottleneck states. A complementary, more explicit line of work does subgoal discovery up front — mining trajectories for the doorway-like states that many successful paths share, then building one option to reach each.

Bottleneck states are the hubs many trajectories must pass through — explore the Markov chain to spot where options naturally want to terminate.

Interactive Markov chain showing states and transition probabilities, with hub-like bottleneck nodes.

Both routes answer the same question — *what is worth being a skill?* — from opposite directions: implicit discovery lets reward shape the options, while explicit discovery seeds the options from graph structure and then refines them.

The failure mode: option collapse

Learned options have a notorious pathology. Without pressure to stay distinct, Option-Critic tends to degenerate in one of two ways: either one option hogs the entire task (the others go unused), or every option terminates almost immediately so the hierarchy quietly collapses back to flat, primitive-level control. Both make the termination machinery pointless.