JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Goals, Not Options: Goal-Conditioned & Feudal RL

A second flavour of hierarchy: instead of selecting skills, a manager sets goals and a worker tries to reach them. Meet UVFAs, hindsight relabeling, and the feudal manager-worker design.

A different way to be hierarchical

Options answer 'which skill?'. There is a second, increasingly popular answer: instead of choosing from a menu of skills, have a high level propose a goal and a low level learn to reach any goal it is handed. This is goal-conditioned RL — a single policy and value function that take the desired goal as an extra input, so one network can pursue thousands of different targets.

This is hierarchy by abstraction of intent rather than abstraction of behaviour: the high level still works at a coarse temporally abstract timescale (it commits to a goal for many steps), but what it emits is a destination, not a hand-named skill.

Universal value functions (UVFA)

To pursue arbitrary goals you need a value function that generalises across them. Universal value function approximators (UVFAs) do exactly that: they approximate `V(state, goal)` or `Q(state, action, goal)` with the goal folded in as input. Train on a handful of goals and a UVFA can interpolate to goals it has never explicitly practised — the bridge that makes goal-conditioned hierarchy scale.

Q^*(s,a,g) = r(s,a,g) + \gamma \max_{a'} Q^*(s',a',g)

A UVFA learns a single value function shared across all goals g, satisfying this goal-conditioned Bellman optimality equation.

Hindsight goal relabeling

Goal-conditioned RL has a chicken-and-egg problem in sparse-reward settings: if the agent almost never reaches the commanded goal, it almost never gets a reward, so it never learns. Hindsight goal relabeling is the elegant fix — and one of the most quietly powerful ideas in modern RL.

On a gridworld, hindsight relabeling treats wherever the agent actually ended up as the goal it 'meant' to reach — so it cannot fail.

Interactive Q-learning gridworld where an agent learns to navigate toward goal cells.

  1. Run an episode aiming for goal g. Suppose you fail and end up at some state *g'* instead.
  2. Relabel the very same trajectory as if g' had been the goal all along. Now it is a success — and a real, dense learning signal.
  3. Train the UVFA on both the original and relabeled goals. Every failure becomes a lesson about some reachable goal.

Feudal RL: managers and workers

Feudal reinforcement learning turns the goal-setting idea into an explicit two-level manager-worker hierarchy. A manager operates at a slow timescale and emits a goal — typically a direction or target in a learned latent space. A worker operates at the fast, primitive timescale and is rewarded for moving the state toward the manager's goal.

The defining principle is reward hiding: the worker never sees the true task reward, only the manager's goal-reaching signal; the manager never issues primitive actions, only goals. This clean division of labour means the manager can learn long-horizon strategy while the worker masters short-horizon control. Modern variants (e.g. FeUdal Networks) set goals in a latent space and train the manager with a transition-direction objective so its goals stay achievable.

r^{I}_t = \cos\!\left(s_{t+c}-s_t,\; g_t\right)

Reward hiding in feudal RL: the worker's intrinsic reward is just how well its movement aligns with the manager's commanded direction g — it never sees the true task reward.

Options or goals — how to choose

  1. Reach for options when there is a small set of clearly reusable, nameable skills and you want them as discrete, transferable building blocks.
  2. Reach for goal-conditioned / feudal designs when the space of desirable sub-targets is large or continuous — a manager proposing points in a goal space scales far better than enumerating thousands of options.
  3. Either way, subgoal discovery resurfaces: a feudal manager that proposes good goals is solving the same problem as an option learner that finds good skills.