manager–worker hierarchy

This is the most common shape a hierarchical agent takes: two coupled policies that run at different clock speeds. The manager (the high-level policy) wakes up occasionally, looks at the situation, and hands down a goal or sub-task. The worker (the low-level policy) then runs flat-out for many steps, taking primitive actions to satisfy whatever the manager asked for, before the manager wakes again to issue the next goal. One brain sets direction; the other does the driving.

What flows down the hierarchy is a goal, and what flows up is the consequence. The worker is typically goal-conditioned — its policy and value function take the manager's goal as an extra input — and is trained on a goal-reaching reward. The manager is trained on the environment's true reward, but its action space is the set of goals rather than primitive moves, so it operates at a coarser, slower time scale that is much easier to plan over.

Almost every named hierarchical method — feudal RL, options with a top-level chooser, goal-conditioned schemes like HIRO — is some instance of this pattern. The recurring engineering challenge is non-stationarity: while the worker is still learning, the same goal produces different outcomes over time, so the manager is aiming at a moving target. Stabilising that two-level interaction is what most of the practical work is about.