feudal reinforcement learning

Feudal RL organises an agent like a medieval chain of command: a lord sets goals for vassals, vassals set goals for their own subordinates, and only the lowest rank touches the actual world. A high-level 'manager' looks at the situation and issues a goal — often a direction to move in some abstract space — and a low-level 'worker' is rewarded purely for satisfying that goal, never for the task's true reward. The manager learns what goals lead to reward; the worker learns how to reach whatever goal it is handed.

Two principles, named in the original 1992 work by Dayan and Hinton, keep the hierarchy clean. Reward hiding means a level is rewarded only for fulfilling its own superior's commands, not the global objective, so each level has a well-defined job. Information hiding means each level sees the world at its own resolution, ignoring the fine detail handled below it and the grand plan above. The modern FeUdal Networks paper revived the idea with deep networks, having the manager emit goals in a learned latent space.

The appeal is strong temporal abstraction and good behaviour on sparse-reward tasks, because the worker always has a dense, immediate goal to chase even when the environment's reward is rare. The difficulty is keeping the two levels in sync: if the manager proposes goals the worker cannot reach, or the worker games its goal reward without helping the real task, the hierarchy quietly falls apart.