options framework

An option is the cleanest formalisation of a 'skill' or a 'macro-behaviour' an agent can call like a subroutine. Imagine a button labelled 'walk to the door': once you press it, a built-in controller takes over, steers the body for as long as needed, and hands control back when you have arrived. The options framework, introduced by Sutton, Precup and Singh, makes that button a first-class object the agent can choose among, alongside or instead of primitive actions.

Formally an option is a triple: an initiation set (the states where you are allowed to start it), an internal policy (what primitive actions it takes while running), and a termination condition (the probability it ends in each state). When the top-level policy selects an option, the option's policy drives the agent until termination fires, then control returns to the top level. Primitive one-step actions are just the special case where the option always terminates after a single step.

The beauty is that planning and learning can happen over options as if they were ordinary actions, turning the underlying problem into a semi-Markov decision process. This shortens horizons, lets you transfer skills between tasks, and gives a clean interface between a planner and its controllers — which is why options remain the backbone of most hierarchical RL even decades later.