The Options Framework

From actions to options

The options framework is the standard formalism for temporal abstraction in RL, built directly on top of the ordinary MDP. An option is a self-contained skill the agent can invoke as if it were a single action — but unlike a primitive action, it can run for many time steps and reacts to what it sees along the way. Crucially, options and primitive actions coexist: the agent can choose a one-step move or a multi-step option at the same decision point.

The three pieces of an option

Formally an option is a triple. Together these three pieces upgrade a one-shot macro-action into a flexible, closed-loop skill described by an option policy and termination condition.

Initiation set — the set of states in which the option is allowed to start. `open-the-door` only makes sense when you are near a door.
Intra-option policy — a closed-loop policy over primitive actions that runs the skill, choosing actions step by step while the option is active.
Termination condition — a probability, for each state, that the option stops now and hands control back to the higher level. This is what makes options react to the world instead of blindly running to the end.

Options make the MDP semi-Markov

Here is the subtle but important consequence. When the high level chooses an option, the option runs for a random number of steps before terminating. So at the high level, transitions no longer happen on a fixed clock — they happen after variable durations. A decision process with variable-duration actions is a semi-Markov decision process (SMDP), and treating options this way gives you semi-Markov options.

The reward and discounting math adjusts cleanly. If an option runs for k steps and collects rewards along the way, the high-level value backup discounts the next state by the discount factor raised to the power k — longer options are discounted more, exactly as a single very-delayed reward would be.

Q(s,o) = \mathbb{E}\!\left[\, r_{t+1} + \gamma\, r_{t+2} + \cdots + \gamma^{k-1} r_{t+k} + \gamma^{k}\max_{o'} Q(s',o') \,\right]

The high-level (SMDP) value backup: an option that runs k steps discounts the next option-value by γ to the power k.

Planning and learning over options

Because the SMDP over options is itself a well-formed decision process, every tool you already know applies — just at the option level. You can run value iteration, Q-learning, or policy gradients over a set of options exactly as you would over primitive actions; the only change is that one 'step' is now one option execution.

# High-level control loop over a fixed set of options
state = env.reset()
while not done:
    option = high_level_policy(state)      # pick an extended skill
    while option.active(state):            # closed-loop execution
        action = option.policy(state)      # intra-option policy
        state, reward = env.step(action)
        accumulate(reward)                 # rewards collected during the option
        if option.terminates(state):       # termination condition
            break
    # SMDP backup happens here, discounting by gamma**duration

The two-level loop: the high level picks an option, the option runs closed-loop until its termination condition fires.

A worked example: the four-room maze

The textbook demo is a grid split into four rooms connected by doorways. Define one option per doorway: its initiation set is the room, its intra-option policy walks toward that doorway, and it terminates upon arrival. Now the high-level agent's job is tiny — chain two or three 'go-to-doorway' options to reach any goal — instead of learning a path through dozens of primitive cells.