intra-option learning

The naive way to learn an option's value is to run it from start to finish, see the total reward, and only then update — one lesson per completed option, which is painfully slow. Intra-option learning is the smarter alternative: it updates an option from every single primitive step it takes, not just at the end. Each tick of the clock becomes a learning signal, so a long option that runs for fifty steps gives you fifty updates instead of one.

The trick rests on a consistency idea. Many options may agree on the same primitive action in a given state, and a step taken while one option is active is also evidence about every other option that would have taken that same step. So you can update all of them at once, off-policy, from the same transition. This makes the data far richer and lets you even learn about options the agent is not currently executing.

The practical effect is dramatically faster credit assignment within the hierarchy and much better sample efficiency, because no experience is wasted waiting for an option to terminate. It is the option-level analogue of the same instinct that makes temporal-difference learning beat Monte Carlo: learn from every step, not just from finished episodes.