option-critic architecture

For years options had to be designed by hand or discovered by separate, often clunky procedures. The option-critic architecture closed that gap by learning everything an option contains directly from the task reward, end to end, with gradients — no subgoals, no auxiliary rewards, just one objective. You specify how many options you want, and the algorithm figures out what each should do and when each should stop.

It works by extending the policy-gradient idea to the inside of options. Two theorems give the gradient of the expected return with respect to the parameters of each option's internal policy and with respect to its termination function. A critic estimates option values to supply the needed advantages, while the actor pieces — the intra-option policies and the termination probabilities — are nudged up or down accordingly. Training the whole stack at once means the options shape themselves to whatever decomposition actually helps maximise reward.

The strength is conceptual cleanliness and the ability to learn temporal abstraction without hand-holding. The well-documented weakness is option collapse: with nothing but task reward to guide them, the learned options often degenerate — one option dominates, or terminations fire almost every step, recovering a flat policy. Taming that, usually with extra regularisers or deliberation costs, is the main practical concern when using it.