return-conditioned policies

Normally a policy maps a state to an action. A return-conditioned policy maps a state and a target return to an action: given that I intend to score 200, what should I do here? At test time you simply request a high return, and the policy acts as if it were the kind of agent that earns it.

Training is supervised. For each logged transition you compute the return-to-go that actually followed, then learn to predict the taken action conditioned on the state and that realized return. This reframes RL as conditional imitation — sometimes called reinforcement learning via supervised learning, or upside-down RL — and needs no value function.

The appeal is simplicity and stability; the weakness is that requesting an out-of-distribution return is itself a form of distributional shift, and these methods inherit imitation's inability to stitch. They work best when the data already contains trajectories near the performance you ask for.