on-policy vs off-policy learning

Two policies are quietly at work in any learning agent: the behavior policy that actually chooses actions and generates the data, and the target policy whose value the agent is trying to learn. On-policy learning means these are the same — you evaluate and improve the very policy you are following, as in SARSA. Off-policy learning means they differ — you learn about one policy while behaving according to another, as in Q-learning, which learns the optimal policy while exploring.

The distinction shapes what is possible. Off-policy learning is more powerful and flexible: it can learn the optimal policy from exploratory or even old, logged data, and reuse experience generated by other agents or earlier versions of itself. The catch is that data drawn from one policy gives a distorted picture of another, so off-policy methods usually need correction — most often importance sampling — and tend to be less stable than their on-policy cousins.