on-policy vs off-policy
/ on-POL-uh-see vurs off-POL-uh-see /
This is a distinction about whose experience an algorithm learns from. An on-policy method learns only from actions taken by the very strategy it is currently improving — it must learn from its own latest behavior. An off-policy method can learn from actions taken by some other strategy: an older version of itself, a clumsy exploring version, even a human demonstrator. On-policy is like only learning to cook from your own dinners; off-policy is like also learning from watching other cooks, including beginners.
The reason it matters comes down to data efficiency versus stability. Off-policy methods (like Q-learning and DQN) can reuse a huge pile of stored past experience over and over, even data generated long ago — far more sample-efficient. On-policy methods (like SARSA and standard PPO) must keep collecting fresh data from the current policy and largely throw it away after each update, which is wasteful but tends to be more stable and easier to reason about.
The classic illustration is SARSA versus Q-learning on the same cliff-edge maze. Q-learning is off-policy: it learns the value of the optimal greedy path even while it's actually stumbling around exploring. SARSA is on-policy: it learns the value of the path it really walks, exploratory missteps included, so it sensibly keeps away from the cliff. Neither label is 'better' — off-policy buys efficiency at the cost of trickier, sometimes unstable learning, while on-policy buys reliability at the cost of thirstier data appetite.
Off-policy in action: DQN keeps a replay buffer of millions of past frames — many played by an earlier, worse version of itself — and trains on random samples of them. An on-policy method like PPO can't do that; after each policy update its old data is stale, so it must go collect fresh experience.
Off-policy reuses old/other data (efficient, trickier); on-policy needs fresh data from itself (wasteful, steadier).
The labels describe a tradeoff, not a quality ranking. Off-policy's ability to recycle data is a real superpower for sample efficiency, but it makes training mathematically harder and prone to instability — which is precisely why off-policy deep RL needs stabilizing tricks that on-policy methods often don't.