off-policy evaluation (OPE)

Before deploying a new policy, you want to know how good it is. Off-policy evaluation answers that from logged data alone — estimating the value of a target policy you have never run, using only experience collected by a different behavior policy. It is the will-this-work question of offline RL.

Three families dominate. Importance sampling reweights logged returns by the ratio of target-to-behavior action probabilities, unbiased but high-variance. Direct methods fit a model or value function and evaluate the policy inside it, low-variance but biased. Doubly robust estimators combine the two, staying accurate if either ingredient is good.

OPE is notoriously hard: importance weights explode over long horizons, and model-based estimates inherit model error. Yet it is indispensable — it is how offline RL is validated, how hyperparameters are chosen without online trials, and how risky deployments are screened before going live.