importance sampling (in RL)

If your data was generated by one policy but you want to evaluate another, the samples are skewed: actions the target policy loves may be rare in the data, and vice versa. Importance sampling fixes this by reweighting. Each observed return is multiplied by the importance ratio — the probability the target policy would have taken that trajectory divided by the probability the behavior policy actually did — so that, on average, the reweighted returns estimate the target policy's value.

It is the workhorse correction behind off-policy Monte Carlo and many off-policy algorithms, but it has a sharp edge: variance. The ratio is a product of per-step factors, so over a long trajectory it can explode toward zero or huge values, making estimates wildly noisy. Practitioners temper this with weighted (rather than ordinary) importance sampling, per-decision ratios, truncation, or clipping to keep the variance manageable.