Offline RL

advantage-weighted regression (AWR)

Advantage-weighted regression is behavior cloning with taste. Instead of imitating every logged action equally, it imitates them in proportion to how good they turned out — copying high-advantage actions strongly and down-weighting poor ones. You learn to mimic the better half of your own data.

It is a weighted maximum-likelihood update: the policy maximizes the log-likelihood of dataset actions, each weighted by the exponential of its estimated advantage. This emerges as the closed-form solution to a KL-constrained policy-improvement step, which is exactly why it naturally stays close to the behavior policy.

This simple, stable recipe underlies many offline and offline-to-online methods, including AWAC and the policy-extraction step of IQL. Its ceiling is, once again, the data: it reweights existing actions but never proposes a genuinely new one.

\pi^*(a\mid s)\ \propto\ \pi_\beta(a\mid s)\,\exp\!\left(\tfrac{1}{\lambda}A(s,a)\right)

Imitate the data, but weight each action by the exponential of its advantage.

Also called

AWRadvantage-weighted actor-criticAWAC