JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Evaluation, Fine-Tuning, and a Practitioner's Playbook

You trained a policy offline — now how do you know it's any good, and how do you safely let it touch the world?

The evaluation trap

Online, you measure a policy by running it. Offline, you can't — running it is the very thing you were trying to avoid, and doing so to compare every checkpoint defeats the purpose. So the hardest, most under-appreciated part of offline RL is often not training the policy but trusting it before deployment.

Off-policy evaluation (OPE)

Off-policy evaluation (OPE) is the science of estimating how good a new policy would be, using only the logged data. The main tools are importance-sampling estimators (reweight logged returns by how likely your policy was to take those actions), fitted value estimators, and doubly-robust combinations that hedge between them. All of them grow shaky exactly where your policy diverges most from the behavior policy — the same distributional shift, now haunting evaluation.

\hat V(\pi)=\frac{1}{n}\sum_{i=1}^{n}\left(\prod_{t=0}^{T}\frac{\pi(a_t^i\mid s_t^i)}{\mu(a_t^i\mid s_t^i)}\right)\sum_{t=0}^{T}\gamma^{t}\,r_t^i

Off-policy evaluation reweights logged returns by the ratio of new-policy to behavior-policy action probabilities — the importance-sampling estimator.

  1. Hold out part of the dataset purely for OPE — never train on it.
  2. Report a range of estimators; if they disagree wildly, distrust all of them.
  3. Sanity-check against the behavior-cloning baseline — beating it offline is the minimum bar.

Offline-to-online fine-tuning

Often the smartest plan is to use offline data as a warm start, then let the agent improve with a small, careful budget of real interaction. This is offline-to-online fine-tuning. The catch is a nasty transition shock: a heavily pessimistic offline policy can collapse when it suddenly faces fresh data, because its over-conservative value estimates were tuned for a world with no new samples. Methods like IQL transition more gracefully than aggressively pessimistic ones like vanilla CQL.

Online fine-tuning re-enters the agent–environment loop, spending a careful budget of real interaction to improve the warm-started policy.

Diagram of the reinforcement-learning loop: the agent takes an action, the environment returns a new state and reward.

A field-tested checklist

  1. Know your data. How was it collected, how broad is the action coverage, how good was the behavior policy? This decides everything downstream.
  2. Always run behavior cloning first. If your fancy method can't beat plain imitation, the fancy method is broken, not impressive.
  3. Match algorithm to data shape: narrow/expert → imitation or AWR; broad/mixed → IQL or CQL; trajectories already strong → a sequence model.
  4. Tune pessimism as your main knob. Too little → distributional shift and divergence; too much → a timid clone of the data.
  5. Never deploy on training-loss alone. Use OPE, held-out estimates, and the smallest safe slice of real interaction you can afford.

Where it's heading

Offline RL is quietly becoming the default way to bring RL out of simulators and into messy reality — and it is the backbone of how large models are tuned from logged human feedback, a close cousin of off-policy learning. Master the three ideas in this track — distributional shift, pessimism, and honest evaluation — and you hold the keys to most of the field. Everything newer is a sharper way to spend the same hard-won data.

Tuning large models from logged human feedback is a close cousin of offline RL: preferences train a reward model that shapes the policy.

Diagram of RLHF: human preferences feed a reward model, which then tunes the policy.