JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Batch and Fitted Methods: The Bridge to Deep RL

Stop updating online — collect a dataset and fit the value function in batches. This single shift turns supervised regression into RL and leads straight to DQN.

From online drips to batch fits

Guides 3 and 4 updated w a tiny bit per transition. Batch (or 'fitted') methods do the opposite: gather a whole dataset of transitions first, build a regression target for every one, then fit a fresh approximator to all of them at once with your favourite supervised learner. Repeat: re-target using the new approximator, re-fit, repeat. Each round is an ordinary regression problem, which is why fitted methods are the cleanest bridge from supervised learning to RL.

The win is stability and data efficiency: a full batch averages out the noise that jitters an online update, and you can reuse the same data many times. The cost is that you have re-introduced bootstrapping (targets use the previous round's estimates) — so the deadly triad is lurking, and the fixes from guide 4 still apply.

Fitted value iteration

Fitted value iteration is value iteration with a regressor in the loop. Each round: for every sampled state, compute a one-step Bellman target using the current approximator for the next state's value; then fit the approximator to those targets. In the language of guide 3 this is a Bellman backup followed by a projection onto value space — the regression is the projection, snapping the backed-up values onto the nearest representable function.

  1. Collect a batch of transitions (state, action, reward, next state).
  2. For each transition, form the target: reward + γ × current estimate of the next state's value.
  3. Fit a fresh approximator (regression) to map states to those targets.
  4. Replace the old approximator with the new one; recompute targets; repeat until stable.
\theta_{k+1} = \arg\min_{\theta} \sum_i \left( r_i + \gamma\,\hat{V}_{\theta_k}(s_i') - \hat{V}_{\theta}(s_i) \right)^2

Each round of fitted value iteration forms Bellman targets, then fits a fresh regressor to them.

Fitted Q iteration: control without a model

Fitted Q iteration (FQI) is the action-value, model-free version and the direct ancestor of deep RL. It fits an approximate action-value function: the target for each transition is reward + γ × the max over next-state actions of the current Q-estimate. Because it learns Q rather than V, it needs no model of the dynamics to pick actions — the greedy action falls straight out of the learned Q. Run it with trees or linear features and it is a complete, practical offline RL algorithm.

Fitted Q iteration fits an action-value function with any regressor — swap in a deep network like this one and you arrive at DQN.

A multilayer neural network mapping an input through hidden layers to an output, serving as the regressor.

The linear closed form: LSTD and LSPI

When the approximator is linear, the inner regression has a closed-form solution — that is exactly least-squares temporal difference from guide 3, used here as the evaluation step inside a fitted loop. Wrap LSTD's policy-evaluation in policy improvement and you get Least-Squares Policy Iteration (LSPI): evaluate the current policy in closed form, act greedily with respect to the result, repeat. It squeezes the most out of a fixed batch and needs no step-size tuning.

With a linear approximator the inner regression has a closed form — ordinary least squares, the heart of LSTD and LSPI.

A straight regression line fitted through scattered points by least squares.

DQN: fitted Q iteration at scale

Swap FQI's regressor for a deep neural network and you have essentially the deep Q-network (DQN). The two famous tricks that made it stable are pure deadly-triad engineering. A replay buffer with experience replay stores past transitions and samples them in mini-batches — recovering the batch averaging and data reuse of fitted methods from an online stream. A target network, a slowly-updated copy used only to compute the bootstrapped targets, freezes the regression target within each round, mimicking the 'fit to fixed targets' structure of fitted Q iteration and damping the divergence feedback loop.

\mathcal{L}(\theta) = \mathbb{E}\!\left[\left( r + \gamma \max_{a'} \hat{Q}_{\theta^-}(s', a') - \hat{Q}_{\theta}(s, a) \right)^2\right]

DQN is fitted Q iteration at scale: the same max-target regression, stabilized by a frozen target network with weights theta-minus.

Seen this way, deep RL is not a break from classical function approximation — it is fitted Q iteration plus two stabilizers that quietly tame the same triad you met in guide 4. Everything in this track — features, the value error, the projected fixed point, the triad, overestimation — is still operating under the hood of a modern agent. That is the real reward of studying approximation: the failures and fixes generalize all the way up.