JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Actor-Critic: Learning a Critic to Guide the Policy

Stop waiting for whole episodes. Train a value estimator alongside the policy so it can learn step by step — and scale it across many parallel environments with A2C and A3C.

The actor and the critic

The actor-critic architecture runs two learners side by side. The actor is the policy — it chooses actions and is improved by policy gradient. The critic estimates a value function and supplies the baseline (or full advantage) the actor needs. Each helps the other: the critic gives the actor a low-variance learning signal, and the actor's experience trains the critic.

Crucially, the critic lets us replace the slow Monte-Carlo return with a temporal-difference estimate. The one-step TD error δ = r + γ·V(s') − V(s) is itself an unbiased-in-expectation sample of the advantage. Now the actor can learn from a single step instead of waiting for the episode to end.

\delta = r + \gamma\, V(s') - V(s)

The one-step TD error delta: the critic's estimate of how much better an outcome was than expected, replacing the slow Monte-Carlo return.

Advantage Actor-Critic (A2C)

Advantage actor-critic is the clean, standard recipe: estimate the advantage (often with GAE over a short rollout), update the actor by advantage-weighted policy gradient, and update the critic by regression toward its TD or return targets. Add the entropy bonus from the previous guide and you have the workhorse that countless deep RL agents are built on.

\nabla_{\theta} J(\theta) = \mathbb{E}\!\left[\nabla_{\theta} \log \pi_{\theta}(a \mid s)\, \hat{A}(s,a)\right]

A2C updates the actor by advantage-weighted policy gradient, pushing up actions that beat the critic's baseline.

Going parallel: A3C and A2C

A single agent's consecutive samples are highly correlated, which hurts gradient estimates. A3C (Asynchronous Advantage Actor-Critic) solved this by running many actor-learners in parallel, each on its own environment copy, each pushing gradients to a shared set of parameters asynchronously. The diversity of simultaneous experiences decorrelates updates without a replay buffer.

A3C and A2C run many copies of this agent–environment loop in parallel, decorrelating the samples that train the shared actor-critic.

Diagram of the reinforcement-learning loop, here run as many parallel copies feeding one shared learner.

It turned out the asynchrony was not the secret ingredient — the parallel data was. A2C (the synchronous variant) simply waits for all workers to finish a rollout, averages their gradients, and applies one clean update. It is simpler, makes better use of GPUs, and usually matches or beats A3C. A2C is the default you should reach for first.

When the critic is 'compatible'

There is a beautiful theoretical caveat. A learned critic introduces bias, so does the gradient still point the right way? The theory of compatible function approximation gives the condition: if the critic's features are exactly the policy's score features ∇θ log π, and it is fit by least squares, then using that critic in place of the true advantage yields the exact policy gradient — no bias at all.

The payoff of a compatible critic: the estimated gradient still steps toward the optimum rather than wandering astray.

Gradient descent stepping down a curved loss surface toward its minimum.

In practice we rarely enforce the condition exactly — deep critics are far more expressive than the compatible family. But it explains why actor-critic is principled rather than a lucky hack, and it foreshadows the natural-gradient methods in the next guide, where compatibility connects directly to the geometry of policy space.