actor-critic
/ AK-tur KRIT-ik /
Actor-critic is a clever marriage of the field's two main families. The actor is a policy — it decides what to do. The critic is a value function — it judges how good the situation is. They train together: the actor acts, the critic evaluates, and the actor uses the critic's feedback to improve. It's like a performer with a coach: the coach can't act, but its sharp, immediate judgment makes the performer get better far faster than self-criticism alone.
The point of the partnership is to fix policy gradient's biggest weakness. Plain policy gradient judges each action by the full, noisy outcome of an entire episode — feedback that arrives late and bounces around wildly. The critic supplies a fast, learned estimate instead, so the actor can be told 'that move was better than expected here' right away, without waiting for the episode to end. That trade — accepting a little bias from the critic's estimates in exchange for far less noise — usually makes learning dramatically smoother.
Almost every modern reinforcement-learning workhorse, including PPO, is an actor-critic at heart. The honest caveat is that you now have two learners that must improve in lockstep: if the critic's judgments are wrong, it can steer the actor astray, and if either races ahead of the other, training can wobble or collapse. Two interacting networks are simply harder to keep stable than one — a recurring theme in why deep RL is finicky.
A robot arm reaching for a cup: the actor decides how to move each joint; the critic estimates 'how close to success is this pose?' If a move makes the critic's estimate jump up, the actor is rewarded for it immediately — no need to wait until the cup is grasped to learn that the move helped.
Actor decides; critic judges. The critic's fast feedback tames policy gradient's noisy, end-of-episode signal.
Actor-critic trades variance for bias: the critic's estimates are smoother than raw episode rewards but can be systematically wrong, and a misleading critic will steer the actor in the wrong direction. Keeping the two networks improving in balance — neither racing ahead — is the practical art of getting it to work.