soft actor-critic (SAC)

SAC is the other modern workhorse of continuous control, and its core idea is a change of goal: don't just maximise reward, maximise reward plus the randomness of the policy. Concretely it rewards the agent for keeping its actions as varied as it can while still doing well. That sounds like a quirk, but it produces an agent that explores thoroughly, hedges against a single brittle strategy, and tends to be far more robust to hyperparameters than DDPG or TD3 — which is much of why people reach for it.

Mechanically SAC is an off-policy actor-critic with a stochastic, usually squashed-Gaussian, policy. It inherits clipped double-Q critics from the TD3 lineage to fight overestimation, and it trains the actor with the reparameterisation trick so the entropy-augmented objective has low-variance gradients. The value targets carry an extra entropy bonus, and a temperature parameter sets how much that bonus counts. Each piece — the entropy term, the twin critics, the reparameterised sampling — is a separate term elsewhere in this field, and SAC is where they assemble.

In practice SAC is sample-efficient and forgiving, often the first thing tried on a new robotics or control task, and its automatic temperature tuning removes one of the last fiddly knobs. Its philosophical opposite is the deterministic TD3: SAC explores by being intrinsically random, TD3 by injecting external noise into a deterministic policy.