Soft Actor-Critic: Rewarding Curiosity

A new goal: succeed and stay random

DDPG and TD3 maximize expected reward, full stop. Soft Actor-Critic (SAC) changes the objective itself. It maximizes reward plus the entropy of the policy — a measure of how random the action choice is. This is the maximum-entropy RL objective: be as successful as you can while staying as unpredictable as the task allows.

J(\pi)=\sum_{t}\mathbb{E}_{(s_t,a_t)\sim\rho_\pi}\!\left[\,r(s_t,a_t)+\alpha\,\mathcal{H}\big(\pi(\cdot\mid s_t)\big)\right]

SAC's maximum-entropy objective: maximize reward plus the policy's entropy, weighted by the temperature alpha.

Why reward randomness? Because a policy pushed to stay broad keeps exploring on its own, resists collapsing onto a single brittle behavior, and often finds multiple ways to solve a task. This is entropy regularization promoted from a side-bonus to a first-class citizen of the objective.

A stochastic, squashed actor

To have entropy at all, SAC's actor must be a stochastic policy — it outputs a distribution over actions, not a single action. The standard choice is a squashed Gaussian policy: the network outputs a mean and a standard deviation, you sample from that Gaussian, then pass the sample through a tanh to fold it into the action bounds.

The tanh squashing keeps actions legal but bends the probabilities, so SAC applies a small correction to the entropy calculation to account for it. It is a known piece of bookkeeping — worth flagging because forgetting it is a classic implementation bug.

The reparameterization gradient

Now there is a puzzle: how do you take a gradient through a random sample? If the actor samples its action, the sampling step seems to block the gradient. The fix is the reparameterization gradient: write the action as mean + std * epsilon, where epsilon is fixed noise drawn from a standard Gaussian. The randomness now lives in epsilon, which has no parameters, so gradients flow smoothly through the mean and std the network produces.

SAC also borrows TD3's clipped double-Q: it trains two critics and uses their minimum in the target, for the same anti-overestimation reason. The pieces from the previous guides keep paying off.

Auto-tuning the temperature

The one hyperparameter people hated tuning was the temperature alpha — too high and the agent flails randomly, too low and it collapses early. The fix is automatic temperature tuning: instead of fixing alpha, you fix a target entropy (a desired level of randomness, often set from the number of action dimensions) and let gradient descent adjust alpha to hit it. The agent stays exactly as exploratory as you asked, automatically, throughout training.

J(\alpha)=\mathbb{E}_{a_t\sim\pi}\!\left[-\alpha\big(\log\pi(a_t\mid s_t)+\bar{\mathcal{H}}\big)\right]

Auto-tuning the temperature: alpha is adjusted so the policy's entropy stays near a target level bar-H.

# SAC actor update (reparameterized)
eps = randn_like(mean)
a = tanh(mean + std * eps)             # reparameterized, bounded sample
logp = gaussian_logprob(mean, std, ...) - tanh_correction(a)
q = min(critic1(s, a), critic2(s, a)) # clipped double-Q
actor_loss = (alpha * logp - q).mean()        # maximize  q - alpha*logp

# temperature follows a target entropy
alpha_loss = -(log_alpha * (logp + target_entropy).detach()).mean()

SAC's actor maximizes value minus alpha-weighted log-probability; alpha self-adjusts to a target entropy.

Why SAC became the default

SAC is off-policy (so it is sample-efficient, reusing a replay buffer), stable across seeds, and largely free of fiddly tuning thanks to auto-temperature. On the MuJoCo suite it matched or beat TD3 while being more robust. For most new continuous-control problems today, SAC is the sensible first thing to try.