Maximum Entropy and Soft Actor-Critic

Why add entropy to the reward

Standard reinforcement learning maximizes expected reward, and the optimal policy is usually deterministic — it collapses onto a single best action per state. Maximum-entropy RL changes the objective: maximize reward plus the entropy of the policy, weighted by a temperature α. The agent is now rewarded for keeping its action distribution as random as it can while still doing well.

J(\pi)=\sum_{t}\mathbb{E}_{(s_t,a_t)\sim\rho_\pi}\!\left[\,r(s_t,a_t)+\alpha\,\mathcal{H}\big(\pi(\cdot\mid s_t)\big)\right]

The maximum-entropy objective: every step's reward is augmented by α times the policy's own entropy.

This is not a hack — it buys three concrete things. First, exploration becomes part of the objective rather than a bolt-on heuristic, directly engaging the exploration–exploitation tradeoff. Second, the policy learns all the ways to succeed, not just one, which makes it robust when the environment shifts and gives downstream fine-tuning more to work with. Third, the entropy term smooths the optimization landscape, helping avoid premature collapse onto a brittle sub-optimal action.

Soft value functions and the soft Bellman backup

Folding entropy into the return changes the Bellman machinery in one clean way. The soft state value becomes a log-sum-exp (a softmax-with-temperature) over actions instead of a hard max, and the soft Q-update adds the policy's entropy to the bootstrap. As α→0 the log-sum-exp sharpens back into the ordinary max and you recover classical value functions; as α grows the agent acts more randomly.

Q(s,a)=r(s,a)+\gamma\,\mathbb{E}_{s'}\big[V(s')\big],\qquad V(s)=\mathbb{E}_{a\sim\pi}\big[\,Q(s,a)-\alpha\log\pi(a\mid s)\big]

The soft Bellman backup: the bootstrapped value carries an −α·log π entropy bonus at every step.

# soft Bellman backup for the entropy-regularized objective
V_soft(s) = E_{a~pi}[ Q(s,a) - alpha * log pi(a|s) ]   # log-sum-exp over actions
Q_target  = r + gamma * V_soft(s')                      # bootstrap with soft value

Entropy enters the backup as an −α·log π bonus on every step.

The fixed point of this soft backup is a Boltzmann (energy-based) policy: π(a|s) ∝ exp(Q(s,a)/α). The optimal action distribution is proportional to exponentiated value, which is exactly why the optimal policy is stochastic and why the temperature controls how peaked it is.

Soft Actor-Critic

Soft actor-critic (SAC) is the practical algorithm built on this objective for continuous action spaces. It is an off-policy actor-critic: a critic learns the soft Q-function from a replay buffer, and an actor — a squashed Gaussian whose mean and variance are network outputs — is trained to match the Boltzmann policy by minimizing KL to exp(Q/α). The actor is updated through the reparameterization trick, so the policy gradient flows through the sampled action rather than relying on a high-variance score-function estimate.

Sample a mini-batch of transitions from the replay buffer (off-policy, so old data is fine).
Update twin Q-critics toward the soft Bellman target, taking the min of the two to curb overestimation.
Update the actor by reparameterized gradient to minimize KL to the Boltzmann policy (maximize Q + entropy).
Soft-update the target critics with Polyak averaging.

Tuning the temperature automatically

The temperature α is the single most important — and most annoying — hyperparameter: too low and the policy collapses and stops exploring; too high and it never commits. The standard fix turns α into a learned variable by setting a target entropy (often −dim(action) as a rule of thumb) and adjusting α to hit it, via a dual optimization that nudges α up when the policy is too deterministic and down when it is too random.

J(\alpha)=\mathbb{E}_{a_t\sim\pi}\!\left[-\alpha\big(\log\pi(a_t\mid s_t)+\bar{\mathcal{H}}\big)\right]

Auto-tuning α: minimize this loss so the policy's expected entropy tracks a fixed target H̄.

Why SAC is sample-efficient — and where it strains

Because it is off-policy, SAC reuses every transition many times from the replay buffer, making it dramatically more sample-efficient than on-policy PPO — often the difference between thousands and millions of environment steps on the same robot. The maximum-entropy objective also makes it unusually stable and forgiving of hyperparameters.

The strains are honest ones. Off-policy bootstrapping with function approximation can still diverge in hostile cases; the entropy bonus subtly changes the objective, so the policy you converge to is optimal for reward-plus-entropy, not pure reward, which matters if you need a sharply deterministic controller at deployment. And like all replay-based methods, SAC inherits sensitivity to the buffer's composition — a theme we will confront head-on when we reach offline RL.