automatic temperature tuning (SAC)

The maximum-entropy objective has one crucial knob: the temperature α, which sets how much the agent values randomness against reward. Get it wrong and SAC misbehaves — too high and the policy stays needlessly random and never sharpens; too low and it collapses early, loses exploration, and gets stuck. Worse, the right value drifts during training and differs wildly across tasks and reward scales, so hand-tuning it is a thankless chore. Automatic temperature tuning removes that chore.

The idea is to stop picking how much entropy to want and instead pick a target entropy — a desired level of randomness — and let α adjust itself to hit it. This is framed as a constrained optimisation: maximise reward subject to the policy's average entropy staying at least at the target. Solving it gives a simple update that raises α when the policy's entropy falls below target and lowers α when it climbs above, so the temperature self-regulates the entire run.

This was the single change between the first SAC paper and its more practical successor, and it mattered enormously — it turned SAC from a strong-but-finicky method into one that works out of the box across very different tasks. A sensible default target is to set it proportional to the action dimension (often −dim(A)), which removes one of the last hyperparameters a practitioner has to sweat over.