maximum-entropy RL objective

Standard RL wants one thing: the policy that collects the most reward. Maximum-entropy RL wants two things at once — collect reward, and stay as random as you can while doing so. Entropy is just a number that measures how spread-out a distribution is: a policy that always picks the same action has zero entropy, one that picks evenly among options has high entropy. The objective adds that entropy, scaled by a temperature, to the reward, so the agent is paid a little extra for keeping its options open.

Why deliberately add randomness? Three reasons that matter in control. It explores better, because it never prematurely commits to a single action and keeps probing alternatives. It is more robust, because a policy that succeeds while staying varied has not balanced on a knife-edge that small disturbances knock over. And it captures multiple good solutions instead of arbitrarily collapsing to one. The temperature dials the trade-off: high temperature favours randomness, low temperature recovers ordinary reward-maximising RL.

This objective is the theoretical heart of soft actor-critic, and the word 'soft' refers to it: the value functions become 'soft' because their Bellman backups include the entropy bonus. It connects RL to probabilistic inference, where acting well is recast as sampling from a distribution over good behaviours rather than picking a single best one.