maximum-entropy reinforcement learning
Maximum-entropy reinforcement learning reframes the objective so the agent is rewarded both for return and for acting as unpredictably as it can while still doing well. The payoff is behavior that is not one brittle optimum but a family of near-optimal strategies, which tends to explore better and degrade gracefully when conditions change.
Formally the objective adds a temperature times the policy's entropy at each step to the reward. The optimal policy is then a Boltzmann distribution over soft Q-values: it picks actions in proportion to the exponential of the soft Q divided by the temperature. This induces soft value functions and a soft Bellman equation, and it connects RL to probabilistic inference through the control-as-inference view and to energy-based policies. Soft actor-critic is its practical off-policy instantiation.
The temperature trades reward against entropy: as it shrinks toward zero, the soft policy sharpens to a greedy one and standard RL is recovered.
The optimal max-ent policy is a Boltzmann distribution over soft Q-values, sharpened by the temperature alpha.
Max-ent RL is not just an exploration heuristic; it is a different, well-defined optimal objective whose solution is a stochastic Boltzmann policy.