average-reward criterion

Discounting bakes in a preference for reward sooner rather than later, which is natural for episodic tasks but odd for systems meant to run forever — a server, a power grid, a trading loop. The average-reward criterion drops the discount and instead optimizes the long-run reward per step: if you ran the policy indefinitely, what is the steady rate at which it accumulates reward? For continuing, non-terminating problems this is often the more honest objective.

The theory replaces the discounted Bellman equation with an average-reward version centered on a gain (the optimal per-step reward, a scalar) and a bias or differential value function (the transient advantage of starting in one state over another). The optimality equation reads as reward plus expected next differential value equals the gain plus current differential value. There is no γ to provide contraction, so convergence arguments are subtler and usually require assumptions about the chain's structure, such as the MDP being communicating or ergodic.

Average-reward RL is theoretically clean but practically less common, because the missing discount removes the easy contraction that makes discounted algorithms stable; it matters most precisely when discounting would distort a genuinely long-horizon objective.