Expected SARSA

SARSA forms its target by sampling one next action and using its value; Q-learning takes the single best next action. Expected SARSA splits the difference: it uses the expected value of the next state under the policy, averaging the values of all possible next actions weighted by how likely the policy is to pick each. It replaces a noisy single sample with a smooth expectation.

Removing that sampling step removes a source of variance, so Expected SARSA tends to learn more stably and can take larger step sizes than ordinary SARSA, often at modest extra computation. It is also flexible: average under the same policy you are following and it is on-policy; average under a greedy target policy and it becomes exactly Q-learning, which is why it is sometimes seen as the unifying view of one-step TD control.