optimal value function

Every policy has its own value function, so there are as many value functions as there are ways to behave. The optimal value function is the very best of them: for each state, it records the highest expected return achievable by any policy at all. Think of it as the ceiling — the score you could reach from each state if you played perfectly from then on.

It comes in two flavours, V-star for states and Q-star for state-action pairs, where the star marks the best possible. A deep fact makes this useful: a single value function is simultaneously optimal for every state, so you never have to trade off doing well from one state against doing well from another. Knowing Q-star essentially solves the task, because acting greedily with respect to it is optimal.