belief state

When you cannot see exactly where you are, the next best thing is a well-calibrated hunch. A belief state is that hunch made rigorous: instead of a single state, the agent holds a probability distribution over all the states it might currently be in, given everything it has observed so far. A robot might be ninety percent sure it is in the kitchen and ten percent sure it is in the hall.

The key fact is that the belief state is itself Markov: it summarises the entire past, so the agent can plan and choose actions from the belief alone, turning a hard POMDP back into an MDP over beliefs. Each new observation updates the belief by Bayes' rule. The catch is that the space of beliefs is continuous and high-dimensional, so exact belief planning is usually intractable and is approximated, often by a recurrent network that learns its own internal summary of history.