state space

The state space is the catalogue of every situation the world can be in. For a chess agent it is every legal board; for a thermostat it might be the current temperature; for a self-driving car it is a vast bundle of positions, speeds and camera pixels. Whatever the agent might need to know to act well should live inside the state.

States can be a handful of discrete cells, a list of continuous numbers, or raw images. The size and shape of this space largely decides which algorithm is feasible: a tiny finite space can be solved with a table, while a high-dimensional or continuous space forces you to approximate with features or neural networks. Designing a compact state that still satisfies the Markov property is often the hardest part of setting up an RL problem.