Mamba and state-space models

Instead of comparing every token to every other one, a state-space model reads the sequence like a person taking notes: it keeps a small running memory, a fixed-size hidden state, and updates it token by token, deciding what to keep and what to forget as it goes. Mamba is the LLM-grade version of this idea, and because the memory size is fixed, processing each new token costs the same no matter how long the sequence already is.

The mathematics is a linear recurrence borrowed from control theory, but Mamba makes its update gates depend on the current input, this selectivity is what lets it choose what to remember, and a clever parallel scan lets it train as fast as a transformer despite being recurrent at heart. The payoff is constant memory per token and linear-time generation, ideal for very long sequences. The tradeoff is that a single compressed state can struggle with tasks needing exact recall of arbitrary earlier tokens, which is why pure Mamba is often blended with a few attention layers.