The quadratic wall
Self-attention is wonderful and expensive. Every token compares itself to every other token, so the cost grows with the square of the sequence length — double the context and you quadruple the work, and the KV cache grows linearly without bound. GQA and sliding windows from guide 3 soften this, but they don't change the underlying law. To truly process book-length or repository-length input, some architects ask a sharper question: what if a layer didn't compare every pair at all?
The attention scores QK^T form an n-by-n matrix — the term whose cost grows with the square of the sequence length.
Linear attention: trade exactness for scale
Linear attention rewrites the math so the cost grows linearly with length instead of quadratically. The trick is to drop the softmax that couples every query to every key, and reorder the multiplication so the model maintains a fixed-size running summary of the past rather than an ever-growing list of keys. The payoff is huge — constant memory per step, no exploding cache. The price is that the summary is lossy: it cannot recall an arbitrary token from a million ago as crisply as true attention can.
Diagram of query, key and value vectors combined through softmax attention weights.
Mamba: a learned memory that scans
Mamba and state-space models (SSMs) take that idea and make it principled. An SSM carries a hidden state — a compressed memory of everything seen so far — and at each token it updates the state and emits an output, exactly like reading left to right. The decisive upgrade in Mamba is that the update is input-dependent: the model learns, per token, what to write into memory and what to forget, so it can hold onto a name mentioned pages ago while discarding filler.
Diagram of a recurrent network unrolled into a chain of repeated time steps.
Hybrids: keep a little attention
Pure SSMs are fast but their lossy memory struggles with tasks that need exact recall — copying a long ID, doing precise in-context lookups. Pure attention recalls perfectly but pays the quadratic tax. So the most promising designs are hybrid attention–SSM models: stack mostly cheap SSM or linear-attention layers, and sprinkle in a few full-attention layers where exact recall matters. You get near-linear cost with an attention safety net — the best of both ledgers.
This is the same instinct as the windowed-plus-global interleaving from guide 3, taken further: cheap-but-local layers for the bulk of the work, expensive-but-global layers as occasional anchors. Whether the future is dense attention, MoE, SSM, or some hybrid, the organizing question of modern architecture is constant — *spend compute and memory only where they buy the most capability.*