JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Attention That Fits in Memory

The KV cache is the real bottleneck at serving time. Multi-query, grouped-query, and sliding-window attention are the architectural answers.

The cache nobody warned you about

When a model generates text, it produces one token at a time, and for each new token it must attend back over every previous token. Recomputing all of that each step would be hopeless, so models keep a KV cache: the keys and values for every past token, stored in memory and reused. This makes generation fast — but the cache grows with every token, and for long conversations it, not the model weights, becomes the thing that won't fit on the GPU.

M_{\text{KV}} = 2 \cdot L \cdot H \cdot d_h \cdot S \cdot B

KV-cache memory grows with every factor at once — layers, heads, head-dimension, sequence length and batch — which is why cutting the head count is the prize.

Multi-query: one set of keys for all

Standard multi-head attention gives every head its own queries, keys, and values — say 32 heads, 32 separate key/value sets to cache. Multi-query attention (MQA) keeps the 32 query heads but makes them all share a single key head and a single value head. The cache shrinks by a factor of 32 overnight, and generation speeds up because there is far less to read from memory each step.

Standard multi-head attention gives every head its own keys and values — 32 heads means 32 K/V sets to cache, exactly the cost multi-query attention attacks.

Diagram of multi-head attention: parallel heads each producing outputs that are concatenated and mixed.

The catch: collapsing all keys and values into one head throws away representational room, and pure MQA models lose a little quality and can become unstable to train. It was the right idea pointing slightly too far.

Grouped-query: the sweet spot

Grouped-query attention (GQA) is the compromise that everyone now ships. Instead of 32 key/value heads (full multi-head) or 1 (MQA), use a handful — say 8 — and let each key/value head be shared by a group of 4 query heads. You recover almost all of the quality of full attention while cutting the cache by 4×. It is the default in Llama, Mistral, Qwen, and most current open models because it sits squarely on the quality-versus-memory sweet spot.

Grouped-query attention keeps many query heads but shares each key/value set across a group — the soft query–key–value lookup that GQA economizes.

Diagram of attention as a query matched against keys via softmax to weight values.

# config knobs you will actually see
num_attention_heads: 32   # query heads
num_key_value_heads: 8    # GQA: 32/8 = 4 query heads share each KV head
# num_key_value_heads == 32 -> plain multi-head
# num_key_value_heads == 1  -> multi-query
One number — the count of key/value heads — slides you between full attention, GQA, and MQA.

Sliding windows: don't look at everything

GQA shrinks the cache per token, but the cache still grows with sequence length. Sliding-window attention attacks length directly: instead of letting each token attend back over the entire history, restrict it to the most recent W tokens — a fixed window of, say, 4096. The cache stops growing once you pass the window, because old keys can be dropped.

Click a word to see what it attends to — sliding-window attention simply restricts that reach to a fixed band of recent tokens.

Interactive self-attention: selecting a token highlights the other tokens it attends to.

Doesn't that lose long-range information? Less than you'd think. Stack many windowed layers and the effective reach compounds: a token can influence one 4096 tokens away through a chain of overlapping windows, the same way a deep convolutional network sees a wide image through small filters. Many models interleave windowed and full-attention layers to get cheap locality and occasional global reach — a first taste of the long-context architectures in the final guide.