Why decoding is memory-bound
When an LLM generates token by token, it recomputes attention over every previous token at each step. To avoid redoing that work it stores each layer's keys and values in the KV cache. The cache grows with sequence length and never shrinks during a generation, so for long prompts the bottleneck is not arithmetic but moving the cache through memory. This is the central fact of LLM serving: throughput is governed by bytes, not FLOPs.
Standard multi-head attention keeps a separate key and value vector for every head at every position. With dozens of heads and dozens of layers, the cache for a single long conversation can dwarf the model's own weights in memory traffic. Everything in this guide is a different answer to one question: how do we make that cache cheaper to store or to move?
The KV-cache footprint grows linearly with sequence length, layers, and heads — the factor of 2 is keys plus values.
MQA and GQA: share the keys and values
Multi-query attention (MQA) takes the radical step of giving all query heads a single shared key/value head. The KV cache shrinks by the number of heads — often 8x to 64x — and decoding speeds up dramatically. The cost is some quality loss and occasional training instability, because all that representational diversity now has to live in the queries alone.
Diagram of multi-head attention: parallel heads are computed, concatenated, and mixed.
Grouped-query attention (GQA) is the now-standard compromise: partition the query heads into a handful of groups and give each group its own shared key/value head. With, say, 8 groups you recover almost all of multi-head quality while keeping most of MQA's cache savings. GQA is why modern open models can serve long contexts without exploding memory — it is the default in the Llama 2/3 and Mistral lineages.
# heads = 32 query heads; choose how many KV heads to keep # MHA: kv_heads = 32 (full cache, best quality) # GQA: kv_heads = 8 (4x smaller cache, ~MHA quality) # MQA: kv_heads = 1 (32x smaller cache, some quality cost) kv_cache_bytes = kv_heads * head_dim * 2 * n_layers * seq_len * dtype_size
FlashAttention-2: stop touching slow memory
MQA/GQA shrink the cache; FlashAttention-2 attacks how attention itself moves data. Naive attention materializes the full N-by-N score matrix in slow GPU memory, reads it back to apply softmax, and writes it again — three round trips over a matrix that grows quadratically. Vol I's FlashAttention fused these steps with tiling and an online softmax so the score matrix never fully leaves fast on-chip memory. The result is exact attention with far less data movement.
FlashAttention-2 computes this exact softmax attention tile by tile, never materializing the full N×N score matrix in slow memory.
The *-2* version is an engineering re-think of the same idea: it reduces non-matmul work, parallelizes across the sequence dimension as well as batch and heads, and partitions work between GPU warps to cut shared-memory traffic. In practice it roughly doubles throughput over the original and pushes hardware utilization toward its theoretical peak. Crucially it changes no math — outputs are identical, so it is a free win you should almost always enable.
Sliding-window attention: cap the reach
The cheapest token is the one you never attend to. Sliding-window attention lets each token attend only to the last W tokens — a fixed local window — turning attention's cost from quadratic to linear in sequence length and capping the active KV cache at W. Information still travels far because windows stack across layers: after L layers a token can be influenced by roughly L-times-W tokens back, the way a deep CNN grows its receptive field.