grouped-query attention (GQA)

In standard multi-head attention every query head has its own key and value head, so the KV cache you must store during generation grows with the number of heads. GQA strikes a middle ground: it splits the query heads into a few groups and lets all heads within a group share one key/value head. With 32 query heads and 8 groups you store only 8 sets of keys and values instead of 32, a fourfold smaller cache, while each token still attends with its full set of distinct query projections.

Formally, with H query heads and G groups where G divides H, GQA keeps H query projections but only G key/value projections; query head i uses KV head floor(i divided by H/G). Setting G = H recovers full multi-head attention and G = 1 recovers multi-query attention, so GQA interpolates between the two. Models are often uptrained from a multi-head checkpoint by mean-pooling the original KV heads into groups, recovering near-baseline quality with only a short amount of extra training.

The payoff is memory bandwidth at decode time: autoregressive generation is dominated by reading the KV cache, so shrinking it delivers most of multi-query attention's speedup while losing far less quality. GQA is now standard in LLaMA-2 70B, Mistral, and many production-scale models.