multi-query attention (MQA)

Multi-query attention is the extreme version of key/value sharing: all query heads attend against a single shared key head and a single shared value head. The queries stay diverse, each head asking a different question, but they all look up the same key/value table. This collapses the KV cache from one-per-head down to one in total, which is the single largest memory cost during long-sequence decoding.

With H heads and head dimension d_h, standard attention caches 2 times H times d_h numbers per token; MQA caches only 2 times d_h, an H-fold reduction. The attention computation is otherwise unchanged: each query head computes softmax of (q_i times K transpose divided by square root of d_h) times V using the shared K and V. Because arithmetic intensity rises, with more floating-point operations per byte loaded, MQA shifts decoding from memory-bound toward compute-bound, the regime where accelerators run efficiently.

The cost is representational: a single KV head can become a quality bottleneck and training can be less stable, which is why grouped-query attention, with a few KV heads rather than one, has largely superseded pure MQA. MQA remains valuable when memory or latency is the dominant constraint, for example very long contexts or tight serving budgets.