multi-query attention (MQA)

Multi-query attention takes the cache-saving idea to its limit: all the query heads share one single key head and one single value head. Where standard attention keeps, say, 32 separate copies of keys and values, MQA keeps exactly one of each. The reference-card analogy becomes one card on the table that every reader consults. The result is the smallest possible KV cache and the fastest token generation, because far less memory has to be read for each new token.

The cost is a small drop in quality and sometimes less stable training, since collapsing to a single key/value head removes a lot of representational freedom. That tradeoff is exactly why grouped-query attention emerged as a softer version, keeping a handful of key/value heads instead of one. MQA still appears where memory and latency dominate, but for most large models GQA is preferred as the sweeter spot between speed and accuracy.