grouped-query attention (GQA)

When a model generates text it must remember the keys and values of every past token, a store called the KV cache, and during long generations that cache, not the math, is what fills memory and slows things down. Standard multi-head attention gives every query head its own key and value heads, which is expensive to store. GQA shrinks the bill by letting several query heads share one key/value pair. Picture eight readers all consulting the same two reference cards instead of eight private copies.

It is the middle ground between full multi-head attention and the extreme of one shared key/value head. You pick a number of groups, say the 32 query heads form 8 groups of 4, and each group keeps a single key/value head. The cache and the memory bandwidth needed per token drop by the group factor, which is the main bottleneck at inference, while quality stays almost identical to full attention. This balance is why GQA is now the standard attention layout in large open models.