ring attention

Ring attention computes exact attention over sequences so long that the keys and values do not fit on a single device. It shards the sequence across a ring of devices, each holding one block of queries, keys, and values, and then passes the key/value blocks around the ring step by step. As each device receives the next block it accumulates that block's contribution to its local queries' attention, so after one full loop every query has attended to every key, without any single device ever holding the whole sequence.

The key engineering insight is overlap: it uses a blockwise online softmax in the FlashAttention style so partial attention can be combined incrementally, and it schedules the ring's communication to hide behind the attention computation. Because each device sends and receives only one KV block at a time, per-device memory is constant in the number of devices and total context length scales linearly with the ring size. There is no approximation, the result is identical to full attention.

This lets context windows reach into the millions of tokens by simply adding devices, and it composes with FlashAttention and sequence parallelism in modern long-context training. Its cost is the communication volume and the requirement for a low-latency interconnect; the ring's latency is bounded by the slowest hop, so topology and bandwidth matter.