flash attention
/ flash uh-TEN-shun /
Flash attention is a clever way to compute the exact same attention result much faster and with far less memory, by being smart about how data moves around inside the chip. The key realization is that on a modern GPU, the slow part is usually not the arithmetic — it is shuttling data between the chip's tiny, lightning-fast on-chip memory and its larger but much slower main memory. Standard attention writes out a giant scores matrix to slow memory and reads it back, and that traffic is the real bottleneck.
Flash attention avoids ever building that full matrix in slow memory. It processes the sequence in small tiles that fit in the fast on-chip memory, computes attention for each tile, and cleverly combines the partial results on the fly using a running, numerically careful version of softmax. The output is mathematically identical to ordinary attention — it is not an approximation — but because it never materializes the huge intermediate matrix, it uses memory that grows only linearly with sequence length instead of quadratically, and it runs several times faster in practice.
This is a textbook example of an idea that changed what is possible without changing the math at all. By slashing the memory cost, flash attention made it practical to train and run models on much longer sequences than before, directly enabling many of the long-context models people use today. The honest framing: it does not break the fundamental quadratic compute of attention — the number of operations is still there — it removes the memory wall and the slow-memory traffic that were the actual practical limit. It is a systems victory, not a new algorithm for attention itself.
Standard attention on a 16,000-token sequence tries to hold a 16,000-by-16,000 score matrix in memory — hundreds of millions of numbers — and chokes. Flash attention never builds it; it streams through small tiles and produces the identical output while fitting comfortably in fast on-chip memory.
Same answer, no giant matrix — a memory trick, not a new formula.
A common overstatement is that flash attention "solved" attention's quadratic cost. It didn't — the number of arithmetic operations is unchanged. What it removed was the memory bottleneck and slow-memory traffic, which were the binding practical limit. It computes the exact same result, just far more efficiently.