FlashAttention

A naive attention implementation builds the full score matrix — every token compared against every token — in slow GPU memory, then reads it back to apply softmax. For long sequences that matrix is enormous, and it is the trips to memory, not the arithmetic, that become the bottleneck.

FlashAttention is an IO-aware kernel that never writes the full attention matrix to main memory at all. It streams the computation in tiles small enough to fit in fast on-chip SRAM, fuses the score, softmax, and value-weighting into one pass, and uses an online-softmax trick to keep running totals correct across tiles. The result is mathematically the exact same attention output, but with far fewer memory reads and writes — large speedups and much lower memory use. In serving this is what makes long prompts and long contexts practical to prefill at all.