GPU kernel fusion

When you run a chain of elementary operations on a GPU — say a matrix multiply, then a bias add, then a GELU — the naive path launches one kernel per operation, and each kernel reads its inputs from global memory and writes its output back. Those round trips to slow off-chip memory dominate the cost of cheap pointwise work. Kernel fusion stitches the chain into a single kernel so the intermediate values never leave the chip, staying in registers or shared memory until the final result is written exactly once.

Fusion pays off mainly for memory-bound operations, where it cuts both the bytes moved and the per-kernel launch overhead. The canonical pattern is fusing an elementwise epilogue (bias, activation, dropout, residual add) into the tail of a preceding matmul or normalization. It is bounded by on-chip resources: a fused kernel must hold all live intermediates in registers and shared memory, so fusing too aggressively spills back to memory or drops occupancy. Reductions complicate fusion because they introduce cross-thread dependencies a simple elementwise fuser cannot absorb.

In transformer training and inference, fusing normalization, the attention epilogue, and the MLP activations recovers arithmetic intensity that would otherwise be wasted shuttling activations to and from memory; flash attention is itself an aggressively hand-fused kernel. Modern compilers automate much of this, but the highest-value fusions in production stacks are still frequently written by hand.