The Roofline: Where Did the FLOPs Go?

The first question a systems engineer asks

You profile a training step and find the GPU is at 30% utilization. The instinct is to add more compute. That instinct is usually wrong. A modern accelerator can do far more arithmetic per second than it can read data from memory, so most real kernels spend their lives waiting for bytes, not waiting for math. The roofline framework gives you a one-line diagnosis: is this kernel limited by compute or by memory bandwidth? Everything you do next depends on that answer.

Arithmetic intensity: FLOPs per byte

Arithmetic intensity is the ratio of useful floating-point operations to bytes moved between the compute units and memory. It is a property of the algorithm and its data layout, not of the hardware. A large matrix multiply of two N-by-N matrices does about 2N³ FLOPs while moving about 3N² floats, so intensity grows with N — that is why big GEMMs are the friendliest workload on a GPU. An element-wise activation, by contrast, reads a value, does one multiply, and writes it back: intensity near 1.

Arithmetic intensity: useful floating-point work divided by the bytes moved between compute and memory.

intensity = useful_flops / bytes_moved
# N=4096 matmul:  ~2.7e11 flops / ~2e8 bytes  -> ~1300 flops/byte (compute-bound)
# elementwise gelu: ~1 flop / 4 bytes          -> ~0.25 flops/byte (bandwidth-bound)

The roofline: one picture, one verdict

The roofline model plots achievable performance (FLOPs/s) against arithmetic intensity on log-log axes. There are two ceilings. A slanted line — the memory roof — equals (peak bandwidth × intensity): when bytes are the bottleneck, doubling intensity doubles speed. A flat line — the compute roof — is the chip's peak FLOPs/s. Where they meet is the ridge point. A kernel to the left of the ridge is memory-bandwidth-bound; to the right it is compute-bound. Plot your kernel's measured intensity and throughput, and the gap to the roof tells you exactly how much you are leaving on the table.

The roofline verdict: attainable FLOPs/s is the lesser of the compute ceiling and intensity times peak bandwidth.

Why bandwidth is the real scarce resource

Accelerators feed their tensor cores from high-bandwidth memory (HBM) — stacked DRAM that delivers terabytes per second, an order of magnitude beyond ordinary DDR. Yet even HBM cannot keep pace with the math units, which is why the ridge point sits at hundreds of FLOPs/byte. The lesson generalizes far beyond one kernel: LLM training is dominated by big GEMMs and tends to be compute-bound, while LLM decode reads the whole weight matrix to produce a single token and is brutally bandwidth-bound. Same model, opposite regimes — and opposite optimization playbooks.

This is also why FlashAttention was such a landmark. The naive attention kernel materializes the full N-by-N score matrix in HBM, making it spectacularly bandwidth-bound. FlashAttention keeps the scores in fast on-chip SRAM and never writes them out — raising arithmetic intensity, sliding the kernel rightward on the roofline, and turning a memory-starved operation into a near-compute-bound one. That is the roofline in action.

Using the roofline in practice

1. Profile the kernel: read achieved FLOPs/s and bytes/s from the GPU profiler (e.g. Nsight Compute, the PyTorch profiler).
2. Compute its arithmetic intensity = FLOPs / bytes, and place the point on the chip's roofline.
3. Read the verdict: left of the ridge means bandwidth-bound — chase data reuse and fusion; right means compute-bound — chase occupancy and precision.
4. Estimate the headroom: distance to the relevant roof is your maximum possible speedup; if it is small, stop and optimize elsewhere.
5. Re-profile after each change and watch the point move — optimization you cannot see on the roofline probably did nothing.