memory-bandwidth bound

A kernel is memory-bandwidth bound when it spends most of its time waiting for data to arrive from memory rather than doing arithmetic — the compute units sit idle, drumming their fingers, because the pipes feeding them are full. Adding more FLOPs of capability does nothing; only moving fewer bytes, or moving them faster, helps.

Formally, a kernel is bandwidth-bound when its arithmetic intensity sits to the left of the roofline's ridge point, so its achievable rate equals intensity times memory bandwidth and falls short of peak FLOP/s. The classic ML case is autoregressive LLM decoding: each generated token must read the entire weight matrix, plus the growing KV cache, out of HBM, yet does only a vector-matrix multiply at batch one, so the FLOPs-per-byte ratio is tiny. Throughput is then set by how fast weights and cache stream out of HBM, not by the GPU's tensor-core peak.

This regime explains why so much LLM systems work targets bytes rather than flops: weight and KV-cache quantization, paged and shared KV caches, speculative decoding, and large request batching that amortizes each weight read over many sequences to push intensity back toward the compute roof.