arithmetic intensity
Think of carrying groceries up many flights of stairs. The exhausting part is the climb (moving the bags); the cooking once you arrive is comparatively quick. If each trip up the stairs lets you cook a whole feast, the climbing was worth it; if each trip only lets you boil one egg, you spend your life on the stairs. Arithmetic intensity measures exactly this ratio for a program: how much useful arithmetic you get done per unit of data you haul up from memory. High intensity means lots of cooking per climb; low intensity means lots of climbing for little cooking.
Precisely, arithmetic intensity is the number of arithmetic operations a computation performs divided by the number of bytes it transfers to and from main memory — operations per byte. Adding two arrays, c[i] = a[i] + b[i], reads 8 bytes and writes 4 to do a single add: roughly 1 operation per 12 bytes, very low intensity. A dense matrix multiply, by contrast, can do thousands of multiply-adds while a value sits reused in fast registers or on-chip memory, pushing its intensity high. The same algorithm can also be made more intense by reusing loaded data (cache blocking, tiling) so each expensive trip to memory feeds far more arithmetic.
Arithmetic intensity is the single number that decides which side of the roofline you live on, which is why it is the honest heart of GPU and vector performance. Low intensity means memory-bound: the chip's arithmetic units sit idle waiting for data, and a faster or wider arithmetic unit buys nothing — only more bandwidth or more data reuse helps. High intensity means compute-bound: data arrives fast enough to keep the arithmetic units busy, and now raw arithmetic throughput is the limit worth chasing. The practical lesson for beginners is counter-intuitive but vital: most real speedups on throughput hardware come not from doing arithmetic faster but from raising arithmetic intensity — restructuring the computation so each byte fetched from memory is used many more times before it is thrown away.
A naive matrix multiply that reads a row and a column from memory for every output element has low intensity and is memory-bound. Blocking it — loading a small tile of each matrix into fast on-chip memory and reusing it for many multiply-adds before fetching more — sharply raises arithmetic intensity, moving the kernel from the memory roof toward the compute roof and often speeding it up many times with identical results.
Arithmetic intensity = operations per byte moved. Raising it (by reusing data) is usually the real path to speed on throughput hardware.
Low arithmetic intensity caps you no matter how powerful the arithmetic units are: you are memory-bound, and buying more compute is wasted money. The usual remedy is to raise intensity by reusing data, not to make the math faster.