memory bandwidth

Think of a water pipe. Two different questions describe it: how long does it take one drop to travel from end to end (that is latency), and how many litres per minute can it deliver when fully flowing (that is bandwidth). A wide firehose has huge bandwidth even if the journey for any single drop is no faster. Memory bandwidth is the litres-per-minute of memory: how many bytes per second the memory system can stream to and from the processor.

Concretely, peak memory bandwidth is built from a simple product: bus width x transfer rate x number of channels. A 64-bit (8-byte) channel running at DDR4-3200 (3200 mega-transfers per second) gives 8 x 3200 x 10^6 = about 25.6 GB/s; two such channels give about 51.2 GB/s; eight channels of fast DDR5 give hundreds of GB/s. That is the theoretical peak — real sustained bandwidth is lower because of row conflicts, refresh, read/write turnarounds, and imperfect access patterns. Bandwidth answers 'how much can I move per second,' which is a completely different axis from latency's 'how long does one access take.'

Why it matters and the honest headline: for enormous classes of modern workloads — dense linear algebra, deep-learning training, graph processing, video, scientific simulation — it is memory bandwidth, not the cores, that sets the performance ceiling. A processor that can compute far faster than memory can feed it simply waits, idle, for data. The roofline model captures this: below a certain arithmetic intensity (operations per byte fetched), you are bandwidth-bound and adding compute does nothing. This is why high-bandwidth memory exists, and why bandwidth, not GHz, is often the number that matters.