high-bandwidth memory (HBM)

An accelerator's compute units are useless if they starve for data. High-bandwidth memory (HBM) is the stacked DRAM mounted right next to the GPU die that feeds those units. Instead of a few memory chips on a distant board, HBM stacks many DRAM dies vertically and connects them over an extremely wide bus through a silicon interposer — trading the long, narrow path of conventional memory for a short, very wide one.

HBM uses through-silicon vias to stack DRAM dies and exposes a 1024-bit-wide interface per stack, so even at modest clock speeds it delivers terabytes per second of bandwidth; recent HBM3 and HBM3e stacks exceed a terabyte per second each, with several stacks per accelerator. The cost is capacity and price: HBM is far more expensive per gigabyte than commodity GDDR or DDR, and stacking limits total capacity, which is why frontier accelerators carry tens of gigabytes rather than hundreds.

Because so many ML kernels — LLM decoding above all — are memory-bandwidth bound, HBM bandwidth rather than raw FLOPs sets their ceiling. The widening gap between how fast compute and memory grow, the so-called memory wall, is exactly why arithmetic intensity, fusion, and quantization carry so much weight in modern systems work.