The workload is one shape: matrix multiply
The previous guide gave you four rules for designing a domain-specific architecture: spend transistors on more arithmetic units, feed them from dedicated software-managed memories instead of caches, exploit the parallelism the domain hands you, and drop to the lowest precision the application can tolerate. This guide watches all four rules land at once in the marquee example — the deep-learning accelerator — and the reason they land so cleanly is a single fact about the workload. Almost everything a neural network does, in training and in serving, reduces to multiplying matrices. A layer that takes a vector of inputs and produces a vector of outputs is a matrix-vector product; do a whole batch at once and it becomes a matrix-matrix product. Convolutions, attention, the big fully-connected layers — under the hood they are all the same arithmetic.
This is exactly the gift the domain-specific guidelines hope for: a workload that is large (millions of multiplies per layer), stable (the same operation, layer after layer, model after model), and massively parallel (every output element is an independent dot product). A general CPU treats each of those multiplies as a separate instruction to fetch, decode, and schedule; a GPU does far better with thousands of lanes but still spends real energy moving operands in and out of register files. The accelerator's bet is that if the work is only matrix multiply, you can build silicon that does nothing else — and does it with almost no overhead per multiply.
The real enemy is data movement, not arithmetic
Here is the subtlety that shapes the whole design. A multiply is astonishingly cheap in modern silicon — a small multiplier is tiny and burns very little energy. What is expensive is fetching the two numbers to multiply. Reading an operand from off-chip DRAM can cost hundreds of times the energy of the multiply it feeds, and even reading from on-chip storage dwarfs the arithmetic. This is the memory wall you met long ago, now wearing an energy hat rather than a latency one. So the figure of merit for an accelerator is not how many multipliers it has but its arithmetic intensity — how many operations it performs per byte it pulls from memory.
Frame the goal that way and the whole design rewrites itself: the trick is to fetch each number once and then squeeze as many multiplies out of it as you possibly can before it leaves the chip. In a matrix multiply this reuse is enormous — every input row feeds into hundreds of output columns, so each value should be multiplied hundreds of times. The question is purely structural: how do you wire up the multipliers so that a value, having been fetched once into the array, naturally flows past every multiplier that needs it, with no extra trip back to memory? That structural answer is the systolic array.
How the grid actually multiplies
Picture a two-dimensional grid of identical tiny cells — Google's first Tensor Processing Unit famously held a 256-by-256 grid, 65,536 multiply cells. Each cell does one humble thing per clock tick: it multiplies two numbers, adds the product to a running total it holds, and passes its two inputs along to its neighbours — one to the right, one downward. That 'multiply, accumulate, pass it on' is a multiply-accumulate (MAC), the atom of a dot product, and it is the only operation the cell knows. There is no instruction fetch, no decode, no register renaming, no branch prediction — none of the machinery the earlier rungs spent so long building. The cell is almost pure arithmetic. That austerity is the whole point: with no overhead, you can afford 65,536 of them.
Now the flow. The accelerator first loads one of the two matrices — say the layer's weights — into the grid, one weight pinned in each cell, where it stays put for the whole matrix multiply. Then it streams the other matrix, the activations, in from the left edge: each row of input marches rightward across the grid, one column of cells per clock tick, like a wave. As an input value passes through a cell, the cell multiplies it by its resident weight, adds it into the partial sum flowing downward, and hands the input on to the next cell to its right. Because the data enters skewed (each row started one tick later than the row above), every cell is busy on every tick once the array fills, and a finished column-sum drops out of the bottom edge each cycle.
A 3x3 systolic array, weights W pinned in each cell.
Activations a enter from the LEFT, skewed by one tick per row.
Partial sums flow DOWN. Each cell does: sum += a * W; pass a right.
a0 ->[W00]->[W01]->[W02] (row 0 enters at tick 0)
| | |
a1 ->[W10]->[W11]->[W12] (row 1 enters at tick 1)
| | |
a2 ->[W20]->[W21]->[W22] (row 2 enters at tick 2)
v v v
out0 out1 out2 <- finished dot-products fall out the bottom
Each activation, fetched ONCE into the left edge, is reused by every
cell in its row. Each weight, loaded ONCE, is reused by every
activation that streams past it. One fetch -> hundreds of multiplies.Trace one input value to feel the win. The number a0 enters the top-left cell, gets multiplied there, then flows right and gets multiplied again in the next cell, and the next — one fetch, then a whole row of multiplies with no further memory access. Meanwhile each pinned weight is reused by every activation that streams past it. This is the arithmetic intensity goal made physical: a value enters the array once and participates in a multiply at every cell on its path. The data movement that dominated the energy budget has been replaced by short hops between adjacent cells — picometres of wire instead of a trip to DRAM.
Lower precision: the fourth rule pays double
The last design rule was 'use the lowest precision the application tolerates,' and neural networks tolerate a lot. A trained model is statistical and noisy; it does not need the full 32-bit floating-point that scientific computing demands. So accelerators run in reduced precision: bfloat16, a 16-bit float that keeps the full 8-bit exponent range of 32-bit float (so it never overflows where the big one wouldn't) but throws away most of the mantissa bits the network does not miss, and int8, an 8-bit integer used for inference once a model has been quantized down from floats.
This pays off twice, and that is easy to miss. First, a multiplier for 8-bit numbers is far smaller and lower-energy than a 32-bit one — multiplier area grows roughly with the square of the bit-width, so going from 32-bit to 8-bit can shrink each cell by an order of magnitude, letting you pack many more MAC cells into the same silicon. Second, every operand is now a quarter or an eighth the bytes, so the same precious memory bandwidth carries four to eight times as many numbers per second. Lower precision attacks both the arithmetic cost and the data-movement cost — the two things this whole architecture exists to minimize.
Software is half the machine, and so are its limits
A systolic array is brutally simple hardware, which means it pushes a lot of work onto software. Nothing inside the array decides what to fetch, when to load the weights, or how to chop a layer too big for the grid into tiles that fit — that scratchpad is software-managed, so the compiler must orchestrate every byte's arrival and departure. This is hardware-software co-design in its purest form: the chip is deliberately dumb so the compiler can be smart, and the two are designed together as one system. A weak compiler leaves those 65,536 cells idle; a good one keeps the array fed and humming. The accelerator is genuinely only half the product.
And be honest about the cost of all this specialization. The systolic array is breathtaking at dense matrix multiply and helpless at almost everything else — it cannot run your operating system, sort a list, or parse text. Worse, its efficiency depends on keeping that grid full: feed it small matrices, or shapes that do not tile neatly onto a 256-by-256 array, and most cells sit idle while you still pay their power. This is exactly the domain-specific bargain restated — the accelerator only wins when the workload is large, stable, and parallel. The moment the work turns small, irregular, or branchy, a GPU or a plain CPU is the better tool. There is no universal accelerator, only a right one for a workload that earns it.