the tensor processing unit (TPU)
/ T-P-U /
When Google realized that if everyone used voice search for a few minutes a day they would need to double their entire fleet of computers just to keep up, they did not buy more general servers — they built a chip that does only the math neural networks need, and does it astonishingly well. The tensor processing unit (TPU) is that chip: a domain-specific accelerator (an ASIC) designed from the ground up to run deep-learning workloads, which are overwhelmingly large matrix multiplications.
Concretely, the heart of a TPU is a systolic array: a large grid of small multiply-accumulate units (the first TPU had a 256 by 256 grid, 65,536 of them) through which the numbers of a matrix multiplication flow in a rhythmic, pipelined wave, each unit multiplying a pair of values and adding to a running total as data marches past it. To feed this array the TPU uses big software-managed scratchpad memories rather than automatic caches, and it works in reduced precision — early TPUs multiplied 8-bit integers (int8), later ones use bfloat16 — because neural networks tolerate low precision well. The freed transistors all go into more arithmetic units, so a TPU can do tens of trillions of multiply-adds per second.
The honest framing: the TPU is the textbook example of a domain-specific architecture, and it illustrates every guideline — exploit the domain's parallelism (matrix multiply), drop to the precision the domain tolerates (int8/bfloat16), use scratchpads instead of caches, and spend the saved silicon on arithmetic. But it is useless outside deep learning, and its true bottleneck is usually not arithmetic but moving the weights and activations in and out of memory fast enough to keep that vast array busy. A TPU only made sense because Google had a workload big and stable enough to justify designing custom silicon.
A neural-network layer computes output = weights times input, a matrix multiply with millions of multiply-adds. A TPU loads the weights into its systolic array once, then streams the inputs through; each multiply-add unit fires every cycle, so the whole array produces a steady torrent of results — far more arithmetic per watt than a CPU walking the same math instruction by instruction.
The TPU: a systolic array of multiply-add units, fed in low precision from scratchpads — a DSA built for matrix multiply.
A TPU is not a faster general CPU or even a faster GPU — it is a fixed-function matrix-multiply engine. Its published peak rate is rarely reached in practice, because keeping the array fed with data, not raw multiplying, is usually the limit.