a deep-learning accelerator
A modern neural network is, under the hood, a tower of the same simple operation repeated billions of times: multiply a big grid of numbers (the inputs) by another big grid (the learned weights), add them up, squash the result, and pass it on. Doing that on a general CPU is like using a fine fountain pen to fill in a thousand-page form by hand — possible, but absurdly slow and wasteful. A deep-learning accelerator (also called a neural-network processor or NPU) is hardware purpose-built to do that one repeated operation, matrix multiplication, at staggering speed and efficiency.
Concretely, a deep-learning accelerator follows the domain-specific-architecture playbook. Since the work is dominated by matrix multiply, it packs the chip with many multiply-accumulate units — often arranged as a systolic array. Since neural networks tolerate low precision, it computes in reduced-precision formats such as bfloat16 or int8, which lets it fit far more arithmetic units in the same silicon and burn less energy per operation. Since the access pattern is predictable, it uses large software-managed scratchpad memories instead of relying on automatic caches, so the compiler can orchestrate exactly which data sits where. Google's TPU is the famous example, but NPUs now appear in phones, laptops, and datacenters from many vendors.
The honest core insight is that arithmetic is the easy part. A chip can multiply faster than memory can supply numbers, so the real bottleneck for a deep-learning accelerator is almost always data movement — getting the weights and activations from memory into the arithmetic units fast enough to keep them busy. This is why these chips obsess over memory bandwidth, on-chip scratchpads, and high-bandwidth memory. And accelerating training or inference still pays off only when the model is large and the workload is steady; for a tiny, one-off computation the overhead of using the accelerator can outweigh the gain.
Running a large language model's attention layer means multiplying huge matrices over and over. A CPU might issue these multiplies one or a few at a time; a deep-learning accelerator runs thousands of int8 or bfloat16 multiply-adds every cycle through its array — but only stays at peak if memory keeps the weights flowing, which is why such chips are paired with high-bandwidth memory.
A neural-network processor: many low-precision multiply-adds, fed from scratchpads — limited in practice by memory, not math.
The headline 'TOPS' (trillions of operations per second) figure is a peak that assumes the arithmetic units never starve. Real performance is usually capped by memory bandwidth and by how much of the model actually fits the accelerator's preferred shapes.