From one core to a team of specialists
The earlier guides in this rung made the case for specialization: with Dennard scaling over and Moore's law slowing, you can no longer just wait for a faster general-purpose core, and the power wall means much of a chip must sit dark anyway. The answer is to stop building one machine that does everything adequately and instead build several machines, each superb at one thing, on the same piece of silicon. That mixed chip is a heterogeneous system, and learning to use it well is the central skill of modern architecture.
Think of a hospital instead of a single family doctor. The CPU is the doctor who can handle anything that walks in — branchy, serial, unpredictable code, the kind of work where the genius generalist beats the thousand interns. The GPU is the team that excels when the same procedure must be run on a flood of data at once. And a hardware accelerator is the specialist surgeon who does exactly one operation, fast and cheap, but nothing else. A heterogeneous system is the hospital that keeps all of them on staff and routes each patient to the right one.
The flexibility spectrum you met earlier sets the trade-off. A CPU runs any program but spends most of its energy on overhead — fetching, decoding, predicting branches, reordering — rather than on the actual arithmetic. An FPGA is reconfigurable hardware, faster and more efficient than a CPU but slower than fixed silicon. An ASIC is frozen into one function and is the fastest and most efficient of all, at the cost of being unchangeable and expensive to design. The further right you move, the more efficiency you gain and the more flexibility you surrender. Heterogeneity lets you place each part of a workload at the right point on that spectrum instead of paying one fixed price for everything.
The same recipe, every time
Guide 3 distilled the domain-specific architecture guidelines, and the striking thing is how universal they are. Whether the target is deep learning, video, or cryptography, the same four moves keep reappearing — because they all attack the same enemy the general-purpose core could never beat: wasted energy and the memory wall. Keep these four in mind and almost any accelerator stops being mysterious.
- Use dedicated, software-managed memories. Replace the deep general memory hierarchy with small scratchpads the compiler controls explicitly, so data lands exactly where it is needed and you never pay for a cache miss or speculative prefetch you did not ask for.
- Spend the saved transistors on more arithmetic units. A CPU devotes most of its area to control and caches; an accelerator floods the die with adders and multipliers running in parallel, because that is where real work gets done.
- Exploit the parallelism the domain actually has. If the workload is naturally data-parallel like a matrix multiply, build the hardware to match that exact shape rather than rediscovering parallelism at run time the way out-of-order execution must.
- Drop to the lowest precision the domain tolerates. If the application is happy with 8-bit integers or 16-bit floats, reduced-precision units are smaller, faster, and cheaper to move data for — and a smaller number is a smaller energy bill on every wire it crosses.
The marquee example: deep-learning accelerators
The clearest place to watch all four guidelines at once is the deep-learning accelerator, and the most famous is Google's Tensor Processing Unit (TPU). Neural networks spend the overwhelming majority of their time on one operation: multiplying matrices, which is just a vast pile of multiply-then-add steps. Guide 4 opened up the TPU's heart, the systolic array — a grid of tiny multiply-accumulate cells through which the data rhythmically pumps, so each number is read from memory once and then reused by every cell it flows past, instead of being fetched again and again.
Lay the TPU against the four moves and every one is present. Dedicated memory: a large software-managed scratchpad feeds the array, with no cache to second-guess. More arithmetic: where a CPU has a handful of multipliers, the array packs tens of thousands. Domain parallelism: the grid's shape is literally the shape of a matrix multiply, so no run-time scheduling is needed. And reduced precision: weights and activations run in 8-bit integers (int8) or bfloat16 — a 16-bit float that keeps the full exponent range of a 32-bit float but throws away mantissa bits the network does not miss. The result is a chip that delivers far more useful arithmetic per watt than any general-purpose core could on the same workload.
The deepest lesson here is that the bottleneck is rarely the multiply itself — it is feeding the array. A systolic array of tens of thousands of cells is useless if it starves for data, so most of the design effort goes into memory bandwidth and reuse: stacking high-bandwidth memory right next to the compute, choreographing exactly when each weight and activation arrives, and reusing each loaded number across as many multiplies as possible. The roofline model from earlier captures this precisely: a kernel with low arithmetic intensity (few operations per byte fetched) is bandwidth-bound, and no amount of extra multipliers will speed it up. Accelerator design is, in the end, mostly the art of moving data less.
The other specialists on the chip
Machine learning gets the headlines, but accelerators are old and everywhere, and your phone is full of them. A digital signal processor (DSP) is tuned for the multiply-accumulate-heavy filtering of audio, radio, and sensor signals. A video codec is fixed-function hardware that compresses and decompresses streams; doing 4K video in software would flatten a battery, so a tiny dedicated block does it for a sliver of the energy. A cryptography engine runs AES encryption and hashing in hardware so secure connections do not tax the CPU. And a SmartNIC or DPU is a small computer on the network card that handles packet processing, encryption, and storage traffic, freeing the main CPU for actual application work.
All these specialists live together on one System-on-Chip — a SoC — the single die at the heart of every phone and most laptops now. The SoC binds the CPU cores, the GPU, and the accelerators around a shared on-chip interconnect and a common path to memory, so a block can hand a result to its neighbour without a costly trip off the chip. The art is in the wiring: who shares which memory, who talks over the network-on-chip, and how the DMA engines move bulk data between blocks while the cores get on with something else.
A heterogeneous chip is not automatically fast, though. Its blocks still talk over shared interconnect and memory, so the same hazards from the multicore guides apply: data must stay consistent as it passes between CPU and accelerator, and a host-device transfer that copies data back and forth can erase the speedup entirely. Worse, the dark-silicon reality means not every block can run at once within the power budget — a chip may carry far more accelerators than it can ever light up simultaneously, lighting whichever specialist the current job needs and leaving the rest dark.
Co-design, and when an accelerator actually pays
A general-purpose CPU honours a frozen contract: the ISA is fixed, and the compiler must make any program fit it. Accelerators throw that separation away. Hardware-software co-design means designing the silicon and the software that drives it together, as one project, each shaped to fit the other. The TPU's hardware was built around how neural-network frameworks express a model, and those frameworks were extended to emit code the TPU runs natively. Neither makes sense alone; the speedup lives in the seam between them.
But the honest question every architect must answer first is whether to build an accelerator at all, and the answer is usually no. A custom block costs a fortune to design and verify and can never be changed once it ships, so it only earns that price back under three conditions at once. The workload must be large — enough total work to amortize the design cost over. It must be stable — the algorithm cannot keep changing, or your silicon is obsolete before it tapes out. And it must be parallel and regular — full of independent, identically-shaped operations the hardware can lay out in a grid. Miss any one of those and a flexible CPU or GPU is the wiser choice.
That is where this rung, and much of the whole ladder, comes to rest. The story of architecture since around 2005 has been a forced migration: a single ever-faster core was no longer on offer, so the field moved first to multicore, then to GPUs, and now to heterogeneous chips full of specialists. The craft is no longer squeezing one general machine faster — it is knowing your workload deeply enough to decide which parts deserve their own silicon and which are better left flexible. Co-design is that craft made concrete: hardware and software, shaped to each other, in the post-Moore world where free speed has run out and cleverness is all that is left.