From one knob to four points
The previous guide made the case for why specializing pays now: with Dennard scaling gone and Moore's law limping, you can no longer just wait for a faster clock, so you buy efficiency by tailoring the silicon to the work. But 'specialize' is not a switch — it is a dial. Turn it one way and you keep total flexibility but waste energy; turn it the other and you gain stunning efficiency but lose the ability to run anything else. Every important class of chip is just a different setting of that one dial, the flexibility-versus-efficiency spectrum.
Four landmarks sit along that dial. The CPU is the most flexible: it runs any program written to its instruction set, branching and deciding cheaply, the one genius who can do anything but only a few things at once. The GPU trades some of that flexibility for raw throughput — a thousand simple lanes doing the same arithmetic. The FPGA (field-programmable gate array) lets you wire up custom hardware after manufacture, re-shaping the chip itself. The ASIC (application-specific integrated circuit) bakes one fixed function into permanent silicon. Left to right, you trade 'can run anything' for 'runs one thing magnificently'.
Walking the dial: what each point buys and costs
Start at the CPU. Its flexibility is not free: an enormous fraction of its energy is spent not on your arithmetic but on the overhead of being general — fetching and decoding instructions, predicting branches, renaming registers, reordering for out-of-order execution. For a serial, branchy program full of decisions, that overhead is exactly what you want and the CPU wins handily. For a wall of identical multiply-adds, that same machinery is pure waste — the chip burns most of its power deciding what to do next instead of doing it.
Slide to the GPU and that waste shrinks. A GPU amortizes one instruction fetch across a whole warp of lanes executing in lockstep (SIMT), so the decode overhead is paid once for thirty-two arithmetic results instead of one. The price is honesty's first casualty in the last guide, sharpened: a GPU is not a faster CPU. Give it serial, branchy code and the lanes diverge, most sit idle, and it crawls. It only shines on wide, regular data-level parallelism where thousands of elements take the same path.
Push further and you reach the FPGA and ASIC, which abandon instructions altogether. Instead of fetching commands that steer a fixed datapath, they wire the computation directly into hardware — the data flows through gates laid out to match the problem. An FPGA does this with a reconfigurable fabric you can rewire in seconds, so it is reprogrammable but slower and larger than custom silicon. An ASIC freezes the same idea into a mask, reaching the best speed and energy of all — at the cost of millions of dollars and many months to design, and zero flexibility once made. The dial's far end is gloriously efficient and utterly inflexible.
The domain-specific playbook
When architects build a domain-specific architecture — an accelerator aimed at one workload — they reach for the same four moves again and again. Each move spends silicon that a general CPU spends elsewhere. Together they are the playbook for turning a known workload into efficient hardware.
- Use dedicated, software-managed memories. Replace the CPU's automatic cache hierarchy with scratchpad memory the compiler controls explicitly. The accelerator knows its access pattern in advance, so it can place data perfectly and skip the tag-matching and guessing a cache does — no cache misses to suffer because nothing is left to chance.
- Spend the saved area on more arithmetic units. With the fetch/decode/branch machinery gone, pour that silicon into hundreds or thousands of multiply-add units, all kept busy. Raw arithmetic, not control, is now what the chip is mostly made of.
- Exploit the domain's specific parallelism. Match the hardware's shape to the problem's shape — a grid of units for a grid of data — instead of relying on a general scheduler to find parallelism at run time.
- Lower the precision. If the workload tolerates it, use narrower numbers (reduced precision like 8-bit integers or 16-bit floats) instead of 32- or 64-bit. Narrower data means smaller multipliers, less memory traffic, and more results per joule — a huge lever when the application can absorb the lost bits.
Notice what every move has in common: it trades generality for a guarantee. A CPU caches because it cannot predict your accesses; an accelerator skips the cache because it can. A CPU keeps 64-bit precision because it must serve everyone; an accelerator drops to 8 bits because it serves one workload that proved it could spare them. The playbook only works when the domain is narrow enough that these guarantees hold — which is exactly the honest limit we return to at the end.
The marquee example: the TPU's systolic array
The clearest place to watch the playbook in action is the deep-learning accelerator, and the most famous one is Google's Tensor Processing Unit (TPU). Deep learning, it turns out, is dominated by one operation: matrix multiply, which is itself a vast pile of multiply-and-accumulate. If you can build hardware that does dense matrix multiply astonishingly fast and cheap, you have accelerated the whole field. That is precisely the bet the TPU makes.
Its engine is a systolic array — a square grid of tiny multiply-add cells, in the original TPU a 256-by-256 mesh of them. The name comes from the heart: data pulses rhythmically through the grid like blood. Weights are loaded into the cells; then input values flow in from the left and partial sums flow down from the top. At each tick, every cell multiplies the value passing through by its stored weight, adds the partial sum arriving from above, and passes both onward. A full matrix multiply falls out as the data marches across the grid — tens of thousands of multiply-adds per clock, with almost no instruction overhead.
systolic array: a value, once read, is reused as it walks the grid
inputs -> [x]->[x]->[x]->[x] each [x] = one multiply-add cell
[x]->[x]->[x]->[x] weights sit inside the cells
[x]->[x]->[x]->[x] partial sums flow DOWN
[x]->[x]->[x]->[x] input data flows RIGHT
| | | |
v v v v
column results (accumulated dot products)
key trick: read each operand from memory ONCE, then let it
do useful work in every cell it passes through on its way across.
fewer trips to memory == the win.Two playbook moves make this fly. First, reduced precision: the TPU multiplies in 8-bit integers (int8) or 16-bit floating point (bfloat16, which keeps a float's wide exponent range but throws away precision bits that training does not need). Smaller multipliers pack the grid denser and cut energy hard. Second — and this is the deep point — the bottleneck in such a machine is not the arithmetic but the data movement: feeding the array. The systolic design is clever exactly because, once a value is read from memory, it is reused in every cell it walks past, so one memory read does many multiplies. Minimizing trips to memory, not adding more multipliers, is the real game.
Beyond ML, and the honest limit
Deep-learning accelerators are the loud example, but the spectrum is everywhere once you look. A DSP is tuned for the multiply-accumulate of filtering audio and radio. The video codec in every phone is a fixed-function block that decodes a stream in a fraction of the power a CPU would burn. A cryptography engine does AES and hashing in dedicated hardware. A SmartNIC or DPU offloads network packet processing off the main CPU. Each is the same idea: take a hot, stable workload off the general-purpose core and give it custom silicon.
This is why the system-on-chip dominates modern devices: a phone SoC is a city of specialized blocks — CPU clusters, a GPU, an ML accelerator, a DSP, codecs, a crypto engine — sharing one die and one memory. And it forces a discipline called hardware-software co-design: you no longer design the chip and the software separately. The accelerator's shape, the data layout, and the program that drives it are designed together, each bending to fit the other, because a perfect array starved of well-laid-out data is just expensive idle silicon.