JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Designing a Domain-Specific Architecture

If specialization is the bet and the CPU-GPU-FPGA-ASIC spectrum is the board, this guide hands you the playbook: the four design guidelines every domain-specific architecture follows, why deep-learning accelerators are the marquee example, and the honest rule that an accelerator only pays when the workload is large, stable, and parallel.

From the spectrum to a blueprint

The last two guides set the stage. We saw why to specialize — with Dennard scaling gone and Moore's law slowing, you cannot just wait for a faster clock — and we laid out the flexibility spectrum from the general-purpose CPU, through the GPU and the reconfigurable FPGA, down to the hard-wired ASIC. Now comes the question that matters: once you have chosen to build a domain-specific architecture, how do you actually design one? A general CPU is built to run anything tolerably; a hardware accelerator is built to run one kind of thing superbly, and that single change of goal rewrites every design decision.

Here is the central trade you are making. A CPU spends a startling fraction of its silicon and energy not on arithmetic but on flexibility: caches that guess what any program might touch, branch predictors betting on which fork the code takes, out-of-order machinery finding parallelism in code that was written serially. All of that is overhead you pay on every instruction so the chip can run code it has never seen. An accelerator knows its workload in advance, so it can delete that overhead and spend the reclaimed transistors on raw compute. Specialization is, at heart, trading generality for efficiency — and the four guidelines below are simply the disciplined ways to spend the savings.

The four guidelines

Across the accelerators that actually shipped and won, the same four moves keep appearing. They are not a recipe you follow blindly; they are four levers, and the art is knowing how hard to pull each for your domain. Read them as 'where did the CPU waste effort, and how does a specialist reclaim it?'

  1. Use dedicated, software-managed memories. A CPU cache guesses what to keep, paying tag-and-compare overhead and sometimes guessing wrong. An accelerator instead uses a scratchpad — fast on-chip memory the compiler explicitly fills and drains, with no tags, no eviction roulette, and no cache misses. Because the access pattern of the domain is known ahead of time, software, not hardware, decides exactly what sits close to the arithmetic.
  2. Spend the saved transistors on more arithmetic units. Every transistor you don't burn on branch prediction or out-of-order logic can become another adder or multiplier. Where a CPU core has a handful of arithmetic lanes, an accelerator can pack hundreds or thousands, because it no longer pays the generality tax that crowds them out.
  3. Exploit the parallelism the domain already has. Don't fish for hidden instruction-level parallelism the way a CPU does — match the hardware to the shape of parallelism the problem hands you for free. If the domain is full of independent vector operations, lay out wide SIMD lanes; if it is a stream of independent data items, build a pipeline; if it is matrix multiply, build a 2-D grid. The structure of the workload is the structure of the chip.
  4. Use the lowest precision the domain can tolerate. Numerical precision is not free — a 32-bit multiplier is far larger and hungrier than an 8-bit one. Many domains simply do not need full precision: graphics, signal processing, and neural networks all stay accurate enough at reduced precision. Dropping from 32-bit floats to 16-bit or 8-bit lets you fit several times more arithmetic units in the same area and move several times more numbers per memory access.

The marquee example: deep-learning accelerators

No domain shows off these guidelines better than deep learning, and Google's Tensor Processing Unit (TPU) is the textbook case. Strip a neural network down and almost all the work is one operation repeated billions of times: matrix multiply, which is a flood of multiply-then-add steps. That is a near-perfect target — the operation is simple, the same for every layer, and overwhelmingly parallel. So a deep-learning accelerator throws away the CPU's generality and builds a machine that does essentially nothing but multiply-accumulate, at enormous scale.

The TPU's signature move is the systolic array — the star of the next guide, so here we only sketch why it fits the guidelines. Imagine a square grid of tiny multiply-add cells. Numbers flow in from the edges and march across the grid in lockstep, like water pulsing through a heart (that is what 'systolic' means). Each cell multiplies the two values passing through it, adds the running total, and passes the result onward — so one number loaded from memory is reused by a whole row and column of cells instead of being fetched again and again. That single trick attacks the quiet villain of the whole field: data movement. Reading a number from off-chip DRAM costs far more energy than the multiply you do with it, so the accelerator that moves data least wins, almost regardless of how many multipliers it has.

And precision is where the fourth guideline pays its biggest dividend. Neural networks are forgiving of noise, so accelerators run training in bfloat16 — a 16-bit float that keeps the wide range of a 32-bit number but spends fewer bits on the mantissa — and run inference in int8, plain 8-bit integers. An 8-bit multiplier is a fraction of the size and energy of a 32-bit one, so the same silicon holds vastly more of them and each byte of bandwidth carries four times as many values. The honest caveat: this works because deep learning tolerates it; you would never run a bank ledger or a physics simulation in int8. Low precision is a domain-specific licence, not a universal upgrade.

It's not just neural nets

Deep learning grabs the headlines, but accelerators predate it by decades and surround you right now. The phone in your pocket is a small zoo of them. A digital signal processor (DSP) is a chip tuned for the multiply-add-heavy math of audio and radio — filtering your voice on a call, decoding a Wi-Fi signal — for which it is far more power-efficient than the main CPU. A video codec block decodes a 4K stream in hardware; doing the same in software would drain your battery in an hour, which is why it earned its own dedicated silicon.

Two more show how broad the idea goes. A cryptography engine does AES encryption in hardware, so every disk write and every HTTPS connection is encrypted with almost no CPU cost — the same workload in software would be both slower and a juicier side-channel target. And in the data centre, a SmartNIC or DPU is a network card that has grown its own processors and accelerators, taking over packet processing, encryption, and storage chores so the expensive server CPUs are freed to run paying customer code. Different domains, same playbook: find a hot, stable, parallel task and give it dedicated hardware.

When an accelerator actually pays

Now the honest counterweight, because specialization is not free. An accelerator does not live alone — it sits on a system on chip (SoC) beside the CPU, GPU, DSP, and memory controllers, all on one die. Work that does not fit the accelerator still runs on the CPU, and crucially, data must be moved between them. If you ship a small job to an accelerator, the cost of copying the data over, kicking off the hardware, and copying the result back can dwarf the work itself. The accelerator can be ten times faster on the math and still lose, because the math was never the bottleneck.

There is a second, slower tax: building the thing. An ASIC costs a fortune and years to design, fabricate, and verify before the first chip exists. That upfront cost has to be paid back over the workload it runs — the principle of amortization. So an accelerator earns its keep only when three conditions all hold. The workload must be large (so the per-call overhead and the design cost are dwarfed by the volume), stable (so the hardware, frozen in silicon, is not obsolete before it ships — this is exactly why reconfigurable FPGAs exist for workloads still in flux), and parallel (so the army of arithmetic units has work to keep them all busy).

This is why the real craft is hardware-software co-design: the chip and the software that drives it are designed together, each shaped to the other. The TPU's bfloat16 format, its scratchpad sizes, and its instruction set were chosen to match how its compiler lays out a neural network — and the compiler was written to keep that systolic array fed. Designing a domain-specific architecture, then, is never just drawing a faster datapath. It is the harder, more honest job of choosing which workload deserves silicon, proving it is large, stable, and parallel, and co-designing the hardware and software so the speedup you measure is the speedup you actually keep.