hardware-software co-design
A great race car and a great driver are designed for each other — the car's controls, the gearing, the seat are all shaped around how the driver works, and the driver trains around what the car can do. Hand either one to a stranger and the magic is gone. Hardware-software co-design is building a chip and the software that runs on it together, as one coupled effort, instead of designing the hardware first and hoping software can use it. For accelerators this is not a nicety — it is essential, because a specialized chip is worthless without a compiler and runtime that can actually target it.
Concretely, an accelerator usually exposes an unusual hardware structure — a systolic array, software-managed scratchpads, reduced-precision number formats, explicit data-movement instructions. None of that is useful unless software knows how to map a problem onto it. So co-design means the hardware architects and the compiler/runtime/library writers work in lockstep: the hardware team decides what operations the chip does natively, and the software team builds the compiler that lowers a programmer's high-level model (say, a neural network described in a framework) into exactly those operations, schedules the data movement through the scratchpads, and quantizes numbers to the chip's precision. Decisions flow both ways — if a hardware feature is too hard for the compiler to use, it gets redesigned; if the software needs an operation, the hardware may add it.
Why it matters and the honest catch: hardware is rigid once fabricated, so a mistake in the hardware/software contract cannot be patched later in the way a pure software bug can. A brilliant accelerator with a poor compiler will sit idle, unable to be fed real workloads — which has killed many promising chips. The flip side is that this tight coupling is also a strength of the domain-specific approach: because the team controls both sides, they can make trade-offs (this precision, this memory layout, this instruction) that a general-purpose CPU, which must run software written by strangers for an open standard, never could.
A TPU is useless on its own; it ships with a compiler (such as XLA) that takes a neural network written in a framework, fuses operations, lowers them onto the systolic array, lays out the data in the scratchpads, and quantizes to bfloat16. The hardware and that compiler were designed together — change one and the other must change too.
Chip and compiler designed as one: the accelerator is only as good as the software that can target it.
An accelerator's peak hardware numbers mean nothing without a mature compiler and runtime; many technically impressive chips failed because their software stack could not feed them. Co-design is the often-underestimated half of the work.