hardware accelerator

A general-purpose CPU does a few things at a time, very flexibly. Vision models instead do the same simple operation — a multiply-accumulate — billions of times in parallel. A hardware accelerator is a chip specialized for exactly that workload: GPUs, TPUs, and NPUs pack thousands of arithmetic units so they can crunch matrix multiplies and convolutions far faster, and far more energy-efficiently per operation, than a CPU could.

Mechanically, the workhorse is massive parallelism over matrix multiplication. GPUs use thousands of cores plus tensor cores — small matrix-multiply units that do, for example, a 4×4 FP16 or int8 multiply-accumulate every cycle. TPUs and many NPUs use systolic arrays, a grid of MAC cells through which data streams so each value is reused many times before leaving the chip. Crucially, performance is bounded by either compute (peak TOPS/FLOPS) or memory bandwidth (how fast you can feed the units) — the roofline model — so on-chip SRAM, HBM bandwidth, and data reuse matter as much as raw MAC count.

The landscape is a zoo: NVIDIA datacenter GPUs (H100/H200) and edge ones (Jetson), Google TPUs, mobile NPUs (Apple Neural Engine, Qualcomm Hexagon, Google Tensor, Coral Edge TPU), and FPGAs and custom ASICs. They strongly prefer low precision (FP16, BF16, FP8, int8) and large, regular, batched workloads. This shapes how you design models — favor dense regular convolutions and matmuls over scattered sparsity, choose sizes that tile cleanly onto the array — and is the reason hardware-aware neural architecture search and hardware/software co-design matter so much.