inference throughput

Throughput is how many inputs you can process per unit time — images per second, or frames per second (FPS) for video, or queries per second (QPS) for a server. Where latency cares about how long one input waits, throughput cares about total volume: how many camera streams a server can handle, or how many photos you can label per hour. The two are related but distinctly different goals.

Throughput is approximately batch_size divided by the per-batch latency when you process inputs in batches. Batching amortizes fixed overheads — kernel launches, weight loading — and fills the parallel hardware (a GPU's tensor cores, an NPU's MAC array), so larger batches usually raise throughput up to a saturation point where the units are fully busy. But each individual input now waits for the whole batch to finish, so latency rises. This is the fundamental latency–throughput tradeoff; pipelining and running concurrent streams also raise throughput.

Always report throughput at a stated batch size and precision, because the number is meaningless without them. The genuinely useful artifact is the throughput-versus-latency curve as batch size grows, and in particular "throughput under a latency SLA" — the maximum QPS you can sustain while keeping p99 latency below a bound. Serving systems such as Triton use dynamic batching to group incoming requests for throughput without blowing the per-request latency budget.