inference latency

Latency is the wall-clock time to get a result for a single input: you give the model one image, how long until you have the answer? It is what a user actually feels — the lag before a face unlocks a phone, or before a self-driving car registers a pedestrian. Lower is better, and many real-time systems impose a hard budget, for example 33 ms per frame to sustain 30 frames per second.

Latency is properly reported as a distribution, not a single number. The median (p50) tells the typical case, but tail percentiles (p95, p99) matter most for real-time and service-level agreements, because the occasional slow response is what breaks a deadline. To measure it honestly: time the full pipeline end-to-end (preprocessing, host-to-device transfer, compute, postprocessing); warm up first so caches, autotuning, and clock ramp settle; synchronize the device, because GPUs run asynchronously and naive timing measures only the kernel launch; and average many runs. Keep batch-size-1 latency separate from throughput-oriented batched timing.

What dominates is often surprising: inference is frequently memory-bandwidth-bound (streaming weights and activations) rather than compute-bound, which is why FLOPs can mislead. Other major costs are kernel-launch overhead when a model has many tiny operators, data transfer (host↔device copies, image decode), and synchronization stalls. You reduce latency by fusing operators, quantizing to cut bandwidth, optimizing the batch-1 path, and removing host-side stalls.