flops

FLOPs (floating-point operations) count how many arithmetic operations a model performs for one forward pass — a hardware-independent estimate of "how much math" it does. It is the standard back-of-the-envelope measure of a model's compute cost, letting you compare architectures (a 4 GFLOP model versus a 40 GFLOP model) on paper before running anything on a real chip.

To count them, tally the multiplies and adds in each layer. A frequent source of confusion: many papers actually report MACs (multiply-accumulate operations) but loosely call them FLOPs, and 1 MAC = 1 multiply + 1 add = 2 FLOPs, so the two differ by a factor of two. A convolution layer costs about H_out · W_out · C_out · C_in · k · k MACs; a matrix multiply of an M×K by a K×N matrix costs M · N · K MACs. "FLOPs" in machine learning almost always means per single inference and counts only the multiply-adds in conv and linear layers, ignoring activations, normalizations, and data movement.

FLOPs is a poor predictor of actual latency and energy, and it is important to know why. Real speed depends on memory bandwidth — captured by arithmetic intensity in the roofline model — plus parallelism, kernel efficiency, and launch overheads, none of which a raw operation count reflects. Depthwise convolutions have very few FLOPs yet are memory-bound and run at low efficiency; two networks with identical FLOPs can differ severalfold in wall-clock time. Treat FLOPs as a rough sanity check and use measured latency and energy as ground truth.