model profiling

Before you optimize a model you need to know where the time and memory actually go — and guessing is almost always wrong. Profiling measures the real per-layer (per-operator) execution time, memory use, and hardware utilization on the target device, turning a vague "the model is slow" into a concrete "this attention operator and that host-to-device copy are 70% of the time." It is the diagnosis that should guide every other optimization.

A profiler records, for each operator, its wall-clock time, call count, and memory allocation, and often hardware counters such as GPU SM occupancy, tensor-core utilization, memory-bandwidth percentage, and cache hit rates. The common tools are PyTorch's torch.profiler, NVIDIA Nsight Systems and Nsight Compute (nsys and ncu), TensorRT's per-layer timing, the TFLite benchmark tool, Chrome trace timelines, and roofline analysis to tell whether a given kernel is compute-bound or memory-bound. Watch host↔device transfers and idle gaps — the accelerator sitting idle — as carefully as you watch kernel time.

In practice: profile on the real target hardware with realistic input sizes and a proper warm-up; read the timeline, not just summed totals, so you catch synchronization stalls and CPU-side bottlenecks; and apply Amdahl's law by optimizing the largest contributor first. Re-profile after every change, because bottlenecks shift — kill the compute hotspot and you may suddenly become memory-bound or launch-bound instead.