a warp

On a GPU, threads do not run truly one by one — they run in small platoons that march together, all stepping on the same beat. A warp is one such platoon: a fixed-size group of threads (32 on NVIDIA GPUs; AMD's equivalent of 64 is a wavefront) that the hardware schedules and executes as a single unit. Picture a rowing eight where everyone must pull on the same stroke: a warp is a crew of threads that fetch and execute the same instruction together, each pulling on its own slice of data.

Why bundle threads at all? Because sharing makes the hardware cheap. The 32 threads of a warp share one instruction-fetch and one decode unit; one instruction is fetched per cycle and all 32 lanes execute it on their own registers. This is exactly how a SIMT machine turns a comfortable thread programming model into efficient SIMD execution underneath. The warp is also the unit of scheduling: when a warp stalls waiting for slow memory, the hardware instantly swaps in another ready warp to keep the arithmetic units busy — GPUs hide memory latency by keeping many warps in flight rather than by big caches, which is why a GPU wants thousands of threads available, not just a few.

Two honest consequences for a beginner. First, because a warp shares one instruction stream, a branch that splits the warp (some threads true, some false) forces the hardware to run both sides one after the other with half the threads idle each time — branch divergence, and it is most painful within a warp. Second, the warp size is a real, fixed number you should respect: launching 33 threads still occupies two warps (one full, one with a single active thread and 31 wasted lanes), so good GPU code organizes work in multiples of the warp size. The warp is the seam where the friendly thread abstraction meets the unforgiving SIMD hardware.