From SIMD to SIMT: the GPU's quiet trick
You already met two ways to do the same operation on many data items. Plain SIMD makes the programmer think in vectors: one instruction explicitly names a wide vector register and the hardware applies it across lanes. The earlier guide showed the GPU as a throughput machine — a thousand interns each doing one tiny sum instead of one genius CPU racing through them one by one. But how does it actually run those thousand interns without a thousand separate control units? That is the question this guide answers, and the answer has a name: SIMT, single instruction, multiple threads.
The genius of SIMT is a change of viewpoint. Instead of writing one vector instruction, you write a small scalar-looking program — a kernel — that describes the work of one thread, as if it were the only thread in the world. You then launch thousands of copies of it, and each copy gets its own thread index so it knows which slice of data is its own. Add two arrays of a million elements? You write `c[i] = a[i] + b[i]` for a single `i`, launch a million threads, and let thread number `i` handle element `i`. The vectorness disappears from your code and moves into the hardware.
Here is the honest catch, and it is the whole point. Those thousands of threads are not truly independent under the hood. The hardware grabs them in fixed-size bundles and runs every thread in a bundle in lockstep, all executing the very same instruction at the very same time, just on different data. So it programs like many independent threads but executes like one wide SIMD instruction. That bundle is the unit everything in this guide turns on — and on NVIDIA hardware it is called a warp.
The warp: lockstep in bundles of 32
A warp is a group of threads — 32 of them on NVIDIA GPUs — that the hardware schedules and executes as one. They share a single program counter and a single instruction fetch: when the warp executes an add, all 32 threads add; when it executes a load, all 32 load. You can picture a warp as one SIMD instruction wearing 32 thread-shaped costumes. The reason the GPU bothers is brutally economical. Fetching and decoding an instruction is expensive, so amortizing one fetch over 32 lanes means the chip spends almost all its transistors on arithmetic and almost none on control — the exact opposite of the out-of-order CPU you studied earlier.
Warps are also how the GPU hides memory latency, and this is its second superpower. When one warp issues a load that must travel to off-chip memory — hundreds of cycles away — the scheduler does not stall waiting for it. It simply parks that warp and switches to another warp that is ready to run, with zero switching cost because every warp's state already lives in the giant register file. With enough warps in flight, there is always one ready to compute while others wait on memory, so the arithmetic units stay busy. A CPU fights latency by predicting and speculating; a GPU just drowns it in spare warps.
Branch divergence: the if-statement that costs double
Now the crack in the foundation. The 32 threads of a warp share one program counter, so they can only ever be at one instruction at a time. That is fine until your kernel hits an `if` whose condition is true for some threads and false for others. The threads no longer agree on where to go — and a single program counter cannot point two places at once. This is branch divergence, and it is the single most important performance idea in GPU programming.
The hardware's honest, slightly painful solution is to run both paths, one after the other, masking off the threads that should not be active. First it runs the `then`-side with the threads whose condition was true switched on and the rest switched off (their results thrown away). Then it runs the `else`-side with the masks flipped. Only after both sides finish do the threads reconverge and march on together. So a warp that splits down the middle pays for both branches in sequence — the work doubles, and the lanes that are masked off are sitting idle, burning time on nothing.
Warp of 8 lanes (real warps are 32). Each runs:
if (x > 0) { A; } else { B; }
lane: 0 1 2 3 4 5 6 7
x > 0 ? T F T T F F T F
step 1 A: [A] . [A] [A] . . [A] . <- 'else' lanes idle
step 2 B: . [B] . . [B] [B] . [B] <- 'then' lanes idle
\__________ both run, in sequence __________/
Result: this warp takes time(A) + time(B), not max of the two.
A warp where ALL lanes agree runs only one side -> no penalty.Two reassurances keep this honest. First, divergence is per warp, not global: if all 32 threads in a warp happen to agree on the branch, the warp runs only the taken side at full speed and pays nothing. The damage is purely when threads within one warp disagree. Second, this is not a branch prediction problem like on a CPU — the GPU does not guess a direction and roll back; it deterministically runs both sides. The cost is not a misprediction flush, it is plain serialized work. Loops with data-dependent trip counts diverge the same way: the warp keeps looping until the last thread is done, while the early finishers sit idle.
Writing for the grain of the machine
Knowing the warp lets you write code that flows with the grain of the hardware instead of against it. The first lesson is to arrange data so that threads in the same warp take the same branch. If you must split work by some condition, try to make the split fall on warp boundaries — for example, sorting elements so all the 'true' cases sit together — so each warp is uniform and never diverges. Tiny, balanced `if`s where both sides are cheap often cost less than the divergence to avoid them, so measure rather than fear every branch.
The same warp lens explains why memory access pattern matters so much. When the 32 threads of a warp read 32 neighbouring addresses, the hardware can fuse them into one wide memory transaction — this is coalescing, and it is spatial locality wearing GPU clothes. When the threads instead read scattered addresses (a gather), the warp's one load fractures into many separate transactions and throughput collapses. So 'threads in a warp should touch adjacent data' is the GPU cousin of the cache-friendly-code lesson from the memory rung: the rule is the same shape, only the unit changed from a cache line to a warp.
Where this fits, and what comes next
Step back and SIMT stops being jargon. It is one honest bargain: let the programmer write simple per-thread scalar code, and let the hardware reclaim SIMD efficiency by running threads in lockstep bundles. The CUDA programming model is just the vocabulary for that bargain — threads grouped into blocks, blocks into a grid, every thread handed an index. Underneath, those threads ride warps, the warps hide latency by swapping for one another, and the whole machine pours its transistors into arithmetic because the SIMT structure spares it from a control unit per thread.
- Write a kernel as scalar per-thread code; launch thousands of copies, each with its own index — that is the SIMT viewpoint.
- The hardware runs threads in lockstep bundles (warps, 32 on NVIDIA): one fetch, one PC, many data.
- An if-statement that splits a warp diverges: both sides run in sequence with half the lanes masked off, so the work doubles.
- Write for the warp: keep threads in a warp on the same branch and reading neighbouring memory (coalescing).
Carry one honest sentence forward: a GPU is not a faster CPU, it is a different bargain. It is breathtaking when thousands of threads agree on what to do and touch tidy, neighbouring data, and it is genuinely poor at serial, branchy, latency-sensitive code where warps diverge and there is no data-level parallelism to feed them. The final guide of this rung makes that trade explicit — pairing the GPU's throughput against the CPU's low-latency strength, with the roofline idea to tell you which one a given workload will actually run faster on.