the CUDA programming model
/ KOO-dah /
Suppose you have a job that is the same small task done a million times — colour a million pixels, add a million array elements. CUDA is a way to tell a GPU: 'here is the task for one element; now run a million copies of it.' You write one short function describing what one thread does to one piece of data, and you launch a huge grid of those threads with a single call. CUDA is NVIDIA's framework (language extensions plus libraries) for writing such programs; OpenCL is the cross-vendor equivalent with the same shape and different names. This entry is orientation only — the mental model, not the syntax.
The model has a clean three-level hierarchy you should picture. The function you write is a kernel — the per-thread program. When you launch it, you choose a grid of thread blocks, and each block holds many threads. Threads are cheap and numerous; each one computes its own global index and uses it to pick the data element it owns. Threads within the same block can cooperate: they can share a small, fast on-chip scratchpad memory and synchronize at a barrier, which is how they pool partial results. Under the hood each block's threads are chopped into warps and executed SIMT-style, so the comfortable thread picture sits on top of the warp hardware from the previous entries.
The honest orientation a beginner needs. CUDA makes launching a million threads look easy, but the model still obeys the GPU's nature: the threads must mostly do independent, regular work to go fast, branch divergence within a warp still costs you, and there is a memory hierarchy you cannot ignore — global device memory is large but relatively slow, the per-block shared scratchpad is tiny but fast, and using it well is most of the art. Above all, your data starts in CPU (host) memory and must be copied across to the GPU (device) before the kernel runs and copied back after; on small problems that transfer can cost more than the computation it enables. CUDA gives you the keys to the throughput machine, but it does not change what that machine is good and bad at.
Vector add in spirit: write a kernel where thread with global index i does c[i] = a[i] + b[i]; launch it as, say, 4096 blocks of 256 threads to cover a million elements. Each thread handles exactly one element; the runtime maps blocks onto the GPU's cores and forms warps for execution.
Write one thread's code, launch a grid of blocks of threads; the model is easy to enter but still obeys warp, divergence, and memory-transfer realities.
CUDA's easy thread launch hides hard truths, not removes them. Divergence, the global-versus-shared memory gap, and the host-to-device copy all still bite. Easy-to-write is not the same as fast — and a kernel only helps if the data was worth shipping to the device.