Data-Level Parallelism: SIMD, Vector & GPUs

a vector processor

Think of an old-fashioned ditch-digging crew. A scalar processor is one worker with one shovel, digging one hole, then the next, then the next. A vector processor is a single foreman who gives one order — 'dig the whole row' — to a line of identical diggers who each take one spot and all dig together. One command moves a whole row of dirt. A vector processor is built around that idea: a single instruction commands an operation over an entire array (a 'vector') of numbers.

The machinery has two key pieces. First, vector registers — wide registers that each hold not one number but a whole list, say 64 elements. Second, deeply pipelined lanes — instead of waiting for one addition to finish before starting the next, the elements stream through an assembly-line adder so a new result pops out almost every cycle. A single vector-add instruction says 'add register V1 to V2, element by element, into V3', and the hardware grinds through all 64 elements with one instruction fetch, often pulling and storing the operands from memory in long efficient bursts. The classic Cray supercomputers of the 1970s and 1980s were built this way.

The honest distinction from the SIMD extensions you find in today's CPUs is subtle but real. SIMD extensions bake a fixed width into the instruction (add exactly 8 floats); a true vector instruction names a vector length the program can set, so the same code runs on hardware with wider or narrower vector units — this is the philosophy behind modern scalable vector ISAs like RISC-V's vector extension and ARM's SVE. Either way, the win and the limitation are the same family as all data-level parallelism: glorious on long, regular, independent arrays; useless when the data is short, scattered, or each step depends on the last.

A vector add of length 64: one instruction, vadd V3, V1, V2, computes V3[i] = V1[i] + V2[i] for i from 0 to 63. A scalar machine would need a 64-iteration loop with a load, an add, a store, and a branch each time — four-plus instructions per element instead of one instruction for all 64.

One vector instruction does the work of a whole scalar loop, and the deep pipelined lanes keep results streaming out.

A vector processor is not 'just a wide ALU'. Its real win is amortizing one instruction over many elements and streaming memory in long bursts; on short or scattered data those benefits vanish and a scalar core can be faster.

Also called
vector machinevector architecture向量機器向量架構