The bottleneck was never compute — it was the trip
Earlier in this rung you met the uncomfortable arithmetic of the post-Moore era: transistors keep shrinking a little, but the easy clock and power gains ended, and the gap between a fast processor and slow memory — the memory wall — keeps widening. Now we follow that wall to its sharpest conclusion. In the classic von Neumann architecture, compute lives in the CPU and data lives in DRAM across a bus, and every instruction and every datum must squeeze through that one doorway — the von Neumann bottleneck. For a long time we treated this as a fact of life and built ever-cleverer caches to hide it. This guide is about giving up on hiding the trip and instead questioning whether the trip needs to happen at all.
Why does moving data dominate? Two numbers tell the story. First, bandwidth: a CPU core can crunch far more numbers per second than the memory channel can deliver, so for data-hungry work the processor spends most of its time waiting, not computing. Second, and more decisively, energy: on modern chips, fetching a word from off-chip DRAM costs on the order of a hundred times the energy of the arithmetic you then perform on it. Read that again — the addition is nearly free; the commute to fetch the operands is what burns the battery. When the data is huge and the math is simple — scanning a database, summing a giant vector, multiplying sparse matrices — you are paying a fortune in energy and time just to shuttle bytes back and forth through that single doorway.
Near-data computing: move the worker closer
The gentler half of the idea is near-data computing: don't move the memory into the processor, just put some modest compute near the data so it travels a shorter distance — or doesn't leave the chip at all. Think of it as a spectrum of how far the worker has to walk. At one end, the data sits in DRAM and the worker is the distant CPU. Near-data computing slides the worker down toward the data: maybe a small accelerator beside the memory controller, maybe logic inside the storage device itself.
A concrete, shipping example is computational storage: an SSD-class device with a small processor inside it. Suppose you have a billion records on disk and want only the few thousand that match a filter. The traditional path drags all billion records up through the bus into the CPU, where almost all are immediately discarded — a colossal commute for almost no kept work. With near-data filtering, you send the predicate down to the device, its little processor scans the records where they already live, and only the matching few thousand make the trip back. You moved a tiny program to the data instead of moving an ocean of data to the program. The win is largest exactly when the filter is selective — when most of what you fetched would have been thrown away.
Processing in memory: compute where the bits already are
The bolder half is processing-in-memory (PIM): put real arithmetic inside the memory itself, so for the right operations the data never leaves the chip it is stored on. DRAM is built as a grid of cells organised into banks and rows; to read, the chip activates a whole row at once into a wide internal buffer — thousands of bits in parallel — and only then squeezes a narrow slice of that out through the pins to the rest of the system. The pins are the bottleneck. The insight behind PIM is that inside the chip the data is enormously wide and cheap to access; it is only the journey out through the narrow pins that is expensive. So why not do some of the work on that wide internal data, right there, before it ever reaches the pins?
Where does a sum of a billion numbers happen?
von Neumann (classic): PIM (compute near the array):
+-----------+ narrow bus +--------------------------+
| DRAM | =====================> | DRAM bank |
| (data) | every number must | row buffer: 1000s wide |
+-----------+ cross the pins, one | +-----+ tiny ALU adds |
| slice at a time | | ALU | the row locally |
v | +-----+ |
+-----------+ +--------------------------+
| CPU | adds them (cheap math, |
| one ALU | but it WAITED forever) v only the small
+-----------+ partial sums leave the chip
Same answer. The PIM version barely uses the expensive doorway.PIM comes in a few flavours, and honesty matters here because the term gets used loosely. The most practical near-term form is processing-near-memory: small, ordinary ALUs placed in the logic layer of a stacked-memory package (recall from the previous guide how 3D stacking and a silicon interposer let a logic die sit right under the DRAM dies, sharing a fat, short connection like high-bandwidth memory). A more radical form is in-DRAM computing, which abuses the analog behaviour of the memory array — activating two or three rows together so the bitlines naturally compute a bitwise AND/OR, turning the storage array itself into a wide, dumb-but-massively-parallel arithmetic engine. The first is engineering you can buy soon; the second is mostly research, with real precision and reliability caveats.
Why now, and why it is hard
The idea of computing in memory is decades old — so why is it suddenly serious? Three forces converged, all of which you have already met climbing this rung. First, the technology to physically co-locate logic and DRAM finally exists and is affordable, thanks to advanced packaging — stacking, interposers, and the chiplets from the packaging guide. Second, the workloads driving everything now are exactly the memory-bound, low-intensity kind that PIM helps most: machine-learning inference, recommendation systems, graph analytics, and database scans all move oceans of data and do little math per byte. Third, with Dennard scaling dead, energy is the hard limit on every chip, and since the data commute dominates that energy, attacking the commute is now the highest-leverage thing an architect can do.
Now the honest caveats, because PIM is genuinely hard and the breathless version oversells it. (1) It is not a general-purpose CPU. The little engines near memory are good at simple, regular, parallel operations — add, compare, mask, accumulate — and bad at branchy, serial, irregular code, much as a GPU is. Workloads have to fit. (2) DRAM is fabricated on a process tuned for cheap dense storage, not fast logic, so logic built in it is slower and clumsier than logic on a normal processor die — there is a real manufacturing tension. (3) The hardest part is rarely the silicon; it is the software. We have decades of compilers, languages, and operating systems built on the von Neumann assumption that compute and memory are separate places. Deciding what to offload, keeping the CPU's caches coherent with results computed inside memory, and exposing all this to programmers are unsolved-at-scale problems.
The bigger shift, and the craft that endures
Step back and notice what PIM and near-data computing really represent. They are part of a broader rebellion against treating memory as a passive, dumb store on the far side of a bus. The whole memory side of the system is becoming active and varied: high-bandwidth memory stacked on the package, non-volatile memory and storage-class memory that blur the old hard line between fast-but-forgetful RAM and slow-but-permanent storage, and now compute migrating into the memory and storage themselves. The clean von Neumann picture — one CPU, one flat memory, one bus — is fragmenting into a rich, heterogeneous landscape where computation happens in many places, each chosen to sit near the data it works on.
And yet — this is the quiet lesson of the whole rung — the craft underneath does not change. Whether the era's technology is bipolar transistors or stacked DRAM, the architect's job is the same: understand where the time and energy actually go, make the common case fast, exploit locality, respect Amdahl's law, and weigh every gain against its cost in complexity, power, and the software burden it imposes. PIM is not a magic escape from the memory wall; it is one more disciplined, tradeoff-laden answer to the oldest question in computer architecture — how do we get the right data and the right compute to the same place at the right time? The technologies will keep changing. That question, and the careful judgement it demands, will outlast every one of them.