shared vs distributed memory
Imagine a team writing one big report. In one office they all share a single whiteboard: anyone can read or change anything instantly, but they have to coordinate so two people do not scribble over the same spot at once. In the other arrangement the writers sit in separate buildings, each with their own private notepad, and they cooperate only by mailing notes back and forth. These are the two fundamental models of parallel computing — shared memory and distributed memory — and almost every parallel program is built on one, the other, or a blend of both.
In shared memory, several execution streams (threads) all see the same single address space — the same arrays in the same RAM. Communication is implicit: one thread writes a value, another simply reads it. This is the world of multicore CPUs and of OpenMP, where you take an existing loop and annotate it to split its iterations across threads. It is convenient, but it scales only as far as one machine's cores and shared RAM, and it brings hazards: two threads touching the same location need synchronisation, or you get race conditions. In distributed memory, each process has its own private memory and cannot see another's; to share data they must explicitly send and receive messages over a network. This is the world of compute clusters and of MPI. It is more work to program — you must partition the data and choreograph every exchange — but it scales to thousands or millions of cores across many machines, because you just add more nodes.
The distinction governs the whole shape of a parallel algorithm. Shared memory makes fine-grained, irregular parallelism easy but hits a ceiling at one node; distributed memory forces you to think about which data lives where and to pay an explicit cost for every byte that crosses the network, but it is the only way to reach the largest scales. Real supercomputer codes are usually hybrid: MPI between nodes to span the cluster, and OpenMP or GPU threads within each node to use its cores. Knowing which model you are in tells you what your bottleneck will be — synchronisation and memory bandwidth in the shared world, communication latency and load balance in the distributed one.
To sum a billion-element array on one 64-core node, shared memory wins: split the array among 64 threads, each sums its slice, then one final reduction combines them — no data is copied. To sum across a 1000-node cluster, each node owns its own chunk in private memory, sums it locally, then the partial sums are combined by MPI messages — the only data that crosses the network is 1000 numbers, not the billion.
Shared = one address space, implicit sharing; distributed = private memory, explicit messages.
Shared memory is easier to write but does not magically scale past one machine, and its convenience hides the trap of race conditions. Distributed memory scales but every shared value costs an explicit, latency-bearing message — so a poor data partition that forces lots of communication can run slower than the serial code.