message passing (MPI)
/ MPI is said letter by letter, 'em-pee-eye' /
When workers cannot share a desk — they sit in separate buildings with no common whiteboard — the only way they can cooperate is to send each other notes: 'here are my numbers', 'I need yours'. Message passing is that style of parallel programming, and MPI, the Message Passing Interface, is the standard library that almost every large scientific simulation uses to do it. It is how a calculation is spread across the thousands of separate computers of a cluster or supercomputer, each with its own private memory.
In the MPI model you launch many copies of your program at once, called processes or ranks, numbered 0, 1, 2, and so on. Each rank runs the same code but on its own slice of the data, in its own private memory it cannot share. To exchange data they call explicit operations: a send on one rank paired with a receive on another moves a buffer of numbers between them; collective operations move data among the whole group at once, such as a broadcast (rank 0 ships a value to all), a scatter (hand each rank its piece), a gather (collect the pieces back), and the workhorse reduce / allreduce (combine one value from every rank — sum, max, and so on — into a single result). A typical step is: each rank computes on its local data, exchanges boundary values with its neighbours, and the loop repeats. The programmer is fully responsible for deciding who holds what and who talks to whom.
MPI matters because it is the lingua franca of large-scale computing: it is how codes reach hundreds of thousands of cores, and the model maps cleanly onto distributed-memory hardware. But its cost model is unforgiving and explicit. Every message has a fixed latency (the time to start it, regardless of size) plus a bandwidth term (proportional to its length), so many tiny messages are far worse than a few big ones. Communication is pure overhead — no science gets done while data is in flight — so the art of MPI programming is to partition the work so each rank has plenty to compute, communicates rarely and in bulk, and stays balanced with its peers. Communication cost, not flop count, usually decides whether a large MPI code scales.
To compute a global dot product across 1000 ranks, each rank multiplies its local slices and sums them into one partial number, then a single MPI_Allreduce with the sum operation combines all 1000 partials and hands the total back to every rank. Total network traffic: about 1000 numbers, in one collective call — versus disaster if each rank instead messaged every other rank pairwise.
Each rank computes locally, then one collective (allreduce) combines partial results across all ranks.
Every message carries a fixed start-up latency, so a code that sends many small messages can be dominated by latency even with tiny payloads — bundle communication into fewer, larger transfers. And mismatched sends and receives, or a wrong message order, cause deadlock: ranks waiting forever for each other.