data parallelism (LLM)

Data parallelism is the simplest way to train across many devices: put a full copy of the model on each one, then split the batch so every device processes a different slice of the data. Each computes gradients on its own slice, the devices average their gradients together, and all copies apply the same update — staying perfectly synchronised. More devices means a larger total batch processed per step, so training finishes sooner.

The averaging step is a collective operation called all-reduce, and it is the main overhead: every device must share its gradients with all the others on each step. The classic limitation is memory — every device holds the entire model, gradients, and optimizer state, which is impossible once models grow huge. Sharded variants such as fully-sharded data parallel and ZeRO fix this by splitting those states across devices and gathering each piece only when needed.

Because of that simplicity and the sharded upgrades, data parallelism is the backbone almost every large run is built on, then combined with tensor and pipeline parallelism to handle models too big for a single device to hold even one copy of.