activation recomputation

Backpropagation needs the forward-pass activations of every layer to compute its gradients, so the naive approach stores them all — and for a deep transformer with long sequences that activation memory dwarfs the weights. Activation recomputation breaks the assumption that you must keep them. Instead of saving every intermediate tensor, you save only a few checkpoints (typically the input to each transformer block) and, during the backward pass, recompute the discarded activations on the fly by re-running the forward math from the nearest checkpoint.

The trade is explicit: you pay one extra forward pass over the recomputed regions — roughly a third more compute for a transformer — in exchange for cutting activation memory from linear in depth to nearly constant per checkpoint. Selective recomputation refines the bargain by saving the cheap-to-store, expensive-to-recompute tensors (like the attention softmax output) and recomputing only the cheap-to-recompute ones, recovering most of the memory for a fraction of the overhead. It composes with every parallelism strategy and is essential to pipeline parallelism, whose in-flight micro-batches would otherwise pin too many activation sets at once.

Activation recomputation is the standard memory-for-compute knob in large training. The honest caveat is that the extra forward work directly lowers hardware utilization, so practitioners recompute only as much as the memory budget forces — full recomputation when desperate, selective when there is room.