gradient noise scale

How large should your batch be? Too small and each gradient is mostly noise, so you waste steps; too large and you are spending compute to shave off noise that was already negligible. The gradient noise scale answers this by measuring where the crossover sits, the so-called critical batch size, from the ratio between the true signal in the gradient and the per-example noise around it.

Formally the noise scale is B_noise = tr(Sigma) over the squared norm of the true gradient G, where Sigma is the covariance of the per-example gradients. It can be estimated cheaply by comparing gradient norms at two different batch sizes, or across data-parallel workers. The interpretation is clean: for batch sizes up to roughly B_noise you get a near-linear reduction in the number of steps needed, while beyond it the returns diminish and you are mostly buying redundant gradient estimates.

The noise scale explains why bigger models, and later stages of any single training run, tolerate and even benefit from larger batches, since their gradients are relatively cleaner. It guides data-parallel scaling decisions and the joint tuning of learning rate and batch size. It should be read as a statistical estimate that drifts upward over training, not as a hard threshold to be enforced.