wasserstein gan
Wasserstein GAN changes what "distance between the real and fake distributions" means, in order to fix the GAN's most painful symptoms: vanishing gradients and uninformative loss curves. The standard GAN effectively minimizes the Jensen–Shannon divergence, which gives no useful gradient when the real and fake distributions barely overlap (common early in training). WGAN instead uses the Earth-Mover (Wasserstein-1) distance — intuitively, the minimum total "work" to reshape one pile of dirt (the fake distribution) into another (the real one), where work equals mass times distance moved. This distance varies smoothly even when the distributions do not overlap, so the generator always gets a useful gradient.
By the Kantorovich–Rubinstein duality, the Wasserstein-1 distance equals the supremum, over all 1-Lipschitz functions f, of E over real x of f(x) minus E over generated x of f(x). WGAN parameterizes f by a neural critic (no sigmoid, outputting an unbounded real score, not a probability) and maximizes that difference; the generator minimizes the negative of E[f(G(z))], i.e. tries to raise the critic's score on fakes. The critic must be 1-Lipschitz (its output cannot change faster than its input). The original WGAN enforced this crudely by weight clipping — clamping weights to [−c, c] — which is sensitive and can underuse capacity.
WGAN-GP (Gulrajani et al., 2017) enforces the Lipschitz constraint far better with a gradient penalty: add a term λ times E of (the norm of the critic's input-gradient minus 1) squared, evaluated at points sampled along straight lines between real and fake samples. The benefits are substantial: much more stable training (less sensitivity to architecture and hyperparameters), strong resistance to mode collapse, and — uniquely useful — a loss that is a meaningful quality signal, since the critic's estimate of the Wasserstein distance decreases as samples improve. Practically the critic is updated several times per generator step. Note that WGAN-GP is incompatible with batch normalization in the critic (the penalty is per-sample), so use layer normalization instead.
WGAN does not magically fix everything — the Lipschitz constraint is only approximately enforced, the gradient penalty adds compute, and the FID of WGAN-GP is not always better than a well-tuned non-saturating GAN with spectral normalization (another, cheaper way to bound the Lipschitz constant). Its lasting contribution is the principle: choose a distance that gives good gradients and a readable loss.