Training GANs That Actually Work

Why GANs Fight Back: The Instability Problem

Up to now in this track, training a network meant one thing: pick a loss, then push it downhill until it stops dropping. A generative adversarial network throws that comfort away. It pits two networks against each other — a generator trying to fake convincing images and a discriminator trying to catch the fakes — and there is no single loss to minimize. Instead the two are chasing a fragile balance point, an equilibrium where the generator's fakes are good enough that the discriminator can do no better than a coin flip. The trouble is that equilibria are easy to overshoot and easy to miss entirely.

What does this feel like in practice? With a normal network the loss curve slides down and flattens — you can watch it converge. With a GAN the two adversarial losses usually oscillate forever: the generator's loss drops while the discriminator's rises, then they swap, again and again, like a tug-of-war that never ends. Sample quality swings with them — gorgeous faces one epoch, melted nightmares the next. And sometimes training simply diverges: one player overpowers the other so completely that the gradients explode or vanish, and the run is dead. None of these are bugs in your code; they are the native behaviour of a two-player game.

The rest of this guide is a guided tour of the specific ways this balance breaks, paired with the fix for each. We start with the most infamous failure — mode collapse, where the generator gets lazy. Then we dig into the deeper mathematical reason GANs give bad training signals early on, which leads naturally to the Wasserstein GAN, the single most important stabilizing idea. After that comes a toolbox of smaller practical tricks and how to actually measure success. We finish with conditional GANs, the upgrade that lets you tell the generator what to make — and the bridge to the final guide.

Mode Collapse: When the Forger Gets Lazy

Imagine a counterfeiter whose only job is to slip fake banknotes past one inspector. Through trial and error the forger discovers a single note — say, a particular fake 20 — that the inspector waves through every single time. What does a rational forger do next? Not branch out into twenties, fifties, and hundreds. They print that one magic 20 over and over. This is exactly mode collapse: the generator discovers a handful of outputs (sometimes literally one) that reliably fool the detective, and it stops bothering to produce anything else.

Why does the game actively reward this laziness? Look closely at what the generator is optimized for: at every step it only needs to fool the discriminator right now. Nothing in its loss says 'cover the full variety of real data.' If the discriminator currently has a blind spot — one region of image-space it scores as very real — the fastest way to lower the generator's loss is to pile all its output into that blind spot. The generator has no memory of the diversity it used to produce and no penalty for abandoning it. It is myopically exploiting the discriminator, not honestly modelling the world.

Diagnosing it is straightforward once you know the tell. Feed the trained generator many different random noise vectors and look at the outputs side by side. A healthy generator gives you a varied gallery; a collapsed one gives you near-identical images no matter how you change the input — the noise goes in, but the variety does not come out. A partial collapse is subtler: you might get only frontal faces and never a profile, or a digit-generator that produces 1s, 7s and 9s but never a single 4.

Recall the manifold idea from the first guide in this track: real images of a given kind don't fill all of pixel-space — they lie on a thin, curved surface (a manifold) inside it, and 'generating' means learning to land anywhere on that surface. In that language, mode collapse means the generator has learned to land on only a tiny patch of the real manifold. It produces points that are on the surface (so they look individually real), but it has thrown away almost all of the surface's extent. Realistic-but-not-diverse is the signature of collapse.

The Root Cause: When Distributions Don't Overlap

Mode collapse is the visible symptom; now let's go under the hood to the deeper instability, because it explains why the original GAN loss is so hard to train. Early in training the generator is terrible — its fakes look nothing like real images. So the fake distribution and the real distribution sit in almost completely separate corners of image-space, barely overlapping. Against two such well-separated piles the discriminator has an easy job: it can draw a clean boundary that classifies real as real and fake as fake with near-perfect, near-100%-confident accuracy.

Here is the cruel twist. The generator improves by following the gradient that flows back through the discriminator — that signal is its only compass for which direction makes a fake look more real. But once the discriminator is essentially perfect and supremely confident, its output is pinned at 'fake = 0' across the entire region where the generator currently lives. A function that is flat at 0 has (almost) zero slope, so the gradient handed back to the generator is vanishingly small. It is told 'wrong' with total certainty, but given no hint about which way is warmer. Imagine a teacher who marks every answer 'wrong' and refuses to say whether you were close — you would have no idea how to improve. A discriminator that is too good is exactly that unhelpful teacher.

There is a precise way to say this. The standard GAN objective, when the discriminator is trained to optimality, is mathematically equivalent to pushing the generator to minimize the Jensen–Shannon divergence between the fake and real distributions — a particular measure of how different two probability distributions are. The fatal property of that measure: when the two distributions barely overlap, it saturates to a constant (its maximum value) and stops changing. A quantity stuck at a constant has zero gradient, so as the supports drift apart the loss tells the generator nothing about how to bring them together. This is the mathematical root of the flat-gradient problem, and it is also fertile ground for mode collapse, since a confused, low-signal generator readily slides into the cheap win of repeating one good output.

Wasserstein GAN: A Smoother Distance to Learn

The Wasserstein GAN (WGAN) replaces the troublesome distance with a far gentler one. Picture each distribution as a pile of dirt sitting on the ground: the real-data pile in one shape, the generated pile in another. The Wasserstein distance — nicknamed the earth-mover's distance — is the least total work needed to reshape the fake pile into the real pile, where 'work' means how much dirt you move multiplied by how far you push each shovelful. Two piles far apart cost a lot to merge; two piles nearly on top of each other cost almost nothing. The Wasserstein GAN trains the generator to shrink this earth-moving cost.

The magic is what happens when the piles don't overlap at all — the exact case that broke the original loss. Jensen–Shannon divergence just reports 'maximally different' and flatlines. The earth-mover distance instead reports an actual number that grows smoothly with how far apart the piles are: nudge the fake pile a little closer and the cost drops a little, every time. That smooth, always-informative response is precisely the non-zero gradient the generator was starving for. The generator now always knows which way is 'warmer,' even on the very first step.

The WGAN value the critic D tries to maximize and the generator G tries to minimize.

Let's read this slowly. D is no longer a probability-spitting detective; in WGAN we call it a critic, and it outputs a plain real-valued score — 'how real does this look' — with no 0-to-1 sigmoid squashing it. The first term is the average of D(x) over real images x drawn from the data distribution p_data: simply the critic's mean score on real photos. The second term is the average of D(G(z)) over noise vectors z drawn from the prior p_z: its mean score on fakes, where z is a random noise vector and G(z) is the image the generator builds from it. W is the gap between those two averages. The critic tries to maximize the gap — score real high, fakes low — while the generator tries to minimize it by making fakes that earn high scores. Concretely, if the critic gives real photos an average of +8 and current fakes an average of −5, then W is about 13; as the generator improves and its fakes start scoring, say, +6, the gap shrinks toward 0 — and that shrinking number is the generator's progress. A welcome bonus falls out of this: because the score is a genuine distance estimate rather than a saturating probability, its value actually tracks sample quality, so for the first time the loss number trends downward in step with the images getting better — a real progress meter instead of meaningless oscillation.

There is one catch that makes the earth-mover interpretation valid: the critic must be 1-Lipschitz. In plain words, the critic is not allowed to change its score too fast as the input changes — if two images are close, their scores must be close; the score can rise at most one unit per unit of input change. Without this leash the critic could blow its scores up toward plus or minus infinity to make the gap meaningless, and the smooth-distance guarantee evaporates. The first WGAN enforced it crudely by weight clipping (forcing every weight into a tiny range like −0.01 to 0.01). The better, now-standard method is a gradient penalty: add a term to the critic's loss that punishes it whenever the magnitude of its input-gradient strays from 1. That softly pins the critic to the 1-Lipschitz condition, keeps the earth-mover meaning honest, and is what keeps the gradients smooth and trainable.

# One WGAN-GP step. The critic is updated several times per generator update.
for _ in range(n_critic):                 # e.g. 5 critic updates per G update
    z    = sample_noise(batch)            # latent vectors   z ~ p_z
    real = sample_data(batch)             # real images      x ~ p_data
    fake = G(z)                           # generator images G(z)

    # Critic wants real scores HIGH and fake scores LOW -> maximize the gap,
    # so we minimize the NEGATIVE gap.
    gap = critic(real).mean() - critic(fake).mean()

    # Gradient penalty: pull the input-gradient norm toward 1 (the 1-Lipschitz leash)
    x_hat = interpolate(real, fake)       # random points between real and fake
    grad  = input_gradient(critic, x_hat) # d critic(x_hat) / d x_hat
    gp    = ((grad.norm(dim=1) - 1.0) ** 2).mean()

    critic_loss = -gap + lambda_gp * gp   # lambda_gp ~ 10
    update(critic, critic_loss)

# Now update the generator: it wants its fakes to score HIGH
z = sample_noise(batch)
gen_loss = -critic(G(z)).mean()
update(G, gen_loss)

A Toolbox of Stabilization Tricks

The Wasserstein GAN tames the biggest source of instability, but seasoned practitioners reach for a small toolbox of complementary tricks. None is magic alone; together they smooth out the remaining wobble. Here are the ones that earn their keep, each with the reason it helps.

1. One-sided label smoothing — instead of training the discriminator with the hard target '1.0 = real,' use a softened target like 0.9. Pinning it at perfect 1.0 invites the over-confidence we saw in the root-cause section (saturated outputs, dead gradients); a slightly softer target keeps the discriminator humble and its gradients alive. 'One-sided' means you soften only the real label, not the fake one (which stays 0).
2. Spectral normalization — another, very clean way to bound the discriminator. It rescales each layer's weight matrix by its largest singular value, which caps how much that layer can amplify its input and so controls the network's Lipschitz constant. It pursues the same 'don't let the critic change too fast' goal as the gradient penalty, but achieves it by construction rather than by an added loss term, and it plays nicely with both standard and Wasserstein losses.
3. The two-timescale update rule (TTUR) — give the generator and discriminator different learning rates (commonly a faster one for the discriminator). Because the two are co-adapting, letting one settle a little ahead of the other keeps the chase from spiralling, and it comes with theory suggesting it helps the pair converge.
4. Balancing update frequency — closely related, you also choose how often each player steps. WGAN, for instance, typically updates the critic several times (often five) per single generator update, so the critic stays a good-enough judge to give the generator a trustworthy signal. Too few critic steps and the signal is noisy; too many and the generator falls behind.

Now the harder question: how do you even know it's working? We've established that a single adversarial loss number is untrustworthy — it oscillates and, in a plain GAN, doesn't track quality at all. So the field judges generators with dedicated metrics that ask the two things we actually care about: do the generated images look real, and are they varied (no mode collapse hiding in the average)?

Fréchet Inception Distance (FID) is today's default. The idea: pass both your real images and your generated images through a pretrained image network (an 'Inception' classifier) and read off a high-level feature vector for each — a compact numeric summary of what's in the picture. Now you have two clouds of feature vectors, one real, one fake. FID measures the distance between those two clouds, treating each as a bell-shaped blob and comparing both their centres and their spreads. Lower is better: a low FID means the fake feature-cloud sits right on top of the real one — generated images that are both realistic and as varied as the real set. Crucially, if the generator mode-collapses, its feature cloud shrinks, the spreads stop matching, and FID rises — so FID punishes the failure a plain loss hides.

The older Inception Score (IS) uses the same pretrained network but asks a cleverer question with no need for real images at hand. For each generated image it looks at the classifier's label prediction: a realistic image should get a confident, peaked prediction ('clearly a dog'), while the collection of all generated images should span many different labels ('dogs and cats and cars and...'). High IS rewards exactly that combination — sharp per-image confidence plus broad variety across the batch. Its blind spot is that it never looks at your real data, so it can be fooled; that's why FID, which compares directly to real images, has largely become the gold standard. In practice you report both, and you still eyeball the samples.

Conditional GANs: Telling the Generator What to Make

So far the generator is a slot machine: feed it random noise and it returns a realistic image — but you can't ask for a specific one. The conditional GAN (cGAN) fixes that by handing both networks an extra input: a condition y, such as a class label like 'cat.' Now the generator turns noise plus a request into an image, and you can steer it — set y to 'cat' and you get cats, set it to 'dog' and you get dogs, all from the same trained model.

Both players receive the condition y: the generator builds from it, the discriminator judges against it.

Reading the two pieces: x_fake = G(z, y) says the generator now takes two inputs — the random noise z (which provides variety, so you still get many different cats) and the condition y (which fixes what kind of image to make) — and produces the fake image x_fake. And D(x, y) says the discriminator also sees the condition: it judges the image x together with the label y it's supposed to match. In a digit generator, y might be the one-hot code for '7'; z then decides whether you get a slanted 7 or a crossed one.

Why must both players get y? Suppose only the generator saw the label and the discriminator did not. Then the discriminator's only question is 'does this look like a real digit?' — it never checks whether the digit matches the request. The generator could happily produce a beautiful '3' when you asked for a '7' and pay no penalty, because the only judge doesn't care about the label. The condition would be silently ignored. By feeding y to the discriminator too, you change its question to 'is this a real image and does it match y?' A '3' offered under the label '7' is now flagged as fake — a mismatched pair — which forces the generator to actually obey the condition.

Make it concrete with the classic demo: a cGAN trained on handwritten digits gives you a dial from 0 to 9. Set it to '4' and resample the noise, and you get an endless variety of distinct, realistic 4s; set it to '8' and you get 8s — control and diversity at once. And the condition need not be a class label. If y is itself an image — a sketch, a segmentation map, a daytime photo — the conditional GAN learns to translate one picture into another. That single generalization is the seed of the image-to-image translation models, pix2pix and CycleGAN, that the final guide in this track builds on top of everything you've now learned about keeping training stable.