GANs: The Forger and the Detective

The Forger and the Detective

Picture a counterfeiter in a back room, painstakingly printing fake banknotes, and a detective at the bank whose entire job is to catch forgeries. The first fakes are laughable — wrong colour, blurry seal, no watermark — and the detective spots them instantly. But the forger learns from every rejection: each time a note is caught, they work out what gave it away and fix it. The detective, in turn, gets sharper, hunting for ever subtler tells. Round after round the fakes improve and the detective grows pickier, until one day the counterfeit is so perfect that the detective can do no better than flip a coin. That escalating duel is the whole idea behind a generative adversarial network.

In the previous guide, the VAE generated images by reconstructing them and measuring success pixel by pixel against the original. That pixel-matching loss is exactly what made VAE samples look soft and blurry: when the model is unsure which sharp picture is right, averaging many plausible pixel values minimises the error — and the average of many sharp pictures is a blur. A GAN throws that loss away entirely. There is no instruction to match any specific target image. Instead, quality is judged by a second network — a learned critic — that asks a single question: does this look real? A blurry image is an easy giveaway, so the forger is pushed toward crisp, decisive images, because those are the ones that fool the critic.

The counterfeiter is a neural network called the generator, and the detective is a neural network called the discriminator. Crucially, they are trained together, in opposition. The generator never sees a real image directly; it learns only from how the discriminator reacts to its fakes. The discriminator learns from a mix of real photos and the generator's latest forgeries. Their fortunes are linked: each one's improvement makes the other's job harder, and that mutual pressure is what drags both toward mastery.

The Generator: Turning Noise into Pictures

Let's open up the forger. The generator has a strange and wonderful job: it must conjure a full image out of pure randomness. Its input is not a photo, not a sketch, not a label — it is a short list of random numbers drawn from noise. We call this list the latent vector, written z. From this seed of randomness the generator grows a complete picture, expanding the handful of numbers into the thousands or millions of pixels of an image through layer after layer of computation.

Noise in, image out — the generator's entire contract.

Read this as: to make a fake image, draw a random vector z, then push it through the generator G. The symbol z is the latent variable — the random noise vector, our seed. The notation z ~ N(0, I) means z is sampled from a standard normal distribution: each component is an independent random number centred at 0 with variance 1, exactly the prior we met with the VAE. The bold I is the identity covariance, a compact way of saying 'the components are independent and each has variance 1'. G is the generator network, and x_fake = G(z) is the image it outputs. The single most important thing to notice: z is the only input. Randomness goes in, an image comes out — there is nothing else feeding the generator, no target to copy, no real photo to lean on.

This is where a GAN's generator differs sharply from the VAE. The VAE had two halves: an encoder that squeezed a real image down into a code, and a decoder that rebuilt an image from that code. A GAN's generator is like the decoder alone — there is no encoder, ever. The generator only travels in one direction: noise → image. It never goes image → code, because it never looks at real images at all. Think of an artist who paints only from imagination and has never once copied from life: hand them different daydreams (different z vectors) and they produce different paintings, but they cannot, even in principle, take a real photograph and reproduce it — they have no eyes for the real world, only a brush.

Because z is random, every fresh draw gives a different latent vector and therefore a different image. One z might decode into a face with glasses, another into a face turned to the side. The generator's task is brutally demanding: not just some of its outputs, but every plausible z it could ever be handed must decode into something that looks real. In effect it is defining a whole generative model of images — a recipe that turns the simple bell-curve cloud of noise into the rich, structured world of believable pictures. Internally it works much like the CNNs from earlier tracks run in reverse: instead of pooling a big image down to a small summary, it uses upsampling and (transposed) convolutions to grow a small vector up into a big image, learning at each layer how to add finer and finer detail.

The Discriminator: A Trained Skeptic

Now meet the detective. Here is the good news: you already know exactly what the discriminator is. It is an ordinary binary image classifier — the same kind of CNN you built in earlier classification tracks, stacking convolutions and nonlinearities to boil an image down to a single decision. The only unusual thing is the two classes it sorts images into: not 'cat versus dog' or 'digit 0–9', but simply real versus fake.

One image in, one probability out.

This says the discriminator D takes any image x and returns a single number between 0 and 1. We interpret that number as a probability: D(x) is how strongly the discriminator believes 'x is a real image'. A value near 1 means 'I'm confident this is genuine'; a value near 0 means 'I'm confident this is a forgery'. So if the detective is doing its job well, it pushes D(x) toward 1 for every real photo x, and pushes D(G(z)) toward 0 for every fake the generator produces. Notice that G(z) is just an image like any other — the discriminator is not told which images are fakes; it has to decide for itself.

What does the discriminator learn from? A training set that is half real photographs drawn from the dataset, and half fresh fakes produced by the generator right now. That second half is what makes this unlike any classifier you have trained before: the 'fake' class is a moving target. As the generator improves, last week's obvious forgeries are replaced by this week's subtle ones, so the discriminator can never settle — it must keep adapting to an opponent that keeps changing its disguise.

The Minimax Game: Writing Down the Contest

Time to write the contest down precisely. A GAN is a two-player game, and like any game we can describe it with a single shared scoreboard — a value function V(D, G) — that one player tries to push up and the other tries to push down. This kind of opposed objective is called a minimax game: one side maximises, the other minimises, over the very same quantity. Let's build the scoreboard piece by piece before assembling it whole.

The discriminator earns points for being right. On a real image x, being right means outputting a high D(x), so we reward it with log D(x): this term is large (close to 0, its maximum) when D(x) is near 1, and plunges toward minus infinity as D(x) falls toward 0. On a fake image G(z), being right means outputting a low D(G(z)) — equivalently a high value of 1 − D(G(z)) — so we reward it with log(1 − D(G(z))). Averaging the first reward over all real images, averaging the second over all noise draws, and adding them, gives the discriminator's total score, which is exactly the value function.

The GAN value function: one term for real images, one for fakes.

Let's name every piece. The min_G max_D is the two-player tug-of-war: the discriminator D chooses its weights to maximise V (classify as well as possible), while the generator G chooses its weights to minimise V (wreck the discriminator's score). The term E_{x~p_data}[…] means 'the average over real images', where p_data is the true distribution of photographs in the dataset; inside it, D wants log D(x) large — it wants to call real things real. The term E_{z~p_z}[…] means 'the average over noise vectors', where p_z is the prior N(0, I) we draw z from; inside it, log(1 − D(G(z))) is large when D(G(z)) is small, i.e. when D correctly calls the fake fake. Here the two players pull opposite ways on the very same term: D wants 1 − D(G(z)) big (spot the fake), while G wants it small — G wants D(G(z)) near 1, the fake mistaken for real. That shared, contested second term is the adversarial loss at the heart of the generator-versus-discriminator duel.

Picture the extremes. Early on, the generator's fakes are garbage; the discriminator easily sets D(x) ≈ 1 on reals and D(G(z)) ≈ 0 on fakes, so both terms sit near their maximum and V is high — the detective is winning. As the generator improves, D(G(z)) creeps up toward 1, dragging log(1 − D(G(z))) down toward minus infinity and pulling V down — the forger is clawing the score back. The ideal endpoint is equilibrium: the fakes become statistically indistinguishable from reals, the discriminator can do no better than guess, and it outputs D ≈ 0.5 on everything. At that point neither player can improve by changing alone — the forger has won, and a coin flip is the best the detective can manage.

Training the Duel: Taking Turns

How do you actually train two networks that want opposite things? You let them spar, taking strict turns. You never update both at once: you freeze one, improve the other against it, then swap. One full iteration has two moves.

1. Update the detective. Freeze the generator's weights. Sample a batch of real images from the dataset and generate a batch of fakes from random z. Show both to the discriminator and update only its weights so it raises D on the reals and lowers D on the fakes — getting better at telling them apart.
2. Update the forger. Now freeze the discriminator's weights. Generate a fresh batch of fakes and pass them through the frozen discriminator. Update only the generator's weights so the discriminator's verdict on these fakes rises toward 'real' (using the non-saturating −log D(G(z))). The generator learns by borrowing gradients that flow back through the frozen detective.

The catch is balance, and it is delicate. The two players must improve in lockstep, like sparring partners of similar skill. If the discriminator becomes too strong too fast, it rejects every fake with total confidence — D(G(z)) ≈ 0 — and the gradient handed back to the generator vanishes; the forger gets a flat 'no' with no hint of how to improve, and learning stalls. If the generator races ahead, the discriminator's feedback becomes noise and quality drifts. Healthy training keeps the contest close.

# One training iteration of a GAN (minibatch SGD).
# G = generator, D = discriminator.
for each training step:

    # ---- Step 1: train the discriminator (G frozen) ----
    x_real = sample_batch(dataset)        # a batch of real images
    z      = sample_normal(batch, dim)    # noise z ~ N(0, I)
    x_fake = G(z).detach()                # fakes; detach so G is NOT updated here
    # D should output 1 on real images and 0 on fakes:
    loss_D = -(log(D(x_real)) + log(1 - D(x_fake))).mean()
    update(D, loss_D)                     # adjust D's weights only

    # ---- Step 2: train the generator (D frozen) ----
    z      = sample_normal(batch, dim)    # fresh noise
    x_fake = G(z)                         # do NOT detach: we want G's gradient
    # Non-saturating loss: push D toward calling the fake "real":
    loss_G = -log(D(x_fake)).mean()
    update(G, loss_G)                     # adjust G's weights only

Keeping that balance is exactly why GANs earned a reputation as some of the trickiest models to train. When the duel tips too far one way, training can collapse in spectacular and frustrating ways — the subject of the very next guide. For now, hold on to the core picture: two networks, taking turns, dragging each other uphill.

DCGAN: Making GANs Work with Convolutions

The minimax game is beautiful in theory, but the earliest attempts to play it on images were wildly unstable. The breakthrough that made image GANs reliably trainable was an architecture called DCGAN — Deep Convolutional GAN. It is less a new idea than a hard-won recipe: a set of concrete architectural rules that, together, keep the duel stable enough to actually converge. Treat it as practical engineering wisdom, each rule earned by trial and error.

1. Replace pooling with strided convolutions. Drop fixed max-pooling (downsampling) in the discriminator and fixed upsampling in the generator; use strided convolutions and transposed convolutions instead. Why: this lets each network learn its own up- and down-sampling rather than having it hard-coded, so it can choose how to shrink or grow the spatial size in a way that helps the task.
2. Use batch normalization in both networks. Normalise each layer's activations to a stable scale during training. Why: GAN losses swing violently as the opponent shifts, and batchnorm keeps activations from exploding or collapsing, smoothing the flow of gradients and making the whole duel far less likely to diverge.
3. Drop the fully-connected hidden layers. Go fully convolutional, top to bottom. Why: convolutions preserve spatial structure and use far fewer parameters than dense layers, which both stabilises training and keeps the model focused on local image patterns rather than memorising flat vectors.
4. Pick activations deliberately. Use ReLU inside the generator, LeakyReLU inside the discriminator, and tanh on the generator's final output. Why: LeakyReLU lets a small gradient through even for negative inputs, so the discriminator never fully starves the generator of signal; tanh maps the output into [−1, 1] to match images scaled to that range, giving crisp, well-bounded pixels.

Step back and the shape is clear: the DCGAN generator is essentially a classification CNN run in reverse. A normal classifier starts with a big image and, through strided convolutions, shrinks it step by step into a small, deep feature stack that ends in a class. The generator does the mirror image: it starts with a small, deep block reshaped from z and, through transposed convolutions, grows it step by step into a big, shallow image. The discriminator is just the ordinary classifier direction. Batch normalization, which you met when training deep classifiers, is the glue that keeps both of these tall stacks numerically healthy while they fight.

DCGAN did more than stabilise training — it revealed that the latent space had learned real structure. Researchers found you could do arithmetic on the z vectors themselves. Take the average latent vector for 'man with glasses', subtract the average for 'man without glasses', add the average for 'woman without glasses', then decode the result — and out comes a woman wearing glasses. The famous 'smiling woman − neutral woman + neutral man = smiling man' demonstrations showed that directions in z correspond to meaningful, smoothly varying concepts, not random noise. This latent structure is the theme the whole track keeps circling back to, and it is exactly what the final guide will exploit to give us deliberate, fine-grained control over what the generator paints.