From Recognizing to Imagining: What It Means to Generate an Image

Two Ways to Look at Data: Discriminative vs Generative

Stop and notice something about every computer-vision track you have finished so far. Image classification asked 'is this a cat or a dog?'. Object detection asked 'where are the cars in this street?'. Segmentation asked 'which exact pixels belong to the road?'. Different shapes, same underlying question: given this image, what is in it? You always handed the machine a picture and asked it to read off an answer. Tasks of this kind are called discriminative — they discriminate, in the old neutral sense of 'tell apart', between possibilities the image might contain.

Now flip the question on its head. Instead of 'what is in this image?', ask 'what does a plausible image even look like in the first place?' That is a generative question, and it is the heart of this whole track. Here is the analogy to keep in your pocket: a discriminative model is an art critic. Show the critic a painting and they will tell you it is a Monet, or that the perspective is off — they judge. A generative model is the artist. The artist can sit down at a blank canvas and produce a brand-new painting that never existed before. Judging and creating are profoundly different skills, and we are now asking the machine to make exactly that jump.

What a classifier learns, versus what a generator learns.

Let us read those two expressions slowly, because the gap between them is the entire reason this track exists. Here x is an image (think of it as the whole grid of pixels), y is a label (like 'cat'), and p(\cdot) means 'the probability of'. The left side, p(y \mid x), reads 'the probability of label y given the image x' — the vertical bar means 'given that we already have'. That is exactly what every classifier you built was estimating: hand it the pixels, it tells you how likely each label is. The right side, p(x), reads 'the probability that the image x exists at all' — how likely this particular arrangement of pixels is to be a real, natural image rather than random static. Modeling p(x) means capturing what makes an image look realistic in the first place: that skies are blue-ish on top, that faces have two eyes above a nose, that edges are usually crisp. A classifier never needs to know any of that; it only needs a boundary that separates cats from dogs. A generator needs the whole shape of reality.

The Space of All Images and Its Hidden Knobs

To build intuition for p(x), picture the space of all possible images. A 256×256 colour image has 256 × 256 pixels, each with a red, green and blue value — that is 196,608 numbers. You can think of one image as a single point whose address has 196,608 coordinates. The 'space of all images' is the unimaginably huge room containing every possible such point: every photo ever taken, but also every screen of random colour-noise you could ever generate. That room is the stage on which p(x) lives.

Here is the crucial fact: almost every point in that room is meaningless. If you set those 196,608 numbers at random, you get TV static — never a face, never a beach, never anything. Real, natural images are vanishingly rare in the space, and they are not scattered evenly. They cluster together on a thin, curved sheet floating inside the giant room. Mathematicians call such a sheet a manifold — a low-dimensional surface bending through a high-dimensional space, the way a crumpled sheet of paper is a flat 2D thing living inside 3D. The whole game of generation is to learn to stay on that sheet: to produce points that land on the manifold of real images instead of falling off into the ocean of noise around it.

But here is the hopeful twist. Even though the room has 196,608 dimensions, the manifold of real images is much thinner than that — you don't need that many independent knobs to describe a realistic picture. Think of a video-game character creator. On screen the character is thousands of pixels, but you, the player, only ever touch a handful of sliders: face width, hair colour, eye size, how big the smile is. Slide those few and every one of the thousands of pixels updates in a coordinated, sensible way; you can never accidentally produce static, because the sliders only ever travel along the manifold of valid faces. Those few hidden controls are what we call latent variables — the small set of underlying factors that secretly govern everything you see.

And now you can feel the whole plan of this track forming. If we could discover those hidden knobs — if a generative model could learn the handful of latent variables that ride along the manifold — then generating an image would collapse into something almost trivial: set the knobs to some values, turn the crank, and out comes a fresh, realistic image. Every method we study (autoencoders, VAEs, GANs) is ultimately a different strategy for finding those knobs and making the crank-turning reliable. Hold on to that image; it is the spine of everything that follows.

Autoencoders: Squeeze It Down, Build It Back

So how could a machine discover the hidden knobs on its own, when nobody knows in advance what they should be? The first concrete answer is a beautifully simple machine called an autoencoder. Its trick is to force the network through a deliberate bottleneck and then dare it to recover what it lost. If the network can rebuild the image after being squeezed through a tiny opening, then whatever survived the squeeze must be the truly essential description of that image — the knobs themselves.

An autoencoder has three parts in a row. First the encoder: it takes the full image and progressively crushes it down into a short list of numbers. Second the bottleneck: that short list, called the code or latent vector, is the narrowest point — maybe just 32 or 128 numbers standing in for nearly 200,000 pixels. Those numbers are the network's own private latent variables. Third the decoder: it takes the tiny code and expands it back out, trying to reconstruct the original image. The whole thing is trained end to end with one goal: make the output match the input. Crucially, nobody tells the encoder what the knobs should mean. We never label one number 'hair colour'. The bottleneck simply isn't wide enough to memorize the picture, so the only way to win is to keep what actually matters and throw away the rest — and 'what actually matters' is exactly the set of hidden factors we were hunting for.

You already have all the parts to build this. Remember from the CNN tracks that convolutional layers with stride or pooling downsample — they shrink the spatial size of a feature map while distilling its content. That is precisely the encoder: a CNN that downsamples an image into a code. The decoder is just the same idea run backwards — layers that upsample, growing a small code back up into a full-resolution image (a 'transposed' or 'up-' convolution). An analogy makes it tangible: describing a friend's face to someone over the phone forces you to compress it into a few words ('round face, curly black hair, big smile') — that is the encoder squeezing an image into a code. The person on the other end is a sketch artist who hears those words and draws the face back — that is the decoder reconstructing the image from the code.

# An autoencoder as three function calls, trained to copy its own input.
# encoder and decoder are both small CNNs (downsample / upsample).

z       = encoder(x)        # x: the input image -> z: the tiny latent code (the 'knobs')
x_hat   = decoder(z)        # z -> x_hat: the network's reconstruction of x

loss    = mean((x - x_hat) ** 2)   # per-pixel squared difference, averaged
# Training nudges encoder+decoder so x_hat looks more and more like x.
# The bottleneck (z is tiny) is what forces it to learn a useful, compact code.

The reconstruction loss: how far the rebuilt image is from the original.

Let us unpack this symbol by symbol, because it is the engine that trains the whole machine. x is the input image. \text{encode}(x) runs it through the encoder to produce the latent code, which we call z. Then \text{decode}(z) = \hat{x} — read 'x-hat' — is the decoder's reconstruction, its best attempt to rebuild x from the code. The expression \lVert x - \hat{x}\rVert^2 means: subtract the rebuilt image from the original pixel by pixel, square each difference (so positive and negative errors both count as error), and add them all up across every pixel. \mathcal{L} is the resulting loss — a single number measuring total wrongness. In plain words: the network is rewarded for making its rebuilt image match the original, and punished for every pixel it gets wrong. A tiny worked example: if a single pixel should be brightness 0.8 but the decoder outputs 0.5, that pixel contributes (0.8 - 0.5)^2 = 0.09 to the loss; get it exactly right and it contributes 0. And notice — this is nothing new. It is exactly the mean-squared error you already met for regression, just applied per pixel and summed over the image. Training simply pushes \mathcal{L} downward, which is the same as saying 'make the reconstruction better'.

Why a Plain Autoencoder Can't Dream

Here comes the disappointment that powers the rest of this track. A trained autoencoder is genuinely impressive at reconstruction: feed it a real face, it squeezes it to a code and rebuilds a recognizable face. So you get an exciting idea — why not skip the encoder entirely, make up a random code out of thin air, hand it straight to the decoder, and let it dream up a brand-new face? You try it. And out comes garbage: smeared colours, broken shapes, nothing like a face at all.

Why does this fail? Because a plain autoencoder is only ever rewarded for one thing: rebuilding the specific images it was trained on. Nothing in the reconstruction loss ever tells the encoder where to place each code. So it scatters them wherever is convenient — this face's code lands over here, that face's code lands way over there, and the vast space between them is left empty. The result is a latent space riddled with holes and gaps: regions that correspond to no training image and that the decoder has never once been asked to handle. A code you pick at random almost certainly lands in one of those voids, and the decoder, faced with an input it has never seen, produces nonsense. There is no rule in the system that says what a valid code even looks like — the latent variables are organized only by accident.

Picture a library where books are shelved at completely random positions, with no catalogue system. If you already know the exact shelf and slot of the book you want, you can walk straight to it and pull it out — that is reconstruction, where the encoder hands you the precise code. But try to browse — pick a shelf at random and reach for whatever sits there — and you'll grab empty air or a meaningless jumble most of the time. A plain autoencoder is exactly that library: superb at retrieval when you have the call number, useless for wandering in and discovering something new. And generation is fundamentally an act of browsing — we want to reach into the space, grab any point, and find something meaningful there.

Two Roads to Generation: VAEs and GANs

We now have a sharp goal and a clear obstacle. The goal: learn the hidden knobs of the image manifold so that turning them produces realistic pictures. The obstacle: a plain autoencoder leaves the latent space full of holes, so random codes decode into garbage. The rest of this track follows two great families that fix this in opposite spirits. Both deserve a one-line teaser now, so you have a map in your head before the details arrive.

The first road is the variational autoencoder, or VAE — the subject of the very next guide. Its move is to keep the encoder-decoder shape you just learned, but add a rule of probability that forces the latent space to be tidy and gap-free: instead of mapping each image to one rigid point, it maps each to a small fuzzy cloud, and it gently presses all those clouds toward one shared, well-behaved region. Once the space is packed neatly with no holes, you can pick a random point, decode it, and reliably get a plausible image. The VAE makes 'turn the crank' actually work, in a principled, mathematically grounded way.

The second road is the generative adversarial network, or GAN — guides 3 and 4. It throws away the encoder and the reconstruction loss altogether and stages a contest between two networks. One, the generator, takes a random code and tries to paint a convincing fake image. The other, the discriminator — remember the art critic from the very first section? — tries to tell real images from the generator's fakes. They train against each other: every time the critic gets sharper, the forger is forced to get better, and round after round the fakes climb toward photorealism. No one ever writes down 'what makes an image realistic'; that knowledge emerges from the duel itself.

It is worth knowing their personalities in advance, because they are a genuine trade-off. VAEs are principled but blurry: their probabilistic tidiness gives smooth, well-organized latent spaces and stable training, but the averaging that keeps the space neat tends to soften fine detail, so samples can look a little washed out. GANs are sharp but temperamental: the adversarial game can produce stunningly crisp, photorealistic images, but training is famously fickle — it can oscillate, collapse, or refuse to converge, which is exactly why a whole guide ('Training GANs That Actually Work') is devoted to taming it. There is even a third road we will only name here: diffusion models, which generate by starting from pure noise and removing it step by step; they now power many state-of-the-art image generators and get their own later track. For this track, VAEs and GANs are the two summits we will climb.