From Noise to Pictures: Why Diffusion Models Work

What it means to 'generate' an image

Most of the vision systems you have met so far are recognisers. You hand them an image and they hand back a label: 'cat', 'dog', 'a stop sign'. The arrow points one way — picture in, answer out. A generative model runs that arrow backwards. You give it (almost) nothing — a fistful of random numbers — and it hands back a brand-new picture that looks like it belongs in the real world. Nobody ever photographed that image; the model imagined it.

How can a machine imagine something it has never seen? Think of a skilled art forger. They do not memorise one painting and copy it stroke for stroke — that would just be a photocopy. Instead they study thousands of works until they have absorbed the style: how this painter loads the brush, mixes colour, places light. Having soaked up the pattern behind all those paintings, the forger can sit at a blank canvas and produce a convincing new piece that the original artist never made, yet that feels unmistakably theirs. A generative vision model is that forger, scaled up to millions of photographs.

Here is the one mental model to carry through this whole track. The set of all believable real-world images forms a probability distribution: photos of faces, cats, and beaches are highly likely; a screen of random static is wildly unlikely. To 'generate' an image is simply to draw a sample from that distribution — to reach into the space of all plausible pictures and pull one out. The model's whole job is to learn the shape of that distribution well enough to sample from it. The family of models we will study, which has dominated image generation since 2021, is the diffusion model.

The core trick: destroy, then rebuild

Sampling from 'the distribution of all natural images' sounds impossibly hard — that space is astronomically large and tangled. Diffusion models sidestep the difficulty with one beautifully simple idea, and if you remember nothing else from this track, remember this: break the hard problem into many tiny easy ones by first learning to destroy, then learning to rebuild.

First, the destroying half — the forward diffusion process. Start with a clean photo. Sprinkle a tiny bit of random noise onto it, so it gets very slightly grainier. Do it again. And again — hundreds or a thousand times. Each single step barely changes the picture, but the grain accumulates. After enough steps the photo is gone: what is left is indistinguishable from the snowy static of an untuned television, pure random noise with no trace of the original.

Now the rebuilding half — the reverse denoising process. We train a neural network to undo one step of that noising: given a slightly-noisy image, make it slightly cleaner. That single skill is the whole game. To generate a brand-new picture we start from a fresh sheet of pure random noise — sampling that is trivially easy — and ask the network to clean it one step at a time, again and again, walking backwards along the chain. Out the far end emerges a sharp, coherent image that was never in the training set. We turned the impossible task ('conjure a realistic image from nothing') into a long sequence of the same easy task ('make this a little less noisy').

Two pictures make the asymmetry vivid. (1) Drop a bead of ink into a glass of water. It blooms and spreads until the water is uniformly grey — easy to watch, but you could stare forever and never see the ink gather itself back into a drop. The forward process is that spreading; a trained model performs the seemingly miraculous reverse. (2) Michelangelo reportedly said the statue was already inside the marble and he merely chipped away everything that was not the statue. A diffusion model is that sculptor: handed a block of pure noise, it chips noise away, step by step, until the figure hidden inside is revealed.

A quick tour of the alternatives: GANs and VAEs

Diffusion did not arrive in an empty room. For years two other families generated images, and you will hear their names constantly, so let us meet them fairly. The first is the Generative Adversarial Network (GAN). Picture a contest between two networks: a generator that tries to paint fake images, and a discriminator — a critic — that tries to tell fakes from real photos. They train against each other. The forger gets better at fooling the critic; the critic gets sharper at catching forgeries; round after round the fakes become uncannily realistic.

GANs can produce stunning images, but the contest is notoriously hard to balance. If either side overpowers the other, training collapses or oscillates instead of settling. They also suffer mode collapse: the generator discovers a few image types that reliably fool the critic and quietly stops producing everything else — imagine our forger learning to paint only sunsets because they sell, forgetting how to paint anything else. The result covers only part of the real data's variety.

The second family is the Variational Autoencoder (VAE). It has two halves: an encoder that squeezes an image down into a small set of numbers — a compact latent code — and a decoder that expands a code back into an image. Train them together so that decode(encode(image)) ≈ the original, and you can then generate by feeding the decoder a fresh random code. VAEs train stably and gracefully, which is lovely — but historically their outputs came out blurry, as if the image were viewed through frosted glass, because the compression smooths away fine detail.

So why did diffusion overtake both around 2021–2022? Three reasons. Stable training: the reverse network is trained with a plain regression loss — predict a target, measure the error, nudge the weights — with no adversarial tug-of-war to balance, so it almost never collapses. Sample quality: the slow many-step refinement yields detail that rivals or beats the best GANs. Coverage: because it learns to denoise every kind of image rather than just whatever fools a critic, it captures the full breadth of the data and largely avoids mode collapse.

Walking the chain of noisy images

Let us make the step-by-step idea concrete with a little gentle notation — nothing scary, just names for the pictures in the chain. Call the clean starting image x₀ (read 'x-zero'; the subscript is just a step counter). One noising step later we have x₁, then x₂, and so on, getting noisier each time, all the way to x_T, which is (almost) pure static. The capital T is the total number of steps, and in practice T is often around 1000. So the forward process is a whole staircase of images, x₀ → x₁ → x₂ → … → x_T, from photo to snow.

The chain has a tidy property: each image depends only on the one immediately before it. To make x₅ you take x₄ and add a dab of noise — you do not need to look back at x₃, x₂, or x₀. A process with this 'memoryless' habit is called a Markov chain: the next state needs only the current state, never the whole history. That is what lets us describe the entire forward process with a single, simple rule applied over and over.

One dial controls the whole staircase: the noise schedule — the recipe that says how much noise to add at each step. A sensible schedule adds only a whisper of noise in the early steps (so the picture degrades gently) and more in the later steps (to finish the job and reach full static). Getting this recipe right matters a lot: add noise too fast and the network has too big a leap to undo at each reverse step; too slow and you waste hundreds of steps and compute. Tuning the schedule is one of the practical arts of diffusion, and we will return to it.

Because the whole forward chain is just 'take the previous image and add scheduled noise', we can write it as a tiny loop. The code below is not the real math (that arrives in Guide 2) — it is a faithful sketch of what is happening, so the picture in your head and the program agree.

# Forward diffusion as a tiny loop (conceptual sketch, not the exact math).
# schedule[t] says how strong the noise is at step t; bigger later on.

x = clean_image          # this is x_0
chain = [x]
T = 1000                 # total number of steps

for t in range(1, T + 1):
    noise = random_normal(shape=x.shape)   # fresh random static
    strength = schedule[t]                 # how much to add at this step
    # Markov: the new image depends ONLY on the current one, x.
    x = mix(x, noise, strength)            # blend in a little noise
    chain.append(x)

# chain[0] is the clean photo; chain[T] is (almost) pure noise.
# Note: NOTHING here was learned. It is a fixed recipe.
# The neural network's job (Guide 2-3) is to walk this list BACKWARDS.

What the model actually learns to do

A fair question has been hanging in the air: when we 'train the reverse network', what exactly is it predicting? You might guess it outputs the cleaner image directly. The actual answer, and the surprisingly tidy choice that powers the workhorse formulation, is even simpler: given a noisy image and a number t telling it which step it is on, the network predicts the noise that was added. It points at the grain and says 'that mess — that is the noise'.

Why is 'predict the noise' the same as 'denoise'? Because of plain arithmetic. The noisy image is, roughly, the clean image plus some noise. If the network can tell you what the noise is, you simply subtract it off, and what remains is a cleaner image. Predicting the noise and removing the noise are two faces of the same coin — and predicting noise turns out to be an easier, better-behaved target for the network to learn than predicting the picture directly. Do this once and you step from x_t to a cleaner x_{t-1}; do it all the way down the chain and a full image emerges from static.

An everyday analogy: hand a seasoned photo retoucher a grainy night shot. They do not repaint the scene from scratch; they look at it and point — 'this speckle is sensor noise, that fleck is noise, so is that' — and once those specks are lifted away, the real photo underneath emerges clean. The network is that retoucher, except it estimates the entire field of noise across the whole image at once, and it does so over and over as it walks back up the chain.

This concrete recipe — a fixed forward noising chain, plus a network trained to predict the added noise at each step — has a name: the denoising diffusion probabilistic model, or DDPM. It is the cleanest, most influential formulation of a diffusion model, and it is exactly what the next guide will make precise. Everything else in this track — text prompts, latent space, fine-grained control — is built on top of this one idea: predict the noise, subtract it, repeat.

Here is the road ahead, so you always know where you stand. We have just built the intuition. Next we make the noising precise, then the denoiser and the sampler, then we learn to steer it with words, and finally we take full control:

1. Guide 2 — Adding Noise on Purpose: the forward diffusion process, written down exactly (the schedule, and the neat shortcut that jumps to any step x_t in one move).
2. Guide 3 — Learning to Denoise: the U-Net that predicts the noise, the training loss, and the sampler that runs the chain backwards to generate an image.
3. Guide 4 — Painting from Words: latent diffusion (the VAE trick from Section 3) and text-to-image conditioning, so a prompt can summon a picture.
4. Guide 5 — Mastering Control: inpainting and editing, ControlNet, and the current frontier of guided, controllable generation.