Adding Noise on Purpose: The Forward Diffusion Process

One small step of noise

In the first guide we watched a diffusion model do something close to magic: start from a screen of pure static and, step by patient step, sculpt it into a sharp picture. We took that on faith. Now we open the hood. Before a model can ever remove noise, we have to define exactly how noise gets added in the first place — and, surprisingly, that adding recipe is the easy half. It is fixed, hand-designed, and involves no learning at all. This is the forward diffusion process, and getting it precisely right is what makes the whole method trainable.

One ingredient appears in every line below, so let's pin it down in a sentence: a Gaussian (or normal) distribution is just random values clustered around a central mean, with their spread set by the variance — picture the classic bell curve, tall and dense in the middle and thinning out smoothly on both sides. A small variance is a narrow, sharp bell (values hug the mean); a large variance is a wide, flat bell (values wander far). When we 'add Gaussian noise' to a pixel, we are nudging its value by a random amount drawn from such a bell.

Diffusion adds noise gradually, over many tiny steps, rather than all at once. Call the clean image x_0 and the fully noisy end-state x_T (with T often around 1000). Each step takes the current image x_{t-1} and produces a slightly noisier x_t. The rule for one step is written as a conditional Gaussian — read it as 'given the previous image, the next one is drawn from this bell':

Let's name every piece. q(\cdot) is the forward (noising) distribution — the lowercase q flags it as fixed and hand-designed, not the learned model. x_t is the image at step t; x_{t-1} is the slightly cleaner image one step before it. The notation \mathcal{N}(x_t;\,\mu,\,\Sigma) says 'x_t is Gaussian with mean \mu and covariance \Sigma'. Here the mean is \mu = \sqrt{1-\beta_t}\,x_{t-1} — a shrunk copy of the previous image: we multiply every pixel by the factor \sqrt{1-\beta_t}, which is just under 1, so the picture is dimmed a hair. The covariance is \Sigma = \beta_t\mathbf{I}, where \beta_t (a small positive number, the step's 'noise variance') sets how much fresh randomness we sprinkle in, and \mathbf{I} is the identity matrix — meaning the noise is added independently to every pixel, each with the same variance \beta_t and no correlation between pixels. So one step = dim the image a touch, then dust it with independent Gaussian grain. Why the shrink factor? It keeps the total 'energy' steady: the surviving signal carries (1-\beta_t) of its variance and we add back \beta_t of noise, so a pixel that started at variance 1 stays at variance 1 — the image never blows up or fades to nothing, it just trades signal for noise. A worked feel: with a typical early-step \beta_t = 0.0001, the factor \sqrt{1-\beta_t} \approx 0.99995 and we add only a whisper of noise, so x_t looks essentially identical to x_{t-1}, just a faint dusting of grain. Crucially, every number here is chosen by us in advance; nothing in this rule is learned.

The noise schedule: how much, and when

A single step barely changes the image — that is the point — but to travel from a photo all the way to pure static we chain hundreds of them. The full list of step sizes \beta_1, \beta_2, \ldots, \beta_T is called the noise schedule. It is the recipe that decides how much noise we add and when. Almost always the \beta_t start tiny and grow as t increases: gentle nudges early, bigger gulps of noise late.

Why does the shape of this schedule matter so much? Think about what the model must learn at each step — to undo exactly the noise just added. If we dump in too much noise too early, the fine structure of the image (edges, textures, small details) is wiped out almost immediately, leaving the model nothing to grab onto during the hardest, most detailed steps. Add too little, too slowly, and most of the T steps are nearly noiseless duplicates — wasted effort the network learns almost nothing from. A good schedule spends its steps where they count.

The original denoising diffusion probabilistic model used a linear schedule: \beta_t rises in a straight line (say, from 10^{-4} to 0.02). It works, but it tends to destroy the signal too quickly in the back half — by the time you are two-thirds of the way to x_T, the image is already almost pure noise, so the final third of the steps do little useful work. A later improvement, the cosine schedule, bends the curve so that noise is added more gently through the crucial middle steps and only ramps up near the very end. The analogy is a dimmer switch: a cheap dimmer jumps to near-black the instant you turn it, while a good one fades the light down smoothly across its whole travel. The cosine schedule is the smooth dimmer — it keeps more signal alive through the middle, exactly where the model is learning to rebuild the most structure, and in practice it produces noticeably better images.

It will pay to give the surviving part of each step its own name. Define \alpha_t = 1 - \beta_t. Since \beta_t is the fraction of variance we spend on new noise at step t, its complement \alpha_t is the fraction of the signal's variance that survives step t. If \beta_t = 0.01 then \alpha_t = 0.99 — 99% of what was there carries through, 1% is replaced by fresh noise. Notice that \alpha_t sits just under 1 for every step. That tiny detail is the seed of the shortcut in the next section: surviving 99% per step, then 99% of that again, then again... is something we can simply multiply out.

The magic shortcut: jump straight to step t

Here is a practical headache. To train the denoiser we will show it noisy images at random steps — sometimes t=5, sometimes t=700. The definition we have so far is recursive: to reach x_{700} we would have to start from x_0 and crank the single-step rule 700 times, every single training iteration. That would be agonizingly slow. We want a way to *jump straight to step t* in one shot, for any t. Astonishingly, there is one.

The reason it exists is a beautiful property of Gaussians: if you take a Gaussian, scale it, and add another independent Gaussian, the result is still a single Gaussian — you can fold many small noising steps into one. Because every step of the forward diffusion process is Gaussian, chaining t of them collapses into one Gaussian that carries the clean image x_0 all the way to x_t directly. Working out the algebra — just repeatedly substituting the one-step rule into itself — gives this closed form:

Term by term. x_0 is the original clean image from your dataset. \bar\alpha_t (read 'alpha-bar at t') is the cumulative product \alpha_1\alpha_2\cdots\alpha_t — all the per-step surviving fractions from the noise schedule multiplied together, i.e. *how much of the original image survives all the way to step t*. Because each \alpha_s is a little under 1, multiplying many of them makes \bar\alpha_t shrink from near 1 (small t) steadily toward 0 (large t). The image at step t is then a weighted blend of two things: \sqrt{\bar\alpha_t}\,x_0, a faded copy of the real image, plus \sqrt{1-\bar\alpha_t}\,\epsilon, a dose of noise. That single noise term \epsilon is one fresh sample from a standard Gaussian \mathcal{N}(0,\mathbf{I}) — mean 0, variance 1 on every pixel, independent — drawn anew each time. The two weights are designed so their variances add to 1, since \bar\alpha_t + (1-\bar\alpha_t) = 1: the image keeps unit energy, continuously trading picture for noise. Put numbers on it. At a small step, say \bar\alpha_t = 0.98: \sqrt{0.98} \approx 0.99 multiplies the real image while only \sqrt{0.02} \approx 0.14 multiplies the noise — x_t is mostly the real photo with light grain. At a large step where \bar\alpha_t = 0.0001: \sqrt{\bar\alpha_t} \approx 0.01 barely keeps any image while \sqrt{1-\bar\alpha_t} \approx 1.0 — so x_t is almost pure noise \epsilon. In other words \bar\alpha_t is literally a dial that slides the image from 'real' to 'static' as t grows.

import torch

def q_sample(x0, t, alpha_bar):
    """Jump straight to step t in ONE line -- no loop over 1..t."""
    eps = torch.randn_like(x0)         # one standard-normal sample, same shape as x0
    a = alpha_bar[t].sqrt()            # sqrt(alpha_bar_t): how much signal survives
    b = (1.0 - alpha_bar[t]).sqrt()    # sqrt(1 - alpha_bar_t): how much noise mixed in
    x_t = a * x0 + b * eps             # the noisy image at step t
    return x_t, eps                    # return eps too -- it is the training target

Reparameterization: why we predict the noise

Now, what should the network actually compute? The closed form x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon links three quantities: the noisy image x_t, the clean image x_0, and the noise \epsilon. Here is the key observation: if you already hold x_t (the network always sees it), then x_0 and \epsilon are two ways of saying the same thing. Rearranging the equation to isolate \epsilon proves it:

Read it as: given the noisy image x_t and the clean image x_0, the noise \epsilon is completely determined — it is just x_t minus the faded clean image, rescaled. Flip it around and the same holds: given x_t and \epsilon you can recover x_0. They are algebraically interchangeable. This is the friendly version of the reparameterization trick: instead of asking the network to output the clean image directly, we can ask it to output the noise \epsilon, and we lose nothing — either answer hands you the other. It turns out predicting \epsilon is the easier, better-behaved target: \epsilon always has the same tidy scale (mean 0, variance 1) no matter which step t we are on, whereas the correct x_0 to reconstruct looks wildly different early versus late. A consistent target makes the regression far more stable.

So we build a network whose job is to look at a noisy image and guess the noise inside it. Write it \epsilon_\theta(x_t, t). The subscript \theta stands for all of the network's learnable weights (the millions of numbers gradient descent will tune). It takes two inputs: x_t, the noisy image, and t, the timestep — feeding in t is essential, because the network must know how noisy its input is to judge how much noise to predict (a barely-grainy x_5 and a near-static x_{900} demand very different answers). This noise-guesser is the denoising U-Net, and once trained it is exactly the engine the reverse denoising process will run, step by step, to turn static back into a picture in the next guide.

The training objective, in one clean line

We now have everything we need to write the training objective — and it is shockingly plain. No adversarial duel between two networks, no delicate balancing act. Just a single regression: make the predicted noise match the real noise. The simplified loss the DDPM paper actually trains on is one line:

Unpacking it: \mathcal{L} is the loss we minimize. The \mathbb{E}_{x_0, t, \epsilon}[\cdot] is an expectation — a fancy word for 'average over many random draws' — and it averages over three random choices: a clean image x_0 sampled from the training set, a timestep t picked uniformly at random, and a noise sample \epsilon \sim \mathcal{N}(0,\mathbf{I}). From those we build the noisy image x_t using the closed-form shortcut from Section 3. Inside the brackets, \epsilon is the true noise we actually added (the target), \epsilon_\theta(x_t, t) is the network's guess, and \lVert\cdot\rVert^2 is the squared L2 norm — sum the squared difference over every pixel. In plain words, the bracket is simply how wrong the noise guess was, and training drives that average wrongness down. It is a plain regression, with none of the adversarial instability that makes other generative models hard to train — and a well-trained \epsilon_\theta is precisely the tool the next guide picks up to walk the reverse chain backward, from static to picture.

1. Grab a real image x_0 from the dataset.
2. Pick a random timestep t (uniformly between 1 and T).
3. Sample fresh noise \epsilon \sim \mathcal{N}(0,\mathbf{I}) and build the noisy image in one line: x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon.
4. Feed x_t and t to the denoising U-Net and read off its prediction \epsilon_\theta(x_t, t).
5. Compute the squared error \lVert \epsilon - \epsilon_\theta(x_t,t)\rVert^2 and take one gradient-descent step to nudge the weights \theta so the next guess is a little better.

def training_step(model, x0, alpha_bar, T):
    t = torch.randint(0, T, (x0.size(0),))   # 1-2. a random timestep per image
    x_t, eps = q_sample(x0, t, alpha_bar)     # 3. sample noise, build x_t in one shot
    eps_pred = model(x_t, t)                  # 4. ask the network to guess the noise
    loss = ((eps - eps_pred) ** 2).mean()     # 5. mean-squared error: how wrong the guess was
    loss.backward()                           #    gradients -> nudge weights downhill
    return loss