denoising u-net

The denoising U-Net is the actual neural network that does the predicting inside almost every diffusion model — the workhorse that, given a noisy image and a timestep, outputs an estimate of the noise to remove. The U-Net shape (named for its U-like diagram) first shrinks the image through a downsampling encoder into a small, abstract bottleneck, then grows it back through an upsampling decoder, with skip connections that hand each decoder level the matching-resolution features from the encoder. This shape is ideal for denoising because removing noise needs both global context (what is this image of?) and pixel-precise local detail (where exactly is this edge?), and the U-Net carries both.

Two extra inputs make it a diffusion U-Net rather than a plain image-to-image U-Net. First, the timestep t is encoded (typically via sinusoidal embeddings, like positional encodings) and injected into every residual block, so the network knows how much noise it is facing and can behave differently at high versus low noise. Second, the conditioning — text embeddings, class labels, or other guidance — is injected, for text usually through cross-attention layers interleaved with the convolutional blocks, where image features query the text. Self-attention layers at the lower-resolution levels let distant parts of the image coordinate (so a generated face stays symmetric, for instance).

The denoising U-Net has been the standard backbone since DDPM, but it is not the only option: diffusion transformers (DiT) replace the convolutional U-Net with a pure transformer over latent patches and scale better at very large sizes, increasingly favored in newer systems. Whatever the backbone, its contract is the same — one network, reused at every reverse step, taking (noisy latent, timestep, conditioning) and returning a noise (or velocity, or score) prediction. Its capacity and how conditioning is wired into it largely determine the model's fidelity and prompt-following ability.