Mastering Control: Editing, ControlNet, and the Frontier

The model zoo: DALL-E and friends

You now understand one full text-to-image pipeline end to end: a text prompt steers a U-Net that denoises a tiny latent, which a decoder turns into pixels. That mental model is your map key — almost every famous system is a variation on it. So before we learn to control these models, let's take a tour of the zoo and see how the big animals differ. The headline first: they all share the same diffusion core you met in Guides 2 and 3. None of them invented a new way to make pictures; they made different choices about what conditions the denoiser, what data they trained on, and where in the pipeline the diffusion runs.

Take DALL·E 2. Its defining choice is to route everything through CLIP — the same vision-language model that scores image–text agreement. Instead of feeding the text straight to the denoiser, DALL·E 2 first trains a small prior network that maps a CLIP text embedding to a CLIP image embedding (a guess at 'what would a matching picture look like, in CLIP's coordinate system?'). A second diffusion decoder then generates an image conditioned on that CLIP image embedding. This two-stage 'unCLIP' design means the actual picture is conditioned on an image-shaped vector, not raw words — which tends to give clean, coherent compositions and easy image variations (re-decode the same embedding and you get siblings of the same picture).

DALL·E 3 changes a different lever: language understanding. Its big insight is that most failures come from vague or under-described prompts, so it couples the image model tightly with a large language model (LLM). When you type 'a cat on a chair', the LLM silently rewrites it into a long, vivid caption — colour, pose, lighting, background — before the diffusion model ever sees it. It was also trained on images paired with these richer machine-written captions, so it has learned to honour detailed instructions and complex scenes (and even render short text on signs more reliably). The diffusion engine underneath is still doing the Guide-3 job; the LLM is a smart translator sitting in front of it.

Two axes organise the whole zoo. The first is where diffusion runs. Early DALL·E and Imagen denoise in pixel space (often a small image, then separate upscaler models enlarge it), while Stable Diffusion denoises in the compressed latent space you learned about — the latent diffusion trick that makes it cheap enough to run on a laptop GPU. Pixel-space can be crisp but expensive; latent-space is fast and memory-light, at the cost of an autoencoder round-trip. The second axis is how you get to use it: closed API models (DALL·E, Midjourney) hide the weights and serve you over the internet, while open-weights models (Stable Diffusion and its relatives) ship the actual network so anyone can download, fine-tune, and build tools like the ones in this guide.

Inpainting and outpainting: editing inside the picture

Generating from scratch is fun, but real work is editing: 'this is almost perfect — just remove the photobomber in the corner.' That is exactly what diffusion inpainting does. You give the model an image, a mask marking a region to regenerate, and a prompt for what should appear there. The model keeps everything outside the mask untouched and invents new content inside it that blends seamlessly with the surroundings — the right lighting, the matching wall texture, the continuing horizon. Outpainting is the identical idea pointed outward: you extend the canvas beyond its original edges and let the model imagine what was 'just out of frame'.

How does the fill stay consistent with the photo instead of drifting into a hallucinated mess? The trick is beautifully simple and reuses the reverse process from Guide 3 unchanged. We run the normal step-by-step denoising loop, but at every single step we overwrite the known (unmasked) part of the latent with the original image, re-noised to exactly the current timestep. The network is only ever allowed to freely invent inside the mask; everywhere else, reality is pinned back in at each step. Because the denoiser sees a correct, noisy version of the true surroundings on every iteration, it is constantly nudged to make the hole agree with them.

The masked-blend update applied after every denoising step.

Let's unpack it symbol by symbol. x_{t-1} is the latent we carry into the next step. m is the binary mask, the same shape as the latent: a value of 1 means 'keep this pixel' and 0 means 'regenerate it'. The \odot symbol means element-wise multiply — multiply matching positions, not matrix multiplication. So m \odot x_{t-1}^{\text{known}} selects only the keep-pixels of the original image re-noised to step t-1, while (1-m) flips the mask (now 1 inside the hole) so (1-m) \odot x_{t-1}^{\text{gen}} selects only the freshly denoised pixels from the network's output x_{t-1}^{\text{gen}}. Adding the two halves stitches them into one latent. Tiny example: if a pixel has m=1, the update becomes 1\cdot x^{\text{known}} + 0\cdot x^{\text{gen}} = x^{\text{known}} — the original wins. If m=0, it becomes 0 + 1\cdot x^{\text{gen}} = x^{\text{gen}} — the generated content wins. Pinning the known region every step (not just at the end) is what forces the hole to grow up matching its neighbours.

# Diffusion inpainting: pin the known region at every denoising step.
# image0  : the original latent we want to edit
# m       : binary mask, 1 = keep, 0 = regenerate (same shape as the latent)

x_t = random_noise()                       # the masked hole starts as pure noise

for t in reversed(range(T)):               # T -> 0, the Guide-3 reverse process
    # 1) the network freely denoises the WHOLE latent one step.
    #    The prompt + guidance still steer WHAT appears in the hole.
    x_gen = denoise_step(x_t, t, prompt)

    # 2) re-noise the ORIGINAL image to exactly this timestep (Guide 2 forward step)
    x_known = forward_noise(image0, t - 1)

    # 3) blend: keep the known pixels, accept generated pixels only in the hole
    x_t = m * x_known + (1 - m) * x_gen

# Outpainting is the SAME loop: just enlarge the canvas and set the new
# border pixels to 0 in the mask so the model must invent them.
result = decode(x_t)

ControlNet: steering with structure

Prompts are wonderful at saying what — 'a knight in golden armour' — but hopeless at saying where. No amount of words reliably places the knight's left arm raised at this exact angle, or makes the new building match the old one's silhouette. Language is a blurry pointing finger. ControlNet is the tool that gives diffusion models precise spatial control, by letting you hand the model a second input that carries geometry directly: an edge map, a pose skeleton, a depth map, a rough scribble.

The architecture is clever and conservative. You take your already-trained Stable Diffusion U-Net and freeze it — its weights never change. Then you make a trainable copy of its encoder layers and run them as a parallel branch that reads the control image (say, a Canny edge map of a reference photo). At each level, the branch's features are added back into the corresponding layer of the frozen main network. The base model keeps all the world knowledge it already had; the new branch only learns the extra skill of 'bend that knowledge toward this shape'. Because the original is frozen, training one ControlNet is cheap and never degrades the base model — you can keep many (pose, depth, edges) and swap them like lenses.

One detail makes this safe to train, and it has a friendly name: the zero-convolution. The connections joining the new branch to the frozen network start out initialised to all zeros, so on training step one the branch contributes exactly nothing — the system behaves identically to the untouched base model. As training proceeds those connections grow away from zero, so the control 'fades in' gently and gradually, never shocking the pretrained network or wrecking the quality it started with.

What does this buy you in practice? Copy a dancer's exact pose from a reference photo onto a totally different character (extract the pose skeleton, prompt the new character). Turn a five-minute pencil sketch into a finished render while keeping your composition (scribble control). Restyle a building from concrete to glass while its windows and roofline stay put (depth or edge control). Colour a line-art drawing without the AI redrawing the lines. In every case the control map supplies the structure and the prompt supplies the content and style.

Prompt craft and negative prompts

Most of the steering you will ever do happens through plain language, so it pays to write prompts deliberately. A strong prompt is layered, not a single noun. A reliable recipe: name the subject (who/what, doing what, where), then the medium (photo, oil painting, 3D render), the style (an art movement, an era, a named aesthetic), the lighting (golden hour, soft studio light, neon), and finally quality cues (sharp focus, highly detailed). You are not casting spells — each phrase nudges the model toward a region of its training distribution it has actually seen.

# BEFORE - thin prompt -> generic, flat, unpredictable
prompt = "a cat"

# AFTER - layered prompt -> subject, medium, style, lighting, quality
prompt = (
    "a tabby cat sitting on a wooden windowsill, "   # subject + scene
    "oil painting, impressionist style, "            # medium + style
    "warm afternoon light, soft shadows, "           # lighting
    "highly detailed, sharp focus"                   # quality cues
)

# NEGATIVE prompt - an explicit condition to steer AWAY from
negative = "blurry, lowres, deformed hands, extra limbs, watermark, text"

# Emphasis syntax (tool-dependent): (cat:1.3) up-weights, (text:0.5) down-weights.
# guidance_scale ~7 balances prompt-fidelity against natural-looking freedom.

Now the part that looks like folklore but is pure mechanics: the negative prompt. Recall classifier-free guidance from Guide 4. To generate, the model runs its denoiser twice each step — once conditioned on your prompt, once on an empty condition — then extrapolates away from the empty result toward the conditioned one. Classifier-free guidance is literally 'push in the direction that the prompt adds.' A negative prompt simply replaces that empty condition with an explicit unwanted condition. Instead of pushing away from 'nothing', the model now pushes away from 'blurry, extra fingers'.

Negative-prompt guidance: extrapolate from the unwanted condition toward the wanted one.

Symbol by symbol: \hat{\epsilon} is the final, guided noise estimate the sampler will use this step. \epsilon_\theta(x_t, c) is the U-Net's predicted noise for the current noisy latent x_t under condition c. Here c_{\text{pos}} is your real prompt and c_{\text{neg}} is the negative prompt (in ordinary CFG, c_{\text{neg}} is just the empty string). The bracket \big(\epsilon_\theta(x_t, c_{\text{pos}}) - \epsilon_\theta(x_t, c_{\text{neg}})\big) is a direction vector: it points from 'what the unwanted condition predicts' toward 'what the wanted condition predicts'. The guidance scale w (see guidance scale) is how many steps we take along that direction. Concretely, if w=1 you just get the positive prediction; w=7 shoves the result seven times further along the 'toward wanted, away from unwanted' arrow — so a bigger w both obeys the prompt harder and flees the negative prompt harder. The whole reason a negative prompt works is that we swapped the start of that arrow from 'nothing' to 'exactly the thing we hate', so the arrow now actively repels it.

Limits, failure modes, and ethics

Mastery means knowing where the tool breaks. Diffusion models have a recognisable set of failures. Hands and fingers come out mangled (too many, fused, bent the wrong way). Text on signs and labels turns to gibberish unless the model was specially trained for it. Counting is unreliable — ask for 'exactly five birds' and you may get four or seven. Prompt-binding errors attach attributes to the wrong object: 'a red cube on a blue sphere' sometimes yields a blue cube on a red sphere. And plain old speed: a many-step sampler can take seconds per image, which is the bottleneck the frontier in the next section attacks.

Why these specific failures? They trace straight back to how the model learns. Hands are small, high-variance, and rarely the captioned subject, so the network gets weak signal about their structure. Counting and binding fail because the text encoder squashes a sentence into a vector that captures gist better than precise relations — 'red', 'blue', 'cube', 'sphere' are all in there, but the leash tying each adjective to its noun is loose. Knowing the cause tells you the fix: be redundant and explicit ('a single red cube resting on top of a large blue sphere'), use ControlNet or inpainting to nail layout, and pick a model trained with better captions.

The deeper issues are societal, not arithmetic. Bias: trained on a slice of the internet, models absorb its stereotypes. Prompt 'a CEO' or 'a nurse' and you often get a narrow, predictable demographic — the model is reflecting and amplifying who was photographed in those roles, not who can hold them. This is not a fringe bug; it shapes how millions of generated images portray the world, which is exactly why fairness deserves explicit measurement and mitigation rather than a shrug.

Then provenance and consent. These models were trained on billions of images scraped from the web, many copyrighted, many of real people who never agreed to it — raising live legal and moral questions about whether a generated style 'in the manner of living artist X' is homage or appropriation. And misinformation: photorealistic text-to-image generation makes convincing fake images of real people and events cheap to produce at scale — non-consensual imagery, fabricated 'evidence', political deepfakes. The same openness that lets you build with Stable Diffusion also removes the guard rails a closed API can enforce.

1. Curate the data: filter training sets to reduce illegal, harmful, and heavily skewed content before the model ever learns it.
2. Watermark and label outputs: embed invisible provenance signals (e.g. C2PA-style content credentials) so generated images can be detected and traced downstream.
3. Apply safety filters: block prompts and outputs for clearly harmful categories at the API or app layer.
4. Honour opt-outs: support registries and 'do-not-train' signals so artists and individuals can remove their work and likeness from future datasets.
5. Disclose: label synthetic media to your audience, especially anything depicting real people or events.

Where to go next: the diffusion frontier

You now own the whole pipeline and the tools to direct it — so let's scan the horizon. The hottest frontier is speed. Standard sampling needs many steps, but distillation and consistency models train a network to leap most of the way in just 1–4 steps. Connect this to Guide 3: there, DDIM let us predict the clean image x_0 from a noisy latent and take big strides. A consistency model takes that idea to its limit — it is trained so that from any point on the noising path it jumps straight to the same x_0 prediction. Squeeze that to a single jump and you get near-real-time generation: images that update live as you type.

A second thread is mathematical tidying. Flow matching and rectified flow reformulate diffusion so that, instead of a wandering noisy trajectory, the model learns to follow nearly straight paths from noise to data. The diffusion model you learned is one way to describe that journey; flow matching is a cleaner, more general description of the same ODE that the DDIM sampling solver already hinted at. Straighter paths are easier to integrate, which is part of why these newer models sample so well in just a few steps. Latent consistency models marry this with the latent trick from Guide 4, powering the live, paint-as-you-go tools you may have seen.

And the biggest leap is into new dimensions. Video diffusion adds time: the network denoises a stack of frames jointly, learning consistency across motion so the cat doesn't grow a sixth leg between frames. 3D generation (NeRF-style and its successors) goes spatial: instead of a flat picture, the model learns a scene you can fly a camera around, optimising a continuous function that returns colour and density at every point in space. Everything you learned about conditioning, guidance, and latents carries over — the canvas just gained an axis.

1. Read the two foundational papers now that you have the intuition: 'Denoising Diffusion Probabilistic Models' (DDPM) for the core training/sampling, and 'High-Resolution Image Synthesis with Latent Diffusion Models' for the latent trick behind Stable Diffusion.
2. Run an open pipeline locally: install the diffusers library, generate from a prompt, then add a negative prompt and watch the change.
3. Add ControlNet: feed a pose or Canny-edge map and reproduce a reference layout with your own prompt.
4. Try inpainting: mask out an object in a photo and prompt a clean replacement, watching the masked-blend trick keep the edges seamless.
5. Then chase the frontier: load a latent-consistency or LCM-LoRA model and feel few-step, near-real-time generation for yourself.