JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Frontiers: Few-Step, Distilled, and Discrete Generation

Diffusion's one weakness is its many sampling steps. Meet consistency models, distillation, and discrete diffusion — the research edge pushing generation toward one step and beyond continuous data.

The cost that defines the frontier

Everything good about diffusion comes from amortizing one hard jump into many easy steps — and that is also its one real liability. A classic sampler may call the network hundreds or thousands of times per image, so inference cost dominates deployment. DDIM and rectified flow cut steps by tens; the frontier asks for the limit: can we generate in a single network evaluation without giving back quality?

Diffusion amortizes one hard jump into a long denoising chain — many network calls per sample, the very cost this frontier attacks.

A row of frames denoising step by step from pure noise into a clear image.

Consistency models: collapse the trajectory into one map

Consistency models take the probability-flow ODE from Guide 4 and impose a self-consistency property: a single network *f(x_t, t)* should map any point on a given ODE trajectory directly to the trajectory's clean endpoint *x_0*. If that holds, then evaluating f once at the noisiest level is generation. Training enforces consistency between adjacent times — either by distilling a pretrained diffusion model, or standalone from scratch.

f_\theta(\mathbf{x}_t, t) = f_\theta(\mathbf{x}_{t'}, t') \quad \forall\, t,t' \in [\epsilon, T], \qquad f_\theta(\mathbf{x}_\epsilon, \epsilon) = \mathbf{x}_\epsilon

The self-consistency property: one network maps every point on a probability-flow trajectory to the same origin, pinned by the identity boundary condition at ε.

Distillation: teach a fast student the slow teacher's path

The broader pattern is distillation: train a many-step teacher diffusion model, then compress its sampling trajectory into a student that needs far fewer steps. Progressive distillation repeatedly halves the step count; adversarial and distribution-matching variants add a GAN-style or score-based term so the student's output distribution matches the teacher's, not just individual trajectories. Reflow from rectified flow is a transport-geometry cousin of the same goal.

Be appropriately skeptical: few-step samples can match the teacher on average metrics yet lose rare modes, sharpen textures into artifacts, or amplify the diversity loss already introduced by classifier-free guidance. 'One-step generation' headlines rarely report the tail behavior, and the tail is where distillation hurts most.

Beyond continuous data: discrete diffusion

Gaussian noise and scores assume a continuous state space, but text, code, and molecules live on discrete tokens. Discrete diffusion models redefine the forward corruption as a Markov process over categories — masking tokens or transitioning them through an absorbing or uniform state — and learn to reverse it. The two leading flavors are masked / absorbing-state diffusion (closely related to masked language modeling) and continuous-time discrete-state chains parameterized by transition rate matrices.

q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathrm{Cat}\!\left(\mathbf{x}_t;\, \mathbf{p} = \mathbf{x}_{t-1}\mathbf{Q}_t\right)

Discrete diffusion swaps Gaussian noise for a categorical transition matrix Q_t that corrupts tokens toward a uniform or [MASK] state.

Why bother, when autoregressive language models are so strong? Discrete diffusion decodes in parallel and can revise any position at any step, instead of committing left-to-right token by token — promising for infilling, controllable generation, and latency. The open question is whether it can close the remaining likelihood and quality gap to autoregressive models at scale; this is live, unsettled research.

Discrete diffusion decodes all positions in parallel and can revise any token, unlike this autoregressive loop that commits left-to-right one token at a time.

An autoregressive generation loop emitting one token at a time, left to right.

Where the theory is still open

Step back over the whole track and the unsolved questions sharpen. We can sample beautifully yet still lack tight, practical bounds on what a finite-capacity score network actually learns about the true score in low-density regions. Few-step methods work empirically well before we fully understand which probability paths are easiest to straighten and distill. And evaluation itself is shaky — sample-quality metrics correlate imperfectly with the density coverage the theory cares about.

  1. Theory of learned scores: how does score estimation error in rarely-seen regions propagate into sampling bias and dropped modes?
  2. Optimal transport for generation: which probability paths minimize sampling steps, and how straight can flow matching truly get?
  3. Discrete and multimodal unification: one principled framework that covers continuous, discrete, and mixed data without bolting on special cases.
  4. Trustworthy evaluation: metrics that actually track mode coverage and tail fidelity, not just headline sample sharpness.