Flows and Continuous-Time Transport: From Jacobians to Flow Matching

Discrete flows: exact likelihood, awkward constraints

Recall from Guide 1 that normalizing flows stack invertible layers and read off exact likelihood via the change-of-variables formula. The practical engineering is all about the Jacobian: coupling layers split the input, transform half conditioned on the other half, and leave a triangular Jacobian whose log-determinant is a trivial sum. You get exact density and exact sampling — but the bijective, dimension-preserving constraint makes flows parameter-hungry and historically a step behind on raw image quality.

Continuous normalizing flows: let depth go to infinity

Continuous normalizing flows (CNFs) replace the stack of discrete layers with a neural ODE: a velocity field *vθ(x, t)* whose integration over time transports the base density to the data. The instantaneous change-of-variables formula turns the log-determinant into the time-integral of the velocity field's divergence — no triangular-architecture gymnastics needed, any network can be the velocity. This is maximally flexible transport.

\frac{dx}{dt}=v_\theta(x,t),\qquad \frac{d\,\log p_t(x)}{dt}=-\operatorname{tr}\!\left(\frac{\partial v_\theta}{\partial x}\right)

A CNF integrates a velocity field as a neural ODE; the log-density evolves by the negative trace (divergence) of the velocity's Jacobian.

The pain was training. The original CNF maximized likelihood by simulating the ODE forward and backpropagating through the solver — expensive, and the divergence integral needs stochastic trace estimates. CNFs were elegant theory throttled by their own training cost, and for years that kept them off the leaderboard. The breakthrough was to stop simulating altogether.

Flow matching: regress the velocity, never simulate

Flow matching asks a deceptively simple question: if I fix a target probability path interpolating noise to data, what velocity field generates it — and can I just regress onto that velocity directly? The answer is yes. Pick a per-sample conditional path (e.g. a straight line from a noise point to a data point), compute its closed-form velocity, and train vθ to match it by plain mean-squared error. No ODE solve in the loop, no divergence estimate, no adversary.

\mathcal{L}_{\mathrm{CFM}}(\theta)=\mathbb{E}_{t,\,q(x_1),\,p_t(x\mid x_1)}\left\|\,v_\theta(x,t)-u_t(x\mid x_1)\,\right\|^2

The conditional flow-matching objective: regress the network velocity onto the known per-sample target velocity — no ODE simulation required.

# conditional flow matching (straight-line path)
x0 = randn_like(x1)               # noise
x1 = data_sample()
t  = rand()                       # time in [0,1]
xt = (1 - t) * x0 + t * x1        # point on the path
u  = x1 - x0                      # target velocity (constant on a line)
loss = mse(v_net(xt, t), u)

Flow matching is just regression of a network onto a known per-sample velocity — strikingly simpler than likelihood-based CNF training.

Rectified flow: straighten the paths

Curved transport paths force an ODE solver to take many small steps; straight paths could in principle be crossed in one. Rectified flow trains a flow whose trajectories are as straight as possible by coupling each noise sample to a data sample and regressing onto the constant straight-line velocity — then optionally reflowing: regenerate pairs with the current model and retrain, which provably straightens the paths further. Straighter paths mean far fewer function evaluations at sampling time.

x_t=(1-t)\,x_0+t\,x_1,\qquad \min_\theta\ \mathbb{E}_{t,x_0,x_1}\left\|\,(x_1-x_0)-v_\theta(x_t,t)\,\right\|^2

Rectified flow regresses the velocity onto the straight-line displacement between noise and data, so trajectories straighten toward one-step transport.

This is why several state-of-the-art image generators adopted rectified-flow / flow-matching training: the same high quality as diffusion, but with a geometry deliberately engineered for cheap, few-step sampling. It sets up the central question of the final guide — how few steps can we get away with?