discrete diffusion models

Standard diffusion adds Gaussian noise, which only makes sense for continuous data. But text, molecules, and other symbolic data live in discrete spaces where you cannot add a little bit of Gaussian noise to a token. Discrete diffusion redefines the corruption process for these spaces: instead of nudging values, it randomly mutates symbols, for example flipping tokens to a special mask or to random categories according to a transition matrix, until the sequence is fully corrupted, then learns to reverse those mutations.

The forward process is a Markov chain over categorical states whose one-step transition is a stochastic matrix; with an absorbing mask state it becomes a generalized, any-order masked language model, and with uniform transitions it resembles repeated random replacement. Training optimizes a discrete-state evidence lower bound, and a continuous-time formulation casts the dynamics through a rate matrix whose reverse depends on the ratio of marginal probabilities between states, the discrete analogue of the score. This concrete score can be fit by a denoising objective, giving the field its score-entropy training principle.

Discrete diffusion is the leading non-autoregressive alternative for generating sequences, offering parallel, any-order decoding and iterative refinement, and it is an active research direction for language and biological-sequence generation, though it has not yet matched the strongest autoregressive language models at scale.