The whole pipeline in one picture, split the way the codebase splits a recipe. The information path is what the model sees. The denoising task is what it must predict and how it is trained. Inference is how predictions become text. Click any part to open it in part 2 below. Change it there and the map redraws to match.
Each layer is one component of the codebase. Pick a part and its picture changes. Parts that cannot work with your earlier choices are crossed out; hover one for the reason. Each picture has its own control for whatever that part changes.
Vocabulary: the, cat, sat, on, mat, dog.
No vectors at all: each position is one of six symbols.
Each token becomes a vector of length six with a single 1 at its word.
Each word is a learned point in a vector space (two dimensions shown). Highlighted points are the five tokens.
Masked: each token is independently hidden behind [M]. A hidden token carries no information about its word.
GIDD: one process that mixes both kinds of discrete noise. A corrupted token is usually masked, but sometimes swapped for a random word (red).
Uniform: each token is independently replaced by a random word (red). It looks real, so the model must work out which ones changed.
Gaussian on one-hot vectors: noise is added to every entry. As t grows the true word’s bar sinks into the noise.
Gaussian on embeddings: noise pushes each point off its word. The blob shows how far noise can reach at this t.
Simplex: each token is a probability vector. Noise drifts it from its corner toward the uniform middle while it always sums to one.
Spherical: embeddings live on the unit sphere. Noise picks a random point and moves along the arc toward it, so the state never leaves the sphere.
Blurring: each embedding feature is smoothed along the sequence, so neighboring tokens bleed into each other. Higher features blur first.
Two corruptions run on the same clean sequence, each in its own representation and with its own schedule. The network sees both views at once. (CANDI is masked + Gaussian.)
Predefined schedules never change: the same curve is used from the first training step to the last.
Semi-adaptive: the curve comes from a fixed family (here a Gumbel-shaped one, as in LangFlow) with a few parameters learned alongside the network. Press Play to watch the knobs move.
Fully adaptive: the whole curve is refit from the training loss (CDCD-style warping, or RePlaid’s variational schedule). As the loss concentrates on some noise levels (right), the schedule warps (left) so training spends more time there.
For masked or uniform noise it warps the keep rate.
With two corruptions, each view has its own schedule.
Self-conditioning off: each pass starts fresh from the noisy input.
Self-conditioning on the expected embedding: the previous pass’s probabilities p are multiplied by the embedding table E, giving the average embedding ē the model expects. That vector is fed back in.
Self-conditioning on the hidden state: the previous pass’s last hidden vector h is fed back directly, before it is turned into word probabilities.
Without time conditioning the network must infer the noise level from the input itself.
Time conditioning tells the network how noisy its input is.
For every position, a probability over the six words for the clean token. Slide the input noise up and the bars flatten.
Score ratios: how much more likely each word is than the current token. Not a normalized distribution. SEDD defines them for uniform noise too; the codebase implements the masked version.
A plug-in estimate for uniform noise: the posterior mixed with a uniform floor that grows with noise.
A clean-token posterior shaped by a Gaussian prior on the noisy vector, used to compute the expected clean vector.
Only the next position is predicted, given everything to its left.
Plain cross-entropy: predict the clean token, with every noise level counting the same. Simple, but not a likelihood bound.
The ELBO is a bound on the data likelihood.
For masking the ELBO is cross-entropy weighted by 1/t (as in MDLM), so nearly clean examples count much more.
For uniform noise the ELBO has its own exact form (as in DUO).
For GIDD the ELBO is composed from its masking and uniform parts (gidd_elbo).
For score-ratio outputs the ELBO is score entropy (SEDD).
For Gaussian noise, RePlaid’s three loss terms fall straight out of the ELBO. The endpoint at t = 0 gives a reconstruction cross-entropy, the endpoint at t = 1 gives a prior KL, and everything in between is an integral of a weighted MSE between the expected embedding ē and the true embedding.
Uniform times between ε and 1. The codebase stratifies them: a batch of n examples gets one time in each of n equal slices, so every batch covers the whole range. This is the default for almost every recipe.
D3PM trains on a grid of discrete steps (1000 in the codebase, 10 shown). Each time is a stratified uniform draw, rounded up to the next step.
Loss-derivative sampling spends more training on the noise levels where the loss changes fastest. It keeps a running record of loss against t and refits the density every 50 steps.
The simplex preset leaves time_sampler.adaptive=false, so in practice it draws the same stratified uniform times (shown).
RePlaid splits each batch in two. Some examples sit at t = 0 to train the reconstruction term (red). The rest get stratified diffusion times. The split follows the running spread of the two losses.
With a block-size mixture, single-token blocks skip this and always train at t = 1 (pure noise).
The schedule says what each t means. This says which t the network practices on.
Ancestral sampling reveals tokens a few at a time in random order, one network call per step.
Entropy-bounded sampling commits every token the model is confident about at once, so it needs fewer steps.
A Gaussian solver (Euler, Heun, adaptive ODE or DDPM) moves every vector a little each step; all words sharpen together.
The process’s own reverse step (simplex bridge, spherical flow or un-blurring) sharpens all positions together.
Each view takes its own reverse step: masked tokens are revealed while the continuous view sharpens.
SEDD's analytic step uses the score ratios to jump several masked tokens at once with exact transition probabilities.
Discrete flow matching follows a continuous-time Markov chain: tokens jump between states, and with uniform noise a filled-in word can still change later (red).
DDPM takes stochastic Gaussian reverse steps: every vector sharpens, with a little fresh noise added each step.
Autoregressive decoding: one new token per step, strictly left to right.
New tokens at each step are outlined.
Turn on block diffusion or insertion to see how they change generation.
Block diffusion splits the sequence into blocks. In training the network reads the noisy sequence and the clean sequence together (length 2L) under the mask above. At generation, blocks are produced left to right; inside a block, tokens are denoised in parallel.
Block-causal clean context, as in the BD3-LM paper (block_diffusion.x0_causal=false).
Token-causal clean context: the clean stream is an ordinary causal language model, which lets every block size share one KV cache (the codebase default, x0_causal=true).
Token-causal clean context plus a next-token loss on it, so the same network is also trained as an autoregressive model of the context (block_diffusion.clean_loss=autoregressive). BlockGen gets its AR behavior a different way: see Mixture of block sizes.
Mixture of block sizes, as in BlockGen: one model with a token-causal clean context is trained on block sizes 1 and 16. Size 1 uses pure noise and CE, so it is next-token prediction; the larger size keeps the recipe’s own loss (+experiment=blockgen_mixture, composed after a fixed-block recipe).
Insertion lets the length change: generation starts short and new tokens are inserted between existing ones (outlined), so the model decides the length.
Absorbing posterior, weighted CE, ancestral inference.
Standard state-independent masking; shares MDLM components. Not GenMD4.
Time-independent posterior (no time conditioning). No unchanged-state prediction cache.
Absorbing variant, discrete-time variational objective plus reconstruction.
Absorbing scores, score entropy and analytic sampling.
Masked training with a local DiT and ancestral decoding.
Basic absorbing order-agnostic construction, without upscaling or adaptive-budget machinery.
Posterior CE and base CTMC probability-velocity updates. No extra corrector.
The plug-in uniform diffusion recipe. No paper guidance extensions.
Uniform corruption with ordinary posterior CE; distinct from plug-in UDLM.
Continuous-relaxation / discrete curriculum (uses one-hot relaxations), uniform reverse inference.
Posterior CE and base CTMC probability-velocity updates. No extra corrector.
Mask / uniform mixture with composed ELBO. Not dynamic GIDD+.
The GIDD mask / uniform mixture trained with plain CE, as a baseline. Not dynamic GIDD+.
Masked / Gaussian hybrid with mask-dependent fusion and rank calibration.
Gated Gaussian / masked hybrid. The YAML records missing dropout, sharing / detachment and schedule / sampler alignment.
Learned-embedding Gaussian CE and adaptive clock; the adaptation machinery differs from the paper.
Gaussian one-hot linear flow and error-based clock; finite calibration bounds and local conditioning differ.
Embedding Gaussian CE, Gumbel clock, expected-embedding self-conditioning. Normalization, conditioning and architectural details still differ.
Learned embeddings and schedule, reconstruction / diffusion / prior terms, DDPM sampling. No separate original-PLAID preset.
Dirichlet bridge construction, CE and churn. Not Dirichlet Flow Matching or SSD-LM.
Geodesic sphere path and tangent integration. Not every paper architecture / velocity approximation.
von Mises-Fisher spherical flows with predictor-corrector inference. Paper classifier / damping details differ.
Token-CE language adaptation of Blurring Diffusion.
Block diffusion with a clean prefix, configurable block size, and ancestral sampling. No claim to the optimized paper schedules. Block-causal clean context, as in the paper (x0_causal: false).
BlockGen’s fixed-block baseline: masked blocks of 16, token-causal clean context, plain CE with observed tokens kept, ancestral sampling (16 calls per block). The reference’s fixed-block scripts use a block-causal context instead. A TinyGSM version is tinygsm_blockgen_masked.
The same fixed-block BlockGen baseline with uniform-state blocks and a token-causal context: plug-in output and the uniform ELBO (not posterior CE), uniform ancestral sampling. A TinyGSM version is tinygsm_blockgen_uniform.
Entropy-bounded unmasking on MDLM. This changes inference, not training.
Masked insertion. Uses our count / gap objectives, not the full paper variational objective.
Log-linear schedule: each token survives with probability α(t), which falls in a straight line from 1 to 0.
D3PM trains on a fixed grid of discrete time steps (orange dots) along the same log-linear path.
DUO uses the log-linear schedule plus a curriculum: early in training the uniform noise is applied to a softened relaxation of the tokens, which anneals to hard tokens.
GIDD’s path: a token stays clean, gets swapped for a random word, or gets masked. The random-word share peaks halfway, at pu = 0.2.
CANDI runs two schedules at once. Masking follows a log-linear path, and the Gaussian view is rank-calibrated against it.
CADD-style hybrids also keep one schedule per view; the YAML records where its schedule alignment differs from the paper.
CDCD-style: the base is an exponential VE schedule, and the adaptive clock refits it from the loss as training goes.
FLM-style: time is defined by an error clock. t is the fraction of tokens a greedy argmax would get wrong, and the Gaussian σ (0.2 to 4) is calibrated to produce it.
LangFlow-style: γ(t) follows Gumbel quantiles. Its location and scale are refit from detached cross-entropy feedback, not by the network’s optimizer. Press Play to watch them move.
RePlaid learns its −log SNR curve with a small monotone network, trained to lower the variance of the ELBO estimate. Press Play to watch it reshape.
The simplex process ships its own schedule, paired with a loss-derivative time sampler.
S-FLM moves along the geodesic; the schedule sets how far along the arc the state is at time t.
vMF spherical flows spread noise as a von Mises-Fisher distribution whose concentration falls over time.
Blurring diffusion blurs high-frequency feature bands first and low ones last, as in the image version.
Masked recipes only predict the masked positions; visible tokens are copied through unchanged.
RADD drops time conditioning: the posterior depends only on which tokens are still visible.
CANDI predicts a clean-token posterior at masked positions, fusing what the masked and Gaussian views say.
RePlaid adds a Gaussian prior term to the network’s logits before the softmax: each word gets a bonus for how well its embedding lines up with the current noisy vector.
D3PM’s bound is a sum over discrete steps: a reconstruction term at the start, a KL term per step, and a prior term at the end.
The masked ELBO reduces to cross-entropy on masked tokens, weighted by 1/t under the log-linear schedule.
SEDD trains score ratios with score entropy, using the same ELBO weighting.
GIDD’s ELBO is assembled from its masking and uniform parts. Shapes are illustrative.
Several papers, one construction: absorbing (mask) noise, a clean-token posterior at masked positions, the 1/t-weighted ELBO, and ancestral unmasking. The configs below differ only in details this figure does not draw.
Uniform noise trained with plain posterior cross-entropy. The two configs differ only in how the sampler takes each step.
One construction under many names: mask noise, a clean-token posterior at masked positions, the 1/t-weighted ELBO. Variants change the time steps and loss (D3PM) or the sampler (entropy-bounded). Block variants are under Block diffusion.
Masking noise, but the network predicts score ratios instead of clean tokens.
Tokens are swapped for random words instead of masked. Variants differ in the output and loss, and DUO adds a training curriculum.
The same flow-matching recipe on two kinds of discrete noise; only the corruption changes.
GIDD’s mask/uniform mixture, trained with its ELBO or with plain CE as a baseline.
Masking and Gaussian noise run side by side. CANDI and CADD differ in how the views are arranged and fused, and in the sampler.
Gaussian noise on vectors, solved back to tokens. The variants mostly differ in the schedule, plus self-conditioning, the loss and the sampler.
Noise on the unit sphere of embeddings. The variants differ in the output and the sampler.
Noise on the probability simplex with a Dirichlet bridge.
Blurring of embedding features along the sequence.
BlockGen’s mixture model with masked blocks: one model trained on block sizes 1 and 16 (5% / 95%, stratified across GPUs), token-causal context. Size 1 is pure noise + CE, the AR component. Mixture likelihood evaluation and the ARPC/EIPC samplers are not implemented.
The same 1 + 16 mixture with uniform-state blocks; size 16 keeps the plug-in uniform ELBO. Mixture likelihood evaluation and ARPC/EIPC are not implemented.
Generate blocks left to right and diffuse inside each block. Training reads the noisy and clean sequence together (2L) under a structured mask. The variants differ in how the clean context attends (BD3-LM: block causal; BlockGen: token causal), in the noise inside each block, and in whether one model is trained on several block sizes.
Masked diffusion that can also insert tokens, so the length is decided by the model.
Each recipe is a named config in the codebase (+experiment=<name>). Presets compose existing components; they do not certify paper benchmark parity.