While continuous diffusion has shown remarkable success in continuous domains such as image generation, its direct application to discrete data has underperformed compared to purely discrete formulations. This gap is counterintuitive, given that continuous diffusion learns score functions that enable joint evolution across multiple positions.
To understand this gap, we introduce token identifiability as an analytical framework for understanding how Gaussian noise corrupts discrete data through two mechanisms.
We reveal that these mechanisms scale differently with vocabulary size, creating a temporal dissonance: at noise levels where discrete corruption preserves enough structure for conditional learning, continuous denoising is trivial; at noise levels where continuous denoising is meaningful, discrete corruption destroys nearly all conditional structure.
To solve this, we propose CANDI (Continuous ANd DIiscrete diffusion), a hybrid framework that decouples discrete and continuous corruption, enabling simultaneous learning of both conditional structure and continuous geometry. We empirically validate the temporal dissonance phenomenon and demonstrate that CANDI successfully avoids it.
This unlocks the benefits of continuous diffusion for discrete spaces: on controlled generation, CANDI enables classifier-based guidance with off-the-shelf classifiers through simple gradient addition; on text generation, CANDI outperforms masked diffusion at low NFE, demonstrating the value of learning continuous gradients for discrete spaces.
We introduce a new framework for understanding continuous corruption of discrete data, and discover a "temporal dissonance" - a fundamental misalignment in how continuous noise corrupts discrete data.
We introduce token identifiability as a framework to understand how Gaussian noise corrupts discrete data through two distinct mechanisms:
Discrete identity corruption controls how the model learns conditional dependencies between tokens. Unlike images, where nearby pixels share values due to spatial smoothness, discrete sequences lack geometric continuity: conditionally related tokens such as "New" and "York" have entirely different identities. To capture such co-occurrence patterns, the model must see clean anchor tokens during training.
Continuous rank degradation controls how the model learns the score function via continuous denoising, enabling coordinated refinement across positions. This gradient signal is the key strength of continuous diffusion: it allows multiple positions jointly evolve at once. Without access to a continuous score function, updates rely on independent conditional sampling, which leads to incoherent generations, especially at low NFE where many positions must be updated in parallel.
These mechanisms scale differently with vocabulary size, creating a temporal dissonance: when discrete structure is learnable, continuous denoising is trivial; when continuous denoising is meaningful, discrete structure is destroyed.
Decouple discrete and continuous corruption to get the best of both worlds.
Continuous ANd DIscrete diffusion solves temporal dissonance with one key insight: decouple the corruptions.
By separating discrete and continuous noise schedules, CANDI enables simultaneous learning of:
We introduce a hybrid noising kernel that uses discrete masking corruption to decouple discrete corruption from Gaussian noise. Paradoxically, decoupling discrete corruption from Gaussian noise allows us to better coordinate both forms of corruption with each other. As we show below, this structured kernel allows for a linear relation between discrete identity corruption and continuous rank degradation.
We use frontier analysis to quantify generative quality and demonstrate why this leads to more robust results than single-point analysis.
Temperature is a crucial hyperparameter for text generation, as it allows flexible control between coherence and diversity. As the same autoregressive model may have different optimal temperatures for different tasks, it follows that completely different diffusion models will have different optimal operating regions for temperature. A global temperature of 1.0 may correspend to different regions of the diversity-coherence frontier across differnet models.
To address this, we use frontier analysis to evaluate generative quality across a range of temperatures, plotting the best achievable trade-off between diversity and coherence. We use this methodology across all experiments, as the cost of the temperature sweep is negligible compared to training the models themselves. This allows us to compare models based on their overall capabilities rather than at a single operating point.
Here we compare CANDI, Duo, and MDLM across different temperatures to demonstrate that rankings can change significantly based on the temperatures. All methods lie within the same general range of entropy and perplexity, but the relative performance varies significantly based on temperature choice.
Our theory of token identifiability and the temporal dissonance predict the following:
We validate that temporal dissonance exists in continuous diffusion baselines and demonstrate that CANDI successfully avoids it.
Coordinated refinement across positions beats independent token sampling.
Continuous gradients enable coordinated refinement across positions, rather than independent token sampling. To demonstrate this, we train a 170 million parameter CANDI model with the diffusion transformer architecture on OWT for 1,000,000 steps. We compare against a strong masked diffusion baseline (MDLM) and a strong uniform diffusion baseline (DUO), using the official checkponts provided by the authors and the same model architecture.
Learning continuous gradients allow for plug-and-play guidance with off-the-shelf classifiers.
Prior guidance methods for discrete diffusion require specially trained classifiers for each diffusion model.
Furthermore, the gradient information must be adapted to suit the categorical distributions used at inference for pure discrete diffusion methods
In contrast, CANDI can leverage off-the-shelf continuous classifiers for guidance with simple gradient addition.
We use the QM9 molecular dataset and compare against MDLM and UDLM using diffusion classifiers. We train classifers on two chemical properties: QED and ring count. For CANDI, we train a normal classifier without any diffusion-specific augmentations.
We visualize frontiers in terms of target property value (coherence) and novel molecules (diversity). We ablate over temperature and show the best frontiers for each method.
CANDI: Hybrid Discrete-Continuous Diffusion Models
@article{pynadath2025candi,
author = {Patrick Pynadath, Jiaxin Shi and Ruqi Zhang},
title = {CANDI: Hybrid Discrete-Continuous Diffusion Models},
journal = {arXiv preprint},
year = {2025},
}