CoChat
Sign Up
PreprintCited by 8,712Epigenetics and DNA MethylationGene expression and cancer classificationGenomics and Chromatin Dynamics

Denoising Diffusion Probabilistic Models

Open access
Ho, Jonathan, Jain, Ajay, Abbeel, Pieter·2020·arXiv
DOI 10.48550/arxiv.2006.11239arXiv 2006.11239
Open this paper in CoChat — free accountOpen PDF

Abstract

We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at https://github.com/hojonathanho/diffusion

AI snapshot

Written from the paper’s full text.

TL;DR

The paper shows diffusion probabilistic models can generate high-quality images by training a reverse Markov chain with a simplified objective connected to denoising score matching and Langevin dynamics. On unconditional CIFAR10 it reports IS 9.46 and FID 3.17, and on 256×256 LSUN it reports sample quality similar to ProgressiveGAN.

Key findings

  • On unconditional CIFAR10, the best model (`L_simple`) obtains an Inception score of 9.46 ± 0.11 and a state-of-the-art FID of 3.17, with NLL ≤ 3.75 (3.72) bits/dim. Training on the true variational bound yields better codelengths but worse sample quality.
  • On 256×256 LSUN, FID scores are 4.90 for the large LSUN Bedroom model, 6.36 for the small LSUN Bedroom model, 7.89 for LSUN Church, and 19.75 for LSUN Cat.
  • Ablations show that predicting ε with the simplified objective performs much better than predicting μ̃ with unweighted mean squared error; learning reverse-process variances leads to unstable training and poorer sample quality compared with fixed variances.
  • For progressive coding, the highest-quality CIFAR10 model has a rate of 1.78 bits/dim and distortion of 1.97 bits/dim, corresponding to RMSE 0.95 on a 0–255 scale; more than half of the lossless codelength describes imperceptible distortions.

Methods

  • The model uses a fixed forward Gaussian diffusion process with T = 1000 and variances β_t increasing linearly from 10⁻⁴ to 0.02, and a learned reverse Markov chain with Gaussian transitions.
  • The reverse process is parameterized by a U-Net backbone similar to PixelCNN++/Wide ResNet, with group normalization, self-attention at 16×16 resolution, and Transformer sinusoidal time embeddings; CIFAR10 uses 35.7M parameters, LSUN/CelebA-HQ use 114M, and a larger LSUN Bedroom variant uses approximately 256M.
  • Training optimizes the simplified objective `E ||ε − ε_θ(√ᾱ_t x_0 + √(1−ᾱ_t) ε, t)||²`; experiments are on CIFAR10, CelebA-HQ 256×256, and LSUN 256×256, evaluated with Inception score, FID, NLL, and rate-distortion.

Limitations

  • The paper states that its models do not have competitive log likelihoods compared with other likelihood-based models, and that most lossless codelength is consumed by imperceptible image details.
  • The progressive compression argument is stated as a proof of concept because Algorithms 3 and 4 depend on a procedure such as minimal random coding, which is not tractable for high-dimensional data.
  • Evident from the text: experiments are limited to image datasets (CIFAR10, CelebA-HQ, LSUN), and FID is computed with respect to the training set, with test-set FID for CIFAR10 reported as 5.24.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.11239

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

15 of 23 cited sources checked · 11 supported · 0 flagged · accuracy not yet measured

  • Findingcitation
    § Pieter Abbeel · p. 2

    Diffusion models have log likelihoods better than the large estimates annealed importance sampling has been reported to produce for energy based models and score matching.

    “(our models do, however, have log likelihoods better than the large estimates annealed importance sampling has been reported to produce for energy based models and score matching [11, 55]).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [11] [11] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems , pages 3603-3613, 2019. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [55] [55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. - source

      Section 4.3 (p. 6) mentions annealed importance sampling only as an inspiration for the paper’s sampling method. Section 6 (p. 9) discusses energy-based models and score matching, but gives no AIS log-likelihood estimates or comparison with diffusion models. Searches for those estimates and comparisons found no supporting passage. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • Interpretationcitation
    § 5 Related Work · p. 8

    The ε-prediction reverse process parameterization establishes a connection between diffusion models and denoising score matching over multiple noise levels with annealed Langevin dynamics for sampling.

    “Our ϵ -prediction reverse process parameterization establishes a connection between diffusion models and denoising score matching over multiple noise levels with annealed Langevin dynamics for sampling [55, 56].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [55] [55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. - source
    • cites [56] [56] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011 , 2020. - source

      Section 2.2 (pp. 2–3) describes estimating scores at multiple Gaussian noise levels and sampling with annealed Langevin dynamics. It does not establish the claimed connection to diffusion models or discuss an ε-prediction reverse-process parameterization. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation
    § C Discussion on related work · p. 15

    NCSN omits the scaling factor.

    “Diffusion models scale down the data with each forward process step (by a √ 1 -β t factor) so that variance does not grow when adding noise, thus providing consistently scaled inputs to the neural net reverse process. NCSN omits this scaling factor.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [55] [55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. - source
    • cites [56] [56] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011 , 2020. - source
  • Findingnone given
    § Abstract · p. 1

    On 256x256 LSUN, the model obtains sample quality similar to ProgressiveGAN.

    “On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Pieter Abbeel · p. 2

    Diffusion models do not have competitive log likelihoods compared to other likelihood-based models.

    “Despite their sample quality, our models do not have competitive log likelihoods compared to other likelihood-based models”
    Quote in the paperSubstantial claim
  • May combine two claims. 36
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Interpretationcitation
    § Pieter Abbeel · p. 2

    A certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling.

    “In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling (Section 3.2) [55, 61].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [55] [55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. - source
    • cites [61] [61] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23(7):1661-1674, 2011. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2 Background · p. 2

    The forward process variances β_t can be learned by reparameterization or held constant as hyperparameters.

    “The forward process variances β t can be learned by reparameterization [33] or held constant as hyperparameters”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [33] [33] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 , 2013. - source

      Section 2.4 (p. 4) explains how reparameterization makes an estimate differentiable with respect to distribution parameters, supporting the general learning method. The paper does not address forward-process variances β_t or the choice to hold them constant as hyperparameters, so it does not establish that specific claim. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § C Discussion on related work · p. 15

    NCSN's sampler coefficients are set by hand post-hoc, and their training procedure is not guaranteed to directly optimize a quality metric of their sampler.

    “In contrast, NCSN's sampler coefficients are set by hand post-hoc, and their training procedure is not guaranteed to directly optimize a quality metric of their sampler.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [55] [55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. - source
    • cites [56] [56] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011 , 2020. - source

  • May combine two claims.

    Interpretationnone given
    § C Discussion on related work · p. 15

    The model architecture, forward process definition, and prior differ from NCSN in subtle but important ways that improve sample quality.

    “Our model architecture, forward process definition, and prior differ from NCSN [55, 56] in subtle but important ways that improve sample quality, and, notably, we directly train our sampler as a latent variable model rather than adding it after training post-hoc.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [55] [55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. - source
    • cites [56] [56] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011 , 2020. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § Extra information · p. 13

    The lossy compression argument in Section 4.3 is only a proof of concept.

    “Our lossy compression argument in Section 4.3 is only a proof of concept, because Algorithms 3 and 4 depend on a procedure such as minimal random coding [20], which is not tractable for high dimensional data.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [20] [20] Marton Havasi, Robert Peharz, and José Miguel Hernández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations , 2019. - source

  • May combine two claims.

    Findingcitation
    § 4.3 Progressive coding · p. 6

    The lossless codelengths are better than the large estimates reported for energy based models and score matching using annealed importance sampling, but not competitive with other types of likelihood-based generative models.

    “Still, while our lossless codelengths are better than the large estimates reported for energy based models and score matching using annealed importance sampling [11], they are not competitive with other types of likelihood-based generative models [7].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [11] [11] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems , pages 3603-3613, 2019. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [7] [7] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Experimental details · p. 14

    Without dropout on CIFAR10, the authors obtained poorer samples reminiscent of the overfitting artifacts in an unregularized PixelCNN++.

    “Without dropout on CIFAR10, we obtained poorer samples reminiscent of the overfitting artifacts in an unregularized PixelCNN++ [52].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [52] [52] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations , 2017. - source

  • May combine two claims.

    Premisecitation
    § Extra information · p. 13

    Algorithms 3 and 4 depend on a procedure such as minimal random coding [20], which is not tractable for high dimensional data.

    “Our lossy compression argument in Section 4.3 is only a proof of concept, because Algorithms 3 and 4 depend on a procedure such as minimal random coding [20], which is not tractable for high dimensional data.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [20] [20] Marton Havasi, Robert Peharz, and José Miguel Hernández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations , 2019. - source

      Section 3.1 says the basic minimal random coding algorithm requires a number of samples that grows exponentially with KL divergence, making it infeasible for encoding a practical neural network. But Section 3.3 describes a block-wise implementation to address that obstacle, so the paper does not establish that minimal random coding is generally intractable for high-dimensional data. The source cannot verify the citing paper’s separate statement about its Algorithms 3 and 4. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    On unconditional CIFAR10, the model obtains an Inception score of 9.46 and a state-of-the-art FID score of 3.17.

    “On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 3.2 Reverse process and L 1: T -1 · p. 4

    Predicting x0 leads to worse sample quality early in experiments.

    “(There is also the possibility of predicting x 0 , but we found this to lead to worse sample quality early in our experiments.)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 4.1 Sample quality · p. 5

    The FID score computed with respect to the test set is 5.24, which is still better than many of the training set FID scores in the literature.

    “when we compute it with respect to the test set, the score is 5.24, which is still better than many of the training set FID scores in the literature.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Progressive coding · p. 6

    The CIFAR10 model with the highest quality samples has a rate of 1.78 bits/dim and a distortion of 1.97 bits/dim, which amounts to a root mean squared error of 0.95 on a scale from 0 to 255.

    “Treating the variational bound terms L 1 + · · · + L T as rate and L 0 as distortion, our CIFAR10 model with the highest quality samples has a rate of 1.78 bits/dim and a distortion of 1.97 bits/dim, which amounts to a root mean squared error of 0.95 on a scale from 0 to 255.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § B Experimental details · p. 14

    The 256×256 images seemed unstable to train with the larger learning rate.

    “We set the learning rate to 2 × 10 -4 without any sweeping, and we lowered it to 2 × 10 -5 for the 256 × 256 images, which seemed unstable to train with the larger learning rate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Reverse process parameterization and training objective ablation · p. 6

    Predicting ε performs approximately as well as predicting μ when trained on the variational bound with fixed variances, but much better when trained with the simplified objective.

    “Predicting ϵ , as we proposed, performs approximately as well as predicting ˜ µ when trained on the variational bound with fixed variances, but much better when trained with our simplified objective.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Reverse process parameterization and training objective ablation · p. 6

    Learning reverse process variances leads to unstable training and poorer sample quality compared to fixed variances.

    “We also see that learning reverse process variances (by incorporating a parameterized diagonal Σ θ ( x t ) into the variational bound) leads to unstable training and poorer sample quality compared to fixed variances.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.4 Simplified training objective · p. 4

    Training on the simplified objective variant is beneficial to sample quality and simpler to implement.

    “However, we found it beneficial to sample quality (and simpler to implement) to train on the following variant of the variational bound:”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Algorithm 4 Receiving · p. 6

    Training on the true variational bound yields better codelengths than training on the simplified objective, but the simplified objective yields the best sample quality.

    “We find that training our models on the true variational bound yields better codelengths than training on the simplified objective, as expected, but the latter yields the best sample quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § D Samples · p. 15

    After 1000 diffusion steps, source information is lost and interpolations are novel samples.

    “On the other hand, after 1000 diffusion steps, source information is lost and interpolations are novel samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Progressive coding · p. 8

    The Gaussian diffusion length is not restricted to equal the data dimension; T=1000 is less than the dimension of the 32×32×3 or 256×256×3 images in the experiments.

    “Moreover, the Gaussian diffusion length is not restricted to equal the data dimension; for instance, we use T = 1000 , which is less than the dimension of the 32 × 32 × 3 or 256 × 256 × 3 images in our experiments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Interpolation · p. 8

    Larger t results in coarser and more varied interpolations, with novel samples at t=1000.

    “Larger t results in coarser and more varied interpolations, with novel samples at t = 1000 (Appendix Fig. 9).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Progressive coding · p. 7

    When t is small, all but fine details are preserved, and when t is large, only large scale features are preserved.

    “When t is small, all but fine details are preserved, and when t is large, only large scale features are preserved.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.4 Interpolation · p. 8

    The reverse process produces plausible interpolations of CelebA-HQ 256×256 images at t=500 that smoothly vary attributes such as pose, skin tone, hairstyle, expression and background, but not eyewear.

    “The reverse process produces high-quality reconstructions, and plausible interpolations that smoothly vary attributes such as pose, skin tone, hairstyle, expression and background, but not eyewear.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Experimental details · p. 14

    Random horizontal flips during training improve sample quality slightly on CIFAR10.

    “We used random horizontal flips during training for CIFAR10; we tried training both with and without flips, and found flips to improve sample quality slightly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Extra information · p. 13

    The authors' L_simple model achieves FID 6.36 on LSUN Bedroom, 7.89 on LSUN Church, and 19.75 on LSUN Cat.

    “Ours ( L simple ) | 6.36 | 7.89 | 19.75 |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Progressive coding · p. 7

    The distortion decreases steeply in the low-rate region of the rate-distortion plot, indicating that the majority of the bits are indeed allocated to imperceptible distortions.

    “The distortion decreases steeply in the low-rate region of the rate-distortion plot, indicating that the majority of the bits are indeed allocated to imperceptible distortions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 2 Background · p. 3

    All KL divergences in Eq. (5) are comparisons between Gaussians, so they can be calculated in a Rao-Blackwellized fashion with closed form expressions instead of high variance Monte Carlo estimates.

    “Consequently, all KL divergences in Eq. (5) are comparisons between Gaussians, so they can be calculated in a Rao-Blackwellized fashion with closed form expressions instead of high variance Monte Carlo estimates.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 3.2 Reverse process and L 1: T -1 · p. 3

    The first variance choice is optimal for x0∼N(0,I), and the second is optimal for x0 deterministically set to one point.

    “The first choice is optimal for x 0 ∼ N ( 0 , I ) , and the second is optimal for x 0 deterministically set to one point.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] [53] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265, 2015. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 3 Diffusion models and denoising autoencoders · p. 3

    A new explicit connection between diffusion models and denoising score matching leads to a simplified, weighted variational bound objective for diffusion models.

    “To guide our choices, we establish a new explicit connection between diffusion models and denoising score matching (Section 3.2) that leads to a simplified, weighted variational bound objective for diffusion models (Section 3.4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § C Discussion on related work · p. 15

    Both of these factors prevent distribution shift when sampling.

    “Both of these factors prevent distribution shift when sampling.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.3 Data scaling, reverse process decoder, and L 0 · p. 4

    The discrete decoder choice ensures that the variational bound is a lossless codelength of discrete data, without adding noise to the data or incorporating the Jacobian of the scaling operation into the log likelihood.

    “our choice here ensures that the variational bound is a lossless codelength of discrete data, without need of adding noise to the data or incorporating the Jacobian of the scaling operation into the log likelihood.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Image data consists of integers in {0,1,...,255} scaled linearly to [-1,1].

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 6 Conclusion · p. 9

    Diffusion models seem to have excellent inductive biases for image data.

    “Since diffusion models seem to have excellent inductive biases for image data, we look forward to investigating their utility in other data modalities and as components in other types of generative models and machine learning systems.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The datasets used (CIFAR10, CelebA-HQ, LSUN) are representative of image data for assessing diffusion models' inductive biases.

  • May combine two claims.

    Interpretationnone given
    § C Discussion on related work · p. 15

    Unlike NCSN, the authors' forward process destroys signal (D_KL(q(x_T|x_0) || N(0,I)) ≈ 0), ensuring a close match between the prior and aggregate posterior of x_T.

    “Unlike NCSN, our forward process destroys signal ( D KL ( q ( x T | x 0 ) ‖ N ( 0 , I )) ≈ 0 ), ensuring a close match between the prior and aggregate posterior of x T .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.3 Progressive coding · p. 7

    With the specified choices, D_KL(q(x_T)||p(x_T))=0, and minimizing D_KL(q(x_{t-1}|x_t)||pθ(x_{t-1}|x_t)) trains pθ to copy coordinates t+1,...,T unchanged and to predict the t-th coordinate given t+1,...,T.

    “With these choices, D KL ( q ( x T ) ‖ p ( x T )) = 0 , and minimizing D KL ( q ( x t -1 | x t ) ‖ p θ ( x t -1 | x t )) trains p θ to copy coordinates t +1 , . . . , T unchanged and to predict the t th coordinate given t +1 , . . . , T .”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: For the autoregressive connection, pθ(x_{t-1}|x_t) is taken to be a fully expressive conditional distribution.

  • May combine two claims.

    Interpretationreasoning
    § 6 Conclusion · p. 9

    The paper has found connections among diffusion models and variational inference for training Markov chains, denoising score matching and annealed Langevin dynamics (and energy-based models by extension), autoregressive models, and progressive lossy compression.

    “Wehave presented high quality image samples using diffusion models, and we have found connections among diffusion models and variational inference for training Markov chains, denoising score matching and annealed Langevin dynamics (and energy-based models by extension), autoregressive models, and progressive lossy compression.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § Extra information · p. 13

    These algorithms serve as a compression interpretation of the variational bound (5) of Sohl-Dickstein et al. [53], not yet as a practical compression system.

    “These algorithms serve as a compression interpretation of the variational bound (5) of Sohl-Dickstein et al. [53], not yet as a practical compression system.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53] [53] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265, 2015. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationnone given
    § 3.2 Reverse process and L 1: T -1 · p. 4

    The ε-prediction parameterization both resembles Langevin dynamics and simplifies the diffusion model's variational bound to an objective that resembles denoising score matching.

    “We have shown that the ϵ -prediction parameterization both resembles Langevin dynamics and simplifies the diffusion model's variational bound to an objective that resembles denoising score matching.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § C Discussion on related work · p. 15

    The authors' β_t are very small, which ensures that the forward process is reversible by a Markov chain with conditional Gaussians.

    “Also unlike NCSN, our β t are very small, which ensures that the forward process is reversible by a Markov chain with conditional Gaussians.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given
    § 5 Related Work · p. 8

    Diffusion models are designed so that q has no parameters and the top-level latent x_T has nearly zero mutual information with the data x_0.

    “While diffusion models might resemble flows [9, 46, 10, 32, 5, 16, 23] and VAEs [33, 47, 37], diffusion models are designed so that q has no parameters and the top-level latent x T has nearly zero mutual information with the data x 0 .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § A Extended derivations · p. 14

    The alternate version of L is not tractable to estimate.

    “The following is an alternate version of L . It is not tractable to estimate, but it is useful for our discussion in Section 4.3.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § C Discussion on related work · p. 15

    The authors' training procedure directly trains their sampler to match the data distribution after T steps: it trains the sampler as a latent variable model using variational inference.

    “Thus, our training procedure directly trains our sampler to match the data distribution after T steps: it trains the sampler as a latent variable model using variational inference.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § 4.3 Progressive coding · p. 8

    Gaussian diffusions can be made shorter for fast sampling or longer for model expressiveness.

    “Gaussian diffusions can be made shorter for fast sampling or longer for model expressiveness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given
    § C Discussion on related work · p. 15

    Diffusion models scale down the data with each forward process step (by a √(1-β_t) factor) so that variance does not grow when adding noise, thus providing consistently scaled inputs to the neural net reverse process.

    “Diffusion models scale down the data with each forward process step (by a √ 1 -β t factor) so that variance does not grow when adding noise, thus providing consistently scaled inputs to the neural net reverse process. NCSN omits this scaling factor.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given
    § D Samples · p. 15

    During sampling, both the prior x_T ∼ N(0,I) and Langevin dynamics are stochastic.

    “During sampling, both the prior x T ∼ N ( 0 , I ) and Langevin dynamics are stochastic.”
    Quote in the paperNot a substantial claim
43 of 43

  • The authors state this as an assumption.

    Image data consists of integers in {0,1,...,255} scaled linearly to [-1,1].

    “We assume that image data consists of integers in { 0 , 1 , . . . , 255 } scaled linearly to [ -1 , 1] .”

  • The authors state this as an assumption.

    There exists a procedure, such as minimal random coding, that can transmit a sample x∼q(x) using approximately D_KL(q(x)||p(x)) bits on average for any distributions p and q, for which only p is available to the receiver beforehand.

    “which assume access to a procedure, such as minimal random coding [19, 20], that can transmit a sample x ∼ q ( x ) using approximately D KL ( q ( x ) ‖ p ( x )) bits on average for any distributions p and q , for which only p is available to the receiver beforehand.”

  • The authors state this as an assumption.

    For the autoregressive connection, pθ(x_{t-1}|x_t) is taken to be a fully expressive conditional distribution.

    “and, for the sake of argument, taking p θ ( x t -1 | x t ) to be a fully expressive conditional distribution.”

  • The paper relies on this without support.

    The datasets used (CIFAR10, CelebA-HQ, LSUN) are representative of image data for assessing diffusion models' inductive biases.

    “Since diffusion models seem to have excellent inductive biases for image data, we look forward to investigating their utility in other data modalities and as components in other types of generative models and machine learning systems.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Minimizing the DDPM loss function also minimizes the variational lower bound of the KL-divergence between the data distribution and the distribution of samples drawn from the DDPM.

    “As shown in [18], minimizing the loss function in Eq 3 also minimizes the variational lower bound of the KL-divergence between the data distribution p ( x 0 ) and the distribution of samples drawn from the DDPM q ( x 0 ) using Eq 1.”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as [18]

    The Background (p. 2) says training optimizes a variational bound on negative log likelihood, not a lower bound on KL divergence between the data and sample distributions. It also identifies q(x₀) as the data distribution and pθ(x₀) as the model distribution—the reverse of the citing sentence’s labels. Section 3.4 (p. 5) further says the simplified DDPM objective is a weighted variational bound, distinct from the standard bound. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • An α slightly smaller than 1 has been shown to improve stability.

    “An α slightly smaller than 1 has been shown to improve stability [18].”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as [18]

    The Background (p. 2) defines αₜ = 1 − βₜ, and the discussion of related work (p. 15) says the small βₜ values scale inputs consistently and help prevent distribution shift. The paper does not report that choosing α slightly below 1 was shown to improve stability, so the claimed stability result is not supported. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Diffusion models have had considerable success in image and language generative modelling but limited success when applied to protein modelling, probably due to the complexity of protein backbone geometry and sequence-structure relationships.

    “Diffusion models10,11 have had considerable success in image and language generative modelling but limited success when applied to protein modelling, probably due to the complexity of protein backbone geometry and sequence–structure relationships.”

    From De novo design of protein structure and function with RFdiffusion (2023) · cites this paper as [11]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Diffusion model is a recently popular generative model that can also be utilized for reconstruction-based anomaly detection.

    “Diffusion model [118] is a recently popular generative model that can also be utilized for reconstruction-based anomaly detection.”

    From Deep Industrial Image Anomaly Detection: A Survey (2024) · cites this paper as [118]

    The introduction (p. 1) identifies diffusion models as generative models, and Section 4.4 (p. 8) demonstrates image reconstructions. Our copy does not discuss using those reconstructions for anomaly detection, so it supports the model description but not that application. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Interpolation formula uses clean images instead of noisy latents

    Minor

    The formula for the interpolated latent combines the clean source images x0 and x′0. The accompanying description says the method interpolates corrupted versions of those images, xt and x′t.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. [14][14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , pages 2672-2680, 2014. -
  2. [15][15] Anirudh Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator as a stochastic recurrent net. In Advances in Neural Information Processing Systems , pages 4392-4402, 2017. -
  3. [17][17] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations , 2020. -
  4. [26][26] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning , pages 2410-2419, 2018. -
  5. [27][27] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations , 2018. -
  6. [32][32] Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems , pages 10215-10224, 2018. -
  7. [36][36] Daniel Levy, Matt D. Hoffman, and Jascha Sohl-Dickstein. Generalizing Hamiltonian Monte Carlo with neural networks. In International Conference on Learning Representations , 2018. -
  8. [43][43] Georg Ostrovski, Will Dabney, and Remi Munos. Autoregressive quantile networks for generative modeling. In International Conference on Machine Learning , pages 3936-3945, 2018. -
  9. [45][45] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQVAE-2. In Advances in Neural Information Processing Systems , pages 14837-14847, 2019. -
  10. [66][66] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 3-19, 2018. -

Cited by

Sorted by citations

Related

Authors
Ho, J., Jain, A., & Abbeel, P.
Affiliations
University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.11239
arXiv
2006.11239
OpenAlex
W3036167779
Semantic Scholar
5c126ae3421f05768d8edd97ecd44b1364e2c99a
Cited by
8,712
References
66
Citations by year
2026: 203 · 2025: 2931 · 2024: 3789 · 2023: 1580 · 2022: 125 · 2021: 71 · 2020: 13
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Epigenetics and DNA Methylation
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
Keywords
denoising diffusion probabilistic models, acute myeloid leukemia, myelodysplastic syndrome, whole-genome bisulfite sequencing, reduced representation bisulfite sequencing, long-range interactions, single-cell bisulfite sequencing, HepG2 cells, CD34+ cells, glioblastoma, breast cancer, Docker deployment, GPU acceleration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review