CoChat
Sign Up
PreprintCited by 482Generative Adversarial Networks and Image SynthesisAdversarial Robustness in Machine LearningImage and Signal Denoising Methods

Improved Techniques for Training Score-Based Generative Models

Open access
Song, Yang, Ermon, Stefano·2020·arXiv
DOI 10.48550/arxiv.2006.09011arXiv 2006.09011
Open this paper in CoChat — free accountOpen PDF

Abstract

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.09011

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The ε-prediction reverse process parameterization establishes a connection between diffusion models and denoising score matching over multiple noise levels with annealed Langevin dynamics for sampling.

    “Our ϵ -prediction reverse process parameterization establishes a connection between diffusion models and denoising score matching over multiple noise levels with annealed Langevin dynamics for sampling [55, 56].”

    From Denoising Diffusion Probabilistic Models (2020) · cites this paper as [56]

    Section 2.2 (pp. 2–3) describes estimating scores at multiple Gaussian noise levels and sampling with annealed Langevin dynamics. It does not establish the claimed connection to diffusion models or discuss an ε-prediction reverse-process parameterization. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The model architecture, forward process definition, and prior differ from NCSN in subtle but important ways that improve sample quality.

    “Our model architecture, forward process definition, and prior differ from NCSN [55, 56] in subtle but important ways that improve sample quality, and, notably, we directly train our sampler as a latent variable model rather than adding it after training post-hoc.”

    From Denoising Diffusion Probabilistic Models (2020) · cites this paper as [56]

  • NCSN's sampler coefficients are set by hand post-hoc, and their training procedure is not guaranteed to directly optimize a quality metric of their sampler.

    “In contrast, NCSN's sampler coefficients are set by hand post-hoc, and their training procedure is not guaranteed to directly optimize a quality metric of their sampler.”

    From Denoising Diffusion Probabilistic Models (2020) · cites this paper as [56]

  • NCSN omits the scaling factor.

    “Diffusion models scale down the data with each forward process step (by a √ 1 -β t factor) so that variance does not grow when adding noise, thus providing consistently scaled inputs to the neural net reverse process. NCSN omits this scaling factor.”

    From Denoising Diffusion Probabilistic Models (2020) · cites this paper as [56]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. [1][1] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems , pages 11895-11907, 2019. -
  2. [3][3] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(Apr):695-709, 2005. -
  3. [5][5] Martin Raphan and Eero P Simoncelli. Least squares estimation without priors or supervision. Neural computation , 23(2):374-420, 2011. -
  4. [6][6] Gareth O Roberts, Richard L Tweedie, et al. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli , 2(4):341-363, 1996. -
  5. [7][7] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688, 2011. -
  6. [8][8] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019 , page 204, 2019. -
  7. [9][9] Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Ioannis Mitliagkas, and Rémi Tachet des Combes. Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.05475 , 2020. - · arXiv
  8. [10][10] Saeed Saremi and Aapo Hyvarinen. Neural empirical bayes. Journal of Machine Learning Research , 20:1-23, 2019. -
  9. [11][11] Zengyi Li, Yubei Chen, and Friedrich T Sommer. Learning energy-based models in highdimensional spaces with multi-scale denoising score matching. arXiv , pages arXiv-1910, 2019. -
  10. [12][12] Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640 , 2020. - · arXiv
  11. [14][14] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106(496):1602-1614, 2011. -
  12. [16][16] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015. -
  13. [17][17] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems , pages 4790-4798, 2016. -
  14. [18][18] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689 , 2019. - · arXiv
  15. [19][19] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems , pages 5767-5777, 2017. -
  16. [20][20] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations , 2018. -
  17. [21][21] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973 , 2018. - · arXiv
  18. [22][22] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems , pages 5228-5237, 2018. -
  19. [23][23] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems , pages 14837-14847, 2019. -
  20. [25][25] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015. - · arXiv
  21. [27][27] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015. - · arXiv
  22. [29][29] Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Li F Fei-Fei, and Michael Bernstein. Hype: A benchmark for human eye perceptual evaluation of generative models. In Advances in Neural Information Processing Systems , pages 3444-3456, 2019. -
  23. [30][30] Jiaming Song and Stefano Ermon. Bridging the gap between f -gans and wasserstein gans. arXiv preprint arXiv:1910.09779 , 2019. - · arXiv
  24. [32][32] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017. - · arXiv
  25. [33][33] David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717 , 2017. · arXiv

Cited by

Sorted by citations

Authors
Song, Y., & Ermon, S.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.09011
arXiv
2006.09011
OpenAlex
W3035384201
Cited by
482
References
6
Citations by year
2026: 1 · 2025: 112 · 2024: 167 · 2023: 120 · 2022: 20 · 2021: 47 · 2020: 15
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
Keywords
score-based generative models, image generation, training stability, exponential moving average, failure modes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review