CoChat
Sign Up
PreprintCited by 1Generative Adversarial Networks and Image SynthesisImage and Signal Denoising MethodsAdversarial Robustness in Machine Learning

The DOI matches its DataCite record.

Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion

Open access
Hoogeboom, Emiel, Mensink, Thomas, Heek, Jonathan, Lamerigts, Kay, Gao, Ruiqi, Salimans, Tim·2024·arXiv
DOI 10.48550/arxiv.2410.19324arXiv 2410.19324
Open this paper in CoChat — free accountOpen PDF

Abstract

Latent diffusion models have become the popular choice for scaling up diffusion models for high resolution image synthesis. Compared to pixel-space models that are trained end-to-end, latent models are perceived to be more efficient and to produce higher image quality at high resolution. Here we challenge these notions, and show that pixel-space models can be very competitive to latent models both in quality and efficiency, achieving 1.5 FID on ImageNet512 and new SOTA results on ImageNet128, ImageNet256 and Kinetics600. We present a simple recipe for scaling end-to-end pixel-space diffusion models to high resolutions. 1: Use the sigmoid loss-weighting (Kingma & Gao, 2023) with our prescribed hyper-parameters. 2: Use our simplified memory-efficient architecture with fewer skip-connections. 3: Scale the model to favor processing the image at a high resolution with fewer parameters, rather than using more parameters at a lower resolution. Combining these with guidance intervals, we obtain a family of pixel-space diffusion models we call Simpler Diffusion (SiD2).

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.19324

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Operating in a learned latent space reduces computational costs during both training and inference while simplifying the denoising task.

    “This design choice offers several advantages: it reduces computational costs during both training and inference while simplifying the denoising task (Hoogeboom et al., 2024; Rombach et al., 2022).”

    From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Hoogeboom et al., 2024

    The Introduction says that encoding images in a lower-dimensional latent space helps models scale better in training compute. “Comparing diffusion models on compute” reports a latent-model advantage in quality versus compute for both training and sampling. Neither passage says that the latent space simplifies the denoising task. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 50

  1. Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Ji- aming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff- i: Text-to-image diffusion models with an ensemble of expert denoisers. CoRR, abs/2211.01324, 2022. · arXiv
  2. Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023. 5
  3. Ting Chen. On the importance of noise scheduling for diffu- sion models. arxiv, 2023. 1, 4, 7
  4. Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. CoRR, abs/2105.05233, 2021. 4, 7 · arXiv
  5. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 5
  6. Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Ángel Bautista, and Josh M. Susskind. f-dm: A multi-stage dif- fusion model via progressive signal transformation. CoRR, abs/2210.04955, 2022. 4, 6 · arXiv
  7. Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh M. Susskind, and Navdeep Jaitly. Matryoshka diffusion models. CoRR, abs/2310.15111, 2023. 4, 6, 7 · arXiv
  8. Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. In Computer Vision -ECCV 2024 -18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXIX, pages 393-411. Springer, 2024. 8
  9. Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. CoRR, abs/2312.02139, 2023. 7 · arXiv
  10. Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Mul- tistep consistency models. CoRR, abs/2403.06807, 2024. 3 · arXiv
  11. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Infor- mation Processing Systems 2020, NeurIPS, 2020. 1, 5
  12. Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffu- sion models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS, 2022. 8
  13. Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, ICML, pages 13213-13232. PMLR, 2023. 1, 3, 4, 5, 6, 7, 8
  14. Zhongzhan Huang, Zhou Pan, Shuicheng Yan, and Liang Lin. Scalelong: Towards more stable training of diffusion model via scaling network long skip connection. In NeurIPS, 2023. 5
  15. Allan Jabri, David J. Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. CoRR, abs/2212.11972, 2022. 7, 8 · arXiv
  16. Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, and Jingfeng Zhang. Scedit: Efficient and controllable image diffusion generation via skip connection editing. Technical Report 2312.11392, arXiv, 2023. 5
  17. Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distri- bution augmentation for generative modeling. In Proceedings of the 37th International Conference on Machine Learning, ICML, 2020. 3, 7
  18. Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elu- cidating the design space of diffusion-based generative mod- els. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS, 2022. 5, 6
  19. Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hell- sten, Timo Aila, and Samuli Laine. Analyzing and im- proving the training dynamics of diffusion models. CoRR, abs/2312.02696, 2023. 1, 4, 6, 8, 2, 5 · arXiv
  20. Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. CoRR, abs/2406.02507, 2024. 2 · arXiv
  21. Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Pagoda: Progressive growing of a one- step generator from a low-resolution diffusion teacher. CoRR, abs/2405.14822, 2024. 7, 4 · arXiv
  22. Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Mu- rata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Represen- tations, 2024. 4
  23. Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018. 1
  24. Diederik P. Kingma and Ruiqi Gao. Understanding the dif- fusion objective as a weighted integral of elbos. CoRR, abs/2303.00848, 2023. 1, 2, 3, 4, 7 · arXiv
  25. Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. CoRR, abs/2107.00630, 2021. 2, 5 · arXiv
  26. Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. CoRR, abs/2404.07724, 2024. 7, 2 · arXiv
  27. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxi- milian Nickel, and Matthew Le. Flow matching for gener- ative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda. OpenReview.net, 2023. 2
  28. Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal ap- proach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 4
  29. Jiajun Ma, Shuchen Xue, Tianyang Hu, Wenjia Wang, Zhao- qiang Liu, Zhenguo Li, Zhi-Ming Ma, and Kenji Kawaguchi. The surprising effectiveness of skip-tuning in diffusion sam- pling. In Proceedings of the 41st International Conference on Machine Learning, pages 34053-34074, 2024. 5
  30. William Peebles and Saining Xie. Scalable diffusion models with transformers. CoRR, abs/2212.09748, 2022. 4, 7 · arXiv
  31. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674- 10685. IEEE, 2022. 1, 3, 4, 5, 7, 8
  32. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Technical report, ArXiV, 2015. 4
  33. David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hooge- boom. Rolling diffusion models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024. OpenReview.net, 2024. 8
  34. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. CoRR, abs/2205.11487, 2022. 8 · arXiv
  35. Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations, ICLR. OpenRe- view.net, 2022. 4
  36. Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching. arXiv preprint arXiv:2406.04103, 2024. 4, 8 · arXiv
  37. Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, pages 49:1-49:10. ACM, 2022. 7, 4
  38. Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):640-651, 2016. 4
  39. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. 2
  40. Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jian- qiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unifying diffusion process across resolutions for image syn- thesis. In The Twelfth International Conference on Learning Representations, ICLR. OpenReview.net, 2024. 4, 7
  41. Haoning Wu, Shaocheng Shen, Qiang Hu, Xiaoyun Zhang, Ya Zhang, and Yanfeng Wang. Megafusion: Extend diffusion models towards higher-resolution image generation without further tuning. CoRR, abs/2408.11001, 2024. 4 · arXiv
  42. Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2021. 4
  43. Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufo- gen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:2311.09257, 2023. 4 · arXiv
  44. Yilun Xu, Gabriele Corso, Tommi S. Jaakkola, Arash Vah- dat, and Karsten Kreis. Disco-diff: Enhancing continuous diffusion models with discrete latents. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. 7
  45. Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One- step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2023. 4 · arXiv
  46. Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Frédo Durand, and William T. Freeman. Im- proved distribution matching distillation for fast image syn- thesis. In NeurIPS, 2024. 8
  47. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to- image generation. CoRR, abs/2206.10789, 2022. 8 · arXiv
  48. Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: masked generative video transformer. CoRR, abs/2212.05199, 2022. 8 · arXiv
  49. Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion -tokenizer is key to visual generation. In The Twelfth International Conference on Learn- ing Representations, ICLR. OpenReview.net, 2024. 7, 8
  50. Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. In Computer Vision -ECCV Proceedings, Part LXXVII, pages 1-22. Springer, 2024.

Cited by

Sorted by citations

Related

Metadata

Authors
Hoogeboom, E., Mensink, T., Heek, J., Lamerigts, K., Gao, R., & Salimans, T.
Affiliations
Google (United States) (US)
ORCID
Mensink, Thomas
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.19324
arXiv
2410.19324
OpenAlex
W4404311570
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
latent diffusion models, high-resolution image generation, skip connections, model scaling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review