CoChat
Sign Up
PreprintCited by 1,829Advanced Image Processing TechniquesGenerative Adversarial Networks and Image SynthesisImage and Signal Denoising Methods

Image Super-Resolution via Iterative Refinement

Open access
Saharia, Chitwan, Ho, Jonathan, Chan, William, Salimans, Tim, Fleet, David J., Norouzi, Mohammad·2021·arXiv
DOI 10.48550/arxiv.2104.07636arXiv 2104.07636
Open this paper in CoChat — free accountOpen PDF

Abstract

We present SR3, an approach to image Super-Resolution via Repeated Refinement. SR3 adapts denoising diffusion probabilistic models to conditional image generation and performs super-resolution through a stochastic denoising process. Inference starts with pure Gaussian noise and iteratively refines the noisy output using a U-Net model trained on denoising at various noise levels. SR3 exhibits strong performance on super-resolution tasks at different magnification factors, on faces and natural images. We conduct human evaluation on a standard 8X face super-resolution task on CelebA-HQ, comparing with SOTA GAN methods. SR3 achieves a fool rate close to 50%, suggesting photo-realistic outputs, while GANs do not exceed a fool rate of 34%. We further show the effectiveness of SR3 in cascaded image generation, where generative models are chained with super-resolution models, yielding a competitive FID score of 11.3 on ImageNet.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.07636

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Diffusion models define the state-of-the-art in class-conditional image synthesis and super-resolution.

    “and define the state-of-the-art in class-conditional image synthesis [15,31] and super-resolution [72].”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [72]

  • LDM-SR, trained only with bicubicly downsampled conditioning as in [72], does not generalize well to images that do not follow this pre-processing.

    “Interestingly, we observe that LDM-SR, trained only with a bicubicly downsampled conditioning as in [72], does not generalize well to images which do not follow this pre-processing.”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [72]

  • The user study results in Tab. 4 were obtained by following the protocol of [72] and using the 2-alternative force-choice paradigm to assess human preference scores for two distinct tasks.

    “For the results of the user study presented in Tab. 4 we followed the protocoll of [72] and and use the 2-alternative force-choice paradigm to assess human preference scores for two distinct tasks.”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [72]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 76

  1. J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," in NeurIPS, 2020.
  2. J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Gan- guli, "Deep Unsupervised Learning using Nonequilibrium Ther- modynamics," in ICML, 2015.
  3. I. Sutskever, O. Vinyals, and Q. Le, "Sequence to Sequence Learn- ing with Neural Networks," in NIPS, 2014.
  4. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention Is All You Need," in NIPS, 2017.
  5. A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, "WaveNet: A Generative Model for Raw Audio," arXiv 1609.03499, 2016. · arXiv
  6. A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, "Conditional Image Generation with Pixel- CNN Decoders," in NIPS, 2016.
  7. D. P. Kingma and M. Welling, "Auto-Encoding Variational Bayes," in ICLR, 2013.
  8. A. Vahdat and J. Kautz, "NVAE: A deep hierarchical variational autoencoder," in NeurIPS, 2020.
  9. L. Dinh, J. Sohl-Dickstein, and S. Bengio, "Density estimation using real NVP," arXiv:1605.08803, 2016. · arXiv
  10. D. P. Kingma and P. Dhariwal, "Glow: Generative Flow with Invertible 1x1 Convolutions," in NIPS, 2018.
  11. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative Adver- sarial Networks," NIPS, 2014.
  12. T. Karras, T. Aila, S. Laine, and J. Lehtinen, "Progressive growing of gans for improved quality, stability, and variation," in ICLR, 2018.
  13. A. Radford, L. Metz, and S. Chintala, "Unsupervised represen- tation learning with deep convolutional generative adversarial networks," arXiv preprint arXiv:1511.06434, 2015. · arXiv
  14. Y. Chen, Y. Tai, X. Liu, C. Shen, and J. Yang, "FSRNet: End-to-end learning face super-resolution with facial priors," in CVPR, 2018, pp. 2492-2501.
  15. R. Dahl, M. Norouzi, and J. Shlens, "Pixel recursive super resolu- tion," in ICCV, 2017.
  16. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., "Photo- realistic single image super-resolution using a generative adver- sarial network," in ICCV, 2017.
  17. S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin, "PULSE: Self-supervised photo upsampling via latent space exploration of generative models," in CVPR, 2020.
  18. N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, "Image transformer," in ICML, 2018.
  19. M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein GAN," in arXiv, 2017.
  20. I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, "Improved training of wasserstein gans," arXiv preprint arXiv:1704.00028, 2017. · arXiv
  21. L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, "Unrolled gener- ative adversarial networks," arXiv preprint arXiv:1611.02163, 2016. · arXiv
  22. S. Ravuri and O. Vinyals, "Classification accuracy score for condi- tional generative models," arXiv preprint arXiv:1905.10887, 2019. · arXiv
  23. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermody- namics," in ICML. PMLR, 2015, pp. 2256-2265.
  24. Y. Song and S. Ermon, "Generative Modeling by Estimating Gra- dients of the Data Distribution," in NeurIPS, 2019.
  25. O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in MICCAI, 2015.
  26. A. Dosovitskiy and T. Brox, "Generating images with perceptual similarity metrics based on deep networks," Advances in neural information processing systems (NeurIPS) 29, pp. 658-666.
  27. D. Berthelot, P. Milanfar, and I. Goodfellow, "Creating high reso- lution images with a latent adversarial generator," arXiv preprint 2003.02365, 2020. · arXiv
  28. Z. Kadkhodaie and E. P. Simoncelli, "Solving linear inverse problems using the prior implicit in a denoiser," arXiv preprint 2007.13640, 2021. · arXiv
  29. R. Zhang, P. Isola, and A. A. Efros, "Colorful image colorization," in ECCV, 2016.
  30. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. C. Loy, "ESRGAN: Enhanced super-resolution generative adversarial networks," in The European Conference on Computer Vision Work- shops (ECCVW), September 2018.
  31. M. S. Sajjadi, B. Scholkopf, and M. Hirsch, "Enhancenet: Single image super-resolution through automated texture synthesis," in ICCV, 2017, pp. 4491-4500.
  32. A. Lugmayr, M. Danelljan, L. Van Gool, and R. Timofte, "SRFlow: Learning the super-resolution space with normalizing flow," in ECCV, 2020.
  33. N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, "WaveGrad: Estimating Gradients for Waveform Generation," in ICLR, 2021.
  34. C. Saharia, W. Chan, H. Chang, C. A. Lee, J. Ho, T. Salimans, D. J. Fleet, and M. Norouzi, "Palette: Image-to-image diffusion models," arXiv 2111.05826, 2021. · arXiv
  35. A. Hyvärinen and P. Dayan, "Estimation of non-normalized sta- tistical models by score matching." JMLR, vol. 6, no. 4, 2005.
  36. P. Vincent, "A connection between score matching and denoising autoencoders," Neural Comp., vol. 23, no. 7, pp. 1661-1674, 2011.
  37. M. Raphan and E. P. Simoncelli, "Least squares estimation without priors or supervision," Neural Comp., vol. 23, pp. 374-420, 2011.
  38. S. Saremi, A. Mehrjou, B. Sch ölkopf, and A. Hyvärinen, "Deep En- ergy Estimator Networks," arXiv preprint arXiv:1805.08306, 2018. · arXiv
  39. Y. Song and S. Ermon, "Improved Techniques for Training Score- Based Generative Models," arXiv preprint arXiv:2006.09011, 2020. · arXiv
  40. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-Based Generative Modeling through Stochastic Differential Equations," in ICLR, 2021.
  41. A. Brock, J. Donahue, and K. Simonyan, "Large scale GAN training for high fidelity natural image synthesis," arXiv preprint arXiv:1809.11096, 2018. · arXiv
  42. E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, "FiLM: Visual Reasoning with a General Conditioning Layer ," in AAAI, 2018.
  43. A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu, "Pixel recurrent neural networks," ICML, 2016.
  44. T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma, "Pixel- CNN++: Improving the PixelCNN with discretized logistic mix- ture likelihood and other modifications," in ICLR, 2017.
  45. D. Rezende and S. Mohamed, "Variational inference with normal- izing flows," in ICML, 2015, pp. 1530-1538.
  46. D. J. Rezende, S. Mohamed, and D. Wierstra, "Stochastic back- propagation and approximate inference in deep generative mod- els," in ICML, 2014, pp. 1278-1286.
  47. E. Denton, S. Chintala, A. Szlam, and R. Fergus, "Deep Gener- ative Image Models using a Laplacian Pyramid of Adversarial Networks," in NIPS, 2015.
  48. R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. Belongie, N. Snavely, and B. Hariharan, "Learning Gradient Fields for Shape Genera- tion," in ECCV, 2020.
  49. C. Dong, C. C. Loy, K. He, and X. Tang, "Learning a deep convo- lutional network for image super-resolution," in ECCV, 2014, pp. 184-199.
  50. J. Kim, J. K. Lee, and K. M. Lee, "Deeply-recursive convolutional network for image super-resolution," in CVPR, 2016, pp. 1637- 1645.
  51. Y. Tai, J. Yang, and X. Liu, "Image super-resolution via deep recursive residual network," in CVPR, 2017, pp. 3147-3155.
  52. N. Ahn, B. Kang, and K.-A. Sohn, "Image Super-resolution via Progressive Cascading Residual Network," in CVPR, 2018.
  53. X. Li, C. Chen, S. Zhou, X. Lin, W. Zuo, and L. Zhang, "Blind face restoration via deep multi-scale component dictionaries," in ECCV, 2020.
  54. Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, "Deep networks for image super-resolution with sparse prior," in ICCV, 2015, pp. 370-378.
  55. C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE Trans PAMI, vol. 38, no. 2, pp. 295-307, 2015.
  56. J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," in CVPR, 2016.
  57. J. Menick and N. Kalchbrenner, "Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling," in ICLR, 2019.
  58. A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, "Conditional image generation with PixelCNN decoders," in NIPS.
  59. L. Yang, C. Liu, P. Wang, S. Wang, P. Ren, S. Ma, and W. Gao, "HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment," in arXiv, 2020.
  60. J. J. Yu, K. G. Derpanis, and M. A. Brubaker, "Wavelet Flow: Fast Training of High Resolution Normalizing Flows," in arXiv, 2020.
  61. T. Karras, S. Laine, and T. Aila, "A Style-Based Generator Archi- tecture for Generative Adversarial Networks," in CVPR, 2019.
  62. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," International Journal of Computer Vision, vol. 115, no. 3, pp. 211-252, 2015.
  63. Z. Wang, A. Bovik, H. Sheikh., and E. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.
  64. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, "Image super- resolution using very deep residual channel attention networks," in ECCV, 2018, pp. 286-301.
  65. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in CVPR, 2016, pp. 770-778.
  66. E. Pérez-Pellitero, J. Salvador, J. Hidalgo, and B. Rosenhahn, "Psyco: Manifold span reduction for super resolution," CVPR, pp. 1837-1845, 2016.
  67. C. Dong, C. C. Loy, and X. Tang, "Accelerating the super- resolution convolutional neural network," in ECCV, 2016, pp. 391- 407.
  68. S. Anwar and N. Barnes, "Densely residual laplacian super- resolution," IEEE Trans PAMI, 2020.
  69. R. K. Mantiuk, A. Tomaszewska, and R. Mantiuk, "Comparison of four subjective methods for image quality assessment," in Computer graphics forum, vol. 31, no. 8. Wiley Online Library, 2012, pp. 2478-2491.
  70. A. Nichol and P. Dhariwal, "Improved denoising diffusion proba- bilistic models," arXiv 2102.09672, 2021. · arXiv
  71. A. Razavi, A. v. d. Oord, and O. Vinyals, "Generating diverse high- fidelity images with vq-vae-2," arXiv 1906.00446, 2019. · arXiv
  72. J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, "Cascaded diffusion models for high fidelity image generation," JMLR, vol. 23, no. 47, pp. 1-33, 2022.
  73. D. Watson, J. Ho, M. Norouzi, and W. Chan, "Learning to efficiently sample from diffusion probabilistic models," arXiv 2106.03802, 2021. · arXiv
  74. A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas, "Gotta go fast when generating data with score- based models," in NeurIPS Workshops, 2021.
  75. T. Salimans and J. Ho, "Progressive distillation for fast sampling of diffusion models," in ICLR, 2022.
  76. J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," arXiv 2010.02502, 2020. · arXiv

Cited by

Sorted by citations

Authors
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
45
Issue
4
Pages
1-14
ISSN
0162-8828 (ISSN-L)
DOI
10.48550/arxiv.2104.07636
arXiv
2104.07636
OpenAlex
W3155072588
Cited by
1,829
References
0
Citations by year
2026: 234 · 2025: 580 · 2024: 662 · 2023: 301 · 2022: 40 · 2021: 12
FWCI
129.48
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
Keywords
image super-resolution, iterative refinement, denoising diffusion probabilistic models, image-to-image translation, U-Net, face super-resolution, high-resolution image generation, human evaluation, image classification accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review