CoChat
Sign Up
Cited by 1,717Generative Adversarial Networks and Image SynthesisImage and Signal Denoising MethodsAdvanced Image Processing Techniques

Palette: Image-to-Image Diffusion Models

Open access
Saharia, Chitwan, Chan, William, Chang, Huiwen, Lee, Chris, Ho, Jonathan, Salimans, Tim, Fleet, David, Norouzi, Mohammad·2022·Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings
DOI 10.1145/3528233.3530757
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io/ for an overview of the results and code.

AI snapshot

Written from the paper’s full text.

TL;DR

Palette is a unified conditional diffusion framework for image-to-image translation that, without task-specific tuning, outperforms GAN and regression baselines on colorization, inpainting, uncropping, and JPEG restoration. A single multi-task Palette model matches or exceeds task-specific specialists on some tasks, and the paper advocates a standardized ImageNet-based evaluation protocol.

Key findings

  • Palette beats strong baselines across all four tasks. Colorization: FID-5K 15.78 vs. regression 17.89 and ColTran 19.37; human fool rate 47.80% vs. ColTran 36.55% and PixColor 29.90%, approaching the ideal 50%.
  • Inpainting (20–30% free-form, ImageNet): FID 5.2 vs. DeepFillv2 9.4 and HiFill 12.4; Places2 FID 11.7 vs. DeepFillv2 13.5, HiFill 15.7, and Co-ModGAN 12.4. Uncropping: ImageNet FID 5.8 vs. Boundless 18.7; human fool rate 40% vs. Boundless 25% and InfinityGAN 15%.
  • JPEG restoration at QF=5: Palette FID 8.3 vs. regression 29.0, CA 64.2% vs. 52.8%; at QF=20: FID 4.3 vs. 11.5. L2 vs. L1 denoising loss gives similar FID but greater sample diversity (e.g., colorization LPIPS 0.15 vs. 0.09; inpainting LPIPS 0.13 vs. 0.11).
  • Global self-attention outperforms fully convolutional alternatives for inpainting; a multi-task generalist Palette outperforms a task-specific JPEG specialist but slightly lags task-specific Palette on inpainting and colorization.

Methods

  • Conditional diffusion model \(p(y \mid x)\) using a U-Net with source-image concatenation and no class conditioning; L2 denoising loss; 256×256 RGB inputs/outputs; trained for 500K steps with batch size 512; sampling requires 0.8 sec./image on a TPUv4 for 1000 test images.
  • Tasks: colorization, inpainting, uncropping, and JPEG restoration, with no task-specific hyper-parameter tuning, architecture customization, or auxiliary loss. Inpainting/uncropping fill masked regions with Gaussian noise; inpainting loss considers only masked pixels.
  • Evaluation on ImageNet ctest10k and Places2 places10k using FID, Inception Score, ResNet-50 classification accuracy, perceptual distance, and 2AFC human fool rate; ablations cover L1 vs. L2 loss, self-attention variants, and multi-task training.

Limitations

  • The paper states: sampling speed is slower than GAN models; the multi-task generalist slightly lags task-specific Palette on inpainting and colorization, though performance is expected to improve with more training; the authors could not compare with all prior inpainting works; and comparison of sample diversity with GAN-based techniques is left to future work.
  • Evident from the text: evaluation is limited to 256×256 RGB images and four tasks, so generality to other resolutions or tasks is not established; no statistical significance tests or confidence intervals are reported in the provided text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3528233.3530757

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 16 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    L2 loss in the denoising diffusion objective leads to higher sample diversity than L1 loss.

    “We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity”
    • Also stated in: § 1 INTRODUCTION · plain
      “We find that while L 2 [Ho et al. 2020] and L 1 [Chen et al. 2021a] losses in the denoising objective yield similar sample-quality scores, L 2 leads to a higher degree of diversity in model samples, whereas L 1 [Chen et al. 2021a] produces more conservative outputs.”
    • Also stated in: § 5.5 Self-attention in model architectures · boosted
      “The results in Table 6 thus clearly show that models trained with the L 2 loss have greater sample diversity than those trained with the L 1 loss.”
    Quote in the paperSubstantial claim
  • Findingown data

    L1 loss produces more conservative outputs than L2.

    “We find that while L 2 [Ho et al. 2020] and L 1 [Chen et al. 2021a] losses in the denoising objective yield similar sample-quality scores, L 2 leads to a higher degree of diversity in model samples, whereas L 1 [Chen et al. 2021a] produces more conservative outputs.”
    Quote in the paperSubstantial claim
  • Findingown data

    The multi-task generalist Palette slightly lags behind task-specific Palette models on inpainting and colorization.

    “It indicates that multi-task generalist Palette outperforms the task-specific JPEG restoration specialist model, but slightly lags behind task-specific Palette models on inpainting and colorization.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    The performance gap between Palette and the regression baseline widens with decreasing quality factor in JPEG restoration.

    “As expected, the performance gap between Palette and the regression baseline widens with decreasing quality factor.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For 20-30% free-form masks, the FID scores for Palette outputs are extremely close to FID scores on the original images from which the masked test inputs were created.

    “Importantly, like the colorization task above, the FID scores for Palette outputs in the case of 20-30% free-form masks, are extremely close to FID scores on the original images from which we created the masked test inputs.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 10
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Findingown data

    Palette, a unified conditional diffusion framework, outperforms strong GAN and regression baselines on colorization, inpainting, uncropping, and JPEG restoration without task-specific hyper-parameter tuning, architecture customization, auxiliary losses, or new techniques.

    “Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed.”
    • Also stated in: § 1 INTRODUCTION · plain
      “We show that Palette, with no task-specific architecture customization, nor changes to hyper-parameters or the loss, delivers high-fidelity outputs across all four tasks.It outperforms task-specific baselines and a strong regression baseline with an identical neural architecture.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: It is not clear whether diffusion models rival GANs in offering a versatile and general framework for image manipulation.

  • May combine two claims.

    Findingown data

    Palette establishes a new state of the art on colorization, outperforming existing works by a large margin.

    “Palette establishes a new SoTA, outperforming existing works by a large margin.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In JPEG restoration at quality factor 5, the regression model produces blurry outputs, while Palette produces sharper images.

    “It is easy to see that the regression model produces blurry outputs, while Palette produces sharper images.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Palette exhibits strong performance across inpainting datasets and mask configurations, outperforming DeepFillv2, HiFill, and Co-ModGAN by a large margin.

    “Palette exhibits strong performance across inpainting datasets and mask configurations, outperforming DeepFillv2, HiFill and Co-ModGAN by a large margin.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On human evaluation for uncropping, Palette has a 40% fool rate, compared to 25% for Boundless and 15% for InfinityGAN.

    “Palette has a 40% fool rate, compared to 25% and 15% for Boundless and InfinityGAN (see Fig. C.2 for details).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ultimate evaluation of image-to-image translation models is human evaluation; i.e., whether or not humans can discriminate model outputs from natural images.

  • May combine two claims.

    Findingnone given

    On human evaluation, Palette improves upon human raters' fool rate of ColTran by more than 10%, approaching an ideal fool rate of 50%.

    “On human evaluation, Palette improves upon human raters' fool rate of ColTran by more than 10%, approaching an ideal fool rate of 50%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ultimate evaluation of image-to-image translation models is human evaluation; i.e., whether or not humans can discriminate model outputs from natural images.

  • May combine two claims.

    Findingown data

    Palette exhibits strong performance across all quality factors in JPEG restoration, outperforming the regression baseline.

    “Table 4 summarizes the Im-ageNet results, with Palette exhibiting strong performance across all quality factors, outperforming the regression baseline.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Palette is equally effective in YCbCr and RGB color spaces for colorization.

    “Preliminary experiments indicated that Palette is equally effective in YCbCr and RGB spaces.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Palette produces high fidelity outputs, unlike many GAN models.

    “Unlike many GAN models, Palette produces diverse and high fidelity outputs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For 1000 test images, Palette requires 0.8 sec./image on a TPUv4.

    “There is a large overhead for loading models and the initial jit compilation, but for 1000 test images, Palette requires 0.8 sec./image on a TPUv4.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Palette's sampling speed is slower than GAN models.

    “Sampling speed is slower than GAN models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Diffusion models have delivered impressive face enhancement results on image super-resolution, outperforming GANs.

    “On image super-resolution, they have delivered impressive face enhancement results, outperforming GANs [Saharia et al. 2021].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Saharia et al. 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Sample diversity is challenging and has been a key limitation of many existing GAN-based methods.

    “Sample diversity is challenging and has been a key limitation of many existing GAN-based methods [Yang et al. 2019b;Zhu et al. 2017b].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Yang et al. 2019b

      Not checked in the free run.

    • cites Zhu et al. 2017b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Diffusion models have outperformed strong GAN baselines in terms of FID scores on the class-conditional ImageNet generation challenge.

    “On the class-conditional ImageNet generation challenge they have outperformed strong GAN baselines in terms of FID scores [Dhariwal and Nichol 2021;Ho et al. 2021].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Dhariwal and Nichol 2021

      Not checked in the free run.

    • cites Ho et al. 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Diffusion models have achieved human evaluation scores on par with SoTA autoregressive models on speech synthesis.

    “On speech synthesis, diffusion models have achieved human evaluation scores on par with SoTA autoregressive models [Chen et al. 2021a,b;Kong et al. 2021].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chen et al. 2021a,b

      Not checked in the free run.

    • cites Kong et al. 2021

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    GANs can be challenging to train.

    “Nevertheless, GANs can be challenging to train [Arjovsky et al. 2017;Gulrajani et al. 2017], and often drop modes in the output distribution [Metz et al. 2016;Ravuri and Vinyals 2019].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Arjovsky et al. 2017

      Not checked in the free run.

    • cites Gulrajani et al. 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The results in Fig.9 are typical.

    “The results in Fig.9are typical; the re-purposed unconditional model does not perform well, in part because it is hard to learn a good unconditional model on diverse datasets like ImageNet, and also because, during iterative refinement, noise is added to all pixels, including the observed pixels.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    All models use the same architecture, training data and number of training steps.

    “All models use the same architecture, training data and number of training steps.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Existing conditional diffusion models SR3 and WaveGrad have found L1 norm to perform better than the conventional L2 loss.

    “While existing conditional diffusion models, SR3 [Saharia et al. 2021] and WaveGrad [Chen et al. 2021a], have found L 1 norm to perform better than the conventional L 2 loss, there has not been a detailed comparison of the two.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Saharia et al., 2021

      Not checked in the free run.

    • cites Chen et al., 2021a

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    There has not been a detailed comparison of L1 versus L2 loss on sample diversity.

    “there has not been a detailed comparison of the two”
    Quote in the paperNot a substantial claim
20 of 20

  • The paper relies on this without support.

    It is not clear whether diffusion models rival GANs in offering a versatile and general framework for image manipulation.

    “Despite these results, it is not clear whether diffusion models rival GANs in offering a versatile and general framework for image manipulation.”

  • The paper relies on this without support.

    Unconditional tasks are often more difficult than conditional tasks.

    “However, unconditional tasks are often more difficult than conditional tasks.”

  • The paper relies on this without support.

    Image uncropping is considered more challenging than inpainting as it entails generating open-ended content with less context.

    “Image uncropping (a.k.a.outpainting) is considered more challenging than inpainting as it entails generating open-ended content with less context.”

  • The paper relies on this without support.

    JPEG restoration is a non-linear inverse problem; it requires a good local model of natural image statistics to detect and correct compression artifacts.

    “While the other tasks are linear in nature, JPEG restoration is a non-linear inverse problem; it requires a good local model of natural image statistics to detect and correct compression artifacts.”

  • The paper relies on this without support.

    Self-attention layers prevent generalization to unseen image resolutions.

    “While self-attention layers provide a direct form of global dependency, they prevent generalization to unseen image resolutions.”

  • The paper relies on this without support.

    The ultimate evaluation of image-to-image translation models is human evaluation; i.e., whether or not humans can discriminate model outputs from natural images.

    “The ultimate evaluation of image-to-image translation models is human evaluation; i.e., whether or not humans can discriminate model outputs from natural images.”

  • The paper relies on this without support.

    Multi-task training is a natural approach to learning a single model for multiple image-to-image tasks.

    “Multi-task training is a natural approach to learning a single model for multiple image-to-image tasks, i.e., blind image enhancement.”

  • The paper relies on this without support.

    Many image-to-image tasks are complex inverse problems where multiple output images are consistent with a single input.

    “Many such tasks, like those in Fig. 1, are complex inverse problems, where multiple output images are consistent with a single input.”

  • The paper relies on this without support.

    The four image-to-image translation tasks (colorization, inpainting, uncropping, and JPEG restoration) are challenging.

    “Palette achieves strong results on four challenging image-to-image translation tasks (colorization, inpainting, uncropping, and JPEG restoration), outperforming strong GAN and regression baselines.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DDPMs have several properties well suited to protein design, including generating highly diverse outputs, being guidable at each step, and being able to model 3D structures in a rotationally equivariant manner.

    “Denoising diffusion probabilistic models (DDPMs), a powerful class of machine learning models recently demonstrated to generate new photorealistic images in response to text prompts14,15, have several properties well suited to protein design.”

    From De novo design of protein structure and function with RFdiffusion (2023) · cites this paper as [15]

    The Conclusion reports that Palette produces diverse, high-fidelity images, and Section 3 describes an iterative denoising process conditioned on an input image. These passages support image generation and output diversity, but not the citing sentence’s text-prompt condition. The paper also does not establish step-by-step guidance, rotationally equivariant 3D modeling, or suitability for protein design. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Palette employs conditional diffusion models to create a unified framework for four image generation tasks: colorization, inpainting, uncropping, and JPEG restoration.

    “Meanwhile, Palette [198] employs conditional diffusion models to create a unified framework for four image generation tasks: colorization, inpainting, uncropping, and JPEG restoration.”

    From Diffusion Models: A Comprehensive Survey of Methods and Applications (2024) · cites this paper as [198]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Multi-task model is claimed to match specialists but lags on two tasks

    Major

    The abstract says the generalist performs as well as or better than task-specific models, but the paper later says it performs worse on inpainting and colorization.

  • Inpainting changes the loss despite a claim of no loss changes

    Minor

    The paper claims the loss is unchanged across all four tasks, then specifies an inpainting loss that considers only masked pixels rather than the entire image.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Martin Arjovsky Soumith Chintala and Léon Bottou. 2017. Wasserstein GAN. In arXiv. Martin Arjovsky Soumith Chintala and Léon Bottou. 2017. Wasserstein GAN. In arXiv.
  2. Ruojin Cai Guandao Yang Hadar Averbuch-Elor Zekun Hao Serge Belongie Noah Snavely and Bharath Hariharan. 2020. Learning Gradient Fields for Shape Generation. In ECCV. Ruojin Cai Guandao Yang Hadar Averbuch-Elor Zekun Hao Serge Belongie Noah Snavely and Bharath Hariharan. 2020. Learning Gradient Fields for Shape Generation. In ECCV.
  3. Nanxin Chen Yu Zhang Heiga Zen Ron J. Weiss Mohammad Norouzi and William Chan. 2021a. WaveGrad: Estimating Gradients for Waveform Generation. In ICLR. Nanxin Chen Yu Zhang Heiga Zen Ron J. Weiss Mohammad Norouzi and William Chan. 2021a. WaveGrad: Estimating Gradients for Waveform Generation. In ICLR.
  4. Alexey Dosovitskiy and Thomas Brox . 2016. Generating Images with Perceptual Similarity Metrics based on Deep Networks. arXiv 1602.0264 ( 2016 ). Alexey Dosovitskiy and Thomas Brox. 2016. Generating Images with Perceptual Similarity Metrics based on Deep Networks. arXiv 1602.0264 (2016).
  5. Max Ehrlich , Larry Davis , Ser-Nam Lim , and Abhinav Shrivastava . 2020 . Quantization guided jpeg artifact correction. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020 , Proceedings, Part VIII 16 . Springer, 293–309. Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava. 2020. Quantization guided jpeg artifact correction. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16. Springer, 293–309.
  6. Ian  J Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair , Aaron Courville , and Yoshua Bengio . 2014. Generative Adversarial Networks. NIPS ( 2014 ). Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Networks. NIPS (2014).
  7. Sergio Guadarrama , Ryan Dahl , David Bieber , Mohammad Norouzi , Jonathon Shlens , and Kevin Murphy . 2017 . Pixcolor: Pixel recursive colorization. arXiv preprint arXiv:1705.07208(2017). Sergio Guadarrama, Ryan Dahl, David Bieber, Mohammad Norouzi, Jonathon Shlens, and Kevin Murphy. 2017. Pixcolor: Pixel recursive colorization. arXiv preprint arXiv:1705.07208(2017). · arXiv
  8. Jonathan Ho Chitwan Saharia William Chan David J. Fleet Mohammad Norouzi and Tim Salimans. 2021. Cascaded Diffusion Models for High Fidelity Image Generation. In arXiv. Jonathan Ho Chitwan Saharia William Chan David J. Fleet Mohammad Norouzi and Tim Salimans. 2021. Cascaded Diffusion Models for High Fidelity Image Generation. In arXiv.
  9. 10.1109/CVPR.2017.632 · DOI · Paper page
  10. Diederik  P. Kingma and Prafulla Dhariwal . 2018 . Glow : Generative Flow with Invertible 1x1 Convolutions. In NIPS. Diederik P. Kingma and Prafulla Dhariwal. 2018. Glow: Generative Flow with Invertible 1x1 Convolutions. In NIPS.
  11. Diederik P Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. In ICLR. Diederik P Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. In ICLR.
  12. Zhifeng Kong , Wei Ping , Jiaji Huang , Kexin Zhao , and Bryan Catanzaro . 2021. DiffWave: A Versatile Diffusion Model for Audio Synthesis. ICLR ( 2021 ). Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2021. DiffWave: A Versatile Diffusion Model for Audio Synthesis. ICLR (2021).
  13. Manoj Kumar , Dirk Weissenborn , and Nal Kalchbrenner . 2021 . Colorization Transformer. In ICLR 2021. https://openreview.net/forum?id=5NA1PinlGFu Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. 2021. Colorization Transformer. In ICLR 2021. https://openreview.net/forum?id=5NA1PinlGFu
  14. Hongyu Liu , Bin Jiang , Yibing Song , Wei Huang , and Chao Yang . 2020 . Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020 , Proceedings, Part II 16 . Springer, 725–741. Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. 2020. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer, 725–741.
  15. Sachit Menon , Alexandru Damian , Shijia Hu , Nikhil Ravi , and Cynthia Rudin . 2020 . PULSE: Self-supervised photo upsampling via latent space exploration of generative models. In CVPR. Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. 2020. PULSE: Self-supervised photo upsampling via latent space exploration of generative models. In CVPR.
  16. Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam Shazeer Alexander Ku and Dustin Tran. 2018. Image transformer. In ICML. Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam Shazeer Alexander Ku and Dustin Tran. 2018. Image transformer. In ICML.
  17. Tim Salimans Andrej Karpathy Xi Chen and Diederik P. Kingma. 2017. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. In ICLR. Tim Salimans Andrej Karpathy Xi Chen and Diederik P. Kingma. 2017. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. In ICLR.
  18. Jascha Sohl-Dickstein Eric Weiss Niru Maheswaranathan and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML. PMLR 2256–2265. Jascha Sohl-Dickstein Eric Weiss Niru Maheswaranathan and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML. PMLR 2256–2265.
  19. Yang Song Jascha Sohl-Dickstein Diederik P. Kingma Abhishek Kumar Stefano Ermon and Ben Poole. 2021. Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR. Yang Song Jascha Sohl-Dickstein Diederik P. Kingma Abhishek Kumar Stefano Ermon and Ben Poole. 2021. Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR.
  20. Piotr Teterwak , Aaron Sarna , Dilip Krishnan , Aaron Maschinot , David Belanger , Ce Liu , and William  T Freeman . 2019 . Boundless: Generative adversarial networks for image extension . In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10521–10530 . Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. 2019. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10521–10530.
  21. Arash Vahdat and Jan Kautz . 2020 . NVAE: A Deep Hierarchical Variational Autoencoder. In NeurIPS. Arash Vahdat and Jan Kautz. 2020. NVAE: A Deep Hierarchical Variational Autoencoder. In NeurIPS.
  22. Aaron van den Oord Nal Kalchbrenner Oriol Vinyals Lasse Espeholt Alex Graves and Koray Kavukcuoglu. 2016. Conditional image generation with PixelCNN decoders. In NIPS. 4790–4798. Aaron van den Oord Nal Kalchbrenner Oriol Vinyals Lasse Espeholt Alex Graves and Koray Kavukcuoglu. 2016. Conditional image generation with PixelCNN decoders. In NIPS. 4790–4798.
  23. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In NIPS. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In NIPS.
  24. Yi Wang , Xin Tao , Xiaoyong Shen , and Jiaya Jia . 2019 a. Wide-Context Semantic Image Extrapolation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1399–1408 . Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019a. Wide-Context Semantic Image Extrapolation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1399–1408.
  25. Jun-Yan Zhu Richard Zhang Deepak Pathak Trevor Darrell Alexei A Efros Oliver Wang and Eli Shechtman. 2017b. Multimodal Image-to-Image Translation by Enforcing Bi-Cycle Consistency. In Advances in neural information processing systems. 465–476. Jun-Yan Zhu Richard Zhang Deepak Pathak Trevor Darrell Alexei A Efros Oliver Wang and Eli Shechtman. 2017b. Multimodal Image-to-Image Translation by Enforcing Bi-Cycle Consistency. In Advances in neural information processing systems. 465–476.

Cited by

Sorted by citations

Related

Authors
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M.
Affiliations
Google (United States) (US)
ORCID
Venue
Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings
Publisher
ACM
Published
2022-08-07
Pages
1-10
DOI
10.1145/3528233.3530757
OpenAlex
W3212516020
Semantic Scholar
37c9c4e7648f639c0b36f150fc6c6c90b3682f4a
Cited by
1,717
References
70
Citations by year
2026: 269 · 2025: 495 · 2024: 598 · 2023: 320 · 2022: 34 · 2021: 1
FWCI
131.20
Type
conference paper
Language
en
Open access
Licence
Locations
ACM SIGGRAPH Conference (SIGGRAPH) (publishedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
Keywords
image-to-image translation, conditional diffusion models, image colorization, image inpainting, self-attention, sample diversity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review