The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models
Open accessAbstract
Image restoration under adverse weather conditions has been of significant interest for various computer vision applications. Recent successful methods rely on the current progress in deep neural network architectural designs (e.g., with vision transformers). Motivated by the recent progress achieved with state-of-the-art conditional generative models, we present a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models. Our patch-based diffusion modeling approach enables size-agnostic image restoration by using a guided denoising process with smoothed noise estimates across overlapping patches during inference. We empirically evaluate our model on benchmark datasets for image desnowing, combined deraining and dehazing, and raindrop removal. We demonstrate our approach to achieve state-of-the-art performances on both weather-specific and multi-weather image restoration, and experimentally show strong generalization to real-world test images.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Making a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766 · cited 3×
“also used two other metrics for reference-free quality assessment of real-world restoration performance, namely the Naturalness Image Quality Evaluator (NIQE) [73], and Integrated Local NIQE (IL-NIQE) [74] scores.”
MethodTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879
SwinIR: Image Restoration Using Swin Transformer
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., & Timofte, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00210 · Cited by 4,734
Restormer: Efficient Transformer for High-Resolution Image Restoration
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., & Yang, M.-H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00564 · Cited by 4,143
Wu, Y., & He, K. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01198-w · Cited by 3,601
DehazeNet: An End-to-End System for Single Image Haze Removal
Cai, B., Xu, X., Jia, K., Qing, C., & Tao, D. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2598681 · Cited by 3,367
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Scope of validity of PSNR in image/video quality assessment
Huynh-Thu, Q., & Ghanbari, M. (2008) · Electronics Letters · doi:10.1049/el:20080522 · Cited by 2,959 · cited 8×
“We evaluated PSNR and SSIM based on the luminance channel Y of the YCbCr color space in accordance with the previous convention [6], [7], [12], [54].”
MethodInfluentialDeep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
Multi-Stage Progressive Image Restoration
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., & Shao, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01458 · Cited by 2,095
Score-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953
Image Super-Resolution Via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204461 · Cited by 1,829 · cited 1×
“Recently, diffusion-based conditional generative models have shown state-of-the-art performance in various tasks such as class-conditional data synthesis with classifier guidance [17], image super-resolution [19], [32], image deblurring [33], textbased image synthesis and editing [18], [20], and general image-”
BackgroundImage Super-Resolution via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2021) · arXiv · doi:10.48550/arxiv.2104.07636 · Cited by 1,829
RePaint: Inpainting using Denoising Diffusion Probabilistic Models
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01117 · Cited by 1,778
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
Improved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550 · cited 3×
“999 was applied during parameter updates, as it was shown to facilitate more stable learning [23], [25].”
BackgroundMethodFrom learning models of natural image patches to whole image restoration
Zoran, D., & Weiss, Y. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126278 · Cited by 1,522
Generative Modeling by Estimating Gradients of the Data Distribution
Song, Y., & Ermon, S. (2019) · arXiv · doi:10.48550/arxiv.1907.05600 · Cited by 1,453
A Feature-Enriched Completely Blind Image Quality Evaluator
Zhang, L., Zhang, L., & Bovik, A. C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2426416 · Cited by 1,311
Removing Rain from Single Images via a Deep Detail Network
Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.186 · Cited by 1,235
Deep Joint Rain Detection and Removal from a Single Image
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.183 · Cited by 1,231
Image De-Raining Using a Conditional Generative Adversarial Network
Zhang, H., Sindagi, V., & Patel, V. M. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2920407 · Cited by 1,214
GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing
Liu, X., Ma, Y., Shi, Z., & Chen, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00741 · Cited by 1,043
A Connection Between Score Matching and Denoising Autoencoders
Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00142 · Cited by 1,011
Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal
Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2691802 · Cited by 940
Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining
Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_16 · Cited by 927
Attentive Generative Adversarial Network for Raindrop Removal from A Single Image
Qian, R., Tan, R. T., Yang, W., Su, J., & Liu, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00263 · Cited by 821
Multi-Scale Progressive Fusion Network for Single Image Deraining
Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., & Jiang, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00837 · Cited by 795
(2005) · doi:10.5555/1046920.1088696 · Cited by 756
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset
Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., & Lau, R. W. H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01255 · Cited by 681
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Choi, J., Kim, S., Jeong, Y., Gwon, Y., & Yoon, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01410 · Cited by 634
MAXIM: Multi-Axis MLP for Image Processing
Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00568 · Cited by 616
Cascaded Diffusion Models for High Fidelity Image Generation
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., & Salimans, T. (2021) · arXiv · doi:10.48550/arxiv.2106.15282 · Cited by 514
Improved Techniques for Training Score-Based Generative Models
Song, Y., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2006.09011 · Cited by 482
DesnowNet: Context-Aware Deep Network for Snow Removal
Liu, Y.-F., Jaw, D.-W., Huang, S.-C., & Hwang, J.-N. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2806202 · Cited by 432
All-In-One Image Restoration for Unknown Corruption
Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01693 · Cited by 432
Optimal Spatial Adaptation for Patch-Based Image Denoising
Kervrann, C., & Boulanger, J. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.877529 · Cited by 431
TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions
Jose Valanarasu, J. M., Yasarla, R., & Patel, V. M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00239 · Cited by 416
How to Train Your Energy-Based ModelsPreprint
Song, Y., & Kingma, D. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03288 · Cited by 407
Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning
Li, R., Cheong, L.-F., & Tan, R. T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00173 · Cited by 402
RefineDNet: A Weakly Supervised Refinement Framework for Single Image Dehazing
Zhao, S., Zhang, L., Shen, Y., & Zhou, Y. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3060873 · Cited by 391
Xiao, J., Fu, X., Liu, A., Wu, F., & Zha, Z.-J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183612 · Cited by 389
Perceptual Adversarial Networks for Image-to-Image Transformation
Wang, C., Xu, C., Wang, C., & Tao, D. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2836316 · Cited by 381
All in One Bad Weather Removal Using Architectural Search
Li, R., Tan, R. T., & Cheong, L.-F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00324 · Cited by 377 · cited 14×
“We compare WeatherDiffp with two state-of-the-art multi-weather image restoration methods: All-in-One [9], which utilizes a multi-encoder and decoder pipeline with a neural architecture search mechanism, and TransWeather [7], which exploits an end-to-end vision transformer.”
BackgroundMethodInfluentialKingma, D. P., Salimans, T., Poole, B., & Ho, J. (2021) · arXiv · doi:10.48550/arxiv.2107.00630 · Cited by 374
Deep Dense Multi-Scale Network for Snow Removal Using Semantic and Depth Priors
Zhang, K., Li, R., Yu, Y., Luo, W., & Li, C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3104166 · Cited by 370
Denoising Diffusion Restoration Models
Kawar, B., Elad, M., Ermon, S., & Song, J. (2022) · neural information processing systems · doi:10.52202/068431-1714 · Cited by 349
Single Image Deraining: From Model-Based to Data-Driven and Beyond
Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2995190 · Cited by 284
Chung, H., Sim, B., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01209 · Cited by 275
Using fast weights to improve persistent contrastive divergence
Tieleman, T., & Hinton, G. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553506 · Cited by 274 · cited 1×
“Notably, energy-based models differ in its training approach which relies on contrastive divergence methods [30], [31], whereas diffusion- and score-based models exploit the sequential forward noising (diffusion) scheme to cover a smoother density across isolated modes of the training data distribution.”
BackgroundDual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration
Liu, X., Suganuma, M., Sun, Z., & Okatani, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00717 · Cited by 272
Multi-Scale Patch-Based Image Restoration
Papyan, V., & Elad, M. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2499698 · Cited by 262
Towards Perceptual Image Dehazing by Physics-Based Disentanglement and Adversarial Training
Yang, X., Xu, Z., & Luo, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12317 · Cited by 261
Deblurring via Stochastic Refinement
Whang, J., Delbracio, M., Talebi, H., Saharia, C., Dimakis, A. G., & Milanfar, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01581 · Cited by 259
Chen, W.-T., Huang, Z.-K., Tsai, C.-C., Yang, H.-H., Ding, J.-J., & Kuo, S.-Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01713 · Cited by 239
Chen, W.-T., Fang, H.-Y., Ding, J.-J., Tsai, C.-C., & Kuo, S.-Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58589-1_45 · Cited by 204
Deep Learning for Seeing Through Window With Raindrops
Quan, Y., Deng, S., Chen, Y., & Ji, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00255 · Cited by 201
Perception Prioritized Training of Diffusion Models
Choi, J., Lee, J., Shin, C., Kim, S., Kim, H., & Yoon, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01118 · Cited by 198
Removing Raindrops and Rain Streaks in One Go
Quan, R., Yu, X., Liang, Y., & Yang, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00903 · Cited by 194
Rain-Free and Residue Hand-in-Hand: A Progressive Coupled Network for Real-Time Image Deraining
Jiang, K., Wang, Z., Yi, P., Chen, C., Wang, Z., Wang, X., Jiang, J., & Lin, C.-W. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3102504 · Cited by 190
Conditional Variational Image Deraining
Du, Y., Xu, J., Zhen, X., Cheng, M., & Shao, L. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2990606 · Cited by 99
Feng, X., Pei, W., Jia, Z., Chen, F., Zhang, D. D., & Lu, G. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3076589 · Cited by 63
Cited by
Diffusion Models in Vision: A Survey
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261988 · Cited by 1,859
Wang, T., Zhang, K., Zhang, Y., Luo, W., Stenger, B., Lu, T., Kim, T.-K., & Liu, W. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111628 · Cited by 150
Metadata
- Authors
- Özdenizci, O., & Legenstein, R.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-08
- Volume
- 45
- Issue
- 8
- Pages
- 10346-10357
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4317496742
- Cited by
- 469
- References
- 74
- FWCI
- 42.64
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.