Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos
Abstract
In this paper, we address the problem of video rain removal by constructing deep recurrent convolutional networks. We visit the rain removal case by considering rain occlusion regions, i.e. the light transmittance of rain streaks is low. Different from additive rain streaks, in such rain occlusion regions, the details of background images are completely lost. Therefore, we propose a hybrid rain model to depict both rain streaks and occlusions. With the wealth of temporal redundancy, we build a Joint Recurrent Rain Removal and Reconstruction Network (J4R-Net) that seamlessly integrates rain degradation classification, spatial texture appearances based rain removal and temporal coherence based background details reconstruction. The rain degradation classification provides a binary map that reveals whether a location is degraded by linear additive streaks or occlusions. With this side information, the gate of the recurrent unit learns to make a trade-off between rain streak removal and background details reconstruction. Extensive experiments on a series of synthetic and real videos with rain streaks verify the superiority of the proposed method over previous state-of-the-art methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For DR(eye)VE testing, Alletto et al.'s proposed algorithm outperformed Vid2Vid and a rain-streak removal algorithm in both PSNR and SSIM metrics.
“Results again showed that the proposed algorithm overperformed the other algorithms, both in the PSNR and SSIM metrics.”
From A Review of Detection and Removal of Raindrops in Automotive Vision Systems (2021) · cites this paper as [68]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937655 · Cited by 8,216 · see citing works
Scope of validity of PSNR in image/video quality assessment
Huynh-Thu, Q., & Ghanbari, M. (2008) · Electronics Letters · doi:10.1049/el:20080522 · Cited by 2,959 · see citing works
Deep Joint Rain Detection and Removal from a Single Image
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.183 · Cited by 1,280 · see citing works
Self-citationRemoving Rain from Single Images via a Deep Detail Network
Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. W. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.186 · Cited by 1,273 · see citing works
Rain Streak Removal Using Layer Priors
Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.299 · Cited by 980 · see citing works
Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition
Kang, L.-W., Lin, C., & Fu, Y.-H. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2179057 · Cited by 913 · see citing works
Removing Rain from a Single Image via Discriminative Sparse Coding
Luo, Y., Xu, Y., & Ji, H. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.388 · Cited by 863 · see citing works
Detection and removal of rain from videos
Garg, K., & Nayar, S. K. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315077 · Cited by 531 · see citing works
A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks
Chen, Y., & Hsu, C.-T. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.247 · Cited by 490 · see citing works
Garg, K., & Nayar, S. K. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-006-0028-6 · Cited by 477 · see citing works
Restoring an Image Taken through a Window Covered with Dirt or Rain
Eigen, D., Krishnan, D., & Fergus, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.84 · Cited by 454 · see citing works
Analysis of Rain and Snow in Frequency Space
Barnum, P. C., Narasimhan, S. G., & Takeo Kanade, A. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0200-2 · Cited by 354 · see citing works
Rain or Snow Detection in Image Sequences Through Use of a Histogram of Orientation of Streaks
Bossu, J., Hautière, N., & Tarel, J. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0421-7 · Cited by 349 · see citing works
Rain Removal in Video by Combining Temporal and Chromatic Properties
Zhang, X., Li, H., Qi, Y., Leow, W. K., & Ng, T. (2006) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2006.262572 · Cited by 298 · see citing works
Single-image deraining using an adaptive nonlocal means filter
Kim, J. H., Lee, C., Sim, J.-Y., et al. (2013) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2013.6738189 · Cited by 278 · see citing works
Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion
Kim, J. H., Sim, J.-Y., & Kim, C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2428933 · Cited by 240 · see citing works
Photorealistic rendering of rain streaks
Garg, K., & Nayar, S. K. (2006) · ACM Transactions on Graphics · doi:10.1145/1141911.1141985 · Cited by 235 · see citing works
Garg, K., & Nayar, S. K. (2005) · doi:10.1109/iccv.2005.253 · Cited by 235 · see citing works
Self-Learning Based Image Decomposition With Applications to Single Image Denoising
Huang, D.-A., Kang, L., Wang, Y.-C. F., & Lin, C. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2284759 · Cited by 214 · see citing works
Utilizing Local Phase Information to Remove Rain from Video
Santhaseelan, V., & Asari, V. K. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0759-8 · Cited by 182 · see citing works
Jiang, T.-X., Huang, T., Zhao, X.-L., Deng, L.-J., & Wang, Y. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.301 · Cited by 169 · see citing works
Should We Encode Rain Streaks in Video as Deterministic or Stochastic?
Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., & Xu, Z. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.275 · Cited by 142 · see citing works
Using the Shape Characteristics of Rain to Identify and Remove Rain from Video
Brewer, N., & Liu, N. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-89689-0_49 · Cited by 122 · see citing works
Context-Aware Single Image Rain Removal
Huang, D.-A., Kang, L., Yang, M.-C., et al. (2012) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2012.92 · Cited by 115 · see citing works
Exploiting image structural similarity for single image rain removal
Sun, S.-H., Fan, S.-P., & Wang, Y.-C. F. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025909 · Cited by 111 · see citing works
A Rain Pixel Recovery Algorithm for Videos With Highly Dynamic Scenes
Chen, J., & Chau, L. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2290595 · Cited by 81 · see citing works
Video post processing: low-latency spatiotemporal approach for detection and removal of rain
Tripathi, A. K., & Mukhopadhyay, S. (2012) · IET Image Processing · doi:10.1049/iet-ipr.2010.0547 · Cited by 62 · see citing works
Pixel Based Temporal Analysis Using Chromatic Property for Removing Rain from Videos
Liu, P., Xu, J., Liu, J., et al. (2009) · Computer and Information Science · doi:10.5539/cis.v2n1p53 · Cited by 50 · see citing works
Self-citationA Probabilistic Approach for Detection and Removal of Rain from Videos
Tripathi, A. K., & Mukhopadhyay, S. (2011) · IETE Journal of Research · doi:10.4103/0377-2063.78382 · Cited by 50 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
All in One Bad Weather Removal Using Architectural Search
Li, R., Tan, R. T., & Cheong, L.-F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00324 · Cited by 377 · see citing works
A Review of Detection and Removal of Raindrops in Automotive Vision Systems
Hamzeh, Y., & Rawashdeh, S. A. (2021) · Journal of Imaging · doi:10.3390/jimaging7030052 · Cited by 21 · see citing works
“For the DR(eye)VE testing, Vid2Vid [67] and a rainstreak removal [68] algorithms were used instead of the Pix2Pix and Eigen et al.”
Method
- Authors
- Liu, J., Yang, W., Yang, S., & Guo, Z.
- Affiliations
- Peking University (CN)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 3233-3242
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2798744505
- Cited by
- 193
- References
- 32
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 20 · 2023: 31 · 2022: 24 · 2021: 47 · 2020: 32 · 2019: 19 · 2018: 8
- FWCI
- 7.99
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Enhancement Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- rain streaks, recurrent convolutional networks, temporal coherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
