Physics-Based Rendering for Improving Robustness to Rain
Open accessAbstract
To improve the robustness to rain, we present a physically-based rain rendering pipeline for realistically inserting rain into clear weather images. Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. We validate our rendering with a user study, proving our rain is judged 40% more realistic that state-of-the-art. Using our generated weather augmented Kitti and Cityscapes dataset, we conduct a thorough evaluation of deep object detection and semantic segmentation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection and 60% for semantic segmentation. Furthermore, we show refining existing networks with our augmented images improves the robustness of both object detection and semantic segmentation algorithms. We experiment on nuScenes and measure an improvement of 15% for object detection and 35% for semantic segmentation compared to original rainy performance. Augmented databases and code are available on the project page.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
(2016) · Cited by 31,590 · see citing works
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · see citing works
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2× · see citing works
“Instead, we refine our untuned models using curriculum learning [4] on rain intensity in ascending order (25, then 50, then 75 and finally 100mm/hr rain).”
MethodR-FCN: Object Detection via Region-based Fully Convolutional NetworksPreprint
Dai, J., Li, Y., He, K., & Sun, J. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06409 · Cited by 5,135 · see citing works
(1986) · Cited by 3,616 · see citing works
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
Learning to Adapt Structured Output Space for Semantic Segmentation
Tsai, Y., Hung, W.-C., Schulter, S., Sohn, K., Yang, M., & Chandraker, M. K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00780 · Cited by 1,753 · see citing works
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709 · see citing works
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632 · see citing works
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,491 · see citing works
Semantic Foggy Scene Understanding with Synthetic Data
Sakaridis, C., Dai, D., & Gool, L. V. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1072-8 · Cited by 1,354 · see citing works
Deep Joint Rain Detection and Removal from a Single Image
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.183 · Cited by 1,280 · see citing works
Image De-Raining Using a Conditional Generative Adversarial Network
Zhang, H., Sindagi, V. A., & Patel, V. M. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2920407 · Cited by 1,232 · see citing works
Density-Aware Single Image De-raining Using a Multi-stream Dense Network
Zhang, H., & Patel, V. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00079 · Cited by 1,066 · see citing works
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., & Hajishirzi, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_34 · Cited by 1,029 · see citing works
Rain Streak Removal Using Layer Priors
Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.299 · Cited by 980 · see citing works
Doppler radar characteristics of precipitation at vertical incidence
Atlas, D., Srivastava, R. C., & Sekhon, R. S. (1973) · Reviews of Geophysics · doi:10.1029/rg011i001p00001 · Cited by 977 · see citing works
Removing Rain from a Single Image via Discriminative Sparse Coding
Luo, Y., Xu, Y., & Ji, H. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.388 · Cited by 863 · see citing works
(2018) · arXiv (Cornell University) · Cited by 679 · see citing works
DSOD: Learning Deeply Supervised Object Detectors from Scratch
Shen, Z., Liu, Z., Li, J., Jiang, Y., Chen, Y., & Xue, X. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.212 · Cited by 674 · see citing works
Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
Johnson‐Roberson, M., Barto, C., Mehta, R., Sridhar, S. N., Rosaen, K., & Vasudevan, R. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989092 · Cited by 649 · see citing works
Yang, F., Choi, W., & Lin, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.234 · Cited by 608 · see citing works
Detection and removal of rain from videos
Garg, K., & Nayar, S. K. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315077 · Cited by 531 · see citing works
A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks
Chen, Y., & Hsu, C.-T. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.247 · Cited by 490 · see citing works
Garg, K., & Nayar, S. K. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-006-0028-6 · Cited by 477 · see citing works
Restoring an Image Taken through a Window Covered with Dirt or Rain
Eigen, D., Krishnan, D., & Fergus, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.84 · Cited by 454 · see citing works
Analysis of Rain and Snow in Frequency Space
Barnum, P. C., Narasimhan, S. G., & Takeo Kanade, A. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0200-2 · Cited by 354 · see citing works
THE PRODUCTION OF RAIN BY A CHAIN REACTION IN CUMULUS CLOUDS AT TEMPERATURES ABOVE FREEZING
Langmuir, I. (1948) · Journal of Meteorology · doi:10.1175/1520-0469(1948)005 · Cited by 351 · see citing works
Barron, J. T., & Poole, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_38 · Cited by 339 · see citing works
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation
Jaritz, M., de Charette, R., Wirbel, É., Perrotton, X., & Nashashibi, F. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00017 · Cited by 311 · see citing works
Self-citationTransient attributes for high-level understanding and editing of outdoor scenes
Laffont, P.-Y., Ren, Z., Tao, X., Qian, C., & Hays, J. H. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601101 · Cited by 271 · see citing works
Image De-raining Using a Conditional Generative Adversarial NetworkPreprint
Zhang, H., Sindagi, V. A., & Patel, V. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.05957 · Cited by 245 · see citing works
Photorealistic rendering of rain streaks
Garg, K., & Nayar, S. K. (2006) · ACM Transactions on Graphics · doi:10.1145/1141911.1141985 · Cited by 235 · see citing works
Garg, K., & Nayar, S. K. (2005) · doi:10.1109/iccv.2005.253 · Cited by 235 · see citing works
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1805.04687 · Cited by 204 · see citing works
A lens and aperture camera model for synthetic image generation
Potmesil, M., & Chakravarty, I. (1981) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/800224.806818 · Cited by 172 · see citing works
Consistent Temporal Variations in Many Outdoor Scenes
Jacobs, N., Roman, N., & Pless, R. B. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383258 · Cited by 171 · see citing works
Atmospheric effects on availability of free space optics systems
Prokeš, A. (2009) · Optical Engineering · doi:10.1117/1.3155431 · Cited by 152 · see citing works
Deep Sky Modeling for Single Image Outdoor Lighting Estimation
Hold-Geoffroy, Y., Athawale, A., & Lalonde, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00709 · Cited by 132 · see citing works
Self-citationA lens and aperture camera model for synthetic image generation
Potmesil, M., & Chakravarty, I. (1981) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965161.806818 · Cited by 117 · see citing works
All the Images of an Outdoor Scene
Narasimhan, S. G., Wang, C., & Nayar, S. K. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47977-5_10 · Cited by 79 · see citing works
Lalonde, J., Efros, A. A., & Narasimhan, S. G. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618477 · Cited by 78 · see citing works
Self-citationHalimeh, J. C., & Roser, M. (2009) · 2009 IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2009.5164347 · Cited by 76 · see citing works
Realistic real-time rain rendering
Rousseau, P., Jolivet, V., & Ghazanfarpour, D. (2006) · Computers & Graphics · doi:10.1016/j.cag.2006.03.013 · Cited by 50 · see citing works
Numerical model for the fall speed of raindrops in a rainfall simulator
Boxel, J. H. V. (1998) · UvA-DARE (University of Amsterdam) · Cited by 49 · see citing works
Robust Optical Flow in Rainy Scenes
Li, R., Tan, R. T., & Cheong, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_18 · Cited by 49 · see citing works
Artist-directable real-time rain rendering in city environments
Tatarchuk, N. (2006) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1185657.1185828 · Cited by 47 · see citing works
Photorealistic rendering of rain streaks
Garg, K., & Nayar, S. K. (2006) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1179352.1141985 · Cited by 33 · see citing works
Fast reactive control for illumination through rain and snow
de Charette, R., Tamburo, R., Barnum, P. C., et al. (2012) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2012.6215217 · Cited by 29 · see citing works
Self-citationA multiscale model for rain rendering in real-time
Weber, Y., Jolivet, V., Gilet, G., et al. (2015) · Computers & Graphics · doi:10.1016/j.cag.2015.04.007 · Cited by 20 · see citing works
R4: Realistic rain rendering in realtime
Creus, C., & Patow, G. (2012) · Computers & Graphics · doi:10.1016/j.cag.2012.12.002 · Cited by 14 · see citing works
Lalonde, J., Efros, A. A., & Narasimhan, S. G. (2009) · ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia) · doi:10.1145/1661412.1618477 · Cited by 11 · see citing works
Self-citationInfluence of Fog on Computer Vision AlgorithmsPreprint
Kahraman, S., & de Charette, R. (2017) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- [13]13. Kshitiz Garg, Gurunandan Krishnan, and Shree K Nayar. Material based splashing of water drops. In Eurographics Conference on Rendering Techniques, 2007. 8
- [36]36. Will Maddern, Geoff Pascoe, Chris Linegar, and Paul New- man. 1 Year, 1000km: The Oxford RobotCar Dataset. In- ternational Journal of Robotics Research, 36(1):3-15, 2017.
- [37]37. John S Marshall and W Mc K Palmer. The distribution of raindrops with size. Journal of meteorology, 5(4):165-166, 1948. 2, 3
- [49]49. Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learn- ing to adapt structured output space for semantic segmenta- tion. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 6
Cited by
Sorted by citations
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
“nuScenes has been used for 3D object detection [83, 60], multi-agent forecasting [9, 68], pedestrian localization [5], weather augmentation [37], and moving pointcloud prediction [27].”
MethodACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding
Sakaridis, C., Dai, D., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01059 · Cited by 572 · see citing works
Hu, J., Li, J., Wang, Z., Li, Z., Chan, K.-C., Ma, X., Jiang, Z., & Zhang, Y. (2026) · AHFE International · doi:10.54941/ahfe1007640
- Authors
- Halder, S., Lalonde, J.-F., & Charette, R. D.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 10202-10211
- OpenAlex
- W2970781471
- Cited by
- 130
- References
- 62
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 21 · 2023: 28 · 2022: 19 · 2021: 28 · 2020: 11 · 2019: 2
- FWCI
- 4.89
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Enhancement Techniques
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- physically based rendering, image degradation, particle simulation, fog, object detection robustness, data augmentation, KITTI, cityscape, nuScenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
