Robust Multimodal Vehicle Detection in Foggy Weather Using Complementary Lidar and Radar Signals
Abstract
Vehicle detection with visual sensors like lidar and camera is one of the critical functions enabling autonomous driving. While they generate fine-grained point clouds or high-resolution images with rich information in good weather conditions, they fail in adverse weather (e.g., fog) where opaque particles distort lights and significantly reduce visibility. Thus, existing methods relying on lidar or camera experience significant performance degradation in rare but critical adverse weather conditions. To remedy this, we resort to exploiting complementary radar, which is less impacted by adverse weather and becomes prevalent on vehicles. In this paper, we present Multimodal Vehicle Detection Network (MVDNet), a two-stage deep fusion detector, which first generates proposals from two sensors and then fuses region-wise features between multimodal sensor streams to improve final detection results. To evaluate MVDNet, we create a procedurally generated training dataset based on the collected raw lidar and radar signals from the open-source Oxford Radar Robotcar. We show that the proposed MVDNet surpasses other state-of-the-art methods, notably in terms of Average Precision (AP), especially in adverse weather conditions. The code and data are available at https://github.com/qiank10/MVDNet.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Point-Fusion proposed a dense fusion network that fuses point cloud features extracted by PointNet and image features extracted by ResNet.
“Point-Fusion [78] proposed a dense fusion network that fuses point cloud features extracted by PointNet [105] and image features extracted by ResNet [106].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [106]
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
(2016) · Cited by 31,590 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 9,767 · see citing works
Single image haze removal using dark channel prior
He, K., Sun, J., & Tang, X. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.168 · Cited by 7,601 · see citing works
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00252 · Cited by 3,437 · see citing works
SECOND: Sparsely Embedded Convolutional Detection
Yan, Y., Mao, Y., & Li, B. (2018) · Sensors · doi:10.3390/s18103337 · Cited by 3,352 · see citing works
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
AOD-Net: All-in-One Dehazing Network
Li, B., Peng, X., Wang, Z., Xu, J., & Feng, D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.511 · Cited by 2,467 · see citing works
Single image haze removal using dark channel prior
He, K., Sun, J., & Tang, X. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206515 · Cited by 2,449 · see citing works
Joint 3D Proposal Generation and Object Detection from View Aggregation
Ku, J., Mozifian, M., Lee, J., Harakeh, A., & Waslander, S. L. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594049 · Cited by 1,697 · see citing works
Semantic Foggy Scene Understanding with Synthetic Data
Sakaridis, C., Dai, D., & Gool, L. V. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1072-8 · Cited by 1,354 · see citing works
PIXOR: Real-time 3D Object Detection from Point Clouds
Yang, B., Luo, W., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00798 · Cited by 1,249 · see citing works
Fattal, R. (2014) · ACM Transactions on Graphics · doi:10.1145/2651362 · Cited by 1,187 · see citing works
GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing
Liu, X., Ma, Y., Shi, Z., & Chen, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00741 · Cited by 1,100 · see citing works
Deep Continuous Fusion for Multi-sensor 3D Object Detection
Liang, M., Yang, B., Wang, S., & Urtasun, R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_39 · Cited by 993 · see citing works
PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
Xu, D., Anguelov, D., & Jain, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00033 · Cited by 856 · see citing works
Making Bertha Drive—An Autonomous Journey on a Historic Route
Ziegler, J., Bender, P., Schreiber, M., Lategahn, H., Strauss, T., Stiller, C., Dang, T., Franke, U., Appenrodt, N., Keller, C. G., et al. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2306552 · Cited by 854 · see citing works
(2018) · arXiv (Cornell University) · Cited by 679 · see citing works
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather
Bijelic, M., Gruber, T., Mannan, F., Kraus, F., Ritter, W., Dietmayer, K., & Heide, F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01170 · Cited by 647 · see citing works
Luo, W., Yang, B., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00376 · Cited by 644 · see citing works
The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset
Barnes, D., Gadd, M., Murcutt, P., Newman, P. M., & Posner, I. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196884 · Cited by 435 · see citing works
Deep Continuous Fusion for Multi-Sensor 3D Object DetectionPreprint
Liang, M. X., Yang, B., Wang, S., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.10992 · Cited by 428 · see citing works
PointCleanNet : Learning to Denoise and Remove Outliers from Dense Point Clouds
Rakotosaona, M., Barbera, V. L., Guerrero, P., Mitra, N. J., & Ovsjanikov, M. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13753 · Cited by 381 · see citing works
A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection
Nobis, F., Geisslinger, M., Weber, M., Betz, J., & Lienkamp, M. (2019) · Sensor Data Fusion: Trends, Solutions, Applications (SDF) · doi:10.1109/sdf.2019.8916629 · Cited by 360 · see citing works
Luo, W., Yang, B., & Urtasun, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12395 · Cited by 345 · see citing works
Vehicle Detection and Tracking in Adverse Weather Using a Deep Learning Framework
Hassaballah, M., Kenk, M. A., Muhammad, K., & Minaee, S. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3014013 · Cited by 281 · see citing works
Weather Influence and Classification with Automotive Lidar Sensors
Heinzler, R., Schindler, P., Seekircher, J., Ritter, W., & Stork, W. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814205 · Cited by 247 · see citing works
Distant Vehicle Detection Using Radar and Vision
Chadwick, S., Maddern, W., & Newman, P. M. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794312 · Cited by 237 · see citing works
CNN-Based Lidar Point Cloud De-Noising in Adverse Weather
Heinzler, R., Piewak, F., Schindler, P., & Stork, W. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2972865 · Cited by 233 · see citing works
Precise Ego-Motion Estimation with Millimeter-Wave Radar Under Diverse and Challenging Conditions
Cen, S. H., & Newman, P. M. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460687 · Cited by 218 · see citing works
A Benchmark for Lidar Sensors in Fog: Is Detection Breaking Down?
Bijelic, M., Gruber, T., & Ritter, W. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500543 · Cited by 209 · see citing works
De-noising of Lidar Point Clouds Corrupted by Snowfall
Charron, N., Phillips, S., & Waslander, S. L. (2018) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2018.00043 · Cited by 204 · see citing works
Rrpn: Radar region proposal network for object detection in autonomous vehicles
Nabati, R., & Qi, H. (2019) · 2019 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2019.8803392 · Cited by 197 · see citing works
AtLoc: Attention Guided Camera Localization
Wang, B., Chen, C., Lu, C. X., Zhao, P., Trigoni, N., & Markham, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i06.6608 · Cited by 179 · see citing works
RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles
Nabati, R., & Hairong, Q. (2019) · IEEE Conference Proceedings · Cited by 176 · see citing works
RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects
Yang, B., Guo, R. B., Liang, M., Casas, S., & Urtasun, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_29 · Cited by 150 · see citing works
What Happens for a ToF LiDAR in Fog?
Li, Y., Duthon, P., Colomb, M., & Ibañez‐Guzmán, J. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2998077 · Cited by 137 · see citing works
Layer Depth Denoising and Completion for Structured-Light RGB-D Cameras
Shen, J., & Cheung, S.-C. S. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.157 · Cited by 135 · see citing works
Total Denoising: Unsupervised Learning of 3D Point Cloud Cleaning
Hermosilla, P., Ritschel, T., & Ropinski, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00014 · Cited by 128 · see citing works
Complex-YOLO: Real-time 3D Object Detection on Point CloudsPreprint
Simón, M., Milz, S., Amende, K., & Groß, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06199 · Cited by 120 · see citing works
Depth Estimation from Monocular Images and Sparse Radar Data
Lin, J.-T., Dai, D., & Gool, L. V. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340998 · Cited by 84 · see citing works
Benchmarking Image Sensors Under Adverse Weather Conditions for Autonomous Driving
Bijelic, M., Gruber, T., & Ritter, W. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500659 · Cited by 72 · see citing works
Deep Learning Based 3D Object Detection for Automotive Radar and Camera
Meyer, M., & Kuschk, G. (2019) · European Radar Conference (EuRAD) · Cited by 72 · see citing works
Golovachev, Y., Etinger, A., Pinhasi, G. A., et al. (2018) · Sensors · doi:10.3390/s18072148 · Cited by 46 · see citing works
Atmospheric and Fog Effects on Ultra-Wide Band Radar Operating at Extremely High Frequencies
Balal, N., Pinhasi, G. A., & Pinhasi, Y. (2016) · Sensors · doi:10.3390/s16050751 · Cited by 34 · see citing works
LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar FusionPreprint
Shah, M., Huang, Z., Laddha, A., Langford, M., Barber, B., Zhang, S., Vallespi-Gonzalez, C., & Urtasun, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00731 · Cited by 28 · see citing works
Multi-modality Cascaded Fusion Technology for Autonomous Driving
Kuang, H., Liu, X., Zhang, J., & Fang, Z. (2020) · International Conference on Robotics and Automation Sciences (ICRAS) · doi:10.1109/icras49812.2020.9135065 · Cited by 15 · see citing works
Radar-Lidar Sensor Fusion Sheme Using Occluded Depth Generation for Pedestrian Detection
Kwon, S. K., Son, S. H., Hyun, E., et al. (2017) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci.2017.322 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
3D Object Detection from Images for Autonomous Driving: A Survey
Ma, X., Ouyang, W., Simonelli, A., & Ricci, E. (2022) · arXiv · doi:10.48550/arxiv.2202.02980 · Cited by 147 · see citing works
Beam Alignment in mmWave V2X Communications: A Survey
Tan, J., Luan, T. H., Guan, W., Wang, Y., Peng, H., Zhang, Y., Zhao, D., & Lu, N. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3383093 · Cited by 75 · see citing works
Towards Robust 3D Object Detection with LiDAR and 4D Radar Fusion in Various Weather Conditions
Chae, Y., Kim, H., & Yoon, K.-J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01436 · Cited by 49 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“LIMO [151] further incorporated IMU data, utilizing sparse visual features to enhance system stability in dynamic environments.”
Self-citation
- Authors
- Qian, K., Zhu, S., Zhang, X., & Li, L. E.
- Affiliations
- University of California San Diego (US)
- Columbia University (US)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 444-453
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3167214600
- Cited by
- 206
- References
- 57
- Citations by year
- 2026: 22 · 2025: 50 · 2024: 56 · 2023: 51 · 2022: 23 · 2021: 4
- FWCI
- 33.16
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Aerospace Engineering › Radar Systems and Signal Processing
- Keywords
- vehicle detection, fog, LiDAR, radar, multimodal sensor fusion, autonomous driving, adverse weather perception, two-stage detection, region proposal generation, feature fusion, synthetic training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
