DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection
Open accessAbstract
Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While prevalent multi-modal methods [34], [36] simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance. However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities. In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion. Based on InverseAug and LearnableAlign, we develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods. For example, DeepFusion improves Point-Pillars, CenterPoint, and 3D-MAN baselines on Pedestrian detection for 6.7,8.9, and 6.2 LEVEL_2 APH, respectively. Notably, our models achieve state-of-the-art performance on Waymo Open Dataset, and show strong model robustness against input corruptions and out-of-distribution data. Code will be publicly available at https://github.com/tensorflow/lingvo.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FusionPainting integrates 2D and 3D segmentation to address boundary blurring.
“FusionPainting [83] integrates 2D and 3D segmentation to address boundary blurring.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [83]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpacePreprint
Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02413 · Cited by 9,834 · see citing works
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 9,767 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00252 · Cited by 3,437 · see citing works
Self-citationSECOND: Sparsely Embedded Convolutional Detection
Yan, Y., Mao, Y., & Li, B. (2018) · Sensors · doi:10.3390/s18103337 · Cited by 3,352 · see citing works
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., & Li, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01054 · Cited by 2,341 · see citing works
Center-based 3D Object Detection and Tracking
Yin, T., Zhou, X., & Krähenbühl, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01161 · Cited by 2,197 · see citing works
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Elfwing, S., Uchibe, E., & Doya, K. (2018) · Neural Networks · doi:10.1016/j.neunet.2017.12.012 · Cited by 2,119 · see citing works
Joint 3D Proposal Generation and Object Detection from View Aggregation
Ku, J., Mozifian, M., Lee, J., Harakeh, A., & Waslander, S. L. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594049 · Cited by 1,697 · see citing works
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335 · see citing works
Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit
Hahnloser, R. H. R., Sarpeshkar, R., Mahowald, M. A., Douglas, R. J., & Seung, H. S. (2000) · Nature · doi:10.1038/35016072 · Cited by 1,293 · see citing works
PIXOR: Real-time 3D Object Detection from Point Clouds
Yang, B., Luo, W., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00798 · Cited by 1,249 · see citing works
PointPainting: Sequential Fusion for 3D Object Detection
Vora, S., Lang, A. H., Helou, B., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00466 · Cited by 1,236 · see citing works
Monocular 3D Object Detection for Autonomous Driving
Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., & Urtasun, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.236 · Cited by 1,019 · see citing works
Deep Continuous Fusion for Multi-sensor 3D Object Detection
Liang, M., Yang, B., Wang, S., & Urtasun, R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01270-0_39 · Cited by 993 · see citing works
Wang, Z., & Jia, K. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968513 · Cited by 547 · see citing works
Epnet: Enhancing point features with image semantics for 3d object detection
Huang, T., Liu, Z., Chen, X., & Bai, X. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58555-6_3 · Cited by 518 · see citing works
PointAugmenting: Cross-Modal Augmentation for 3D Object Detection
Wang, C., Ma, C., Zhu, M. B., & Yang, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01162 · Cited by 481 · see citing works
LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving
Meyer, G. P., Laddha, A., Kee, E., Vallespi-Gonzalez, C., & Wellington, C. K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01296 · Cited by 374 · see citing works
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
Chen, X., Zhang, T., Wang, Y., Wang, Y., & Zhao, H. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00022 · Cited by 337 · see citing works
Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction
Ku, J., Pon, A., & Waslander, S. L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01214 · Cited by 306 · see citing works
Embracing Single Stride 3D Object Detector with Sparse Transformer
Fan, L., Pang, Z., Zhang, T., Wang, Y.-X., Zhao, H., Wang, F., Wang, N., & Zhang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00827 · Cited by 296 · see citing works
Pillar-Based Object Detection for Autonomous Driving
Wang, Y., Fathi, A., Kundu, A., Ross, D. A., Pantofaru, C., Funkhouser, T., & Solomon, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58542-6_2 · Cited by 237 · see citing works
Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-Out Classifiers
Vyas, A., Jammalamadaka, N., Zhu, X., Das, D., Kaul, B., & Willke, T. L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_34 · Cited by 205 · see citing works
End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection
Qian, R., Garg, D., Wang, Y., You, Y., Belongie, S., Hariharan, B., Campbell, M., Weinberger, K. Q., & Chao, W. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00592 · Cited by 204 · see citing works
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence ModelingPreprint
Shen, J., Nguyen, P., Wu, Y., Chen, Z., Chen, M. X., Ye, J., Kannan, A., Sainath, T. N., Cao, Y. J., Chiu, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08295 · Cited by 169 · see citing works
Self-citationRSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection
Sun, P., Wang, W., Chai, Y., Elsayed, G. F., Bewley, A., Zhang, X., Sminchisescu, C., & Anguelov, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00567 · Cited by 167 · see citing works
AFDetV2: Rethinking the Necessity of the Second Stage for Object Detection from Point Clouds
Hu, Y., Ding, Z., Ge, R., Shao, W., Huang, L., Li, K., & Liu, Q. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19980 · Cited by 137 · see citing works
IPOD: Intensive Point-based Object Detector for Point Cloud
Yang, Z., Sun, Y., Liu, S., Shen, X., & Jia, J. (2018) · arXiv · doi:10.48550/arxiv.1812.05276 · Cited by 132 · see citing works
StarNet: Targeted Computation for Object Detection in Point CloudsPreprint
Ngiam, J., Caine, B., Han, W., Yang, B., Chai, Y., Sun, P., Zhou, Y., Yi, X., Alsharif, O., Nguyen, P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11069 · Cited by 119 · see citing works
Self-citation3D-MAN: 3D Multi-frame Attention Network for Object Detection
Yang, Z., Zhou, Y., Chen, Z., & Ngiam, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00190 · Cited by 117 · see citing works
Self-citation4D-Net for Learned Multi-Modal Alignment
Piergiovanni, A., Casser, V., Ryoo, M. S., & Angelova, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01515 · Cited by 64 · see citing works
Improving 3D Object Detection Through Progressive Population Based Augmentation
Cheng, S., Leng, Z., Cubuk, E. D., Zoph, B., Bai, C., Ngiam, J., Song, Y., Caine, B., Vasudevan, V., Li, C., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_17 · Cited by 62 · see citing works
Self-citationChen, Q., Sun, L., Cheung, E. C. H., & Yuille, A. (2020) · neural information processing systems · Cited by 61 · see citing works
Self-citation1st Place Solution for Waymo Open Dataset Challenge -- 3D Detection and Domain AdaptationPreprint
Ding, Z., Hu, Y., Ge, R., Huang, L., Chen, S., Wang, Y., & Liao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15505 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274 · see citing works
Self-citationLearning for Vehicle-to-Vehicle Cooperative Perception Under Lossy Communication
Li, J., Xu, R., Liu, X., Ma, J., Chi, Z., Ma, J., & Yu, H. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3260040 · Cited by 161 · see citing works
ULIP-2: Towards Scalable Multimodal Pre-Training for 3D Understanding
Xue, L., Yu, N., Zhang, S., Panagopoulou, A., Li, J., Martín-Martín, R., Wu, J., Xiong, C., Xu, R., Niebles, J. C., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02558 · Cited by 144 · see citing works
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
Yin, J., Shen, J., Chen, R., Li, W., Yang, R., Frossard, P., & Wang, W. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01412 · Cited by 133 · see citing works
Self-citationFocalFormer3D : Focusing on Hard Instance for 3D Object Detection
Chen, Y., Yu, Z., Chen, Y., Lan, S., Anandkumar, A., Jia, J., & Alvarez, J. M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00771 · Cited by 119 · see citing works
“Recent multi-modal detectors [5–7, 39, 42, 57] follow the similar paradigm of BEV detectors and incorporate the multi-view image features by physical projection or learnable alignments between LiDAR and cameras.”
MethodEmerging trends in autonomous vehicle perception: Multimodal fusion for 3D object detection
Alaba, S. Y., Gurbuz, A. C., & Ball, J. E. (2024) · World Electric Vehicle Journal · doi:10.3390/wevj15010020 · Cited by 82 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Future research will likely focus on improving scalability, real-time capabilities, and developing unified frameworks that integrate localization with high-level perception and decision-making tasks [145].”
A review of research on slam technology based on the fusion of lidar and vision
Chen, P., Zhao, X., Zeng, L., Liu, L., Liu, S., Sun, L., Li, Z., Chen, H., Liu, G., Qiao, Z., et al. (2025) · Sensors · doi:10.3390/s25051447 · Cited by 46 · see citing works
- Authors
- Li, Y., Yu, A. W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J., Lu, Y., Zhou, D., Le, Q. V., Yuille, A., & Tan, M.
- Affiliations
- Google (United States) (US)
- Johns Hopkins University (US)
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 17161-17170
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4225986494
- Cited by
- 563
- References
- 41
- Citations by year
- 2026: 91 · 2025: 162 · 2024: 168 · 2023: 110 · 2022: 32
- FWCI
- 121.16
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- LiDAR-camera fusion, autonomous driving, geometric alignment, cross-attention, feature fusion, pedestrian detection, Waymo Open Dataset, out-of-distribution robustness, point clouds
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
