PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
Open accessAbstract
We present PointFusion, a generic 3D object detection method that leverages both image and 3D point cloud information. Unlike existing methods that either use multistage pipelines or hold sensor and dataset-specific assumptions, PointFusion is conceptually simple and application-agnostic. The image data and the raw point cloud data are independently processed by a CNN and a PointNet architecture, respectively. The resulting outputs are then combined by a novel fusion network, which predicts multiple 3D box hypotheses and their confidences, using the input 3D points as spatial anchors. We evaluate PointFusion on two distinctive datasets: the KITTI dataset that features driving scenes captured with a lidar-camera setup, and the SUN-RGBD dataset that captures indoor environments with RGB-D cameras. Our model is the first one that is able to perform better or on-par with the state-of-the-art on these diverse datasets without any dataset-specific model tuning.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
EP-Net introduces Li-Fusion module that adaptively fuses LiDAR and image features at point level.
“EP-Net [82] introduces a Li-Fusion module that adaptively fuses LiDAR and image features at the point level.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [82]
In autonomous driving, fusion weights of RGB images and LiDAR point clouds can be adaptively adjusted according to ambient lighting.
“For example, in autonomous driving, the fusion weights of RGB images and LiDAR point clouds can be adaptively adjusted according to the ambient lighting conditions [82].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [82]
PointFusion achieves leading results on benchmark datasets, including KITTI for outdoor and SUN-RGBD for indoor environments, outperforming several multi-stage and voxel-based methods.
“Through advanced sensor fusion techniques, PointFusion operates effectively without requiring dataset-specific parameter adjustments and achieves leading results on benchmark datasets, including KITTI for outdoor and SUN-RGBD for indoor environments, outperforming several multi-stage and voxel-based methods.”
From Use and Potential of AI in Assisting Surveyors in Building Retrofit and Demolition—A Scoping Review (2025) · cites this paper as [46]
Xu et al. (2018) had high computation.
“Xu et al. (2018) [46] Deep Learning PointFusion KITTI, SUN-RGBD SoTA on datasets High computation”
From Use and Potential of AI in Assisting Surveyors in Building Retrofit and Demolition—A Scoping Review (2025) · cites this paper as [46]
Xu et al. (2018) achieved state-of-the-art performance on datasets using a deep learning PointFusion method on KITTI and SUN-RGBD.
“Xu et al. (2018) [46] Deep Learning PointFusion KITTI, SUN-RGBD SoTA on datasets High computation”
From Use and Potential of AI in Assisting Surveyors in Building Retrofit and Demolition—A Scoping Review (2025) · cites this paper as [46]
From the tables, most works propose to fuse LiDAR and camera features extracted from 2D convolution neural networks; several works cluster and segment 3D LiDAR points to generate 3D region proposals but still use a LiDAR 2D representation to extract features for fusion; several works project LiDAR points on the camera-plane or RGB camera images on the LiDAR BEV plane to align features, whereas许多许多
“From the tables we have three observations: (1). Most works propose to fuse LiDAR and camera features extracted from 2D convolution neural networks. To do this, they project LiDAR points on the 2D plane and process the feature maps through 2D convolutions. Only a few works extract LiDAR features by PointNet (e.g. [104], [105], [128]) or 3D convolutions (e.g. [123]); (2). Several works on multi”
From Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2021) · cites this paper as [104]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 9,767 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.351 · Cited by 2,743 · see citing works
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · see citing works
Scalable Object Detection Using Deep Neural Networks
Erhan, D., Szegedy, C., Toshev, A., & Anguelov, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.276 · Cited by 1,161 · cited 1× · see citing works
“Instead of directly regressing the 2D box, a common solution is to generate object proposals by using sliding windows [6] or by predicting the box displacement relative to spatial anchors [24, 8, 15, 25].”
MethodSelf-citation3D Bounding Box Estimation Using Deep Learning and Geometry
Mousavian, A., Anguelov, D., Flynn, J., & Kosecka, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.597 · Cited by 1,129 · see citing works
Self-citationUhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924 · see citing works
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
Song, S., & Xiao, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.94 · Cited by 671 · see citing works
(2015) · Cited by 656 · see citing works
Vehicle Detection from 3D Lidar Using Fully Convolutional Network
Li, B., Zhang, T., & Xia, T. (2016) · Robotics: Science and Systems XII · doi:10.15607/rss.2016.xii.042 · Cited by 577 · see citing works
3D fully convolutional network for vehicle detection in point cloud
Li, B. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8205955 · Cited by 488 · see citing works
Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models
Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., & Šivic, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.487 · Cited by 471 · see citing works
A multi-sensor fusion system for moving object detection and tracking in urban driving environments
Cho, H., Seo, Y., Kumar, B. V. K. V., & Rajkumar, R. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907100 · Cited by 441 · see citing works
The MOPED framework: Object recognition and pose estimation for manipulation
Collet, A., Martinez, M., & Srinivasa, S. S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911401765 · Cited by 416 · see citing works
Rothganger, F., Lazebnik, S., Schmid, C., & Ponce, J. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-005-3674-1 · Cited by 413 · see citing works
DenseBox: Unifying Landmark Localization with End to End Object Detection
Huang, L., Yang, Y., Deng, Y., & Yu, Y. (2015) · arXiv · doi:10.48550/arxiv.1509.04874 · Cited by 407 · see citing works
Sliding Shapes for 3D Object Detection in Depth Images
Song, S., & Xiao, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_41 · Cited by 361 · see citing works
Tulsiani, S., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298758 · Cited by 348 · see citing works
Data-driven 3D Voxel Patterns for object category recognition
Yu, X., Choi, W., Lin, Y., & Savarese, S. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298800 · Cited by 319 · see citing works
Subcategory-Aware Convolutional Neural Networks for Object Proposals and Detection
Yu, X., Choi, W., Lin, Y., & Savarese, S. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.108 · Cited by 303 · see citing works
Vehicle Detection from 3D Lidar Using Fully Convolutional NetworkPreprint
Li, B., Zhang, T., & Xia, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07916 · Cited by 275 · see citing works
Single image 3D object detection and pose estimation for grasping
Zhu, M., Derpanis, K. G., Yang, Y., Brahmbhatt, S., Zhang, M. M., Phillips, C. J., Lecce, M., & Daniilidis, K. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907430 · Cited by 217 · see citing works
Simultaneous Object Recognition and Segmentation from Single or Multiple Model Views
Ferrari, V., Tuytelaars, T., & Gool, L. V. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-005-3964-7 · Cited by 200 · see citing works
Aligning 3D models to RGB-D images of cluttered scenes
Gupta, S., Arbeláez, P., Girshick, R., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299105 · Cited by 199 · see citing works
2D-Driven 3D Object Detection in RGB-D Images
Lahoud, J., & Ghanem, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.495 · Cited by 177 · see citing works
A Multilevel Mixture-of-Experts Framework for Pedestrian Classification
Enzweiler, M., & Gavrila, D. M. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2142006 · Cited by 150 · see citing works
Three-Dimensional Object Detection and Layout Prediction Using Clouds of Oriented Gradients
Ren, Z., & Sudderth, E. B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.169 · Cited by 138 · see citing works
Multi-modal sensor registration for vehicle perception via deep neural networks
Giering, M. J., Venugopalan, V., & Reddy, K. (2015) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec.2015.7322485 · Cited by 31 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
“And while there are several works that have proposed fusion methods based on cameras and lidar [48, 14, 64, 52, 81, 75, 29], PointPillars [51] showed a lidar-only method that performed on par with existing fusion based methods.”
BackgroundFeng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
Wang, Y., Chao, W.-L., Garg, D., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00864 · Cited by 1,154 · see citing works
Yoo, J. H., Kim, Y., Kim, J., & Choi, J. W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58583-9_43 · Cited by 520 · see citing works
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection
Nabati, R., & Qi, H. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00157 · Cited by 461 · see citing works
Deep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261 · see citing works
Robust Multimodal Vehicle Detection in Foggy Weather Using Complementary Lidar and Radar Signals
Qian, K., Zhu, S., Zhang, X., & Li, L. E. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00051 · Cited by 206 · see citing works
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
Yin, J., Shen, J., Chen, R., Li, W., Yang, R., Frossard, P., & Wang, W. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01412 · Cited by 133 · see citing works
M3DSSD: Monocular 3D Single Stage Object Detector
Luo, S., Dai, H., Shao, L., & Ding, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00608 · Cited by 93 · see citing works
Emerging trends in autonomous vehicle perception: Multimodal fusion for 3D object detection
Alaba, S. Y., Gurbuz, A. C., & Ball, J. E. (2024) · World Electric Vehicle Journal · doi:10.3390/wevj15010020 · Cited by 82 · see citing works
Multimodal Fusion Object Detection System for Autonomous Vehicles
Person, M., Jensen, M., Smith, A. O., & Gutierrez, H. (2019) · Journal of Dynamic Systems, Measurement, and Control · doi:10.1115/1.4043222 · Cited by 39 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“…and FV (Front View) with RGB features, followed by AVOD [75], which uses FPN to extract high-resolution features for feature fusion; methods such as Pointfusion [81], MVX-Net [82], and Pointpainting [83] have further explored the benefits of different fusion granularity feature fusion advantages.”
Multi-Sem Fusion: Multimodal Semantic Fusion for 3-D Object Detection
Xu, S., Li, F., Song, Z., Fang, J., Wang, S., & Yang, Z.-X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3387732 · Cited by 26 · see citing works
Transformer-based sensor fusion for autonomous vehicles: A comprehensive review
Abdulmaksoud, A., & Ahmed, R. (2025) · IEEE Access · doi:10.1109/access.2025.3545032 · Cited by 25 · see citing works
Use and Potential of AI in Assisting Surveyors in Building Retrofit and Demolition—A Scoping Review
Yin, Y., Zuo, H., Jennings, T., Jain, S., Cartwright, B., Buhagiar, J., Williams, P., Adams, K., Hazeri, K., & Childs, P. (2025) · Buildings · doi:10.3390/buildings15193448 · Cited by 1 · see citing works
- Authors
- Xu, D., Anguelov, D., & Jain, A.
- Affiliations
- Stanford University (US)
- ORCID
- Xu, Danfei
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 244-253
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962888833
- Cited by
- 856
- References
- 38
- Citations by year
- 2026: 44 · 2025: 93 · 2024: 130 · 2023: 138 · 2022: 113 · 2021: 131 · 2020: 120 · 2019: 80 · 2018: 7
- FWCI
- 25.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 3D object detection, 3D bounding box estimation, sensor fusion, point clouds, LiDAR-camera fusion, fusion network, PointNet, spatial anchors, KITTI, indoor environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
