Multimodal Fusion Object Detection System for Autonomous Vehicles
The DOI matches its Crossref record; 41 of the 43 references it cites were found in a registry or an index.
Abstract
In order for autonomous vehicles to safely navigate the road ways, accurate object detection must take place before safe path planning can occur. Currently, general purpose object detection convolutional neural network (CNN) models have the highest detection accuracies of any method. However, there is a gap in the proposed detection frameworks. Specifically, those that provide high detection accuracy necessary for deployment but do not perform inference in realtime, and those that perform inference in realtime but detection accuracy is low. We propose multimodel fusion detection system (MFDS), a sensor fusion system that combines the speed of a fast image detection CNN model along with the accuracy of light detection and range (LiDAR) point cloud data through a decision tree approach. The primary objective is to bridge the tradeoff between performance and accuracy. The motivation for MFDS is to reduce the computational complexity associated with using a CNN model to extract features from an image. To improve efficiency, MFDS extracts complimentary features from the LiDAR point cloud in order to obtain comparable detection accuracy. MFDS is novel by not only using the image detections to aid three-dimensional (3D) LiDAR detection but also using the LiDAR data to jointly bolster the image detections and provide 3D detections. MFDS achieves 3.7% higher accuracy than the base CNN detection model and is able to operate at 10 Hz. Additionally, the memory requirement for MFDS is small enough to fit on the Nvidia Tx1 when deployed on an embedded device.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“a multimodal fusion detection system for autonomous vehicles, that combines visual features from cameras along with data from Light Detection and Ranging (LiDAR) sensors, is able to achieve significantly higher accuracy (3.7% improvement) than a single-modal CNN detection model21.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [21]
In the citing paper
“Multimodal deep learning models that can ingest pixel data along with other data types (fusion) have been successful in applications outside of medicine, such as autonomous driving and video classification.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [21]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
(2016) · Cited by 11,679
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R. Q., Su, H., Kaichun, M., & Guibas, L. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.16 · Cited by 9,237
Robust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813
(2013) · Cited by 3,740
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450
R-FCN: Object Detection via Region-based Fully Convolutional Networks
Dai, J., Li, Y., He, K., & Sun, J. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06409 · Cited by 3,149
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.351 · Cited by 2,743
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632
Autonomous driving in urban environments: Boss and the Urban Challenge
Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M. N., Dolan, J., Duggins, D., Galatali, T., Geyer, C., et al. (2008) · Journal of Field Robotics · doi:10.1002/rob.20255 · Cited by 1,555
cuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943
PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
Xu, D., Anguelov, D., & Jain, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00033 · Cited by 856
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
Song, S., & Xiao, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.94 · Cited by 681
Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks
Engelcke, M., Rao, D., Wang, D. Z., Tong, C. H., & Posner, I. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989161 · Cited by 549
Object Classification Using CNN-Based Fusion of Vision and LIDAR in Autonomous Vehicle Environment
Gao, H., Cheng, B., Wang, J., Li, K., Zhao, J., & Li, D. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2822828 · Cited by 524
Sliding Shapes for 3D Object Detection in Depth Images
Song, S., & Xiao, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_41 · Cited by 359
Voting for Voting in Online Point Cloud Object Detection
Zeng Wang, D., & Posner, I. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.035 · Cited by 337
FusionNet: 3D Object Classification Using Multiple Data Representations
Hegde, V., & Zadeh, R. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05695 · Cited by 151
Object Detection and Classification by Decision-Level Fusion for Intelligent Vehicle Systems
Oh, S.-I., & Kang, H.-B. (2017) · Sensors · doi:10.3390/s17010207 · Cited by 120
Vehicle detection based on LiDAR and camera fusion
Zhang, F., Clarke, D., & Knoll, A. (2014) · 17th International IEEE Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2014.6957925 · Cited by 79
Berger, C., & Rumpe⋆, B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.0413 · Cited by 71
LiDAR/camera sensor fusion technology for pedestrian detection
Wu, T.-E., Tsai, C.-C., & Guo, J.-I. (2017) · 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2017.8282301 · Cited by 42
Approximation with neural networks activated by ramp sigmoids
Cheang, G. H. L. (2010) · Journal of Approximation Theory · doi:10.1016/j.jat.2010.03.004 · Cited by 40
A novel approach for fusing LIDAR and visual camera images in unstructured environment
Yohannan, B., & Chandy, D. A. (2017) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs.2017.8014604 · Cited by 4
(2009) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings
Research works printed but not linked: 3
- 2006, Stanley: The Robot That Won the Darpa Grand Challenge: Research Articles, J. Robot. Syst., 23, 661
- 2015, SSD: Single Shot Multibox Detector
- 2017, Fusing Bird Eye View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection
Cited by
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
“As an example, a multimodal fusion detection system for autonomous vehicles, that combines visual features from cameras along with data from Light Detection and Ranging (LiDAR) sensors, is able to achieve signi fi cantly higher accuracy (3.7% improvement) than a single-modal CNN detection model 21 .”
InfluentialFernandes, D., Silva, A., Névoa, R., Simões, C., Gonzalez, D., Guevara, M., Novais, P., Monteiro, J., & Melo-Pinto, P. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.11.002 · Cited by 268
Multimodal image fusion: A systematic review
Kalamkar, S. P., & Amalanathan, G. M. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100327 · Cited by 112
Rethinking Multi-Modal Object Detection From the Perspective of Mono-Modality Feature Learning
Zhao, T., Liu, B., Gao, Y., Sun, Y., Yuan, M., & Wei, X. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00600 · Cited by 61
Stability Regions of Vehicle Lateral Dynamics: Estimation and Analysis
Huang, Y., Liang, W. M., & Chen, Y. (2020) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.4048932 · Cited by 53
Metadata
- Authors
- Person, M., Jensen, M., Smith, A. O., & Gutierrez, H.
- Affiliations
- Florida Institute of Technology (US)
- Assistant Professor Department of Mechanical and Aerospace Engineering, Florida Institute of Technology, Melbourne, FL 32901 e-mail:
- Corresponding author
- Person, Michael
- Smith, Anthony O.
- Gutierrez, Hector
- Venue
- Journal of Dynamic Systems, Measurement, and Control
- Publisher
- ASME International
- Published
- 2019-05-08
- Volume
- 141
- Issue
- 7
- ISSN
- 0022-0434 (ISSN-L)
- 1528-9028
- OpenAlex
- W2923165220
- Semantic Scholar
- fcd7321732a15660b38c317f21c9a5df0e249e5e
- Cited by
- 39
- References
- 43
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 7 · 2023: 6 · 2022: 6 · 2021: 5 · 2020: 4
- FWCI
- 1.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Dynamic Systems Measurement and Control (publishedVersion)
- Funders
- Nvidia
- Florida Institute of Technology
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Keywords
- object detection, autonomous vehicles, multimodal sensor fusion, LiDAR point cloud, image detection, 3D object detection, real-time inference, embedded deployment, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.