Multimodal Fusion Object Detection System for Autonomous Vehicles
The DOI matches its Crossref record; 41 of the 43 references it cites were found in a registry or an index.
Abstract
In order for autonomous vehicles to safely navigate the road ways, accurate object detection must take place before safe path planning can occur. Currently, general purpose object detection convolutional neural network (CNN) models have the highest detection accuracies of any method. However, there is a gap in the proposed detection frameworks. Specifically, those that provide high detection accuracy necessary for deployment but do not perform inference in realtime, and those that perform inference in realtime but detection accuracy is low. We propose multimodel fusion detection system (MFDS), a sensor fusion system that combines the speed of a fast image detection CNN model along with the accuracy of light detection and range (LiDAR) point cloud data through a decision tree approach. The primary objective is to bridge the tradeoff between performance and accuracy. The motivation for MFDS is to reduce the computational complexity associated with using a CNN model to extract features from an image. To improve efficiency, MFDS extracts complimentary features from the LiDAR point cloud in order to obtain comparable detection accuracy. MFDS is novel by not only using the image detections to aid three-dimensional (3D) LiDAR detection but also using the LiDAR data to jointly bolster the image detections and provide 3D detections. MFDS achieves 3.7% higher accuracy than the base CNN detection model and is able to operate at 10 Hz. Additionally, the memory requirement for MFDS is small enough to fit on the Nvidia Tx1 when deployed on an embedded device.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“a multimodal fusion detection system for autonomous vehicles, that combines visual features from cameras along with data from Light Detection and Ranging (LiDAR) sensors, is able to achieve significantly higher accuracy (3.7% improvement) than a single-modal CNN detection model21.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [21]
In the citing paper
“Multimodal deep learning models that can ingest pixel data along with other data types (fusion) have been successful in applications outside of medicine, such as autonomous driving and video classification.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [21]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 3
- 2006, Stanley: The Robot That Won the Darpa Grand Challenge: Research Articles, J. Robot. Syst., 23, 661
- 2015, SSD: Single Shot Multibox Detector
- 2017, Fusing Bird Eye View LIDAR Point Cloud and Front View Camera Image for Deep Object Detection
Cited by
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
“As an example, a multimodal fusion detection system for autonomous vehicles, that combines visual features from cameras along with data from Light Detection and Ranging (LiDAR) sensors, is able to achieve signi fi cantly higher accuracy (3.7% improvement) than a single-modal CNN detection model 21 .”
InfluentialFernandes, D., Silva, A., Névoa, R., Simões, C., Gonzalez, D., Guevara, M., Novais, P., Monteiro, J., & Melo-Pinto, P. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.11.002 · Cited by 268
Multimodal image fusion: A systematic review
Kalamkar, S. P., & Amalanathan, G. M. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100327 · Cited by 112
Rethinking Multi-Modal Object Detection From the Perspective of Mono-Modality Feature Learning
Zhao, T., Liu, B., Gao, Y., Sun, Y., Yuan, M., & Wei, X. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00600 · Cited by 61
Stability Regions of Vehicle Lateral Dynamics: Estimation and Analysis
Huang, Y., Liang, W. M., & Chen, Y. (2020) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.4048932 · Cited by 53
Survey on Image and Point-Cloud Fusion-Based Object Detection in Autonomous Vehicles
Peng, Y., Qin, Y., Tang, X., Zhang, Z., & Deng, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3206235 · Cited by 45
Yin, M., Lin, J., Wang, Y., Liu, Y., Zhang, R., Duan, W., Zhou, Z., Zhu, S., Gao, J., Liu, L., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105341 · Cited by 44
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 37
Tang, P., Yan, X., Yang, N., Hu, X., Menze, B. H., Krammer, S., & Lasser, T. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110604 · Cited by 36
Hamed, S. K., Aziz, M. J. A., & Yaakub, M. R. (2024) · IEEE Access · doi:10.1109/access.2024.3443092 · Cited by 35
Autonomous Vehicle Driving in Harsh Weather: Adaptive Fusion Alignment Modeling and Analysis
Hasanujjaman, M., Chowdhury, M. Z., Hossan, M. T., & Jang, Y. M. (2023) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-023-08389-1 · Cited by 18
Survey and Evaluation of Neural 3D Shape Classification Approaches
Mirbauer, M., Krabec, M., Křivánek, J., & Šikudová, E. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3102676 · Cited by 18
Object Detection for Connected and Autonomous Vehicles using CNN with Attention Mechanism
Gupta, A., Illanko, K., & Fernando, X. N. (2022) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2022-spring54318.2022.9860676 · Cited by 17
Hamed, S. K., Aziz, M. J. A., & Yaakub, M. R. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140742 · Cited by 15
Multimodal Data Fusion Integrating Text and Medical Imaging Data in Electronic Health Records
Rele, M., Julian, A., Patil, D., & Udaya Krishnan, M. (2024) · Information systems engineering and management · doi:10.1007/978-3-031-69197-3_27 · Cited by 15
Multi-modal Medical Data Fusion using Deep Learning
Haritha, D., & Sandhya, B. (2022) · International Conference on Computing for Sustainable Global Development · doi:10.23919/indiacom54597.2022.9763296 · Cited by 14
Deep Learning, Machine Learning -- Digital Signal and Image Processing: From Theory to ApplicationPreprint
Hsieh, W., Bi, Z., Liu, J., Peng, B., Zhang, S., Pan, X., Xu, J., Wang, J., Chen, K., Yin, C. H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20304 · Cited by 14
Single-sample image-fusion upsampling of fluorescence lifetime images
Kapitány, V., Fatima, A., Žičkus, V., Whitelaw, J., McGhee, E. J., Insall, R. H., Machesky, L. M., & Faccio, D. (2024) · Science Advances · doi:10.1126/sciadv.adn0139 · Cited by 12
Systematic Literature Review: An Intelligent Pulmonary TB Detection from Chest X-Rays
Jimmy, Cenggoro, T. W., & Pardamean, B. (2021) · doi:10.1109/iccsai53272.2021.9609717 · Cited by 10
Wan, L., An, Q., & Su, H. (2022) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2022.09.045 · Cited by 9
Lamichhane, B. R., Paudel, B., Paudel, S., Srijuntongsiri, G., & Horanont, T. (2025) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2025.101571 · Cited by 7
CNN based sensor fusion method for real-time autonomous robotics systems
Yıldız, B., Durdu, A., Kayabaşı, A., & DURAMAZ, M. (2022) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.3906/elk-2008-147 · Cited by 7
On Using a Low-Density Flash Lidar for Road Vehicle Tracking
R., V. K. A., Subramanian, S. C., & Rajamani, R. (2021) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.4050255 · Cited by 6
Prospective study on challenges faced in a perception system
Rolen, L. R., Bhat, S. J., & Santhosh, K. V. (2024) · Cogent Engineering · doi:10.1080/23311916.2024.2353498 · Cited by 5
Adversarial and Cooperation Tasks in Multi- Agent System with Large Language Models
Jia, S., Cui, Z., Wang, K., Ding, Y. J., Han, D., & Ma, T. (2024) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus61736.2024.10839948 · Cited by 5
Representation Space Constrained Learning with Modality Decoupling for Multimodal Object DetectionPreprint
Shao, Y., & Shi, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.15433 · Cited by 5
3D Object Detection for Point Cloud in Virtual Driving Environment
Xu, B., Yu, R., & Zhao, M. (2022) · IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA) · doi:10.1109/ispce-asia57917.2022.9970914 · Cited by 4
Meta-learning for Clinical and Imaging Data Fusion for Improved Deep Learning Inference
Vasilevski, K. (2023) · eScholarship@McGill (McGill) · doi:10.82308/15966 · Cited by 3
Li, H., Tan, X., Zhang, X., & Han, Y. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3626349 · Cited by 3
Multimodal image fusion for enhanced vehicle identification in intelligent transport
Mudawi, N. A., Ahmed, M. W., Alhasson, H. F., Alshassari, N. S., Alazeb, A., Alshehri, M., & Alabdullah, B. I. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3270 · Cited by 2
Nguyen, P., Cao, D., Kha, Q. H., Chu, H., Minh, L. H. N., Pham, Q. T. T., & Le, N. Q. K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.27343 · Cited by 2
Object Detection for Autonomous Vehicles Using Deep Learning Algorithm
Pavan, E. J. S., Ramya, P., Valarmathi, B., Chellatamilan, T., & Santhi, K. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-33-6862-0_28 · Cited by 2
AN OVERVIEW OF MULTIMODAL LEARNING: CONCEPTS, CHALLENGES, APPLICATIONS AND DATASETS
Ateş, A. H., & Yurtsever, U. (2026) · Konya Journal of Engineering Sciences · doi:10.36306/konjes.1716813 · Cited by 1
PACGNet: Pyramidal Adaptive Cross-Gating for Multimodal DetectionPreprint
Gu, Z., & Tian, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5956897 · Cited by 1
PACGNet: Pyramidal Adaptive Cross-Gating Network for Multimodal Object Detection in Aerial ImageryPreprint
Ning, X., & Gu, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.18291 · Cited by 1
Zhao, Y., & Wang, L. (2020) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2020-01-0484 · Cited by 1
3D Object Recognition of Unmanned Surface Vessel Based on Grid-Convolutional Neural Network
Wan, L., Miao, S., & Su, H. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-6613-2_78 · Cited by 1
A Bibliometric View of the Research Development in Autonomous Driving from 2018 to 2021Preprint
Schauer, C. A., Schiebel, E., & Schlögl, C. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4102784
Research on multimodal dense small object detection algorithms guided by cross-modal information
Hu, J., Ma, Y., Xing, Y., Zi, Y., Deng, Y., Wang, M., Liu, H., & Du, J. (2026) · PLoS ONE · doi:10.1371/journal.pone.0348533
Yu, Y., Zhang, Z., Song, Z., & Ou, X. (2026) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2026.104936
Griffith, T. D., Hubbard, J. E., & Balas, M. J. (2023) · Synthesis lectures on biomedical engineering · doi:10.1007/978-3-031-23529-0_1
SFJOF: Mitigating Modal Parasitism in UAV Multispectral Detection via Bidirectional Feature Guidance
Li, X., Yu, Y., Wang, K., & Li, Z. (2026) · Springer proceedings in physics · doi:10.1007/978-981-92-4349-5_45
Auf Deep Learning basierende multimodale Fusionsmethode zur Klassifizierung von Hautläsionen
Tang, P., Lasser, T., & München, T. U. (2024)
Hybrid Multimodal Fusion Network (HMFNet) for Enhanced Perception in Autonomous Vehicles
Mahalakshmi, Ranjini, K. S., Vaishnaw, N. S., & Joseph, J. (2026) · Wiley eBooks · doi:10.1002/9781394311729.ch8
Adaptive Multimodal Fusion for Robust Autonomous Driving Perception with Attention‐Based Learning
Sangeetha, R. (2026) · Wiley eBooks · doi:10.1002/9781394311729.ch4
Dynamically Reconfigurable Multisensor Fusion for Enhanced Object Detection in Autonomous Vehicles
Muthukumaran, V., Kumar, M. E. B., Kumaran, G., Vidya, K., & Alkhayyat, A. (2026) · Wiley eBooks · doi:10.1002/9781394311729.ch10
Błaszczak, M., Knapińska, A., Jabłoński, I., & Walkowiak, K. M. (2026) · IFIP Networking Conference (IFIP Networking) · doi:10.23919/ifipnetworking70592.2026.11579167
Feature-Level Multimodal Fusion for Integrated Vehicle Condition and Performance Prediction
Napoleon, R., & Kavin, B. P. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11689032
Metadata
- Authors
- Person, M., Jensen, M., Smith, A. O., & Gutierrez, H.
- Affiliations
- Florida Institute of Technology (US)
- Assistant Professor Department of Mechanical and Aerospace Engineering, Florida Institute of Technology, Melbourne, FL 32901 e-mail:
- Corresponding author
- Person, Michael
- Smith, Anthony O.
- Gutierrez, Hector
- Venue
- Journal of Dynamic Systems, Measurement, and Control
- Publisher
- ASME International
- Published
- 2019-05-08
- Volume
- 141
- Issue
- 7
- ISSN
- 0022-0434 (ISSN-L)
- 1528-9028
- OpenAlex
- W2923165220
- Semantic Scholar
- fcd7321732a15660b38c317f21c9a5df0e249e5e
- Cited by
- 39
- References
- 43
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 7 · 2023: 6 · 2022: 6 · 2021: 5 · 2020: 4
- FWCI
- 1.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Dynamic Systems Measurement and Control (publishedVersion)
- Funders
- Nvidia
- Florida Institute of Technology
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Keywords
- object detection, autonomous vehicles, multimodal sensor fusion, LiDAR point cloud, image detection, 3D object detection, real-time inference, embedded deployment, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.