The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Deep Multimodal Learning: A Survey on Recent Advances and Trends
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 301–400 of 1,001
Multi-modal lifelog data fusion for improved human activity recognition: A hybrid approach
Oh, Y., & Kim, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102464 · Cited by 25
Multimodal fusion with relational learning for molecular property prediction
Zhou, Z., Li, Y., Hong, P., & Xu, H. (2025) · Communications Chemistry · doi:10.1038/s42004-025-01586-z · Cited by 30
“…enable a multi-view analysis from diverse sources is essential for improving molecule analysis, we can apply the Multi-Modality Fusion (Lahat et al., 2015; Khaleghi et al., 2013; Poria et al., 2015; Ramachandram & Taylor, 2017; Pawłowski et al., 2023; Manzoor et al., 2023; Priessner et al., 2024).”
AMSA: Adaptive Multimodal Learning for Sentiment Analysis
Wang, J., Mou, L., Ма, Л., Huang, T., & Gao, W. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3572915 · Cited by 41
“The emergence of multimodal learning provides an opportunity for further development of afective computing [44].”
BackgroundCompiler Technologies in Deep Learning Co-Design: A Survey
Zhang, H., Xing, M., Wu, Y., & Zhao, C. (2023) · Intelligent Computing · doi:10.34133/icomputing.0040 · Cited by 28
A comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
DGImNet: A deep learning model for photovoltaic soiling loss estimation
Fang, M., Qian, W., Qian, T., Bao, Q., Zhang, H., & Qiu, X. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124335 · Cited by 22
GTP-4o: Modality-Prompted Heterogeneous Graph Learning for Omni-Modal Biomedical Representation
Li, C., Liu, X., Wang, C., Liu, Y., Yu, W., Shao, J., & Yuan, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73235-5_10 · Cited by 54
Sensor-based technologies for motion analysis in sports injuries: a scoping review
Arzehgar, A., Seyedhasani, S. N., Ahmadi, F. B., Baravati, F. B., Hesar, A. S., Kachooei, A. R., & Aalaei, S. (2025) · BMC Sports Science Medicine and Rehabilitation · doi:10.1186/s13102-025-01063-z · Cited by 41
Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
Lisondra, M., Benhabib, B., & Nejat, G. (2026) · Robotics · doi:10.3390/robotics15030055 · Cited by 44
Evaluating robotic-assisted surgery training videos with multi-task convolutional neural networks
Wang, Y., Dai, J. C., Morgan, T., Elsaied, M., Garbens, A., Qu, X., Steinberg, R. L., Gahan, J. C., & Larson, E. C. (2021) · Journal of Robotic Surgery · doi:10.1007/s11701-021-01316-2 · Cited by 23
“In neural networks, this information sharing is achieved using shared weight representations between tasks—it has been shown to increase accuracy and generalization in a number of applications [14].”
BackgroundMulti input–Multi output 3D CNN for dementia severity assessment with incomplete multimodal data
Gravina, M., Garcia-Pedrero, A. M., Gonzalo‐Martín, C., Sansone, C., & Soda, P. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102774 · Cited by 23
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansionPreprint
Douillard, A., Ramé, A., Couairon, G., & Cord, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11326 · Cited by 22
“The independent classifiers paradigm [65] coupled with the sigmoid activation and binary crossentropy loss exclude explicitly a late fusion [57] of the task embeddings resulting in more specialized classifiers.”
MethodRetaining Privileged Information for Multi-Task Learning
Tang, F., Xiao, C., Wang, F., Zhou, J., & Lehman, L.-W. H. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330907 · Cited by 25
“This is quite different from representation fusionmethods [22, 29], which only try to learn a shared representation space for input modalities X1, .”
BackgroundMethodMAVEN: A Memory Augmented Recurrent Approach for Multimodal Fusion
Islam, M. M., Yasar, M. S., & Iqbal, T. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3164261 · Cited by 22
“early, intermediate, and late fusion [21], [23], [25] (see Fig.”
BackgroundMonitoring Amyloidogenesis with a 3D Deep-Learning-Guided Biolaser Imaging Array
Chan, K. K., Shang, L., Qiao, Z., Liao, Y., Kim, M., & Chen, Y. (2022) · Nano Letters · doi:10.1021/acs.nanolett.2c03148 · Cited by 20
Spatiotemporal adaptive attention 3D multiobject tracking for autonomous driving
Zhang, X., Fan, Z., Tan, X., Liu, Q., & Shi, Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110442 · Cited by 22
The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings
Yilma, B. A., & Leiva, L. A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581477 · Cited by 25
Multimodal fusion for sensorimotor control in steering angle prediction
Munir, F., Azam, S., Yow, K., Lee, B., & Jeon, M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107087 · Cited by 27
Multimodal Vigilance Estimation With Modality-Pairwise Contrastive Loss
Zhang, M., Luo, Z., Xie, L., Liu, T., Ye, Y., Yao, D., Zhao, S., & Yin, E. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3328942 · Cited by 22
Peng, C., Chen, K., Shou, L., & Chen, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29374 · Cited by 45
Zeng, L., Fu, J., Sheng, C., Li, B., Guo, Z., Xiang, Q., Wang, J., Shan, X., Fu, X., Deng, Z., et al. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.140225 · Cited by 19
Multi-modal visual tracking based on textual generation
Wang, J., Liu, F., Jiao, L., Wang, H., Li, S., Liu, X., Chen, P., & Liu, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102531 · Cited by 21
Multimodal Approaches Based on Microbial Data for Accurate Postmortem Interval Estimation
Hu, S. C., Zhang, X., Yang, F., Nie, H., Lu, X., Guo, Y., & Zhao, X. (2024) · Microorganisms · doi:10.3390/microorganisms12112193 · Cited by 27
Multimodal Federated Learning in AIoT Systems: Existing Solutions, Applications, and Challenges
Anagnostopoulos, C., Γκίλλας, Α., Mavrokefalidis, C., Pikoulis, E., Piperigkos, N., & Lalos, A. S. (2024) · IEEE Access · doi:10.1109/access.2024.3508030 · Cited by 28
Dong, S., Xie, W., Yang, D., Li, Y., Zhang, J., Tian, J., & Lei, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3520318 · Cited by 26
“Multimodal fusion, according to its stages of implementation [18], [19], [20], can be categorized into three stages: pixel-level fusion, feature-level fusion, and decision-level fusion, as shown in Fig. model training.”
Wang, T., Chen, R., Fan, N., Zang, L., Yuan, S., Du, P., Wu, Q., Wang, A., Li, J., Kong, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/54676 · Cited by 22
“Furthermore, developers can set multiple data types, such as crucial details of patient’s history, physical examination, and imaging tests, as inputs to build a multimodal to improve the clinical value of LSS diagnosis and grading by AI approaches [48].”
Dai, W., Sun, X., Zhang, W., Wang, D., Tang, D., & Peng, Y. (2025) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2025.01.033 · Cited by 19
Multi-Modal Biometric Authentication: Leveraging Shared Layer Architectures for Enhanced Security
Vatchala, S., Yogesh, C. K., Govindarajan, Y., Raja, M., Ganesan, V. P. A., Arul, A. V., & Ramesh, D. (2025) · IEEE Access · doi:10.1109/access.2025.3534223 · Cited by 34
“Lastly, the integration of multimodal deep learning with remote-sensing imagery classification demonstrates how diverse data modalities enhance the identification and classification of geographical features [18].”
BackgroundTeles, A. S., de Moura, I. R., Silva, F., Roberts, A., & Stahl, D. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102981 · Cited by 27
An efficient model-level fusion approach for continuous affect recognition from audiovisual signals
Pei, E., Jiang, D., & Hichem Sahli, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.09.037 · Cited by 21
Haak, H. E., Gao, X., Maas, M., Waktola, S., Benson, S., Beets‐Tan, R. G. H., Beets, G. L., van Leerdam, M. E., & Melenhorst, J. (2021) · Surgical Endoscopy · doi:10.1007/s00464-021-08685-7 · Cited by 22
Zhang, A., Li, Q., Li, Z., & Li, J. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3266872 · Cited by 20
“Multimodal sensing data combines information from different sensors, and different modalities can compensate for each other to yield better performance [35].”
BackgroundHan, W., Chen, H., Kan, M., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.717 · Cited by 27
A Hybrid Deep Learning Emotion Classification System Using Multimodal Data
Kim, D.-H., Son, W.-H., Kwak, S.-S., Yun, T.-H., Park, J.-H., & Lee, J.-D. (2023) · Sensors · doi:10.3390/s23239333 · Cited by 23
“Recent studies [22,23] aim to use data from different sources to address practical issues and improve the performance of machine learning models.”
BackgroundA multimodal dynamical variational autoencoder for audiovisual speech representation learning
Sadok, S., Leglaive, S., Laurent Girin, B., Alameda-Pineda, X., & Séguier, R. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106120 · Cited by 26
A Payload Based Malicious HTTP Traffic Detection Method Using Transfer Semi-Supervised Learning
Chen, T., Chen, Y., Lv, M., He, G., Zhu, T., Wang, T., & Weng, Z. (2021) · Applied Sciences · doi:10.3390/app11167188 · Cited by 21
“Representation transfer attempts to transfer the knowledge of representing the same object from different domains [27].”
BackgroundChatzichristos, C., Kofidis, E., Paesschen, W. V., Lathauwer, L. D., Theodoridis, S., & Huffel, S. V. (2021) · Human Brain Mapping · doi:10.1002/hbm.25717 · Cited by 34
“Different types of fusion can be realized (Karahan et al., 2015; Lahat et al., 2015; Ramachandram & Taylor, 2017) but generally, the definition may differ with regard to the degree of generality and the specific research areas (Cocchi, 2019).”
BackgroundMethodInfluentialYang, H., Zhu, D., He, S., Xu, Z., Liu, Z., Zhang, W., & Cai, J. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.115896 · Cited by 24
Jeong, K., Oh, H., Lee, Y., Seo, H., Cho, G., Jeong, J., Park, G., Choi, J., Seo, Y.-D., Jeong, J.-H., et al. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3461775 · Cited by 20
“Furthermore, data fusion can enhance model predictive power by providing richer and more comprehensive data sets through AI models that handle complex data, such as deep multimodal techniques [155].”
Multimodal hate speech detection: a novel deep learning framework for multilingual text and images
Saddozai, F. K., Badri, S. K., Alghazzawi, D., Khattak, A. M., & Asghar, M. Z. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2801 · Cited by 21
“Different techniques are used for joining features from different modalities, such as concatenation, product rules, and pooling ( Ramachandram & Taylor, 2017 ).”
Multi-scale digital twins for personalized medicine
Vallée, A. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1753906 · Cited by 22
Sun, G., Liu, X., & Yu, X. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106422 · Cited by 23
Guo, K., Zhou, H., Tian, Y., Zhou, W., Ji, Y., & Li, X. (2022) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom48880.2022.9796879 · Cited by 18
Corbin, A., & Marques, O. (2023) · IEEE Access · doi:10.1109/access.2023.3289320 · Cited by 20
“They have adopted a multimodal approach that utilizes a late fusion technique [16] to integrate different image types and text-based metadata into a single solution.”
BackgroundMethodThe role of artificial intelligence in crop improvement
Negus, K. L., Li, X., Welch, S. M., & Yu, J. (2024) · Advances in agronomy · doi:10.1016/bs.agron.2023.11.001 · Cited by 26
Jeong, H., Yun, B., Na, S., Son, M., Chae, S. H., Kim, C., & Cho, K. H. (2024) · Journal of Membrane Science · doi:10.1016/j.memsci.2024.123105 · Cited by 18
Recent Advances in 3D Object Detection for Self-Driving Vehicles: A Survey
Fawole, O. A., & Rawat, D. B. (2024) · AI · doi:10.3390/ai5030061 · Cited by 30
“Utilizing this sensor fusion methodology is crucial in autonomous driving since it is important to maintain precision in various scenarios [68].”
MethodComputer-Aided Hepatocarcinoma Diagnosis Using Multimodal Deep Learning
Menegotto, A. B., Becker, C. D. L., & Cazella, S. C. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-24097-4_1 · Cited by 19
Shao-jun, Y., Wei, H., Zhang, L., & Qin, S. (2021) · Energies · doi:10.3390/en14154387 · Cited by 18
Enhancing Security in Multimodal Biometric Fusion: Analyzing Adversarial Attacks
Alghamdi, S. M., Jarraya, S. K., & Kateb, F. (2024) · IEEE Access · doi:10.1109/access.2024.3435527 · Cited by 17
Lian, W., Lindblad, J., Stark, C. R., Hirsch, J., & Sladoje, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109498 · Cited by 25
Xu, K., Xie, Q., Zhu, Y. P., Cao, W., & Ni, J. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.109924 · Cited by 22
Multimodal Neural Network Acceleration on a Hybrid CPU-FPGA Architecture: A Case Study
Ajili, M. T., & Hara–Azumi, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3144977 · Cited by 20
“, modalities) are called ‘‘multimodal NNs’’ and can yield better detection, classification, or prediction tasks than NNs dealing with a single modality–unimodal NNs [2].”
BackgroundSelf-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth
Jiang, Z., Taira, H., Miyashita, N., & Okutomi, M. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811842 · Cited by 23
Qiao, L., Cui, Y., Jia, Z., Xiao, K., & Su, H. (2022) · Applied Sciences · doi:10.3390/app12157838 · Cited by 19
Cross-Modal Information-Guided Network Using Contrastive Learning for Point Cloud Registration
Xie, Y., Zhu, J., Li, S., & Shi, P. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3331625 · Cited by 17
“Ontheotherhand,pointcloudsprovidecrucial informationonspatialandgeometricdetails,butonlycapture localandtexture-freefeatures.”
Yue, G., Zhuo, G., Zhou, T., Liu, W., Wang, T., & Jiang, Q. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3347556 · Cited by 23
Deep Multimodal Fusion of Data With Heterogeneous Dimensionality via Projective Networks
Morano, J., Aresta, G., Grechenig, C., Schmidt-Erfurth, U. M., & Bogunović, H. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3352970 · Cited by 21
“In ophthalmology, optical coherence tomography (OCT) [9], a non-invasive 3-dimensional (3D) imaging technique that allows obtaining cross-sectional images (B-Scans) of the retina, is often combined with 2D fundus imaging, such as color fundus photography (CFP), confocal scanning laser…”
BackgroundZhukov, A., Rivero, A., Benois‐Pineau, J., Zemmari, A., & Mosbah, M. (2024) · Sensors · doi:10.3390/s24041171 · Cited by 18
“However, Ramachandram et al. [39] argue that there is no conclusive evidence supporting the superiority of late fusion over early fusion.”
BackgroundAutomatic Fine-Grained BIM element classification using Multi-Modal deep learning (MMDL)
Liu, H., Gan, V. J. L., Cheng, J. C. P., & Zhou, S. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102458 · Cited by 17
Choi, J. H., & Lee, J. (2019) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3341162.3344871 · Cited by 17
“Along with the increased number of sensors, many modality fusion methods have been proposed to efficiently handle the characteristics of the multimodal data [16].”
Performance Optimization using Multimodal Modeling and Heterogeneous GNN
Dutta, A., Alcaraz, J., TehraniJamsaz, A., Cesar, E., Sikora, A., & Jannesari, A. (2023) · Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing · doi:10.1145/3588195.3592984 · Cited by 16
Deep learning for land use classification: a systematic review of HS-LiDAR imagery
Rehman, M. Z. U., Islam, S. M. S., Blake, D., Ulhaq, A., & Janjua, N. K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11265-z · Cited by 31
“In some situations, preprocessing techniques can be employed to extract a high-level representation of data, followed by the fusion of modalities at a single level before feeding into a feature extraction algorithm Ramachandram and Taylor ( 2017 ).”
Enhancing the Tracking of Seedling Growth Using RGB-Depth Fusion and Deep Learning
Garbouge, H., Rasti, P., & Rousseau, D. (2021) · Sensors · doi:10.3390/s21248425 · Cited by 16
“Different fusion strategies have been reviewed [28,29].”
Xu, H., & Huang, H. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02207-z · Cited by 17
Guo, Q., Liao, Y., Li, Z., & Liang, S. (2023) · Entropy · doi:10.3390/e25101421 · Cited by 19
“This is achieved by introducing a common layer at the center of the multimodal network, giving rise to what is known as intermediate multimodal fusion [37].”
BackgroundYang, J., Hao, Z., Xu, J., Wang, J., & Jiang, X. (2023) · Dental Materials · doi:10.1016/j.dental.2023.11.013 · Cited by 16
Aeroengine Blades Damage Detection and Measurement Based on Multimodality Fusion Learning
Wu, X., Wei, X., Xu, H., Hou, Y., Li, C., Yin, Y., He, W., & Zhou, L. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3394505 · Cited by 17
“Multimodal fusion technology (MFT) in deep learning is the process of processing different forms of data when the model is in the task of analysis and identification [32].”
BackgroundDeep learning-based geological map generation using geological routes
Li, C., Li, F., Liu, C., Tang, Z., Fu, S., Lin, M., Lv, X., Liu, S., & Liu, Y. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114214 · Cited by 19
Asish, S. M., Kulshreshth, A. K., Borst, C. W., & Sutradhar, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456207 · Cited by 22
“Ramachandram et al. [38] argue that there is no compelling evidence that late fusion is superior to early fusion.”
BackgroundDing, T., & Hirose, A. (2020) · IEEE Access · doi:10.1109/access.2020.3013940 · Cited by 17
“We focus on the fact that the separated channel states c m ( t ) have rotary locus in the complex plane and, then, predict its change in time for obtaining the future channel by using CVNNs.”
BackgroundMethodFei, X., Shen, L., Ying, S., Cai, Y., Zhang, Q., Kong, W., Zhou, W., & Shi, J. (2020) · Cognitive Computation · doi:10.1007/s12559-020-09761-1 · Cited by 16
Articles That Use Artificial Intelligence for Ultrasound: A Reader’s Guide
Kuang, M., Hu, H.-T., Li, W., Chen, S., & Lü, X. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.631813 · Cited by 17
“Augmentation can enrich data diversity, and it can simulate the common causes of image heterogeneity as observed under real-world conditions in ultrasound examinations (18, 19).”
BackgroundHartmann, M. C., Koblet, O., Baer, M. F., & Purves, R. S. (2022) · Journal of Outdoor Recreation and Tourism · doi:10.1016/j.jort.2021.100479 · Cited by 16
Ye, Y., Mao, J., Liu, L., Zhang, S., Shen, L., & Sun, M. (2019) · IEEE Access · doi:10.1109/access.2019.2961418 · Cited by 16
“This simplest fusion approach often cannot fully exploit the complementary nature of the multiple sources of information and may cause information redundancy [18].”
BackgroundOn Single Source Robustness in Deep Fusion ModelsPreprint
Kim, T., & Ghosh, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04691 · Cited by 30
“For extensive literature reviews, please refer to the recent survey papers about deep multi-modal learning methods in general [34] and for autonomous driving [9].”
BackgroundInvestigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning
Yu, Y., Lee, H. J., Kim, B. C., Kim, J. U., & Ro, Y. M. (2020) · arXiv · doi:10.48550/arxiv.2005.10987 · Cited by 15
“Many studies in computer vision took account of multimodal data fusion when researchers want to enhance the performance such as for medical image segmentation [4, 9], semantic segmentation [8, 15], and person re-identification [18, 14].”
Brain Foundation Models: A survey on advancements in neural signal processing and brain discovery
Zhou, X., Liu, C., Chen, Z., Wang, K., Ding, Y., Jia, Z., & Wen, Q. (2025) · IEEE Signal Processing Magazine · doi:10.1109/msp.2025.3592356 · Cited by 42
Wang, H., Wang, B., Li, M., Luo, P., Ma, H., & Ma, F. (2021) · Frontiers in Energy Research · doi:10.3389/fenrg.2021.746378 · Cited by 15
Pace-Adaptive and Noise-Resistant Contrastive Learning for Multimodal Feature Fusion
Guo, X., Kot, A. C., & Kong, A. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3252270 · Cited by 19
“Although combining high-dimensional multimodal signals is nat-ural and easy for humans, it is a challenge for current AI systems [1], [2], [3].”
BackgroundCho, H. S., Koo, W., & Kim, H. (2023) · IEEE Transactions on Semiconductor Manufacturing · doi:10.1109/tsm.2023.3283101 · Cited by 21
“Multimodal fusion strategies are generally divided into early, intermediate, and late fusion [31], [32], [33].”
BackgroundMulti-label movie genre classification based on multimodal fusion
Cai, Z.-H., Ding, H., Wu, J., Xi, Y., Wu, X., & Cui, X. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16121-2 · Cited by 16
“However, directly concatenating features in a shared layer easily aggra-vates overfitting, but applying some dimensionality reduction strategies can alleviate it and improve multimodal fusion performance [28].”
InfluentialCross-Modal Prototype Learning for Zero-Shot Handwriting Recognition
Ao, X., Zhang, X.-Y., Yang, H., Yin, F., & Liu, C. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00100 · Cited by 15
“One perspective on multi-modal learning is the fusion of multiple modalities [13] for better decision-making.”
BackgroundVariational multimodal machine translation with underlying semantic alignment
Liu, X., Zhao, J., Sun, S., Liu, H., & Yang, H. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.11.011 · Cited by 22
Multi-modality Cascaded Fusion Technology for Autonomous Driving
Kuang, H., Liu, X., Zhang, J., & Fang, Z. (2020) · International Conference on Robotics and Automation Sciences (ICRAS) · doi:10.1109/icras49812.2020.9135065 · Cited by 15
“Multi-modality sensors fusion can be divided into decision-level, feature-level and data-level according to the input data [20].”
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 26
Huo, T., Wu, W., Chen, X., Xue, M., Liu, P., Zhang, J., Xie, Y., Wang, H., Zhou, H., Yan, Z., et al. (2025) · Intelligent Oncology · doi:10.1016/j.intonc.2025.06.005 · Cited by 17
M3Sense: Affect-Agnostic Multitask Representation Learning Using Multimodal Wearable Sensors
Samyoun, S., Islam, M. M., Iqbal, T., & Stankovic, J. A. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3534600 · Cited by 26
“Multimodal representation learning approaches:Multimodal representation learning approaches have produced state-of-the-art results for various applications [29, 78, 81], such as human activity recognition [34, 35, 38, 44, 54], gesture recognition [38, 115], video classification [16, 30, 40], image captioning [57, 110]…”
BackgroundInfluentialHou, G., Jia, L., Zhang, Y., Wu, W. D., Zhao, L., Zhao, J., Wang, L., & Qiang, Y. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106140 · Cited by 18
Li, Y., Daho, M. E. H., Conze, P.-H., Zeghlache, R., Boité, H. L., Bonnin, S., Cosette, D., Magazzeni, S., Laÿ, B., Guilcher, A. L., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13172770 · Cited by 14
“In this section, we describe three fusion network structures commonly used in multimodal research: input fusion, feature fusion, and decision fusion [39].”
BackgroundFrom Collision to Verdict: Responsibility Attribution for Autonomous Driving Systems Testing
Zhou, J., Tang, S., Guo, Y. R., Li, Y., & Xue, Y. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00062 · Cited by 14
“Depending on the specific requirements of the task, fusion can transpire at various levels, such as early fusion (feature-level fusion) or late fusion (decision-level fusion) [20].”
BackgroundRizaldy, A., Afifi, A. J., Ghamisi, P., & Gloaguen, R. (2024) · Remote Sensing · doi:10.3390/rs16132336 · Cited by 18
Multi-modal data clustering using deep learning: A systematic review
Raya, S. P., Orabi, M., Afyouni, I., & Aghbari, Z. A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128348 · Cited by 16
Data–Model Fusion Methods and Applications Toward Smart Manufacturing and Digital Engineering
Tao, F., Li, Y., Wei, Y., Zhang, C., & Zuo, Y. N. (2025) · Engineering · doi:10.1016/j.eng.2024.12.034 · Cited by 22
Application of machine learning in LC-MS-based non-targeted analysis
Jin, Z.-L., Chen, L., Wang, Y., Shi, C.-T., Zhou, Y., & Xia, B. (2025) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2025.118243 · Cited by 20
A vector quantized masked autoencoder for audiovisual speech emotion recognition
Sadok, S., Leglaive, S., & Séguier, R. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104362 · Cited by 25
“In recent years, advances in artificial intelligence technology and hardware acceleration have enabled a shift towards multimodal processing in the field of emotion recognition [6, 7, 8].”
Yan, L., Song, H., Guo, Y., Ren, P., Zhou, W., Li, S., Yang, J., & Shen, X. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103163 · Cited by 15
Multimodal Urban Sound Tagging With Spatiotemporal Context
Bai, J., Chen, J., & Wang, M. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3160168 · Cited by 16
“In early fusion strategies, all modalities are used to train a single model, but this can be rather challenging due to information redundancy [31].”
BackgroundDyHealth: making neural networks dynamic for effective healthcare analytics
Zheng, K., Cai, S., Chua, H., Herschel, M., Zhang, M., & Ooi, B. C. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3554821.3554835 · Cited by 17
“A number of studies have investigated multimodality [8, 23, 51], and some of them focus on healthcare analytics [19, 49, 50, 67].”
BackgroundGuo, J., Kurup, U., & Shah, M. (2018) · arXiv · doi:10.48550/arxiv.1811.11277 · Cited by 14
“Data collected from various types of sensors is greatly appreciated in deep multimodal learning, which is believed to enhance the inference performance of deep neural networks [117].”
Background
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.