Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 401–500 of 1,002
Guo, J., Kurup, U., & Shah, M. (2018) · arXiv · doi:10.48550/arxiv.1811.11277 · Cited by 14
“Data collected from various types of sensors is greatly appreciated in deep multimodal learning, which is believed to enhance the inference performance of deep neural networks [117].”
BackgroundMultimodal Deep Learning for Android Malware Classification
Arrowsmith, J., Sušnjak, T., & Jang‐Jaccard, J. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7010023 · Cited by 18
“This benefits from imperfectly correlated errors of base models, which enhance complementarity without explicitly learning multimodal interactions [2].”
InfluentialLuo, X., Wang, A., Zhang, X., Huang, K., Wang, S., Chen, L., & Cui, Y. (2025) · Mathematics · doi:10.3390/math13213382 · Cited by 21
Xin, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22752-2 · Cited by 17
Reliable Deep Learning based Localization with CSI Fingerprints and Multiple Base Stations
Foliadis, A., García, M. H. C., Stirling-Gallacher, R. A., & Thomä, R. S. (2022) · International Conference on Communications · doi:10.1109/icc45855.2022.9838535 · Cited by 13
Zhang, Y., Deng, Y., Zhou, Z., Zhang, X., Jiao, P., & Zhao, Z. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.1021400 · Cited by 18
“Speci fi cally, it can model nonlinearity and cross-modality relationships, which has expanded its range of applications from Computer Vision (CV) to Natural Language Processing (NLP) to the biomedical fi eld (Ramachandram and Taylor, 2017; Bichindaritz et al., 2021).”
BackgroundKhairuddin, M. Z. F., Hasikin, K. B., Razak, N. A. A., Mohshim, S. A., & Ibrahim, S. S. (2023) · IEEE Access · doi:10.1109/access.2023.3304328 · Cited by 18
Pingi, S. T., Zhang, D., Bashar, M. A., & Nayak, R. (2023) · Data Science and Engineering · doi:10.1007/s41019-023-00232-9 · Cited by 19
“This approach operates under the assumption that cross-modal relationships exist at feature level or lower levels of abstraction, thereby ignoring learning of marginal representations [46].”
Teshima, T., Niitsuma, M., & Nishimura, H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123153 · Cited by 15
Tsinganos, N., Fouliras, P., Mavridis, I. K., & Gritzalis, D. A. (2024) · IEEE Access · doi:10.1109/access.2024.3359030 · Cited by 16
Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks
Yang, Y., Pan, H., Jiang, Q.-Y., Xu, Y., & Tang, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3547417 · Cited by 46
Wudarczyk, O. A., Kırtay, M., Kuhlen, A. K., Rahman, R. A., Haynes, J., Hafner, V. V., & Pischedda, D. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.630789 · Cited by 16
“In Robotics, this term indicates the type of sensory data associated with different aspects of the observed phenomenon, such as depth and color data recorded by sensors in an object-recognition experiment (see Ramachandram and Taylor, 2017).”
BackgroundDing, X., Han, T., Fang, Y., & Larson, E. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04095-x · Cited by 15
“Such kind of intermediate fusion using deep learning is also called deep multimodal learning [31].”
BackgroundMulti-Modal Learning for AU Detection Based on Multi-Head Fused Transformers
Zhang, X., & Yin, L. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667030 · Cited by 13
“a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities [29].”
BackgroundPan, J., Li, X., Cai, Z., Sun, B., & Cui, W. (2022) · Remote Sensing · doi:10.3390/rs14092046 · Cited by 13
A Feasible and Explainable Network Traffic Classifier Utilizing DistilBERT
Shin, C.-Y., Park, J.-T., Baek, U.-J., & Kim, M. (2023) · IEEE Access · doi:10.1109/access.2023.3293105 · Cited by 17
“Since there is little limit on the number of features that can be input into the DL model, numerous features have been entered into the DL model at once [14], [31].”
BackgroundConvolutional Neural Networks for Spectroscopic Redshift Estimation on Euclid Data
Stivaktakis, R., Tsagkatakis, G., Moraes, B., Abdalla, F. B., Starck, J., & Tsakalides, P. (2020) · UCL Discovery (University College London) · Cited by 13
Incorporating Depth Information into Few-Shot Semantic Segmentation
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412921 · Cited by 16
Bai, L., Chen, S., Gao, M., Abdelrahman, L., Alghamdi, M., & Abdel-Mottaleb, M. S. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9629607 · Cited by 13
“[12-14] In this paper, we take the patients’ age and gender information and the retinal image as the information in the two fields of text and image respectively.”
MethodBaruch, S. B., Rotman-Nativ, N., Baram, A., Greenspan, H. K., & Shaked, N. T. (2021) · Cells · doi:10.3390/cells10123353 · Cited by 13
“Indeed, ResNet-50 has been successfully used for many medical image classification tasks [32,33].”
MethodTransfer Learning for Depression Screening from Follow-Up Clinical Interview Questions
Flores, R., Tlachac, M., Toto, E., & Rundensteiner, E. A. (2022) · Advances in intelligent systems and computing · doi:10.1007/978-981-19-6153-3_3 · Cited by 15
The Advance of Deep Learning and Attention Mechanism
Chen, X. (2022) · International Conference on Electronics and Devices, Computational Science (ICEDCS) · doi:10.1109/icedcs57360.2022.00078 · Cited by 12
“Another method is to increase the quality instead of the quantity, such as multimodal learning [7, 8].”
MethodLearning to Predict Head Pose in Remotely-Rendered Virtual Reality
Illahi, G. K., Vaishnav, A., Kämäräinen, T., Siekkinen, M., & Francesco, M. D. (2023) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/3587819.3590972 · Cited by 22
“Indeed, sensor fusion is a mature and actively researched field that studies this problem [18] and there has been considerable work to use multimodal data for ML applications [49].”
BackgroundYilma, B. A., Kim, C. M., Cupchik, G. C., & Leiva, L. A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642636 · Cited by 24
“This approach combines information from multiple data sources, like images and associated textual descriptions to create a unified representation space [65].”
Farmanesh, A., Sanchis, R. G., & Ordieres-Meré, J. B. (2025) · Sensors · doi:10.3390/s25103048 · Cited by 17
“Multi-modal approaches that combine different data sources, such as visual inspection, sensor measurements, and process parameters, offer opportunities for more robust defect detection [114,115].”
Towards pedestrian head tracking: A benchmark dataset and a multi-source data fusion network
Sun, K., Wang, X., Liu, S., Zhao, Q., Huang, G., & Liu, C. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111265 · Cited by 21
Gong, Y., Lu, N., & Zhang, J. (2018) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.4779 · Cited by 12
Xu, X., Wang, Y., Jia, J., Chen, B., & Li, D. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10041 · Cited by 16
Multimodal Semantic Consistency-Based Fusion Architecture Search for Land Cover Classification
Li, X., Lei, L., Zhang, C., & Kuang, G. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3193273 · Cited by 13
Wu, X., Zhang, X., Feng, X., López, M. B., & Liu, L. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3220040 · Cited by 21
“More effective and efficient fusion methods are demanded for audio-visual kinship verification, such as multimodal regularization [80], and multimodal joint representation [81] learning the complementary semantics.”
MethodHand gesture recognition based on attentive feature fusion
Yu, B., Luo, Z., Wu, H., & Li, S. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5910 · Cited by 13
Wang, Q., Zhang, K., Zhao, H., Zhang, H., Zhang, L., Yan, X., Liu, P., Fan, L., Yang, Y., & Yao, J. (2022) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2022.110931 · Cited by 13
Multi-agent reinforcement learning based on local communication
Zhang, W., Ma, L., & Li, X. (2018) · Cluster Computing · doi:10.1007/s10586-018-2597-x · Cited by 12
“In order to acquire good adaptability to complex and uncertain environments, learning mechanisms are often employed in multi-agent systems [1].”
BackgroundData-driven geophysics: from dictionary learning to deep learningPreprint
Yu, S., & Ma, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06183 · Cited by 17
“D. Ramachandram and G. W. Taylor (2017).”
BackgroundInfluentialSituation Awareness Discrimination Based on Physiological Features for High-Stress Flight Tasks
Qian, C., Liu, S., Wanyan, X., Feng, C., Zhen, L., Sun, W., & Wang, Y. (2024) · Aerospace · doi:10.3390/aerospace11110897 · Cited by 11
A Multimodal Approach to Improve Performance Evaluation of Call Center Agent
Abdelrahman, A., Shaalan, K. F., Toral, S. L., & Hifny, Y. (2021) · Sensors · doi:10.3390/s21082720 · Cited by 12
Multimodal action recognition: a comprehensive survey on temporal modeling
Shabaninia, E., Nezamabadi–pour, H., & Shafizadegan, F. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17345-y · Cited by 36
Integration of Feature and Decision Fusion With Deep Learning Architectures for Video Classification
Kızıltepe, R. S., Gan, J. Q., & Escobar, J. J. (2024) · IEEE Access · doi:10.1109/access.2024.3360929 · Cited by 17
Úbeda, P. L., Martín‐Noguerol, T., Paulano‐Godino, F., & Luna, A. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108334 · Cited by 11
BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning
Zhao, J., Chen, H., Tian, Q., Chen, J., Yang, B., Zhang, Z., & Liu, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802152 · Cited by 18
“This challenge arises due to the heterogeneity and inconsistency of multimodal information, and the indirect reward feedback in RL training further amplifies its complexity [14], [15].”
Fahad, S. K. A., Zhengkui, D. W., Ng, P. C., Wong, N., Ng, A. B., & See, S. C. W. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-80334-5_19 · Cited by 18
Daly, K., & Olukoya, O. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107959 · Cited by 19
MDFormer: Transformer-Based Multimodal Fusion for Robust Chest Disease Diagnosis
Liu, X., Pan, F., Song, H., Cao, S., Li, C., & Li, T. (2025) · Electronics · doi:10.3390/electronics14101926 · Cited by 21
“In decision-level fusion, the key is to synthesize the decisions of unimodal models rather than directly merging the data or features [26].”
Encrypted Multitask Traffic Classification via Multimodal Deep Learning
Aceto, G., Ciuonzo, D., Montieri, A., Nascita, A., & Pescapè, A. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500316 · Cited by 13
End-to-End Mispronunciation Detection and Diagnosis Using Transfer Learning
Peng, L., Gao, Y., Bao, R., Li, Y., & Zhang, J. (2023) · Applied Sciences · doi:10.3390/app13116793 · Cited by 12
“Moreover, it is also related to multimodal learning [31], which involves training a model on data from multiple modalities to jointly learn representations that capture the relationships between the modalities.”
BackgroundAn Effective Automatic Image Annotation Model Via Attention Model and Data Equilibrium
Vatani, A., Ahvanooey, M. T., & Rahimi, M. (2018) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2018.090338 · Cited by 14
“Recently, many types of research [5], [6], [12], [31] have been conducted on the AIA that can be grouped into two different models [13]; generative models, such as [1], [7], and discrimination or conditional models such as [3], [12].”
BackgroundDeep Learning in Age-invariant Face Recognition: A Comparative Study
Sajid, M., Ali, N., Ratyal, N. I., Usman, M., Butt, F. M., Riaz, I., Mussadiq, U., Baig, M. J. A., Baig, S., & Salaria, U. A. (2020) · The Computer Journal · doi:10.1093/comjnl/bxaa134 · Cited by 12
COMPASS: Contrastive Multimodal Pretraining for Autonomous Systems
Ma, S., Vemprala, S. H., Wang, W., Gupta, J. K., Song, Y., McDufft, D., & Kapoor, A. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982241 · Cited by 15
“Numerous surveys on multimodal learning are also largely focused on vision, text and speech [31], [32], [33].”
BackgroundA Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical ApplicationsPreprint
Guarrasi, V., Aksu, F., Caruso, C. M., Feola, F. D., Rofena, A., Ruffini, F., & Soda, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4952813 · Cited by 21
Dang, M., Liu, B., Chen, Y., & Zhang, Z. (2026) · Psychoradiology · doi:10.1093/psyrad/kkag007 · Cited by 10
MIFNet: multimodal interactive fusion network for medication recommendation
Huo, J., Hong, Z., Chen, M., & Duan, Y. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-05908-1 · Cited by 11
Machine learning-based wood anatomy identification: towards anatomical feature recognition
He, X., Pelt, D. M., Gao, J., Gravendeel, B., Zhu, P., Chen, S., Qiu, J., & Lens, F. (2024) · IAWA Journal - KU Leuven/IAWA Journal · doi:10.1163/22941932-bja10157 · Cited by 17
Liu, Q., & Wang, X. (2024) · Remote Sensing · doi:10.3390/rs16132289 · Cited by 12
Designing deep neural networks for driver intention recognition
Vellenga, K., Steinhauer, H. J., Karlsson, A., Falkman, G., Rhodin, A., & Koppisetty, A. C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109574 · Cited by 11
“Typically fusion strategies for DNNs are categorized into: early, intermediate, or late fusion (31) (see figure 1 for a schematic overview).”
Rudroff, T., Klén, R., Rainio, O., & Tuulari, J. J. (2024) · Brain Sciences · doi:10.3390/brainsci14121209 · Cited by 12
“Flexibility: AI methods can often be fine-tuned for specific tasks, allowing researchers to adapt the dimension reduction process to the particular needs of their Long COVID studies [49].”
A Quantum Multimodal Neural Network Model for Sentiment Analysis on Quantum Circuits
Zheng, J., Gao, Q. Z., Dong, D., Lü, J., & Deng, Y. C. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3511514 · Cited by 21
“This diversification is primarily manifested by the emergence of a large amount of multimodal data, encompassing various forms such as images, text, audio, and video [24], [25].”
LININ: Logic Integrated Neural Inference Network for Explanatory Visual Question Answering
Xue, D., Qian, S., Fang, Q., & Xu, C. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521709 · Cited by 21
Resource-Efficient Federated Multimodal Learning via Layer-Wise and Progressive Training
Tun, Y. L., Thwal, C. M., Nguyen, M. N. H., & Zhang, C. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3554541 · Cited by 12
“On the other hand, supervised multimodal learning aims to fuse information from various modalities to enhance the predictive performance of the model [7], [8], [9].”
Gao, F., & Tang, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-06229-w · Cited by 15
“Multimodal learning fundamentally aims to leverage complementary information from diverse data sources to enhance model performance beyond what individual modalities can achieve independently 45 .”
RetractedMultimodal Alzheimer’s disease recognition from image, text and audio
Lee, B., Song, H. J., Park, Y. J., & Kang, B. (2025) · Scientific Reports · doi:10.1038/s41598-025-14998-7 · Cited by 16
“Multimodal fusion is categorized into early, intermediate, and late fusion 22–24 .”
Together Yet Apart: Multimodal Representation Learning for Personalised Visual Art Recommendation
Yilma, B. A., & Leiva, L. A. (2023) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3565472.3592964 · Cited by 16
Senadheera, I., Larssen, B. C., Mak-Yuen, Y. Y. K., Steinfort, S., Carey, L. M., & Alahakoon, D. (2023) · Brain Sciences · doi:10.3390/brainsci13091253 · Cited by 13
“For now, such analyses are opening the door to new insights and new ways of meaningfully capturing the complexity of functions such as somatosensation and how they might be differentially impacted by brain injury such as stroke.”
IMPRINT: Interactional Dynamics-aware Motion Prediction in Teams using Multimodal Context
Yasar, M. S., Islam, M. M., & Iqbal, T. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3626954 · Cited by 18
“These approaches fuse multimodal information in three ways: early, late, and intermediate fusion [55, 73, 101, 102].”
HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information
Medeiros, H. R., Peña, F. A. G., Aminbeidokhti, M., Dubail, T., Granger, É., & Pedersoli, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00147 · Cited by 17
“By providing high-resolution data, these sensors offer additional observations of common environmental phenomena to aid in the effectiveness of DL algorithms [26].”
BackgroundConvolutional Neural Networks for Spectroscopic Redshift Estimation on Euclid Data
Stivaktakis, R., Tsagkatakis, G., Moraes, B., Abdalla, F. B., Starck, J., & Tsakalides, P. (2019) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2934475 · Cited by 25
Unveiling MIMETIC: Interpreting Deep Learning Traffic Classifiers via XAI Techniques
Nascita, A., Montieri, A., Aceto, G., Ciuonzo, D., Persico, V., & Pescapè, A. (2021) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr51186.2021.9527948 · Cited by 10
“M IMETIC is a multimodal DL traffic classifier that exploits multiple modalities (viz. views) of traffic data, to capitalize their heterogeneous nature via intermediate fusion [16] of (automatically extracted) features.”
BackgroundMultistage Fusion With Dissimilarity Regularization for SAR/IR Target Recognition
Cho, Y., Shin, S., Yim, S.-H., Kong, K., Cho, H.-W., & Song, W.-J. (2018) · IEEE Access · doi:10.1109/access.2018.2885736 · Cited by 10
Deep Multi-biometric Fusion for Audio-Visual User Re-Identification and Verification
Marras, M., Marín-Reyes, P. A., Lorenzo-Navarro, J., Castrillón-Santana, M., & Fenu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-40014-9_7 · Cited by 14
Multimodal Deep Learning for Crop Yield Prediction
Jácome-Galarza, L.-R. (2022) · Communications in Computer and Information Science · doi:10.1007/978-3-031-18347-8_9 · Cited by 9
Li, Z., Li, H., Ralescu, A. L., Dillman, J. R., Altaye, M., Cecil, K. M., Parikh, N. A., & He, L. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102993 · Cited by 10
“Multimodal learning aims to construct artificial intelligence (AI) models that can analyze and integrate relevant features extracted from diverse data modalities, with the goal of performing various tasks such as classification and regression [7].”
BackgroundMAFN: multi-level attention fusion network for multimodal named entity recognition
Zhou, X., Zhang, Y., Wang, Z., Lu, M., & Liu, X. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17376-5 · Cited by 10
“Atpresent,thestudyofdeepmultimodallearningmainlyincludesfiveissues[21]:thereare multimodal representation, multimodal translation, multimodal fusion and modal alignment, and collaborative learning.”
Humbert‐Vidan, L., Patel, V., King, A. P., & Urbano, T. G. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad8290 · Cited by 10
Qin, J., Liu, F., & Zong, L. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107449 · Cited by 13
Shukla, V., Kaarthika, Mathur, A., Narayan, P. B., & Kishor, K. (2025) · IEEE Access · doi:10.1109/access.2025.3575126 · Cited by 12
Li, X., Huang, M., Xu, Y., Cao, Y., Lu, Y., Wang, P., & Xiang, X. (2022) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-022-00108-3 · Cited by 11
“Deep architectures offer the flexibility of implementing multimodal fusion either as early, intermediate, or late fusion (Ramachandram and Taylor 2017).”
BackgroundSyntactic Multi-view Learning for Open Information Extraction
Dong, K., Sun, A., Kim, J., & Li, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.272 · Cited by 22
Powerset Fusion Network for Target Classification in Unattended Ground Sensors
Wang, Y., Cheng, X., Xiang, L., Li, B., & Yuan, X. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3067648 · Cited by 10
“A common strategy is to concatenate synchronized frames before they are input to a network or, alternatively, to train separate modality-wise networks, and then fuse at a deeper stage by using a shared representation layer [12].”
BackgroundHuang, J., He, R., Chen, J., Li, S., Deng, Y., & Wu, X. (2021) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-021-00026-9 · Cited by 11
“In recently years, [19, 20] review advances in deep multimodal learning, as well as the application of stacking technique in this research field.”
BackgroundCMC2R: Cross‐modal collaborative contextual representation for RGBT tracking
Liu, X., Luo, Y., Yan, K., Chen, J., & Lei, Z. (2022) · IET Image Processing · doi:10.1049/ipr2.12427 · Cited by 10
An Adaptive Framework for Anomaly Detection in Time-Series Audio-Visual Data
Kumari, P., & Saini, M. K. (2022) · IEEE Access · doi:10.1109/access.2022.3164439 · Cited by 11
“A simplified assumption of conditional independence between the modalities is considered in early fusion-based models [41], which might not hold true in practice.”
BackgroundChen, S. (2019) · IEEE Multimedia · doi:10.1109/mmul.2019.2897471 · Cited by 9
“Interpreting features generated from medical imaging is a major challenge, considering it requires the validation from highly trained human experts.(8,9) In summary, despite the early achievements of utilizing DL in multimedia applications, there still exist several critical challenges that require more attention in f…”
BackgroundMultiple Model BLE-based Tracking via Validation of RSSI Fluctuations under Different Conditions
Atashi, M., Salimibeni, M., Malekzadeh, P., Barbulescu, M., Plataniotis, K. N., & Mohammadi, A. (2019) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion43075.2019.9011367 · Cited by 9
“Alternatively, some solutions are developed based on machine learning schemes such as Artificial Neural Networks (ANNs) [14], [15].”
BackgroundMass Personalization of Deep LearningPreprint
Schneider, J., & Vlachos, M. (2019) · arXiv (Cornell University) · Cited by 12
Algorithmic Behavioral Science: Machine Learning as a Tool for Scientific DiscoveryPreprint
Ludwig, J., & Mullainathan, S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4164272 · Cited by 17
Multimodal Approach for Harmonized System Code Prediction
Amel, O., Stassin, S., Mahmoudi, S. A., & Siebert, X. (2023) · European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) · doi:10.14428/esann/2023.es2023-165 · Cited by 18
“Next, we merge the modalities with different early fusion methods such as a simple concatenation [11] (Concat) of each modality representation M i or a multimodal low-rank tensor fusion (LMF) [12].”
MethodLim, M., Heo, K.-S., Kim, J.-M., Kang, B., Lin, W., Zhang, H., Shen, D. G., & Kam, T. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3355966 · Cited by 8
A Multimodal Deep Learning Based Approach for Alzheimer’s Disease Diagnosis
Simone, A. D., & Sansone, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-51026-7_12 · Cited by 10
Liu, X., Dong, R., & Yang, H. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3453919 · Cited by 8
Sun, W., Geng, X., Thompson, D., Wang, T., Zhou, J., & Zhang, J. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112587 · Cited by 8
“The process of combining multiple features for deep learning tasks is called feature fusion, also known as multi-modal fusion technology (MFT) [48].”
MMER-LMF: Multi-Modal Emotion Recognition in Lightweight Modality Fusion
Kim, E. H., MyungJin, L., & Shin, J. (2025) · Electronics · doi:10.3390/electronics14112139 · Cited by 8
Bai, L., Zhang, X., Qin, W., Long, J., Wang, H., Dong, X., & Du, S. (2025) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2025.3588505 · Cited by 10
“Specifically, an effective multimodal objective needs to consider the distinct nature and unique properties of each modality as well as harness the intrinsic relationships and interdependencies between them to propel the model training [143], [144].”
Zhang, X., Huang, S. J., & Gao, B. (2026) · Engineering Geology · doi:10.1016/j.enggeo.2025.108520 · Cited by 9
Xue, C., Yu, M. Z., Yan, G., Qin, M., Liu, Y., & Jia, J. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-211697 · Cited by 8
RetractedMultiplayer Tension In the Wild: A Hearthstone Case
Blom, P. M., Melhárt, D., Liapis, A., Yannakakis, G. N., Bakkes, S. C. J., & Spronck, P. H. M. (2023) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3582437.3582440 · Cited by 8
“Through the use of deep learning and multimodal fusion [37] it is possible to capture the patterns both in the pixels of game play [24] and the pixels of the player’s webcam feed and thus better predict emotion.”
BackgroundChen, Z.-Y., Fan, Z., & Sun, M. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3267858 · Cited by 10
“…intricate structures in high-dimensional data [15], [16] and in learning from heterogeneous data through redesigning network architectures [16], [17], and have been successfully applied to many practical problems, e.g., image recognition, medical diagnosis, and vision-language multimodal fusion…”
BackgroundMethodInfluentialDeepGraviLens: a multi-modal architecture for classifying gravitational lensing data
Vago, N. O. P., & Fraternali, P. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08766-9 · Cited by 11
Ramazanova, M., Escorcia, V. A., Heilbron, F. C., Zhao, C., & Ghanem, B. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00516 · Cited by 22
“Deep learning facilitates audiovisual learning as it enables learning per-modality hierarchical representations [39], which are more optimal than designing hand-crafted features.”
BackgroundAwotunde, J. B., Imoize, A. L., Adeniyi, A. E., Abiodun, M. K., Ayo, F. E., Kavitha, K. V. N., Ajamu, G. J., & Ogundokun, R. O. (2023) · Springer eBooks · doi:10.1007/978-3-031-38036-5_2 · Cited by 13
Deep Multimodal Fusion with Corrupted Spatio-Temporal Data Using Fuzzy Regularization
Altinses, D., & Schwung, A. (2023) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon51785.2023.10312522 · Cited by 17
“The process of fusing the transformed raw input to the higher-level representation of various modalities is called intermediate fusion [9].”
BackgroundWang, G., Teng, H., Qiao, L., Yu, H., Cui, Y., & Xiao, K. (2024) · Energies · doi:10.3390/en17112710 · Cited by 8
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.