Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 101–200 of 1,002
Zhang, Y., He, K., Guo, Y., Liu, X., Yang, Q., Zhang, C., Xie, Y., Mu, S., Guo, Y., Fu, Y., et al. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.00457 · Cited by 77
“Although there have been advances in new medical imaging technologies, highly trained experts are still required to interpret these modalities for diagnosis (12).”
BackgroundA multimodal deep learning-based drug repurposing approach for treatment of COVID-19
Hooshmand, S. A., Zarei Ghobadi, M., Hooshmand, S. E., Azimzadeh Jamalkandi, S., Alavi, S. M., & Masoudi-Nejad, A. (2021) · Molecular Diversity · doi:10.1007/s11030-020-10144-9 · Cited by 70
Holobar, A., & Farina, D. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3057051 · Cited by 70
M2FN: An end-to-end multi-task and multi-sensor fusion network for intelligent fault diagnosis
Cui, J., Xie, P., Wang, X., Wang, J., He, Q., & Jiang, G. (2022) · Measurement · doi:10.1016/j.measurement.2022.112085 · Cited by 76
Network intrusion detection using feature fusion with deep learning
Ayantayo, A., Kaur, A., Kour, A., Schmoor, X., Shah, F., Vickers, I., Kearney, P., & Abdelsamea, M. M. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00834-0 · Cited by 118
“Ramachandram et al. [4] explore various deep multimodal learning architectures capable of integrating features from distinct modalities or signals from diverse sensors.”
BackgroundA survey of identity recognition via data fusion and feature learning
Qin, Z., Zhao, P., Zhuang, T., Deng, F., Ding, Y., & Chen, D. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.032 · Cited by 86
Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Electronics · doi:10.3390/electronics12071558 · Cited by 85
“Despite the fact that Multimodal Machine Learning is a popular and young research area that has received much attention, it is still in its infancy [4–6 , 45 ].”
BackgroundMethodInfluentialHarmony: Heterogeneous Multi-Modal Federated Learning through Disentangled Model Training
Ouyang, X., Xie, Z., Fu, H., Cheng, S., Pan, L., Ling, N., Xing, G., Zhou, J., & Huang, J. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596844 · Cited by 109
“Incorporating information from multiple complementary data modalities provides improved model performance [46].”
MethodSentiment analysis of social images via hierarchical deep fusion of content and links
Xu, J. J., Huang, F., Zhang, X., Wang, S., Li, C., Li, Z., & He, Y. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.04.010 · Cited by 76
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 81
Chen, Q., Li, M., Chen, C., Zhou, P., Lv, X., & Chen, C. (2022) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-022-04180-1 · Cited by 76
“According to different data fusion strategies, data fusion is divided into three different fusion methods: feature-fusion, decision-fusion, and hybrid-fusion (Ramachandram and Taylor 2017).”
BackgroundMultimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 128
Liu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66
Deep learning for mining protein data
Shi, Q., Chen, W., Huang, S., Wang, Y., & Xue, Z. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbz156 · Cited by 66
Prediction of mechanical properties of composite materials using multimodal fusion learning
Song, L., Wang, D., Liu, X., Yin, A., & Long, Z. (2023) · Sensors and Actuators A Physical · doi:10.1016/j.sna.2023.114433 · Cited by 72
Artificial intelligence in thyroid ultrasound
Cao, C.-L., Li, Q., Jin, T., Shi, L., Li, W., Ya, X., Cheng, J., Du, T., Li, J., & Cui, X. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1060702 · Cited by 84
“Additionally, most current studies do not consider various clinical data, such as medical history, clinical inspection results, and other relevant information (121).”
BackgroundBilotti, U., Bisogni, C., Marsico, M. D., & Tramonte, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107708 · Cited by 72
Deep Multimodal Learning: Merging Sensory Data for Massive MIMO Channel Prediction
Yang, Y., Gao, F., Xing, C., An, J., & Alkhateeb, A. (2020) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3041383 · Cited by 72
“As opposed to [30] and [31] that mainly study DML in computer vision, speech, and natural language processing areas, this is the first work that explores how DML technologies can be applied to wireless communications to the best of the authors’ knowledge.”
BackgroundMethodInfluentialMultimedia Recommender Systems: Algorithms and Challenges
Deldjoo, Y., Schedl, M., Hidasi, B., Wei, Y., & He, X. (2021) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_25 · Cited by 79
Deep Learning Enabled IRS for 6G Intelligent Transportation Systems: A Comprehensive Study
Song, W., Rajak, S., Dang, S., Liu, R., Li, J., & Chinnadurai, S. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3184314 · Cited by 71
“Reference [122] illustrated the significant methods for a network to deal with different sensor models.”
MethodOpen challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 106
“The data collected through various sensors, while rich, often provides limited insights when examined in isolation due to the specificity of each measurement modality [101].”
Identification of COPD From Multi-View Snapshots of 3D Lung Airway Tree via Deep CNN
Du, R., Qi, S., Feng, J., Xia, S., Kang, Y., Qian, W., & Yao, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2974617 · Cited by 66
“Our fusion strategy of three views is a kind of late or decision-level fusion, which is simpler than the early or data-level fusion and intermediate or feature-level fusion [58].”
BackgroundSmart Wearables for the Detection of Occupational Physical Fatigue: A Literature Review
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2022) · Sensors · doi:10.3390/s22197472 · Cited by 60
“Here, multimodal datasets are defined as data observed with multiple sensors, where the output of each sensor is called a modality and can be associated with a dataset [141].”
BackgroundChen, G., Huang, L., Xiao, S., Zhang, C., & Zhao, H. (2023) · Information Systems Research · doi:10.1287/isre.2021.0292 · Cited by 72
Recent Advances and Trends in Multimodal Deep Learning: A ReviewPreprint
Jabeen, S., Li, X., Amin, M. S., Li, S., & Abdul, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11087 · Cited by 141
“D Ramachandram et al. [119] examined recent developments in deep multimodal learning and limitations and obstacles in this active area of study; also regularization strategies and methods to optimize the structure of multimodal fusion are highlighted.”
RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey
Gao, M., Jiang, J., Zou, G., John, V., & Liu, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2907071 · Cited by 60
“In the literature, early fusion and late fusion are the two most popular fusion schemes [50], [52], [60] in complete multimodal data fusion.”
BackgroundJoint Discriminative Sparse Coding for Robust Hand-Based Multimodal Recognition
Li, S., & Zhang, B. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3074315 · Cited by 60
“Besides this, the deep learning techniques, such as convolutional neural network (CNN), have been proposed and applied to various recognition tasks [34].”
MethodDeep multimodal learning for lymph node metastasis prediction of primary thyroid cancer
Wu, X., Li, M., Cui, X., & Xu, G. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac4c47 · Cited by 63
Advances of deep learning in electrical impedance tomography image reconstruction
Zhang, T., Tian, X. K., Liu, X., Ye, J., Fu, F., Shi, X., Liu, R., & Xu, C. (2022) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2022.1019531 · Cited by 71
Multimodal fusion models for pulmonary embolism mortality prediction
Cahan, N., Klang, E., Marom, E. M., Soffer, S., Barash, Y., Burshtein, E., Konen, E., & Greenspan, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-34303-8 · Cited by 58
“This choice of fusing various representations at different depths is perhaps the most powerful and flexible aspect of deep multimodal fusion as compared to other fusion techniques 39 .”
ResultAI-powered integration of multimodal imaging in precision medicine for neuropsychiatric disorders
Huang, W., & Shu, N. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102132 · Cited by 86
“53 Early fusion integrates information from different modalities before inputting it into a single model.”
Multi-Modal Deep Analysis for Multimedia
Zhu, W., Wang, X. Y., & Li, H. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2940647 · Cited by 67
“Different from the survey paper [43], in this work we focus on multi-modal scenarios in areas related to multimedia, which includes but is not limited to deep neural network based architectures.”
BackgroundRFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation
Deng, L., Yang, M. Z., Li, T., He, Y., & Wang, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00135 · Cited by 54
“The multimodal data fusion has gained long-time attention to exploit the complementary nature of the data of different sources [26], [27].”
BackgroundGuo, H., Zhu, X., Huang, J. J., Zhang, Z., Tian, S., & Chen, Y. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.129466 · Cited by 66
Multimodality Fusion Aspects of Medical Diagnosis: A Comprehensive Review
Kumar, S., Rani, S., Sharma, S., & Min, H. (2024) · Bioengineering · doi:10.3390/bioengineering11121233 · Cited by 85
Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
Kim, M., Hong, J. H., Park, S. J., & Ro, Y. M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00036 · Cited by 63
“Recently, many studies are dealing with diverse information from multiple sources finding relationships among them [40].”
BackgroundSensing and Artificial Perception for Robots in Precision Forestry: A Survey
Ferreira, J. F., Portugal, D., Andrada, M. E., Machado, P., Rocha, R. P., & Peixoto, P. (2023) · Robotics · doi:10.3390/robotics12050139 · Cited by 68
Chen, X., Lin, X., Shen, Q., & Qian, X. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3035789 · Cited by 53
“In most multi-modal deep learning models [15], [16], the extracted features are directly concatenated and then selected and fused in fully connected layers.”
BackgroundMUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17260 · Cited by 52
Mann, P., Paes, A., & Matsushima, E. H. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7313 · Cited by 71
Tiong, L. C. O., Kim, S. T., & Ro, Y. M. (2020) · Image and Vision Computing · doi:10.1016/j.imavis.2020.103977 · Cited by 53
“For deep multimodal learning, several studies have been considered in the literature [12,17], whereby it highlights the importance of a fusion algorithm.”
Small Sample Learning in Big Data Era
Shu, J., Xu, Z., & Meng, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04572 · Cited by 50
“, the lack of visual instances can be compensated by semantic descriptions on the same object (Srivastava and Salakhutdinov, 2012; Ramachandram and Taylor, 2017) (see Fig.”
BackgroundMUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · arXiv · doi:10.48550/arxiv.2102.02340 · Cited by 52
Ahmad, Z., & Khan, N. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3028561 · Cited by 57
“A major challenge in multimodal fusion is to decide when and where to integrate the modalities for optimal result [19].”
BackgroundDecoding EEG Brain Activity for Multi-Modal Natural Language Processing
Hollenstein, N., Renggli, C., Glaus, B., Barrett, M., Tröndle, M., Langer, N., & Zhang, C. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.659410 · Cited by 67
“However, recent multi-modal machine learning work has shown the benefits of late fusion mechanisms (Ramachandram and Taylor, 2017).”
BackgroundMultimodal Information Fusion for Weather Systems and Clouds Identification From Satellite Images
Bai, C., Zhao, D., Zhang, M., & Zhang, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3202246 · Cited by 55
“Multimodal classification [15], [16], [17], [18], [19] has became a hot topic recently.”
BackgroundAsgari-Chenaghlu, M., Feizi‐Derakhshi, M., Farzinvash, L., Balafar, M. A., & Motamed, C. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06488-4 · Cited by 62
Multimodal Representation Learning: Advances, Trends and Challenges
Zhang, S., Zhai, J., Xie, B., Zhan, Y., & Wang, X. (2019) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc48188.2019.8949228 · Cited by 52
“Although there are a few review articles on the multimodal machine learning [2-4], the review papers are all based on the perspective of multimodal data fusion.”
BackgroundMulti-modal visual tracking: Review and experimental comparison
Zhang, P., Wang, D., & Lu, H. (2024) · Computational Visual Media · doi:10.1007/s41095-023-0345-5 · Cited by 61
“3, these methods can be further categorized into methods based on early fusion (EF) and late fusion (LF) [34, 98].”
RadioPathomics: Multimodal learning in non-small cell lung cancer for adaptive radiotherapy
Tortora, M., Cordelli, E., Sicilia, R., Nibid, L., Ippolito, E., Perrone, G., Ramella, S., & Soda, P. (2023) · IEEE Access · doi:10.1109/access.2023.3275126 · Cited by 49
“Definitely, the late fusion approach is a well-suited multimodal strategy when input modalities are significantly uncorrelated and they vary significantly in terms of data dimensionality and sampling rates [12].”
Multimodal vision-based human action recognition using deep learning: a review
Shafizadegan, F., Naghsh‐Nilchi, A. R., & Shabaninia, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10730-5 · Cited by 65
Deriving Design Feature Vectors for Patent Images Using Convolutional Neural Networks
Jiang, S., Luo, J., Pava, G. R., Hu, J., & Magee, C. L. (2020) · Journal of Mechanical Design · doi:10.1115/1.4049214 · Cited by 59
Automatic monitoring of lettuce fresh weight by multi-modal fusion based deep learning
Lin, Z., Fu, R., Ren, G., Zhong, R., Ying, Y. B., & Lin, T. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.980581 · Cited by 51
“Extracting and combining information from multiple modalities considered for a given learning task can produce an improved performance (Ramachandram and Taylor, 2017).”
BackgroundChannel Exchanging Networks for Multimodal and Multitask Dense Image Prediction
Wang, Y., Sun, F., Huang, W., He, F., & Tao, D. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3211086 · Cited by 64
“Joining the success of deep learning, multimodal fusion is recently specified as deep multimodal fusion by introducing end-to-end neural integration of multiple modalities [2], and it has exhibited remarkable benefits against the unimodal paradigm in semantic segmentation [3], [4], action recognition [5], [6], [7], vi…”
BackgroundEarly-stage fusion of EEG and fNIRS improves classification of motor imagery
Li, Y., Zhang, X., & Ming, D. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1062889 · Cited by 65
“…that multimodal data with higher heterogeneity tend to have better performance in late-fusion models, while multimodal data with low heterogeneity tend to perform better in the early fusion in the field of medical image (Ramachandram and Taylor, 2017; Mogadala et al., 2021; Yan et al., 2021).”
BackgroundAlam, M. U., & Rahmani, R. (2023) · Sensors · doi:10.3390/s23020970 · Cited by 64
“Moreover, recent ML and DL methods are well-adjusted to working with multi-modal data to utilize this useful concept in a prediction model for CDSS [15,16].”
MethodCross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers
Zhu, Z., Hou, J., & Wu, D. O. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02015 · Cited by 81
Dense Multimodal Fusion for Hierarchically Joint Representation
Hu, D., Wang, C., Nie, F., & Li, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683898 · Cited by 57
Deep-learning-enabled multimodal data fusion for lung disease classification
Kumar, S., Ivanova, O., Melyokhin, A., & Tiwari, P. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101367 · Cited by 52
A Comprehensive Survey on Deep Learning Techniques in Educational Data Mining
Lin, Y., Chen, H., Xia, W., Lin, F., Wang, Z., & Liu, Y. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00303-z · Cited by 126
“Deep Multimodal Learning algorithms [131], which combine CNNs for image/video data, RNNs or LSTMs for temporal data, and Transformers for textual data, could be used to leverage this wealth of information.”
Rahaman, M. A., Chen, J., Fu, Z., Lewis, N., Iraji, A., & Calhoun, V. D. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9630693 · Cited by 59
Unsupervised Multimodal Anomaly Detection With Missing Sources for Liquid Rocket Engine
Feng, Y., Liu, Z., Chen, J., Lv, H., Wang, J., & Zhang, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3162949 · Cited by 53
“As concerns the concept of multimodality, some scholars define it as data from multiple sensors [24], [52], whereas some emphasize the different types of data [9], such as visual and audio modality.”
BackgroundA novel feature-level fusion scheme with multimodal attention CNN for heart sound classification
Ranipa, K., Zhu, W., & Swamy, M. N. S. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108122 · Cited by 51
Nuha, N. S., Mahmud, T., Rezaoana, N., Hossain, M. S., & Andersson, K. (2023) · IEEE International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE) · doi:10.1109/wiecon-ece60392.2023.10456488 · Cited by 53
“In comparison to traditional approaches, deep neural networks are superior at multi-modal learning [21].”
ResultA multimodal deep learning model for cardiac resynchronisation therapy response prediction
Anton, E. P., Sidhu, B. S., Gould, J., Porter, B., Elliott, M. K., Mehta, V. S., Rinaldi, C. A., & King, A. P. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102465 · Cited by 53
“However, recently, inspired by the success of DL methods in other applications, multimodal deep learning (MMDL) (Ramachandram and Taylor, 2017) has attracted significant research attention due to its ability to learn common features from”
Deep multimodal predictome for studying mental disorders
Rahaman, M. A., Chen, J., Fu, Z., Lewis, N., Iraji, A., van Erp, T. G. M., & Calhoun, V. D. (2022) · Human Brain Mapping · doi:10.1002/hbm.26077 · Cited by 45
“Several data fusion techniques have been introduced in the literature (Ramachandram & Taylor, 2017).”
MethodNew Ideas and Trends in Deep Multimodal Content Understanding: A Review
Chen, W., Wang, W., Liu, L., & Lew, M. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.042 · Cited by 50
A Multimodal Deep Learning-Based Fault Detection Model for a Plastic Injection Molding Process
Kim, G., Choi, J. G., Ku, M., Cho, H., & Lim, S. (2021) · IEEE Access · doi:10.1109/access.2021.3115665 · Cited by 44
“They can learn nonlinear correlations among modalities for information fusions [79] as well as complementary information among modalities [80], thus fully leveraging the potential of multimodal data.”
BackgroundMMTFN : Multi‐modal multi‐scale transformer fusion network for Alzheimer's disease diagnosis
Miao, S., Xu, Q., Li, W., Yang, C., Sheng, B., Liu, F., Bezabih, T. T., & Xiao, Y. (2023) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22970 · Cited by 55
“17 Multi-modal data fusion can increase the amount of information available for model decision-making and, as a result, increase the accuracy of the decision's final outcome.”
Zhang, Y., Xia, K., Jiang, Y., Qian, P., Cai, W., Qiu, C., Lai, K. W., & Wu, D. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3142748 · Cited by 43
“To the best of our knowledge, multi-modality fusion can be organized into three main categories, early fusion, decision-level fusion and intermediate fusion [6], which are illustrated in Fig.”
BackgroundLiang, T., Lin, G., Wan, M., Li, T., Ma, G., & Lv, F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01505 · Cited by 65
“Image-text multimodal classification aims to improve the performance over the unimodal counterpart by integrating information from both the visual and language modalities [22].”
BackgroundDeep learning methods improve genomic prediction of wheat breeding
Montesinos‐López, A., Crespo‐Herrera, L., Dreisigacker, S., Gerard, G., Vitale, P., Pierre, C. S., Govindan, V., Tadesse, Z., Flores, M. C., Pérez‐Rodríguez, P., et al. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1324090 · Cited by 52
Chen, Q., Keenan, T. D. L., Allot, A., Peng, Y., Agrón, E., Domalpally, A., Klaver, C. C. W., Luttikhuizen, D. T., Colyer, M. H., Cukras, C. A., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa302 · Cited by 52
“This is not effective, especially when the modalities are very different.(31,32) In response, we proposed a novel multimodal deep learning framework, which uses a multitask learning technique and a multiattention mechanism for detecting macular features with improved performance.”
Optimizing the OCTA layer fusion option for deep learning classification of diabetic retinopathy
Ebrahimi, B., Le, D., Abtahi, M., Dadzie, A. K., Lim, J. I., Chan, R. V. P., & Yao, X. (2023) · Biomedical Optics Express · doi:10.1364/boe.495999 · Cited by 50
Tschand, A., Rajan, A. T. R., Idgunji, S., Ghosh, A., Holleman, J., Király, C., Ambalkar, P., Borkar, R., Chukka, R., Cockrell, T., et al. (2024) · International Symposium on High-Performance Computer Architecture · doi:10.1109/hpca61900.2025.00092 · Cited by 51
“The computing units in newer hardware versions have features that yield better energy efficiency: support for lower numeric precision, more efficient designs for the same precision, and better handling of memory-compute coordination [34], [35].”
Implicit Fusion by Joint Audiovisual Training for Emotion Recognition in Mono Modality
Han, J., Zhang, Z., Ren, Z. X., & Schuller, B. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682773 · Cited by 54
“In the literature, a number of fusion paradigms have been investigated for multi-modal emotion recognition [3, 15–20].”
BackgroundFlower classification with modified multimodal convolutional neural networks
Bae, K. I., Park, J., Lee, J., Lee, Y.-S., & Lim, C. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113455 · Cited by 49
Multi-modal voice pathology detection architecture based on deep and handcrafted feature fusion
Ömeroğlu, A. N., Mohammed, H. M. A., & Oral, E. A. (2022) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2022.101148 · Cited by 55
Multimodal deep learning methods enhance genomic prediction of wheat breeding
Montesinos-López, A., Rivera, C., Pinto, F., Piñera, F., Gonzalez, D., Reynolds, M., Pérez-Rodríguez, P., Li, H., Montesinos-López, O. A., & Crossa, J. (2023) · G3: Genes, Genomes, Genetics · doi:10.1093/g3journal/jkad045 · Cited by 41
Tiong, L. C. O., Kim, S. T., & Ro, Y. M. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7618-0 · Cited by 44
Personalization of Deep Learning
Schneider, J., & Vlachos, M. (2021) · Springer eBooks · doi:10.1007/978-3-658-32182-6_14 · Cited by 63
“ng of humans [7]. Small sample learning [29] deals with the problem of learning a model using a limited amount of data. Common strategies include data augmentation, using knowledge from other domains [32], transfer learning [8], or prior knowledge on concepts [33]. Transfer learning [8] aims at using existing parts su…”
BackgroundConditioned Source Separation for Musical Instrument Performances
Slizovskaia, O., Haro, G., & Gomez, E. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3082331 · Cited by 48
Multi-Level Confidence Learning for Trustworthy Multimodal Classification
Zheng, X., Tang, C., Wan, Z., Hu, C., & Zhang, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26346 · Cited by 70
“Multimodal learning has been widely investigated in past years and usually obtains remarkable performance improvement when compared to single modal learning models (Ramachandram and Taylor 2017; Baltrušaitis, Ahuja, and Morency 2018).”
ResultLee, D.-J., Shin, D.-H., Son, Y.-H., Han, J.-W., Oh, J.-H., Kim, D.-H., Jeong, J.-H., & Kam, T. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3366662 · Cited by 51
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
Guo, Y., Cheng, Z., Nie, L., Liu, Y., Wang, Y., & Kankanhalli, M. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331186 · Cited by 42
“The flexibility of deep architecture advances the implementation of multimodal fusion either as feature-level fusion or decision-level fusion [31].”
BackgroundMethodObjective Measures of Cognitive Load Using Deep Multi-Modal Learning: A Use-Case in Aviation
Wilson, J. C., Nair, S., Scielzo, S., & Larson, E. C. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3448111 · Cited by 52
“They take advantage of several key concepts from within machine learning: (1) multi-task learning [75, 85], (2) multimodal learning [59, 79], (3) variational autoencoders [55], and (4) x-vectors.”
BackgroundEmploying multimodal co-learning to evaluate the robustness of sensor fusion for industry 5.0 tasks
Rahate, A., Mandaokar, S., Chandel, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2022) · Soft Computing · doi:10.1007/s00500-022-06802-9 · Cited by 40
Evaluating the potential of Distribution of Relaxation Times analysis for plant agriculture
Haeverbeke, M. V., Bernard De Baets, C., & Stock, M. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108249 · Cited by 38
Forecasting backdraft with multimodal method: Fusion of fire image and sensor data
Zhang, T., Ding, F., Wang, Z., Xiao, F., Lu, C. X., & Huang, X. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.107939 · Cited by 41
Machine Learning in Petrology: State-of-the-Art and Future Perspectives
Petrelli, M. (2024) · Journal of Petrology · doi:10.1093/petrology/egae036 · Cited by 55
“This strategy takes the name of unimodal approach (Ramachandram and Taylor, 2017; Stahlschmidt et al., 2022).”
BackgroundInfluentialMultimodal fusion techniques: Review, data representation, information fusion, and application areas
Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 59
STUPEFY: Set-Valued Box Particle Filtering for Bluetooth Low Energy-Based Indoor Localization
Malekzadeh, P., Mohammadi, A., Barbulescu, M., & Plataniotis, K. N. (2019) · IEEE Signal Processing Letters · doi:10.1109/lsp.2019.2945402 · Cited by 38
Cross-Domain Transfer Hashing for Efficient Cross-Modal Retrieval
Li, F., Wang, B., Zhu, L., Li, J., Zhang, Z., & Chang, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3374791 · Cited by 50
Continual Learning for Smart City: A Survey
Yang, L., Luo, Z., Zhang, S., Teng, F., & Li, T. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3447123 · Cited by 57
“Thus multi-modal machine learning (MML) has emerged [185], [186], [187].”
BackgroundA machine learning approach for multimodal data fusion for survival prediction in cancer patients
Νικολάου, Ν., Salazar, D., RaviPrakash, H., Gonçalves, M., Mulla, R., Burlutskiy, N., Markuzon, N., & Jacob, E. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-00917-6 · Cited by 65
Citrus Huanglongbing Detection Based on Multi-Modal Feature Fusion Learning
Yang, D., Wang, F., Hu, Y., Lan, Y., & Deng, X. (2021) · Frontiers in Plant Science · doi:10.3389/fpls.2021.809506 · Cited by 48
“Deep multi-modal learning can reduce the design requirements for feature engineering and deep-learning architectures, and can achieve the required accuracy more simply and quickly (Atrey et al., 2010; Ramachandram and Taylor, 2017; Baltrusaitis et al., 2019).”
BackgroundPrompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction
Hu, X., Chen, J., Liu, A., Meng, S., Wen, L., & Yu, P. S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611899 · Cited by 60
“In multimodal learning, models process and integrate data from multiple modalities [5, 6, 45], with applications in visual and language learning [43], video understanding [46, 47], and natural language understanding [29, 30, 35].”
BackgroundZhou, X., Chen, C., Lv, X., Zuo, E., Li, M., Wu, L., Chen, X., Wu, X., & Chen, C. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103804 · Cited by 45
Why concepts are (probably) vectors
Piantadosi, S. T., Muller, D. C. Y., Rule, J., Kaushik, K., Gorenstein, M. I., Leib, E. R., & Sanford, E. M. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.06.011 · Cited by 87
On Robustness of Multi-Modal Fusion—Robotics Perspective
Bednarek, M., Kicki, P., & Walas, K. (2020) · Electronics · doi:10.3390/electronics9071152 · Cited by 43
“There exist three main types of model agnostic data fusion methods—early (data-level), intermediate (feature-level), and late (decision-level) [14], however it is possible to combine at least two of them into a hybrid fusion [2].”
Background
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.