Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 201–300 of 1,002
Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning
Dolmans, T. C., Poel, M., Klooster, J. W. J. R. V. '., & Veldkamp, B. P. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.609096 · Cited by 39
“Ramachandram and Taylor formulated several key considerations to be made for deep learning with multiple modalities in their overview of deep multimodal learning (Ramachandram and Taylor, 2017).”
BackgroundRecent Advances in Conventional and Deep Learning-Based Depth Completion: A Survey
Xie, Z., Yu, X., Gao, X., Li, K., & Shen, S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3201534 · Cited by 44
“Even so, due to the diversity and flexibility of deep learning model structures, using the hybrid fusion method is a better strategy [96].”
MethodLyu, W., Dong, X., Wong, R., Zheng, S., Abell-Hart, K., Wang, F., & Chen, C. (2022) · arXiv · doi:10.48550/arxiv.2208.10240 · Cited by 46
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram & Taylor, 2017).”
BackgroundImage Aesthetics Assessment With Attribute-Assisted Multimodal Memory Network
Li, L., Zhu, T., Chen, P., Yang, Y., Li, Y., & Lin, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3272984 · Cited by 41
“Multimodal learning has been demonstrated more effective compared to single modal-based methods in various tasks [25], [26].”
BackgroundPose-Aided Video-Based Person Re-Identification via Recurrent Graph Convolutional Network
Pan, H., Liu, Q., Chen, Y., He, Y., Zheng, Y., Zheng, F., & He, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3276996 · Cited by 40
“This conclusion is also demonstrated in other multi-modal learning tasks [70], [71].”
ResultPrediction of ultimate tensile strength of Al‐Si alloys based on multimodal fusion learning
Zhu, L., Luo, Q., Chen, Q., Zhang, Y. X., Zhang, L., Hu, B., Han, Y., & Li, Q. (2024) · Materials Genome Engineering Advances · doi:10.1002/mgea.26 · Cited by 46
A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social Environments
Duncan, J. A., Alambeigi, F., & Pryor, M. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3657030 · Cited by 54
“Model-agnostic fusion techniques can be categorized as follows depending on the phase at which data are fused [25, 128, 180]: — Early (data-level) fusion : Unprocessed data from different modalities are aligned and combined into a single data structure.”
Prediction of Cloud Resources Demand Based on Hierarchical Pythagorean Fuzzy Deep Neural Network
Chen, D., Zhang, X., Wang, L., & Han, Z. (2019) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2019.2906901 · Cited by 36
Jian, J., Xia, W., Zhang, R., Zhao, X., Zhang, J., Wu, X., Li, Y., Qiang, J., & Gao, X. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102194 · Cited by 37
Multimodal Arabic Rumors Detection
Albalawi, R. M., Jamal, A. T., Khadidos, A. O., & Alhothali, A. (2023) · IEEE Access · doi:10.1109/access.2023.3240373 · Cited by 37
“fusion (hybrid fusion), and late fusion (decision-based) [50].”
BackgroundGuida, C., Zhang, M., & Shan, J. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08214-8 · Cited by 38
“Intermediate fusion is performed by combining features from different networks into one representation and then further learning from this joint multimodal representation [23].”
BackgroundMethodPrivacy-Preserving Machine Learning for Healthcare: Open Challenges and Future Perspectives
Guerra-Manzanares, A., López, L. J. L., Maniatakos, M., & Shamout, F. E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39539-0_3 · Cited by 62
“This limits the generalization of the results and neglects the potential improvement in predictive performance that could result from combining different data sources in multi-modal learning settings (Ramachandram & Taylor, 2017).”
BackgroundCommon Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications
Mena, F., Arenas, D., Nuske, M., & Dengel, A. R. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3361556 · Cited by 40
“Recent reviews compared conventional with NN models for MV learning [39], [40].”
Adam, M., Albaseer, A. M., Baroudi, U. A., & Abdallah, M. M. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3554537 · Cited by 61
AlignFusionNet: Efficient Cross-Modal Alignment and Fusion for 3D Semantic Occupancy Prediction
Xu, Z., Qi, L., Du, H.-E., Yang, J., & Chen, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3589858 · Cited by 47
AOMD: An analogy-aware approach to offensive meme detection on social media
Shang, L., Zhang, Y., Zha, Y., Chen, Y., Youn, C., & Wang, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102664 · Cited by 48
A Comprehensive Survey on Multimodal Data Representation and Information Fusion Algorithms
Gaonkar, A., Chukkapalli, Y., Raman, P. J., Srikanth, S., & Gurugopinath, S. (2021) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit51480.2021.9498415 · Cited by 42
Huang, Y., Huang, S., Wang, Y., Li, Y., Gui, Y., & Huang, C. (2022) · Frontiers in Physiology · doi:10.3389/fphys.2022.937546 · Cited by 54
“Since the decision-level fusion strategy used in this study is to fuse the prediction results of different submodels, it can make the errors of different submodels often disconnected from and unaffected by each other without causing further accumulation of errors (Ramachandram and Taylor, 2017; Murphy, 2019).”
MethodXue, Y., Zhu, J., Huang, X., Xu, X., Li, X., Zheng, Y., Zhu, Z., Jin, K., Ye, J., Gong, W., et al. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104233 · Cited by 46
Integrating multiple materials science projects in a single neural network
Hatakeyama‐Sato, K., & Oyaizu, K. (2020) · Communications Materials · doi:10.1038/s43246-020-00052-8 · Cited by 37
“To integrate knowledge, varied information must be inputted and outputted (i.e., multimodal learning) 19,20 .”
BackgroundMultimodal Hand Gesture Classification for the Human–Car Interaction
D’Eusanio, A., Simoni, A., Pini, S., Borghi, G., Vezzani, R., & Cucchiara, R. (2020) · Informatics · doi:10.3390/informatics7030031 · Cited by 33
“However, a general and effective fusion strategy is still not be found, while it has been shown that the best fusion scheme is often task and dataset dependent [48].”
BackgroundUNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation
Tian, J., Cheung, W., Glaser, N., Liu, Y., & Kira, Z. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197266 · Cited by 50
“For more details on the recent trends and architectures for multimodal fusion, we refer to [25].”
BackgroundFINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection
İnceoğlu, A., Aksoy, E. E., Ak, A. C., & Sarıel, S. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636455 · Cited by 59
“Multimodal fusion has a wide variety of applications with different kinds of input modalities [27].”
BackgroundSound-Based Construction Activity Monitoring with Deep Learning
Xiong, W., Xu, X., Chen, L., & Yang, J. (2022) · Buildings · doi:10.3390/buildings12111947 · Cited by 39
A reference architecture for the operationalization of machine learning models in manufacturing
Raffin, T., Reichenstein, T., Werner, J., Kühl, A., & Franke, J. E. (2022) · Procedia CIRP · doi:10.1016/j.procir.2022.10.062 · Cited by 36
Liu, H., Yang, A., Zhong, C., Zhu, X., Hao, M., Feng, Z., Tang, J., Yang, C., Zhang, J., Liu, J., et al. (2025) · ACS Energy Letters · doi:10.1021/acsenergylett.5c00626 · Cited by 34
Hierarchical attention-based multimodal fusion for video captioning
Wu, C., Wei, Y., Chu, X., Sun, W., Su, F., & Wang, L. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.029 · Cited by 38
Fu, D., Weber, C., Yang, G., Kerzel, M., Nan, W., Barros, P. V. A., Wu, H., Liu, X., & Wermter, S. (2020) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2020.00010 · Cited by 42
“Many studies focus on multimodal fusion (Ramachandram and Taylor, 2017), but research about crossmodal selective attention in computer science is limited.”
BackgroundImproving exchange rate forecasting via a new deep multimodal fusion model
Windsor, E., & Cao, W. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03342-5 · Cited by 42
“Next, the hidden state features obtained from the two LSTM modules are concatenated and fed into a shared representation layer, which represents the couplings between the modalities via intermediate multimodal fusion [23].”
MethodDeep CNN for COPD identification by Multi-View snapshot integration of 3D airway tree and lung field
Wu, Y., Du, R., Feng, J., Qi, S., Pang, H., Xia, S., & Qian, W. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.104162 · Cited by 34
Multimodal explainability via latent shift applied to COVID-19 stratification
Guarrasi, V., Tronchin, L., Albano, D., Faiella, E., Fazzini, D., Santucci, D., & Soda, P. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110825 · Cited by 31
“Nevertheless, this availability of information from di ff erent modality makes explainability a key necessity to reduce the opacity of the multimodal deep architectures [7].”
Shen, C., Cai, G., Tian, J., Wu, X., Ding, M., Wang, B., & Liu, D. (2023) · Food Chemistry · doi:10.1016/j.foodchem.2023.138265 · Cited by 31
Perspectives on Advancing Multimodal Learning in Environmental Science and Engineering Studies
Liu, W., Chen, J., Wang, H., Fu, Z., Peijnenburg, W. J. G. M., & Hong, H. (2024) · Environmental Science & Technology · doi:10.1021/acs.est.4c03088 · Cited by 54
Pulmonary Nodule Classification Using Feature and Ensemble Learning-Based Fusion Techniques
Muzammil, M. J., Ali, I., Haq, I. U., Amir, M., & Abdullah, S. (2021) · IEEE Access · doi:10.1109/access.2021.3102707 · Cited by 35
“c) The hybrid approaches, in which DCNNs are utilized to extract the deep features transfer learning techniques, the conventional machine learning (ML) methods are used to combine the hand-crafted features and deep features and determine the final decision [18]–[20].”
MethodGated-Residual Block for Semantic Segmentation Using RGB-D Data
Qian, Y., Deng, L., Li, T., Wang, C., & Yang, M. Z. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3107672 · Cited by 36
Estimating hydroponic lettuce phenotypic parameters for efficient resource allocation
Ojo, M. O., Zahid, A., & Masabni, J. G. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.108642 · Cited by 32
Multi-Modal Fusion Strategies for Ultrasound in Diagnosis of Liver Malignant Tumor
Li, M., Li, W., Lin, M., Lin, X.-X., Hu, H.-T., Wang, Y.-C., Ruan, S., Huang, Z.-R., Lu, R.-F., Lu, R.-F., et al. (2022) · Neurocomputing · doi:10.1016/j.neucom.2024.128257 · Cited by 32
Saputra, M. R. U., Bhaswara, I. D., Nasution, B. I., Ang, M. L. E., Husna, N. L. R., Witra, T., Feliren, V., Owen, J. R., Kemp, D., & Lechner, A. M. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114584 · Cited by 41
Digital Phenotyping Using Multimodal Data
Cohen, A. S., Cox, C. R., Masucci, M. D., Le, T. P., Cowan, T., Coghill, L. M., Holmlund, T. B., & Elvevåg, B. (2020) · Current Behavioral Neuroscience Reports · doi:10.1007/s40473-020-00215-4 · Cited by 35
Emery, K. J., Zannoli, M., Warren, J., Xiao, L., & Talathi, S. S. (2021) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3448018.3457996 · Cited by 40
“Here, we did not apply dimensionality reduction techniques to the features, nor did we explore more complex sensor fusion approaches (i.e. intermediate and late [Atrey et al. 2010; Khaleghi et al. 2009; Ramachandram and Taylor 2017]).”
BackgroundMethodMin, J. H., Ham, S. W., Kim, D., & Lee, E. H. (2022) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221130026 · Cited by 31
Category-Aware Multimodal Attention Network for Fashion Compatibility Modeling
Jing, P., Cui, K., Guan, W., Nie, L., & Su, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3246796 · Cited by 30
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 36
“We can categorize medical data fusion techniques into early fusion, late fusion and joint fusion methods [61].”
MethodXia, F., Lou, Z., Sun, D., Li, H., & Quan, L. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103352 · Cited by 32
Artificial intelligence accelerates multi-modal biomedical process: A Survey
Li, J., Han, X., Qin, Y., Tan, F., Chen, Y., Wang, Z., Song, H., Zhou, X., Zhang, Y., Hu, L., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126720 · Cited by 35
Multimodal fusion recognition for digital twin
Zhou, T., Zhang, X., Kang, B., & Chen, M. (2022) · Digital Communications and Networks · doi:10.1016/j.dcan.2022.10.009 · Cited by 38
Mehari, T., & Strodthoff, N. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3310989 · Cited by 29
“approach in the multimodal learning literature [37].”
Artificial intelligence for drug delivery: Yesterday, today and tomorrow
Wu, Y., Wang, N., Xiong, P., Wang, R., Deng, J., & Ouyang, D. (2025) · Acta Pharmaceutica Sinica B · doi:10.1016/j.apsb.2025.09.022 · Cited by 46
“Furthermore, data alignment and integration techniques, cross-modal learning, and multimodal deep learning frameworks have been developed to handle multimodal data 81 .”
Integration of deep neural networks and ensemble learning machines for missing well logs estimation
Jian, H., Chenghui, L., Cao, Z., & Haiwei, M. (2020) · Flow Measurement and Instrumentation · doi:10.1016/j.flowmeasinst.2020.101748 · Cited by 29
Aligning Multi-Sequence CMR Towards Fully Automated Myocardial Pathology Segmentation
Ding, W., Li, L., Qiu, J., Wang, S., Huang, L., Chen, Y., Yang, S., & Zhuang, X. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3288046 · Cited by 40
MDF-Net for abnormality detection by fusing X-rays with clinical data
Hsieh, C., Nobre, I. B., Sousa, S. C., Ouyang, C., Brereton, M. F., Nascimento, J. C., Jorge, J. A., & Moreira, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-41463-0 · Cited by 33
Intra- and Inter-Modal Curriculum for Multimodal Learning
Zhou, Y., Wang, X. Y., Chen, H., Duan, X., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612468 · Cited by 47
“For example, it can be divided according to fusion strategies [53], model frameworks [27], multimodal challenges [6], etc.”
BackgroundCoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities
Kontras, K., Chatzichristos, C., Phan, H., Suykens, J. A. K., & Vos, M. D. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3354388 · Cited by 40
“They potentially contain complementary information for the identification of the underlying data distribution [24], [49].”
BackgroundA systematic review of artificial neural network techniques for analysis of foot plantar pressure
Wang, C., Evans, K. A., Hartley, D., Morrison, S., Veidt, M., & Wang, G. (2024) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2024.01.005 · Cited by 30
Man, T., Osipov, V. Y., Zhukova, N. A., Subbotin, A., & Ignatov, D. I. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102427 · Cited by 34
On the Gap between Domestic Robotic Applications and Computational Intelligence
Zhong, J., Ling, C., Cangelosi, A., Lotfi, A., & Liu, X. (2021) · Electronics · doi:10.3390/electronics10070793 · Cited by 37
“Computing-wise: How to efficiently utilize the multi-modal sensor signals to identify the emotional status of users? To solve this, we may consider the multi-modal machine learning methods [104].”
BackgroundKIEGLFN: A unified acne grading framework on face images
Yi, L., Jiang, J., Ma, Z., Chen, D., Guan, Y., You, H., Cheng, X., Liu, B., & Luo, G. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106911 · Cited by 36
Real-time detection algorithm of helmet and reflective vest based on improved YOLOv5
Chen, Z., Zhang, F., Liu, H., Wang, L., Zhang, Q., & Guo, L. (2023) · Journal of Real-Time Image Processing · doi:10.1007/s11554-023-01268-w · Cited by 29
“These approaches provide 4 Page 2 of 14 a new perspective for the real-time detection of safety equipment [5–7].”
BackgroundGuarrasi, V., & Soda, P. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106625 · Cited by 28
Multimodal Dual-Embedding Networks for Malware Open-Set Recognition
Guo, J., Wang, H., Xu, Y., Xu, W., Zhan, Y., Sun, Y., & Guo, S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3373809 · Cited by 32
“Specifically, multimodal refers to different data formats such as image, video, text, audio, etc., where each format can correspond to the same or different objects/events [43].”
Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method
Yan, R., Ren, F., Rao, X., Shi, B., Xiang, T., Zhang, L., Liu, Y., Liang, J. S., Zheng, C.-H., & Zhang, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-26763-6_44 · Cited by 30
Deep Multimodal Fusion for Subject-Independent Stress Detection
Radhika, K. R., & Murthy, O. V. R. (2021) · International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence51648.2021.9377132 · Cited by 33
“Much of the deep multimodal fusion research adopts this intermediate-fusion technique [22].”
BackgroundJian, J., Li, Y., Xia, W., Zhang, H., Zhang, R., Li, H., Zhao, X., Zhao, S., Zhang, J., Cai, S., et al. (2021) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.28008 · Cited by 29
“As stated in a study by Ramachandram and Taylor, early fusion may not fully exploit the complementary natures of multiple source data, therefore leading to redundancies of input.(27) In contrast, late fusion works in an ensemble-learning manner, thus boosting performance.”
BackgroundDeep learning in multimedia healthcare applications: a review
Tobón, D. P., Hossain, M. S., Muhammad, G., Bilbao, J., & Saddik, A. E. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00948-0 · Cited by 32
“They found DNNs leverage the EHRs correlations among features in different modalities.”
BackgroundInfluentialAttention-Based Cross-Modal CNN Using Non-Disassembled Files for Malware Classification
Kim, J., Paik, J., & Cho, E.-S. (2023) · IEEE Access · doi:10.1109/access.2023.3253770 · Cited by 27
“In late fusion [28], the outputs of the individual classifiers on different modalities are combined, as shown in Fig.”
BackgroundMethodCau, R., Pisu, F., Suri, J. S., Mannelli, L., Scaglione, M., Masala, S. A., & Saba, L. (2023) · Diagnostics · doi:10.3390/diagnostics13122061 · Cited by 29
“This paradigm, known as multimodal learning, aims to leverage these heterogeneous data in a complementary manner to facilitate the learning of complex tasks [29].”
Feature-Level Fusion Recognition of Space Targets With Composite Micromotion
Zhang, Y., Xie, Y., Kang, L., Li, K., Luo, Y., & Zhang, Q. (2023) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3331339 · Cited by 27
“Compared to pixel-level fusion and decision-level fusion, feature-level fusion offers a more flexible multidomain data fusion approach [30].”
BackgroundOstertag, C., Visani, M., Urruty, T., & Beurton-Aimar, M. (2023) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-023-02866-6 · Cited by 26
“Among the possible data fusion strategies [2], Xu et al.”
Online multi-hypergraph fusion learning for cross-subject emotion recognition
Pan, T., Ye, Y., Zhang, Y., Xiao, K., & Cai, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102338 · Cited by 32
Tian, J., Li, Y., Luo, Y., Zhang, H., & Lü, X. (2024) · Structural Health Monitoring · doi:10.1177/14759217241244549 · Cited by 28
“42,43 Hence, we propose a new multisource information-gated fuser based on the feature fusion technique, 44,45 which is used to further select which parts of the input seepage–deformation coupling features and water level–deformation coupling features are more likely to contribute to the correct…”
Overcoming the modal/amodal dichotomy of concepts
Michel, C. (2020) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-020-09678-y · Cited by 38
Detecting Product Adoption Intentions via Multiview Deep Learning
Zhang, Z., Wei, X., Zheng, X., Li, Q., & Zeng, D. D. (2021) · INFORMS journal on computing · doi:10.1287/ijoc.2021.1083 · Cited by 25
How to leverage the multimodal EHR data for better medical prediction?
Yang, B., & Wu, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.329 · Cited by 26
“resentation is nontrivial, as naively adding more features could result in worse performances (Ramachandram and Taylor, 2017).”
A typhoon trajectory prediction model based on multimodal and multitask learning
Qin, W., Tang, J., Lu, C. D., & Lao, S. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.108804 · Cited by 29
On the role of multimodal learning in the recognition of sign language
Ferreira, P. M., Cardoso, J. S., & Rebelo, A. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6565-5 · Cited by 33
“A very comprehensive and recent survey on deep multimodal learning and regularization can be found in [18].”
BackgroundFacing & mitigating common challenges when working with real-world data: The Data Learning Paradigm
Lever, J., Cheng, S. L., Quilodrán-Casas, C., Liu, C., Fan, H., Platt, R., Rakotoharisoa, A., Johnson, E., Li, S., Shang, Z., et al. (2025) · Journal of Computational Science · doi:10.1016/j.jocs.2024.102523 · Cited by 35
Multi-modal learning-based algae phyla identification using image and particle modalities
Kwon, D. H., Lee, M. J., Jeong, H., Park, S., & Cho, K. H. (2025) · Water Research · doi:10.1016/j.watres.2025.123172 · Cited by 31
FunkR-pDAE: Personalized Project Recommendation Using Deep Learning
Zhang, P., Xiong, F., LEUNG, H. K. N., & Song, W. (2018) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2018.2870734 · Cited by 27
“Therefore, a large number of representative feature information is obtained, and the prediction precision is improved [17].”
BackgroundDifferentiate cavernous hemangioma from schwannoma with artificial intelligence (AI)
Bi, S., Chen, R., Zhang, K., Xiang, Y., Wang, R., Lin, H., & Yang, H. (2020) · Annals of Translational Medicine · doi:10.21037/atm.2020.03.150 · Cited by 25
“Such an approach requires involving multimodal machine learning (35-37), because MRI images with different weighted sequences should be processed as separate modes.”
BackgroundMemory based fusion for multi-modal deep learning
Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.10.005 · Cited by 29
Mekala, M. S., Park, W., Dhiman, G., Srivastava, G., Park, J. H., & Jung, H.-Y. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09670-y · Cited by 27
“For instance, Principal Component Analysis (PCA) and Canonical Correlation Analysis (CCA) [16, 17] methods.”
MethodDeep multimodal learning for municipal solid waste sorting
Lu, G., Wang, Y., Xu, H., Yang, H., & Zou, J. T. (2021) · Science China Technological Sciences · doi:10.1007/s11431-021-1927-9 · Cited by 27
“With the rise of deep learning [8–12], the deep-© learning method was proposed for multimodal fusion, which is called deep multimodal learning [13].”
MethodA Novel Multimodal Radiomics Model for Predicting Prognosis of Resected Hepatocellular Carcinoma
He, Y., Hu, B., Zhu, C., Xu, W., Ge, Y., Hao, X., Dong, B., Chen, X., Dong, Q., & Xianjun, Z. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.745258 · Cited by 30
Multimodal Learning for Multi-omics: A Survey
Tabakhi, S., Suvon, M. N. I., Ahadian, P., & Lu, H. (2023) · World Scientific Annual Review of Artificial Intelligence · doi:10.1142/s2811032322500047 · Cited by 31
“area due to its superior performance in numerous domains of multimodal learning.(156) In this approach, a network is trained with each modality from biological”
BackgroundVariational Causal Inference Network for Explanatory Visual Question Answering
Xue, D., Qian, S., & Xu, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00238 · Cited by 40
“Multimodal reasoning is a vital ability for humans and a fundamental problem for artificial intelligence [27, 39, 8].”
BackgroundDeep learning on multi-view sequential data: a survey
Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 30
“Several surveys reviewed the progress of multi-view deep learning (Wang 2021; Baltrušaitis et al. 2018; Chen et al. 2020c; Summaira et al. 2021; Rahate et al. 2021; Ramachandram and Taylor 2017; Zhao et al. 2017a).”
BackgroundDeep learning algorithms applied to computational chemistry
Pando, A. G., Ramírez-Alonso, G. M. D. J., Arzate-Quintana, C., & Camarillo‐Cisneros, J. (2023) · Molecular Diversity · doi:10.1007/s11030-023-10771-y · Cited by 30
“Incorporating information from different sources has demonstrated significant promise in enhancing the quality results of deep learning algorithms [59, 60].”
Coupled tensor decompositions for data fusion
Chatzichristos, C., Eyndhoven, S. V., Kofidis, E., & Huffel, S. V. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-824447-0.00016-9 · Cited by 30
Multistage Spatio-Temporal Networks for Robust Sketch Recognition
Li, H., Jiang, X., Guan, B., Wang, R., & Magnenat‐Thalmann, N. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3160240 · Cited by 26
Designing CNNs for Multimodal Image Restoration and Fusion via Unfolding the Method of Multipliers
Marivani, I., Tsiligianni, E., Cornelis, B., & Deligiannis, N. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3163649 · Cited by 33
“Therefore, image restoration can be formulated as a problem of the form (11).”
BackgroundA Multimodal Knowledge-Based Deep Learning Approach for MGMT Promoter Methylation Identification
Capuozzo, S., Gravina, M., Gatta, G., Marrone, S., & Sansone, C. (2022) · Journal of Imaging · doi:10.3390/jimaging8120321 · Cited by 24
“In particular, we implement an early fusion technique, in which information coming from multiple sources is merged to highlight different characteristics [26].”
MethodTong, R. M., Zhu, Z., & Ling, J. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16068 · Cited by 28
“…tabular dataset, further research on fully utilizing of multimodal data such as medical documents with nature language and medical imaging for survival analysis via the popular technology of neural network and deep learning seems to be a promising direction to improve the model performance [35].”
BackgroundImproving Lettuce Fresh Weight Estimation Accuracy through RGB-D Fusion
Xu, D., Chen, J., Ba, L., & Ma, J. (2023) · Agronomy · doi:10.3390/agronomy13102617 · Cited by 24
Modelling patient trajectories using multimodal information
Silva, J., & Matos, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104195 · Cited by 29
Tang, M., Li, P., Zhang, H., Deng, L., Liu, S., Zheng, Q., Chang, H., Zhao, C., Wang, M., Zuo, G., et al. (2024) · Biomedical Technology · doi:10.1016/j.bmt.2024.10.003 · Cited by 23
Fusion of EEG and fMRI via Soft Coupled Tensor Decompositions
Chatzichristos, C., Davies, M. E., Escudero, J., Kofidis, E., & Theodoridis, S. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553077 · Cited by 23
Li, X., Lei, L., Zhang, C., & Kuang, G. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3176936 · Cited by 31
“…existing methods either delete extra modality information of train sets and reserved complete modalities to complete the learning of feature extractor and classifiers [15], [16], or introduce a series of effective imputation strategies to incomplete modalities based on available modalities [17].”
MethodUAVM: Towards Unifying Audio and Visual Models
Gong, Y., Liu, A. H., Rouditchenko, A., & Glass, J. R. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3224688 · Cited by 39
“To enable AI systems to have a similar ability, audiovisual multi-modal learning has been extensively studied [1], [2].”
BackgroundFeng, J., Tang, E., Zeng, M., Gu, Z., Kou, P., & Zheng, W. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103427 · Cited by 24
Multimodal Physiological Signals Fusion for Online Emotion Recognition
Pan, T., Ye, Y., Cai, H., Huang, S., Yang, Y., & Wang, G. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612555 · Cited by 36
“To exploit these multiple informative data for decision making, multimodal learning has been explored and applicated recently [3, 32, 43].”
Background
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.