Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
“[10] provide an overview on deep multi-modal learning, and mention its applications in diverse research fields, such as robotic grasping and human action recognition.”
BackgroundSoybean yield prediction from UAV using multimodal data fusion and deep learning
Maimaitijiang, M., Sagan, V., Sidike, P., Hartling, S., Esposito, F., & Fritschi, F. B. (2020) · Remote Sensing of Environment · doi:10.1016/j.rse.2019.111599 · Cited by 971
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
BackgroundInfluentialA Survey of Deep Learning: Platforms, Applications and Emerging Research Trends
Hatcher, W. G., & Yu, W. (2018) · IEEE Access · doi:10.1109/access.2018.2830661 · Cited by 654
“Particularly, other works focus primarily on the advances and needs of a single learning mechanism or modality [19], [109], or towards improvements in a single application [38], [67], [68], [98].”
BackgroundResultDeep Learning for Geophysics: Current and Future Trends
Yu, S., & Ma, J. (2021) · Reviews of Geophysics · doi:10.1029/2021rg000742 · Cited by 571
“In multimodal DL (Ngiam et al., 2011; Ramachandram & Taylor, 2017), inputs are from different sources, such as seismic data and gravity data.”
BackgroundGupta, A., Anpalagan, A., Guan, L., & Khwaja, A. S. (2021) · Array · doi:10.1016/j.array.2021.100057 · Cited by 565
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
“From the perspective of multimodal deep learning, the closest effort to ours is [20] which mainly reviews the models and applications relying on multimodal features fusion (categorized as joint representation in ours).”
BackgroundSocial media data for conservation science: A methodological overview
Toivonen, T., Heikinheimo, V., Fink, C., Hausmann, A., Hiippala, T., Järv, O., Tenkanen, H., & Di Minin, E. (2019) · Biological Conservation · doi:10.1016/j.biocon.2019.01.023 · Cited by 468
Task-Oriented Multi-User Semantic Communications
Xie, H., Qin, Z., Tao, X., & Letaief, K. B. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3191326 · Cited by 458
“…employ more than one user to serve one multimodal intelligent task, which is suitable for the emerging autonomous scenarios in daily life [19] and industry [20], i.e., autonomous checkout at retail stores [21], intelligent control at smart home [22], and human activity recognition in…”
BackgroundBayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456
Deep learning to detect Alzheimer's disease from neuroimaging: A systematic literature review
Ebrahimighahnavieh, M. A., Luo, S., & Chiong, R. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105242 · Cited by 429
Deep Learning for EMG-based Human-Machine Interaction: A Review
Xiong, D., Zhang, D., Zhao, X., & Zhao, Y. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2021.1003865 · Cited by 395
Attention Bottlenecks for Multimodal Fusion
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., & Sun, C. (2021) · arXiv · doi:10.48550/arxiv.2107.00135 · Cited by 400
“Audiovisual Learning: Audiovisual multimodal learning has a rich history, both before and during the deep learning era [49].”
BackgroundDeep Learning for Intelligent Transportation Systems: A Survey of Emerging Trends
Veres, M., & Moussa, M. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2929020 · Cited by 394
“survey: [79])?”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
Douillard, A., Rame, A., Couairon, G., & Cord, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00907 · Cited by 357
“The independent classifiers paradigm [65] coupled with the sigmoid activation and binary crossentropy loss exclude explicitly a late fusion [57] of the task embeddings resulting in more specialized classifiers.”
MethodMultimodal skin lesion classification using deep learning
Yap, J., Yolland, W., & Tschandl, P. (2018) · Experimental Dermatology · doi:10.1111/exd.13777 · Cited by 326
Deep multi-view learning methods: A review
Yan, X., Hu, S., Mao, Y., Ye, Y., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.090 · Cited by 322
“These surveys focus on the problems of specific MVL methods, such as multi-view fusion [10, 27, 28], multi-modal learning [37, 38], multi-view clustering [26] and multi-view representation learning [39, 40].”
BackgroundBota, P. J., Wang, C., Fred, A. L. N., & Placido Da Silva, H. (2019) · IEEE Access · doi:10.1109/access.2019.2944001 · Cited by 322
“For further information on DL, the authors refer the reader to [159], [160].”
BackgroundAn Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation
Michelsanti, D., Tan, Z.-H., Zhang, S.-X., Xu, Y., Yu, M., Yu, D., & Jensen, J. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3066303 · Cited by 316
“based on the processing level at which the fusion occurs [158], [209]: early fusion and late fusion (cf.”
BackgroundMMTM: Multimodal Transfer Module for CNN Fusion
Vaezi Joze, H. R., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01330 · Cited by 259
Attention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
“1 3 multimodality in neural networks is a historical problem widely discussed by the scientific community (Ramachandram and Taylor 2017; Gao et al. 2020b).”
BackgroundMulti-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 291
“task in multi-modal pretrained language models [45].”
MIMETIC: Mobile encrypted traffic classification using multimodal deep learning
Aceto, G., Ciuonzo, D., Montieri, A., & Pescapè, A. (2019) · Computer Networks · doi:10.1016/j.comnet.2019.106944 · Cited by 288
“…reasons, multimodal DL is here fore-seen as an appealing alternative toward a sophisticated form of information fusion, named intermediate fusion [19], overcoming both the limitations of the early (or data ) fusion and late (or score / decision ) fusion , o ff ering a truly-flexible tool for…”
BackgroundA comparative review on multi-modal sensors fusion based on deep learning
Tang, Q., Liang, J., & Zhu, F. (2023) · Signal Processing · doi:10.1016/j.sigpro.2023.109165 · Cited by 280
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274
“Note that some publications refer to input fusion as early fusion, while intermediate fusion may be considered as feature-level fusion, and output fusion is equivalent to decision-level fusion or late fusion (Boulahia et al., 2021; Ramachandram and Taylor, 2017; Li et al., 2022c).”
BackgroundInfluentialOuhami, M., Hafiane, A., Es-Saady, Y., El Hajji, M., & Canals, R. (2021) · Remote Sensing · doi:10.3390/rs13132486 · Cited by 264
Deep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261
“(PCA), Independent Components Analysis (ICA), and Canonical Correlation Analysis (CCA) [25].”
BackgroundMethodMedical deep learning—A systematic meta-review
Egger, J., Gsaxner, C., Pepe, A., Pomykala, K. L., Jonske, F., Kurz, M., Li, J., & Kleesiek, J. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106874 · Cited by 251
Artificial intelligence-based methods for fusion of electronic health records and imaging data
Mohsen, F., Ali, H., El Hajj, N., & Shah, Z. (2022) · Scientific Reports · doi:10.1038/s41598-022-22514-4 · Cited by 250
AI meets UAVs: A survey on AI empowered UAV perception systems for precision agriculture
Su, J., Zhu, X., Li, S., & Chen, W.-H. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.11.020 · Cited by 235
El-Sappagh, S., Abuhmed, T., Riazul Islam, S. M., & Kwak, K. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.05.087 · Cited by 229
“Multimodal data could be fused in different ways, and selecting the best modality combination and suitable fusion scheme is a challenging task [13,23].”
Qiu, S., Cui, X., Ping, Z., Shan, N., Li, Z., Bao, X., & Xu, X. (2023) · Sensors · doi:10.3390/s23031305 · Cited by 221
“…data from heterogeneous platforms, i.e., multimodal fusion, is for the purpose that complementary information could be extracted from each modality, thus yielding a richer representation that could be used to achieve higher-quality intelligent FDP, compared to using only a single modality [165].”
MethodGuo, J., Kurup, U., & Shah, M. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2926042 · Cited by 221
“Data collected from various types of sensors is greatly appreciated in deep multimodal learning, which is believed to enhance the inference performance of deep neural networks [133].”
BackgroundReview of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235
Modern views of machine learning for precision psychiatry
Chen, Z. S., Kulkarni, P. (., Galatzer-Levy, I. R., Bigio, B., Nasca, C., & Zhang, Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100602 · Cited by 210
“Empowered by cutting-edge deep learning techniques, emerging methods have been increasingly developed for deep multimodal fusion [275].”
BackgroundA Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“cross-modality associations and correlations between different modalities [51].”
BackgroundMultimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey
Wang, T., Zheng, P., Li, S., & Wang, L. (2024) · Advanced Intelligent Systems · doi:10.1002/aisy.202300359 · Cited by 195
Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification
Han, Z., Yang, F., Huang, J., Zhang, C., & Yao, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02005 · Cited by 188
“To integrate multiple types of data for decision making, multimodal learning has been widely explored recently [8, 48, 65].”
BackgroundMethodA Review on Methods and Applications in Multimodal Deep Learning
Jabeen, S., Li, X., Amin, M. S., Bourahla, O., Li, S., & Jabbar, A. (2023) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3545572 · Cited by 196
“[118] 2017 IEEE Human Activity Recognition, Medical Applications, Autonomous Systems Ours – – Multimodal Image Description, Multimodal Video Description, Multimodal VQA, Multimodal Speech Synthesis, Multimodal Emotion Recognition, Multimodal Event Detection, Multimodal Action Recognition, Multimodal Cross-Modal Retrie…”
BackgroundZhang, Y.-P., Zhang, X.-Y., Cheng, Y.-T., Li, B., Teng, X.-Z., Zhang, J., Lam, S., Zhou, T., Ma, Z.-R., Sheng, J.-B., et al. (2023) · Military Medical Research · doi:10.1186/s40779-023-00458-8 · Cited by 199
“Generally, there are three time nodes for multi-modality fusion, namely early fusion, intermediate fusion and late fusion [170], and the advantages of each are described in the “Limitations and suggestions” section.”
BackgroundImproving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training
Abavisani, M., Joze, H. R. V., & Patel, V. M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00126 · Cited by 190
“A multimodal recognition system is trained with multiple streams of data and classifies the multimodal observations during testing [35, 4] (Figure 1 (a)).”
BackgroundMethodShaban, A., & Yousefi, S. (2024) · Springer Optimization and Its Applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 189
DISTILLER: Encrypted traffic classification via multimodal multitask deep learning
Aceto, G., Ciuonzo, D., Montieri, A., & Pescapé, A. (2021) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2021.102985 · Cited by 180
“…we foresee multimodal multitask DL as the appealing means toward a sophisticated form of information fusion, termed intermediate fusion [36], overcoming these limitations by o ff ering a flexible tool for practical ET classification, able to solve multiple tasks by processing e ff ectively…”
MethodDeep Multimodal Subspace Clustering Networks
Abavisani, M., & Patel, V. M. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2875385 · Cited by 181
“sion, it is a common practice to aggregate the weak or correlated modalities at earlier stages and combine the remaining strong modalities at the in-depth stages [39].”
BackgroundNascita, A., Montieri, A., Aceto, G., Ciuonzo, D., Persico, V., & Pescapé, A. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3098157 · Cited by 177
Liu, Y., Guo, L., Gao, H., You, Z., Ye, Y., & Zhang, B. (2022) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2021.108068 · Cited by 174
Emotion Recognition From Multiple Modalities: Fundamentals and methodologies
Zhao, S., Jia, G., Yang, J., Ding, G., & Keutzer, K. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3106895 · Cited by 165
“There have been several reviews/surveys on MER related topics [4, 6, 7, 8, 9].”
InfluentialDigital data sources and methods for conservation culturomics
Correia, R. A., Ladle, R., Jarić, I., Malhado, A. C. M., Mittermeier, J. C., Roll, U., Soriano‐Redondo, A., Veríssimo, D., Fink, C., Hausmann, A., et al. (2021) · Conservation Biology · doi:10.1111/cobi.13706 · Cited by 170
“Accounting for multimodality (i.e., the presence of more than 1 data type) is an emerging topic of research in automated content analysis (Ramachandram & Taylor 2017) that is likely to be of relevance for conservation culturomics.”
BackgroundA comprehensive review on deep learning approaches in wind forecasting applications
Wu, Z., Luo, G., Yang, Z., Guo, Y., Li, K., & Xue, Y. (2022) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12076 · Cited by 161
Multimodal learning with graphs
Ektefaie, Y., Dasoulas, G., Noori, A., Farhat, M., & Zitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00624-6 · Cited by 166
Adversarial machine learning: a taxonomy and terminology of attacks and mitigations
Vassilev, A., Oprea, A., Fordyce, A., & Anderson, H. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 147
An overview of deep learning methods for multimodal medical data mining
Behrad, F., & Saniee Abadeh, M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117006 · Cited by 159
Towards Robust Pattern Recognition: A Review
Zhang, X.-Y., Liu, C., & Suen, C. Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2989782 · Cited by 200
“Recently, due to the development of deep learning which learns hierarchical representations, the intermediate-level fusion [223] is used to dynamically integrate modalities at different levels with an automatically learned and optimized fusion structure.”
BackgroundMulti-modality approaches for medical support systems: A systematic review of the last decade
Salvi, M., Loh, H. W., Seoni, S., Barua, P. D., García, S., Molinari, F., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102134 · Cited by 144
“Deep neural networks can learn hierarchical, abstract data representations that embed modality-specific topology and statistical dependencies [13].”
BackgroundXu, Y., & Chen, H. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01942 · Cited by 134
Deep Learning for Inertial Positioning: A Survey
Chen, C., & Pan, X. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3381161 · Cited by 185
Multimodal Categorization of Crisis Events in Social Media
Abavisani, M., Wu, L., Hu, S., Tetreault, J., & Jaimes, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01469 · Cited by 188
FusionM4Net: a multi-stage multi-modal learning algorithm for multi-label skin lesion classification
Tang, P., Yan, X., Nan, Y., Xiang, S., Krammer, S., & Lasser, T. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102307 · Cited by 131
Wang, M., Tan, K., Jia, X., Wang, X., & Chen, Y. (2020) · Remote Sensing · doi:10.3390/rs12020205 · Cited by 141
“Such multi-modal data sets consist of data from different sensors observing a common phenomenon, and the goal is to use the data in a complementary manner in learning a complex task [22,23].”
BackgroundA review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126
“This has been well recognized in various other fields [258,259,303] but deserves more attention in bioimaging and calls for an integrative approach to bioimage analysis and bioinformatics.”
BackgroundMultimodal Classification: Current Landscape, Taxonomy and Future Directions
Sleeman, W. C., Kapoor, R., & Ghosh, P. (2023) · ACM Computing Surveys · doi:10.1145/3543848 · Cited by 124
“For the cross-modality style architectures, Ramachandram and Taylor [77] and Syed et al. [94] used the term intermediate, while Gao et al. [25] used deep shared modality.”
BackgroundMethodInfluentialMulti-Source and Multi-modal Deep Network Embedding for Cross-Network Node Classification
Yang, H., He, H., Zhang, W., Wang, Y., & Jing, L. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3653304 · Cited by 111
Multimodal Movie Recommendation System Using Deep Learning
Mu, Y., & Wu, Y. (2023) · Mathematics · doi:10.3390/math11040895 · Cited by 135
“Therefore, multimodal deep learning is paving the way for better presentations to be extracted from unstructured multiple types of data [30].”
BackgroundSelf-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 221
“Existing survey papers either focus solely on supervised multimodal learning [1], [2], [11], [12] or unimodal self-supervised learning [9], [10], [13], or specific subareas of SSML, e.g., vision-language pretraining [14].”
Semantics for Robotic Mapping, Perception and Interaction: A Survey
Garg, S., Sünderhauf, N., Dayoub, F., Morrison, D., Cosgun, A., Carneiro, G., Wu, Q., Chin, T.-J., Reid, I., Gould, S., et al. (2020) · Foundations and Trends® in Robotics · doi:10.1561/2300000059 · Cited by 105
Shi, Y., Zhao, Y., Liu, X., Zheng, F., Ou, W., You, X., & Peng, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3172716 · Cited by 113
“However, due to the modal93 ity discrepancy, data features of different modalities cannot 94 comprehensively have the same representational ability [2], 95 [3].”
BackgroundDiagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review
Shoeibi, A., Khodatars, M., Jafari, M., Ghassemi, N., Moridian, P., Alizadehsani, R., Ling, S. H., Khosravi, A., Alinejad‐Rokny, H., Lam, H., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.12.010 · Cited by 121
Machine Learning as a Tool for Hypothesis Generation
Ludwig, J., & Mullainathan, S. (2024) · The Quarterly Journal of Economics · doi:10.1093/qje/qjad055 · Cited by 186
Credit risk prediction of SMEs in supply chain finance by fusing demographic and behavioral data
Zhang, W., Yan, S., Li, J., Tian, X., & Yoshida, T. (2022) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2022.102611 · Cited by 125
Barua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 152
“Articles [6], [23], [24], and [25] present an overview of the methods of fusion with challenges and prospects.”
BackgroundMultimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101
“Atrey et al. (2010), Khaleghi et al. (2013), Lahat et al. (2014) and Ramachandram and Taylor (2017) have performed comprehensive reviews of these challenges and we provide these as additional references for the reader.”
BackgroundInfluentialPerformance of deep learning in mapping water quality of Lake Simcoe with long-term Landsat archive
Guo, H., Tian, S., Huang, J. J., Zhu, X., Wang, B., & Zhang, Z. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.11.023 · Cited by 105
Ensemble Learners of Multiple Deep CNNs for Pulmonary Nodules Classification Using CT Images
Zhang, B., Qi, S., Monkam, P., Li, C., Yang, F., Yao, Y., & Qian, W. (2019) · IEEE Access · doi:10.1109/access.2019.2933670 · Cited by 106
“Using deep CNNs as tools of feature extraction, the traditional machine learning (ML) algorithms are applied to fuse the hand-crafted features and CNN learned features and give the final prediction [11], [12].”
MethodArrowsmith, S., Trugman, D. T., MacCarthy, J., Bergen, K. J., Lumley, D., & Magnani, M. B. (2022) · Reviews of Geophysics · doi:10.1029/2021rg000769 · Cited by 124
“Deep neural network architectures are particularly appropriate due to their flexibility, allowing DNNs to accept a range of data types as inputs and learn joint representations that integrate multisource or multimodal data (Ramachandram & Taylor, 2017).”
BackgroundA systematic review of intermediate fusion in multimodal deep learning for biomedical applications
Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105509 · Cited by 95
Identifying bad software changes via multimodal anomaly detection for online service systems
Zhao, N., Chen, J., Yu, Z., Wang, H., Li, J., Qiu, B., Xu, H., Zhang, W., Sui, K., & Pei, D. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468543 · Cited by 89
“To tackle these challenges, we leverage the idea of multimodal learning to identify anomalies in an unsupervised manner, which has been widely used to learn information from cross modality (e.g., text and images) [59].”
InfluentialMultimodal deep learning approaches for single-cell multi-omics data integration
Athaya, T., Ripan, R. C., Li, X., & Hu, H. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad313 · Cited by 108
“They have demonstrated impressive predictive power [4] and offered several advantages over existing computational methods [5, 6].”
BackgroundIslam, M. M., & Iqbal, T. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3059624 · Cited by 92
“Another key challenge of multimodal learning methods is to fuse extracted salient multimodal features [40].”
BackgroundToward Open-World Electroencephalogram Decoding Via Deep Learning: A comprehensive survey
Chen, X., Li, C., Liu, A., McKeown, M. J., Qian, R., & Wang, Z. J. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3134629 · Cited by 109
“Multi-modal learning aims to utilize complementary or supplementary information from different modalities to complete a shared task or multiple related tasks [50].”
BackgroundGkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 117
“Recent advances in multimodal machine learning have been reviewed covering various directions of the field, such as representation, translation, alignment, fusion, and co-learning [19, 18].”
BackgroundMultimodality in meta-learning: A comprehensive survey
Ma, Y., Zhao, S., Wang, W., Li, Y., & King, I. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108976 · Cited by 99
Emotion fusion for mental illness detection from social media: A survey
Zhang, T., Yang, K., Ji, S., & Ananiadou, S. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.031 · Cited by 113
“In addition, directly appending emotion features to word-embedding features may not be efficient because their vector spaces are distinct, which will generate redundant input vectors [152].”
BackgroundDeep Learning in Biomedical Optics
Tian, L., Hunt, B., Bell, M. A. L., Yi, J., Smith, J. T., Ochoa, M., Intes, X., & Durr, N. J. (2021) · Lasers in Surgery and Medicine · doi:10.1002/lsm.23414 · Cited by 93
“DNNs provide powerful frameworks for integrating multimodal and multi‐dimensional data [15].”
BackgroundMethodA review of cancer data fusion methods based on deep learning
Zhao, Y., Li, X., Zhou, C., Peng, H., Zheng, Z., Chen, J., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102361 · Cited by 95
Richer fusion network for breast cancer classification based on multimodal data
Yan, R., Zhang, F., Rao, X., Lv, Z., Li, J., Zhang, L., Liang, S., Li, Y., Ren, F., Zheng, C., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01340-6 · Cited by 82
“From existing research, the correctness of this view is also proved, especially that feature-level fusion is better than decision-level fusion [11].”
BackgroundWang, J., Ji, J., Zhang, M., Lin, J., Zhang, G., Gong, W., Cen, L., Lu, Y., Huang, X., Huang, D., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.8758 · Cited by 97
Dehghan, A., Razzaghi, P., Abbasi, K., & Gharaghani, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120754 · Cited by 93
Multimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 95
“Multimedia Detection: An important research area that heavily utilizes multimedia data is human activity detection [58].”
BackgroundDeep learning–driven multi-omics analysis: enhancing cancer diagnostics and therapeutics
Zhang, J., Che, Y., Liu, R., Wang, Z., & Liu, W. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf440 · Cited by 112
“DL, with its ability to handle high-dimensional, heterogeneous, and unstructured data, as well as its capability to discover complex relationships in multimodal data, offers a potential solution to this challenge [10, 11].”
Dynamic Routing-based Multimodal Neural Network for Multi-sensory Fault Diagnosis of Induction Motor
Fu, P., Wang, J., Zhang, X., Zhang, L., & Gao, R. X. (2020) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2020.04.009 · Cited by 82
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge
Zhao, L., Peng, X., Chen, Y., Kapadia, M., & Metaxas, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00656 · Cited by 96
A Survey on Multi-modal Summarization
Jangra, A., Mukherjee, S., Jatowt, A., Saha, S., & Hasanuzzaman, M. (2023) · ACM Computing Surveys · doi:10.1145/3584700 · Cited by 116
“Although quite a few survey papers were written for uni-modal summarization tasks including surveys on text summarization [45, 49, 64, 120, 165, 181] and video summarization [8, 61, 81, 114, 149], and a few survey papers covering multi-modal research [4, 6, 63, 134, 150, 158].”
BackgroundHu, H., Wang, W., Chen, L., Ruan, S., Chen, S., Li, X., Lu, M., Xie, X., & Kuang, M. (2021) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.15522 · Cited by 82
Li, J., & Wang, Q. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.018 · Cited by 77
Pei, X., Zuo, K., Li, Y., & Pang, Z. (2023) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00225-6 · Cited by 77
Robust Deep Multi-modal Learning Based on Gated Information Fusion Network
Kim, J., Koh, J., Kim, Y., Choi, J., Hwang, Y., & Choi, J. W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_6 · Cited by 102
Ahmad, Z., & Khan, N. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2947446 · Cited by 73
“A major difficulty is in deciding at which level of information fusion: early fusion, feature level fusion and decision/late fusion, the modalities should be fused [10].”
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.