Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 501–600 of 1,002
Scale-wised feature enhancement network for change captioning of remote sensing images
Zhang, F., Zhang, W., Xia, K., & Feng, H. (2024) · International Journal of Remote Sensing · doi:10.1080/01431161.2024.2380544 · Cited by 10
“In addition, as the bridge between image and natural language, study of RSICC can be a reference of image retrieval and multi-modal learning (Ramachandram and Taylor 2017).”
Machine learning predicts pulmonary Long Covid sequelae using clinical data
Cordelli, E., Soda, P., Citter, S., Schiavon, E., Salvatore, C., Fazzini, D., Clementi, G., Cellina, M. I., Cozzi, A., Bortolotto, C., et al. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02745-3 · Cited by 14
“Although deep-learning-based multimodal learning should offer several advantages over conventional machine-learning methods [45], it needs a certain amount of training data that are not available in this application.”
Advancing Forex prediction through multimodal text-driven model and attention mechanisms
Dakalbab, F. M., Kumar, A., Talib, M. W. A., & Nasir, Q. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200518 · Cited by 8
Predicting Brain Degeneration with a Multimodal Siamese Neural Network
Ostertag, C., Beurton‐Aimar, M., Visani, M., Urruty, T., & Bertet, K. (2020) · International Conference on Image Processing Theory, Tools and Applications (IPTA) · doi:10.1109/ipta50016.2020.9286657 · Cited by 9
“’s review [4], focusing on the distinction between early fusion by concatenation or Principal Component Analysis (PCA) before the model, late fusion by pooling individual decisions from”
BackgroundTransmission Line Fault-Cause Identification Based on Hierarchical Multiview Feature Selection
Jian, S., Peng, X., Yuan, H., Lai, C. S., & Lai, L. L. (2021) · Applied Sciences · doi:10.3390/app11177804 · Cited by 8
“Unlike traditional feature selection, multiview feature selection treats multiview data as inherently related and ensures that complementary information from other views is exploited [25,26].”
BackgroundA study on specific learning algorithms pertaining to classify lung cancer disease
Saminathan, M., Manikandan, R., Kumar, A., Kulandaivel, R., Khanna, A. K., & Singh, P. (2021) · Expert Systems · doi:10.1111/exsy.12797 · Cited by 9
“Deep convolutional neural networks (CNNs) are a tool for extracting the features and conventional machine learning (ML) methods were included to synthesize hand-crafted and neural network characteristics and gives the endmost prediction (Ramachandram & Taylor, 2017).”
MethodA Combined System Metrics Approach to Cloud Service Reliability Using Artificial Intelligence
Chhetri, T. R., Dehury, C. K., Lind, A., Srirama, S. N., & Fensel, A. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6010026 · Cited by 14
A Multimodal Data Fusion-Based Deep Learning Approach for Drug-Drug Interaction Prediction
Huang, A., Xie, X., Wang, X., & Peng, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-23198-8_25 · Cited by 8
Multi-antenna intelligent spectrum sensing in the presence of non-Gaussian interference
Liu, M., Zhang, X., Chen, Y., & Tan, H. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2023.104135 · Cited by 8
Joint SNR and Rician K-Factor Estimation Using Multimodal Network Over Mobile Fading Channels
Tamura, K., Kojima, S., Trinh, P. V., Sugiura, S., & Ahn, C.-J. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3412054 · Cited by 8
Literature review on vulnerability detection using NLP technologyPreprint
Wu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11230 · Cited by 21
“CodeBERT uses bimodal (bimodal) [101]–[103] to train the model, where bimodal refers to natural language (NL) and programming language (PL), where NL refers to the program code Natural language annotations.”
BackgroundWang, Y., Wu, Z., Dai, J. C., Morgan, T., Garbens, A., Kominsky, H. D., Gahan, J. C., & Larson, E. C. (2023) · Journal of Robotic Surgery · doi:10.1007/s11701-023-01657-0 · Cited by 7
“It has been shown to increase accuracy and generalization in many applications [16].”
BackgroundZhu, Y., Li, H., Zhen, T., & Li, Z. (2023) · Applied Sciences · doi:10.3390/app13137655 · Cited by 8
Xu, X., Tu, W., & Yang, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102749 · Cited by 21
Data Augmentation-Based Enhancement for Efficient Network Traffic Classification
Shin, C.-Y., Choi, Y., & Kim, M. (2024) · IEEE Access · doi:10.1109/access.2024.3525000 · Cited by 12
Multimodal U-Net: A Novel Approach for 2-D Inversion of Magnetotelluric Data
Li, X., Dong, Y., He, J., Yang, X., Zhou, W., & Hu, X. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3547157 · Cited by 10
“The advantages of MDL have been witnessed in various fields, including medical applications, human activity recognition, and autonomous systems [34].”
Early fusion for Autism classification with biomarker detection using rs-fMRI and phenotypic data
Chawla, V., & Rana, B. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108020 · Cited by 9
A Novel Human-in-the-Loop Multimodal Intention Fusion Method for Human–Robot Interaction
Jiang, S., Zhang, W., Liu, J., Li, H., Wang, Z., Zhou, Y., & He, B. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3580469 · Cited by 11
“Moreover, the fusion e ff ect is a ff ected by the feature extraction method, which has certain limitations [22].”
Performance benchmarking of multimodal data-driven approaches in industrial settings
Altinses, D., & Schwung, A. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100691 · Cited by 10
Karimi, Z., Karamad, M., & Rahmany, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111642 · Cited by 8
Huang, A., Zhang, F., & Song, C. (2025) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2025.3589199 · Cited by 8
HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation
Cheng, H., Wang, T., Shi, G., Zhao, Z., & Fu, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00093 · Cited by 15
“Frameworks using hierarchical and adversarial training [19, 22, 50] have shown promise for joint feature learning, proving effective in enhancing the realism of generated motions.”
Ding, Q., Wan, G., Wang, N., & Ma, X. (2023) · IEEE Microwave and Wireless Technology Letters · doi:10.1109/lmwt.2023.3340105 · Cited by 7
“In DL society, to characterize an entity by the DL model, several modalities with totally different statistical properties could be exploited by multimodal (MM) technology at the same time to enhance the performance of the DL model [15], [16], [17].”
MethodMultimodal data fusion techniques in smart healthcare
Karbout, K., Ghazouani, M. E., Lachgar, M., & Hrimech, H. (2024) · International Conference on Global Aeronautical Engineering and Satellite Technology (GAST) · doi:10.1109/gast60528.2024.10520803 · Cited by 9
“Late fusion methods include ensemble learning [17] [18] [19] and deep late fusion [20] [21] [22].”
BackgroundEnd-to-End Clustering Enhanced Contrastive Learning for Radiology Reports Generation
Liu, X., Xin, J., Shen, Q., Li, C., Huang, Z., & Wang, Z. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3449876 · Cited by 7
Singh, S. K., & Chaturvedi, A. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109225 · Cited by 11
MEMTD: Encrypted Malware Traffic Detection Using Multimodal Deep Learning
Zhang, X., Lu, J., Sun, J., Xiao, R., & Jin, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-09917-5_24 · Cited by 7
Estimation of Rayleigh flat channel coefficients using deep learning
Drakshayini, M. N., & Kounte, M. R. (2022) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.4651 · Cited by 6
Obstacles Avoidance of Self-driving Vehicle using Deep Reinforcement Learning
Radwan, M. O., Sedky, A. A. H., & Mahar, K. (2021) · International Conference on Computer Theory and Applications (ICCTA) · doi:10.1109/iccta54562.2021.9916640 · Cited by 11
“Ramachandram and Taylor [10] provide an overview on deep multi-modal learning, and mention its applications in diverse research fields, such as robotic grasping and human action recognition.”
BackgroundCOVID-19 Detection via Fusion of Modulation Spectrum and Linear Prediction Speech Features
Zhu, Y. M., Tiwari, A. S., Monteiro, J., Kshirsagar, S., & Falk, T. H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3265603 · Cited by 10
“This architecture is favoured when errors made by different systems are mutually exclusive [38].”
BackgroundVisual–tactile fusion object classification method based on adaptive feature weighting
Zhang, P., Bai, L., Shan, D., Wang, X., Li, S., Zou, W., & Chen, Z. (2023) · International Journal of Advanced Robotic Systems · doi:10.1177/17298806231191947 · Cited by 13
Forest Fire Prediction Using Multi-Source Deep Learning
Mutakabbir, A., Lung, C., Ajila, S. A., Zaman, M., Naik, K., Purcell, R., & Sampalli, S. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-52265-9_9 · Cited by 7
Bakhshi, A., Wong, A. S. W., & Chalup, S. K. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200190 · Cited by 9
“…on DNNs. Numerous researchers in academia, but also in industry are interested in this active area of research, and it has resulted in several groundbreaking improvements for many practical problems such as computer vision, human-machine interaction, speech, and natural language processing [29].”
BackgroundNeuralIO: Indoor?Outdoor Detection via Multimodal Sensor Data Fusion on Smartphones
Wang, L., Sommer, L., Zhou, Y., Huang, Y., Wang, J., Riedel, T., & Beigl, M. (2020) · Sensors and Materials · doi:10.18494/sam.2020.2586 · Cited by 7
How to Leverage Multimodal EHR Data for Better Medical Predictions?Preprint
Yang, B., & Wu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15763 · Cited by 7
“resentation is nontrivial, as naively adding more features could result in worse performances (Ramachandram and Taylor, 2017).”
Test method of laser paint removal based on multi-modal feature fusion
Huang, H., Hao, B., Ye, D., Gao, H., & Li, L. (2022) · Journal of Central South University · doi:10.1007/s11771-022-5163-x · Cited by 7
“Multi-modal Fusion Technology (MFT) in deep learning is the process by which models process different forms of data when analyzing and identifying tasks [13].”
BackgroundQuantifying the advantage of multimodal data fusion for survival prediction in cancer patientsPreprint
Νικολάου, Ν., Salazar, D., RaviPrakash, H., Gonçalves, M., Mulla, R., Burlutskiy, N., Markuzon, N., & Jacob, E. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.08.574756 · Cited by 7
Ahmad, I., & Ahmed, T. (2026) · Sustainability · doi:10.3390/su18020794 · Cited by 9
Development of a Bayesian multimodal model to detect biomarkers in neuroimaging studies
Bhaumik, D. K., Wang, Y., Yen, P., & Ajilore, O. (2023) · Frontiers in Neuroimaging · doi:10.3389/fnimg.2023.1147508 · Cited by 7
“We assessed a late-fusion approach (Ramachandram and Taylor, 2017) by collecting individual restingstate average correlations of pairwise BOLD signal for FC and fiber counts for SC.”
MethodShi, Q., Xu, W., & Miao, Z. (2024) · Journal of Electronic Imaging · doi:10.1117/1.jei.33.4.043042 · Cited by 6
“32 This approach has been widely used in various social media projects, including emergency response, 9,32 emotional recognition, 33 and fake news detection.”
Multi-modal deep learning framework for damage detection in social media posts
Zhang, J., Liao, M., Wang, Y., Huang, Y., Chen, F., & Makiko, C. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2262 · Cited by 7
“Multimodal learning addresses this by evolving or tailoring machine learning algorithms to assimilate information from various modalities ( Ramachandram & Taylor, 2017 ).”
BackgroundWang, J., Cai, J., Tang, W., Dudurych, I., van Tuinen, M., Vliegenthart, R., & van Ooijen, P. M. A. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102553 · Cited by 7
Lu, S., Zhang, X., Su, Y., Liu, X., & Yu, L. (2025) · Annals of Operations Research · doi:10.1007/s10479-025-06612-w · Cited by 8
“Multimodal fusion meth-ods can be divided into three categories based on the fusion stage: data-level, feature-level, and decision-level fusion (Ramachandram & Taylor, 2017).”
Zhang, L., Jian, L., Long, Y., Ren, Z., Calhoun, V. D., Passos, I. C., Tian, X., & Xiang, Y. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106201 · Cited by 10
Intelligent analysis of chest X-ray based on multi-modal instruction tuning
Yao, J., Wang, J., Xiao, Z., Hao, X., & Jiang, X. (2025) · Meta-Radiology · doi:10.1016/j.metrad.2025.100172 · Cited by 7
Efficient Multivariate Data Fusion for Misinformation Detection During High Impact Events
Damasceno, L. P., Shafer, A., Japkowicz, N., Cavalcante, C. C., & Boukouvalas, Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18840-4_19 · Cited by 6
Rodrigues, L. D. S., Filho, E. C., Sakiyama, K. M., Santos, M. F., Jank, L., Carromeu, C., Silveira, E., Matsubara, E. T., Marcato, J., & Gonçalves, W. N. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.107957 · Cited by 6
Machine Learning Approaches for Stem Cells
Mazalan, M., Do, T.-D., Zaman, W. S. W. K., & Ramlan, E. I. (2023) · Current Stem Cell Reports · doi:10.1007/s40778-023-00228-1 · Cited by 9
“Another useful avenue to explore is multi-modal learning [90, 91] by combining various types of data such as gene expression, epigenetic modifications, imaging, or physiological measurements.”
BackgroundDeep Learning-Based Positioning With Multi-Task Learning and Uncertainty-Based Fusion
Foliadis, A., Castañeda, M. H., Stirling-Gallacher, R. A., & Thomä, R. S. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3441521 · Cited by 15
“However, when considering changes in the UE-BS channel between the training phase and deployment phase, e.g., due to a blockage of the line of sight (LOS) between a UE and a BS, late fusion can benefit from uncertainty estimation [21].”
BackgroundAlmatrafi, S., Abbas, Q., & Ibrahim, M. E. A. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19104-z · Cited by 7
“In addition, computational power has taken forward according to the rapid improvements in both CPUs and GPUs [33, 34].”
Multi-Modal Data Fusion Based Cardiac Disease Prediction using Late Fusion and 2D CNN Architectures
Patel, K. K., Kanodia, A., Kumar, B., & Gupta, R. K. (2024) · International Conference on Signal Processing and Integrated Networks (SPIN) · doi:10.1109/spin60856.2024.10512195 · Cited by 8
Visual-guided hierarchical iterative fusion for multi-modal video action recognition
Zhang, B., Zhang, Y., Zhang, J., Sun, Q., Wang, R., & Zhang, Q. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.10.003 · Cited by 7
Ortmann, T., Wang, Q., & Putzar, L. (2025) · IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR) · doi:10.1109/aixvr63409.2025.00048 · Cited by 9
“2) Intermediate Fusion: For intermediate fusion, we relied on the EfficientNet-B0’s final GlobalAveragePooling layer, which extracts 1,280 features from an input image, and on the MLP’s first Dense layer, which extracts 128 features from each FEA input.”
Audio–visual self-supervised representation learning: A survey
Alsuwat, M. S., Al-Shareef, S., & Alghamdi, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129750 · Cited by 10
Orthogonal Sequential Fusion in Multimodal Learning
Labbaki, S., & Mináry, P. (2025) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion65864.2025.11124022 · Cited by 9
“Even in the face of increasingly complex models and methodologies, these traditional fusion techniques remain prevalently employed, often serving as the base upon which these more advanced models are built [9].”
NeuralIO: Indoor Outdoor Detection via Multimodal Sensor Data Fusion on Smartphones
Wang, L., Sommer, L., Riedel, T., Beigl, M., Zhou, Y., & Huang, Y. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-51005-3_13 · Cited by 10
Learning Deep Features for Robotic Inference From Physical Interactions
Dehban, A., Zhang, S., Cauli, N., Jamone, L., & Santos-Victor, J. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3152383 · Cited by 8
“In robotics and machine learning, multimodality commonly refers to having more than one sensor modality [30], [31], such as vision, sound, tactile, joint trajectories, etc.”
BackgroundWang, Y., Zhang, W., Sun, J., Wang, L., Song, X., & Zhao, X. (2022) · Computational and Mathematical Methods in Medicine · doi:10.1155/2022/1924906 · Cited by 7
“Deep learning is a new direction in the field of machine learning that models high-level abstractions in input data with hierarchies and multiple layers [32, 33].”
BackgroundEstimating the carbon content of oceans using satellite sensor data
Sooknanan, A., & Hosein, P. A. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00647-7 · Cited by 6
“Future work may potentially explore the usage of multi-modal approaches [44], particularly in the field of deep-learning [33], which has proven to be useful in handling heterogeneous inputs across multiple fields, including remote-sensing [17, 38, 49] Notwithstanding, the relationship between SST and SSS holds as vali…”
BackgroundMultimodal AutoML via Representation Evolution
Škrlj, B., Bevec, M., & Lavrač, N. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010001 · Cited by 6
“Currently, most solutions considering multimodal data are based on deep learning [29], due to its capacity for representation learning of texts and images.”
BackgroundGraMDTA: Multimodal Graph Neural Networks for Predicting Drug-Target Associations
Yella, J., Ghandikota, S. K., & Jegga, A. G. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995245 · Cited by 6
“Multi-modal representation learning on the other hand offers robustness when any of the modality is incomplete or corrupted and further the modalities act complementary to each other [16].”
BackgroundFang, C., Liang, F., Li, T., & Guan, F. (2024) · Future Internet · doi:10.3390/fi16060213 · Cited by 6
SpikeCV: open a continuous computer vision era
Zheng, Y., Zhang, J., Zhao, R., Ding, J., Chen, S., Wu, W., Xiong, R., Yu, Z., & Huang, T. (2026) · Science China Information Sciences · doi:10.1007/s11432-023-4565-6 · Cited by 15
“Deep learning has been successfully applied to various vision/natural language tasks [20], [32], [36], but its training requires a large amount of training data.”
Developments in AI and Machine Learning for Neuroimaging
O’Sullivan, S., Jeanquartier, F., Jean-Quartier, C., Holzinger, A., Shiebler, D., Moon, P., & Angione, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50402-1_18 · Cited by 7
“There are two significant challenges faced by medical applications community when using multimodal deep learning, namely the difficulty in obtaining sufficient labelled data, and class imbalance [56].”
BackgroundMenegotto, A. B., & Cazella, S. C. (2021) · Studies in fuzziness and soft computing · doi:10.1007/978-3-030-70111-6_13 · Cited by 7
Weakly Supervised Multimodal 30-Day All-Cause Mortality Prediction for Pulmonary Embolism Patients
Cahan, N., Marom, E. M., Soffer, S., Barash, Y., Konen, E., Klang, E., & Greenspan, H. K. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761700 · Cited by 7
“This choice of fusing various representations at different depths is perhaps the most powerful and flexible aspect of deep multimodal fusion as opposed to other fusion techniques [15].”
BackgroundA Multi-View learning approach to enhance automatic 12-Lead ECG diagnosis performance
Choi, J.-W., Hong, D.-Y., Jung, C., Hwang, E., Park, S.-H., & Roh, S. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106214 · Cited by 8
XAI-based Data Visualization in Multimodal Medical DataPreprint
Sharma, S., Singh, M., Mcdaid, L. J., & Bhattacharyya, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.11.664302 · Cited by 16
A central multimodal fusion framework for outdoor scene image segmentation
Zhang, Y., Morel, O., Seulin, R., Mériaudeau, F., & Sidibé, D. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10357-y · Cited by 5
Single Modality and Joint Fusion for Emotion Recognition on RAVDESS Dataset
Haddad, S., Dâassi, O., & Belghith, S. M. (2024) · SN Computer Science · doi:10.1007/s42979-024-03020-y · Cited by 5
“Late fusion (decision-level fusion) [31] refers to the process of leveraging predictions from multiple models to make a final decision.”
Multimodal Multitask Control Plane Verification Framework
Dai, Y., Zhang, H., Wang, J., & Liao, J. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3442298 · Cited by 5
“Existing feature-level fusion methods include early feature fusion, late feature fusion, and mixed feature fusion [32], [53].”
Xu, C., Ding, J., Wang, B., Qiao, Y., Zhang, L., & Zhang, Y. F. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100748 · Cited by 14
Dai, M., Jurszyk, J., Gillott, C., Sun, K., Lanau, M., Liu, G., & Tingley, D. D. (2025) · Journal of Industrial Ecology · doi:10.1111/jiec.70048 · Cited by 5
Wang, T., Wang, A., Zhang, Y., Liu, X., Fan, N., Yuan, S., Du, P., Wu, Q., Chen, R., Xi, Y., et al. (2025) · Neurosurgical FOCUS · doi:10.3171/2025.4.focus24670 · Cited by 8
A multi-objective optimization approach for rocket nozzle design based on hybrid surrogate model
Tian, X., Sun, Z., Zhu, Y., Hu, Y., Zhang, Y., & Qin, C. (2025) · Physics Letters A · doi:10.1016/j.physleta.2025.131200 · Cited by 5
Zhang, X., Zhang, H., & Xu, M. (2026) · AI · doi:10.3390/ai7020063 · Cited by 9
“Differences in signal scale, sampling frequency, physiological latency, and data availability across modalities can induce gradient imbalance and modality dominance, leading to unstable training or suboptimal fusion [203].”
MTFR: An universal multimodal fusion method through Modality Transfer and Fusion Refinement
Guo, X., Tian, S., Yu, L., He, X., & Wang, Z. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108844 · Cited by 8
CCFN: Depression Detection via Multimodal Fusion with Complex-valued Capsule Network
Lei, J., Yang, Q., Li, B., & Zhang, W. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650262 · Cited by 8
“The multimodal states of human beings, such as facial expression, voice, and speech, contain a wealth of information for recognizing depression status [28].”
BackgroundXi, Y., Chen, R., Wang, T., Zang, L., Jiao, S., Xie, T., Wu, Q., Wang, A., Fan, N., Yuan, S., et al. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1479187 · Cited by 7
“Next, intermediate multimodal fusion CNN-based classifiers will be designed to integrate the information of axial CT image and sagittal MRI (26).”
BackgroundA Neural Architecture Search for Automated Multimodal Learning
Singh, A., & Nair, H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118051 · Cited by 8
A Survey of Full-Cycle Cross-Modal Retrieval: From a Representation Learning Perspective
Wang, S., Zhu, L., Shi, L., Mo, H., & Tan, S. (2023) · Applied Sciences · doi:10.3390/app13074571 · Cited by 9
A multiparameter radiomic model for accurate prognostic prediction of glioma
Li, Y., Li, B., Yang, C., Deng, Z., Zhang, X., Pin, X., Su, X., Zeng, F., Mehrabi, M. Q. U., Yue, Q., et al. (2023) · MedComm – Future Medicine · doi:10.1002/mef2.41 · Cited by 5
Bridging High-Quality Audio and Video Via Language for Sound Effects Retrieval from Visual Queries
Wilkins, J., Salamon, J., Fuentes, M., Bello, J. P., & Nieto, O. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248113 · Cited by 12
“Multimodal machine learning methods have become popular across the visual, audio, and text domains [3, 4], due to their ability to learn from the correspondence between signals in different modalities, often via contrastive learning [5, 6, 7], and the growing availability of large multimedia datasets [8, 9, 10, 11, 12…”
BackgroundMultimodal fuzzy granular representation and classification
Han, F., Zhang, X., He, L., Kong, L., & Chen, Y. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05080-8 · Cited by 6
“With the advent of the big data era, it is a fact that heterogeneous and multimodal data widely appear [1].”
BackgroundMultimodal Fusion Methods with Vision Transformers for Remote Sensing Semantic Segmentation
Morelli, V. G., Barbato, M. P., Piccoli, F., & Napoletano, P. (2023) · Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing · doi:10.1109/whispers61460.2023.10430788 · Cited by 6
“Multimodal Learning (MML) represents a versatile approach to constructing AI models capable of extracting and correlating information from various data sources, which are commonly referred to as modalities [1].”
BackgroundAdvancements in accurate speech emotion recognition through the integration of CNN-AM model
Adebiyi, M. O., Adeliyi, T. T., Olaniyan, D., & Olaniyan, J. (2024) · TELKOMNIKA (Telecommunication Computing Electronics and Control) · doi:10.12928/telkomnika.v22i3.25708 · Cited by 6
Jameel, S. S., Raazi, S. M. K.-U.-R., & Jameel, S. M. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-62871-9_17 · Cited by 8
Zhang, Z., Zhang, W., He, C., Xie, J., Liang, F., Zhao, Y., Lilian, T., Lai, S., Jiang, X., Wei, X., et al. (2024) · Medical Physics · doi:10.1002/mp.17401 · Cited by 5
Bermudez, C. L., Bao, S., Petersen, K. J., Lopez, A. M., Reid, J. A., Plassard, A. J., Zald, D. H., Claassen, D. O., Dawant, B. M., & Landman, B. A. (2019) · Journal of medical imaging · doi:10.1117/1.jmi.6.4.044007 · Cited by 7
Stress Detection using CNN Fusion
Radhika, K. R., & Murthy, O. V. R. (2021) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon54134.2021.9707438 · Cited by 5
Trustworthiness assessment in multimodal human-robot interaction based on cognitive load
Kırtay, M., Öztop, E., Kuhlen, A. K., Asada, M., & Hafner, V. V. (2022) · IEEE RO-MAN · doi:10.1109/ro-man53752.2022.9900730 · Cited by 10
“In this work, the term modality refers to an employed sensor to record data that are associated with different aspects (e.g., audio and visual) of interactive experiments [26], [27].”
BackgroundBio-Inspired Presentation Attack Detection for Face Biometrics
Tsitiridis, A., Conde, C., Gómez-Ayllón, B., & Cabello, E. (2019) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2019.00034 · Cited by 6
“Multiple lines of research have been explored for CNN architectures in last two decades and a huge number of different methods are proposed in references (Canziani et al., 2016; Ramachandram and Taylor, 2017).”
BackgroundCarvalho, R., Morgado, A. C., Andrade, C., Nedelcu, T., Carreiro, A. V., & Vasconcelos, M. J. M. (2021) · Diagnostics · doi:10.3390/diagnostics12010036 · Cited by 5
“To merge this information, we employ a late feature fusion technique [63].”
MethodEmbraceNet for Activity: A Deep Multimodal Fusion Architecture for Activity RecognitionPreprint
Choi, J. H., & Lee, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13918 · Cited by 5
“Along with the increased number of sensors, many modality fusion methods have been proposed to efficiently handle the characteristics of the multimodal data [16].”
Multimodal Strategy To Defend Mobile Devices Against Vishing Attacks
Mishra, M., Jain, S. K., Suraj Patel, M., Mani, S., Boragule, A., Sahni, N., & Nair, R. C. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3690683 · Cited by 7
“Utilising multimodality[7–10] data to solve a problem is experimented on different domains like voice recognition Figure 1: Illustration of a real world use case where attacker pretends to be a banker over voice call, requests user for one-time password required to unlock his account, shows: a)…”
Multi-modal classification of retinal disease based on convolutional neural network
Pan, H., Miao, J., Yu, J., Li, J., Wang, X., & Feng, J. (2025) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/adeb92 · Cited by 4
An Adaptive Transfer Learning Framework for Multimodal Autism Spectrum Disorder Diagnosis
Malik, W., Fahiem, M. A., Khan, J., Jung, Y., & Alturise, F. (2025) · Life · doi:10.3390/life15101524 · Cited by 7
Wang, C., Zhu, M. S., & Zakaria, S. A. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-17656-0 · Cited by 4
Multimodal Fusion of Chest X-Rays and Blood Biomarkers for Automated Silicosis Staging
Priego, B., Sopo-Lambea, I., Khalili, E., Martín-Carrillo, A., Campos‐Caro, A., León‐Jiménez, A., & Morillo, D. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14228074 · Cited by 7
“Nonetheless, this early fusion is prone to overfitting when samples are limited [40] and may struggle to detect connections between the modalities if these connections only become clear at more abstract levels, as marginal representations are not specifically learned [41–43].”
Toward accurate breast cancer classification: A review of multi-modal machine learning approaches
Mathur, A., Dudhiyawala, A. M., Dey, S. R., & Saha, S. (2025) · Methods · doi:10.1016/j.ymeth.2025.10.011 · Cited by 5
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.