Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 701–800 of 1,002
Time‐Series Land Subsidence Monitoring Based on GeoDistillBERT‐BiLSTM Framework in Mine Areas
Li, M., Li, R., & Sha, Z. (2025) · Transactions in GIS · doi:10.1111/tgis.70101 · Cited by 2
“The fusion of multi-source factors is typically regarded as a multimodal feature fusion problem due to the diverse forms of influencing factors (e.g., text, images) (Ramachandram and Taylor 2017).”
Chen, S., Huang, Z., Zhou, X., Li, H., Wang, C., & Zhou, S. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04978-0_47 · Cited by 4
Multimodal Industrial Anomaly Detection via Geometric Prior
Li, M., He, J., Li, G., Li, J., Wan, J., & Han, D. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3613708 · Cited by 6
“Existing MIAD methods can be categorized into early fusion, intermediate fusion, and late fusion based on the fusion stage [33], [34].”
MOSAIC: multimodal multistakeholder-aware visual art recommendation
Yilma, B. A., & Leiva, L. A. (2025) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-025-09435-3 · Cited by 6
“Traditionally, late fusion (i.e. at post-hoc) has been preferred over early fusion (i.e. at the feature level) because the fused models can be independent of each other, so each can use their own features, numbers of dimensions, etc. Ramachandram and Taylor (2017).”
Ammaiappan, S., Liang, G., Tan, C., & Dong, F. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3613327 · Cited by 2
“The widespread deployment of multisensory technology has created a diverse and sophisticated multi-modal database [21], [22].”
Topology-aware cross-modal learning with dynamic gradient modulation for survival prediction
Yuan, Y., Li, G., Jiang, N., Lv, J., Ye, C., Ji, D., Ren, Y., Luo, G., & Zhang, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130492 · Cited by 3
Huang, W., Zhang, P., Xu, D. S., Hu, J., & Yuan, Y. (2025) · Sustainability · doi:10.3390/su18010019 · Cited by 2
Bottom-up building exposure modeling with multimodal earth vision
Pelizari, P. A., Geiß, C., & Taubenböck, H. (2026) · Isprs Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.10.036 · Cited by 3
Huang, H., & Abbas, H. T. (2026) · Electronics · doi:10.3390/electronics15010204 · Cited by 2
Enhancing Fault Tolerance in Multimodal Learning: A VAE-Based Approach with Probabilistic Fusion
Altinses, D., & Schwung, A. (2025) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit66093.2025.11321387 · Cited by 2
“Ramachandram and Taylor [8] present a survey on deep learning methods for multimodal data fusion, emphasizing the integration of heterogeneous data sources to improve model performance.”
A Radar Echo Extrapolation Method Based on the ARUR Network with Multi-Source Data
Chen, J., Hu, Z., Yuan, S., Hua, J., Chen, X., Wang, S., & Li, G. (2025) · 气象学报:英文版 · doi:10.1007/s13351-025-5040-2 · Cited by 2
“Recently, another method borrowing from multimodal learning has been addressed to provide extrapolated microphysical quantity fields by integrating multisource data (Ramachandram and Taylor, 2017; Liu et al., 2025).”
Nanostructured electrochemical biosensors for real-time and portable health monitoring systems
Chowdhury, A. D., Sarkar, S., Dasgupta, U., & Ghosh, M. (2026) · Biosensors and Bioelectronics X · doi:10.1016/j.biosx.2026.100766 · Cited by 12
Qiu, Y., Chen, H., Xie, J., Liang, C., Hui, Y., Zhao, P., Guo, S., Bhattacharya, S., & Yu, J. (2026) · Fuel Processing Technology · doi:10.1016/j.fuproc.2026.108440 · Cited by 2
A survey on multimodal explainable Artificial Intelligence
Giovannoni, C., Metta, C., Monreale, A., & Rinzivillo, S. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200671 · Cited by 4
Multimodal machine learning in real estate appraisal: models, data, and trends
Huang, C., Liang, B., Li, Z., Wang, J., & Chen, F. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01204-8 · Cited by 3
“…surveys highlight that multi-level fusion mitigates the limitations of shallow concatenation by modelling non-linear cross-modal dependencies [78, 79] Hybrid fusion is frequently instantiated via attention-based or Transformer-based cross-modal blocks interleaved with modality-specific…”
Jiang, G.-F., Zhou, N., Wang, S.-M., & Ni, Y.-Q. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.111903 · Cited by 2
Cross-modal Assisted Training for Abnormal Event Recognition in Elevators
Chen, X., Gong, X., Cheng, M., Deng, Q., & Li, M. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479908 · Cited by 2
Star-Net: a Multi-Branch Convolutional Network for Multiple Source Image Segmentation
Placidi, G., Cinque, L., Nappi, M., Polsinelli, M., Sciarra, A., & Tortora, G. (2022) · International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) · doi:10.1109/sitis57111.2022.00034 · Cited by 3
“It is worth saying that the strategy used for the fusion is strongly task-dependent [6]–[8].”
BackgroundERDBF: Embedding-Regularized Double Branches Fusion for Multi-Modal Age Estimation
Wu, B., Lu, H., Chen, Z., Zhu, C., & Xu, S. (2023) · IEEE Access · doi:10.1109/access.2023.3275765 · Cited by 2
“Multi-modal fusion has been proven effective in some applications [26], such as person verification and emotion recognition.”
BackgroundMulti-Modality Tensor Fusion Based Human Fatigue Detection
Ha, J., Ryu, J., & Ko, J.-H. (2023) · Electronics · doi:10.3390/electronics12153344 · Cited by 2
Hierarchical Spatial-Temporal Network for Skeleton-Based Temporal Action Segmentation
Tan, C., Sun, T., Fu, T., Wang, Y., Xu, M., & Liu, S. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8549-4_3 · Cited by 2
Deep Multimodal-Based Recognition of Thai Finger Spelling with Two-Handed Postures
Vijitkunsawat, W., & Chantagram, P. (2024) · International Electrical Engineering Congress (iEECON) · doi:10.1109/ieecon60677.2024.10537957 · Cited by 2
“Data fusion is a distinct feature of multimodal learning [9] in which information from several data sources must be merged while developing a joint model [11].”
BackgroundA Deep Multimodal Representation Learning Framework for Accurate Molecular Properties Prediction
Yang, Y., Wang, Z., Ahadian, P., Jerger, A., Zucker, J., Feng, S., Cheng, F., & Guan, Q. (2024) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3649476.3660377 · Cited by 2
“This technique encompasses two primary strategies for fusing different modalities of data, known as early fusion and late fusion, distinguished by when the fusion of modalities takes place [21].”
MethodAutoAMS: Automated attention-based multi-modal graph learning architecture search
Al-Sabri, R., Gao, J., Chen, J., Olouladé, B. M., & Wu, Z. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106427 · Cited by 5
Detección de emociones en discursos utilizando machine learning
Miranda-Leon, M., & Toala, R. (2024) · 593 Digital Publisher CEIT · doi:10.33386/593dp.2024.4.2367 · Cited by 2
Novel AI Multimodal Approach for Combating Against Pulmonary Carcinoma
Poornima, G., & Anand, L. G. (2024) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet61516.2024.10593234 · Cited by 2
“Then, “the Foundation for the National Institutes of Health (FNIH)” provided support to expand the database with additional CT scans that were acquired from private businesses [11].”
BackgroundThe Optimal Choice of the Encoder–Decoder Model Components for Image Captioning
Bartosiewicz, M., & Iwanowski, M. (2024) · Information · doi:10.3390/info15080504 · Cited by 3
Enhancing Bioimpedance Tissue Classification with Elastography Sensor Data via Multimodal Learning
Ege, M., Hellwich, E., Krauβ, F., Lovasz, Z., Schüle, J., Veil, C., Sawodny, O., & Tarín, C. (2024) · Proceedings of IEEE Sensors/Proceedings of IEEE Sensors ... · doi:10.1109/sensors60989.2024.10784632 · Cited by 2
“In data-driven sensor fusion, data integration can occur in early, intermediate, or late stages [11], [12].”
A novel RSVP-based system using EEG and eye-movement for classification and localization
Wu, H., Li, F., Chu, W., Li, H., Ji, Y., Li, Y., Niu, Y., Wang, H., Chen, Y., & Shi, G. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107331 · Cited by 2
Random Forest-Supervised Manifold Alignment
Rhodes, J. S., & Rustad, A. G. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825663 · Cited by 7
Few-Shot Learning with Multimodal Fusion for Efficient Cloud–Edge Collaborative Communication
Gao, B., Liu, X., & Zhou, Q. (2025) · Electronics · doi:10.3390/electronics14040804 · Cited by 2
“While computationally efficient, it may miss intricate cross-modal interactions [33].”
Deng, L., Pei, J., Song, Y., Huo, W., Zhang, Y., & Huang, Y. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3550532 · Cited by 4
“Currently, valuable data resources are characterized by multimodality [59].”
Leroy, P., Abisset‐Chavanne, E., Pommier, R., & Montemurro, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111516 · Cited by 4
PAL: Prompting analytic learning with missing modality for multi-modal class-incremental learning
Yue, X., Chen, Y., Zhang, X., Gao, X., Feng, M., Lao, M., Zhuang, H., & Li, H. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113467 · Cited by 10
Multimodal Deep Learning in Semantic Image Segmentation: A Review
Raman, V., & Kumari, M. (2018) · International Conference on Cloud Computing and Internet of Things (CCIOT) · doi:10.1145/3291064.3291067 · Cited by 2
Improved SAR Imaging via Cross-Learning From Camera Images
Gishkori, S., Wright, D., Daniel, L., Gashinova, M. S., & Mulgrew, B. (2021) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2021.3054686 · Cited by 2
“, exploiting the framework of multi-modal fusion [11]–[13].”
PandaDB: Intelligent Management System for Heterogeneous Data
Shen, Z., Zhao, Z., Wang, H., Liu, Z., Hu, C., & Zhou, C. (2021) · International Journal of Software and Informatics · doi:10.21655/ijsi.1673-7288.00241 · Cited by 2
Wang, L., Wang, X., Hawbani, A., Xiong, Y., & Zhang, X. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11247-7 · Cited by 2
“Intermediate fusion [22] combines information in the middle part of the whole multimodal network, and employs it for multimodal reconstruction, which can be implemented by learning a new shared representation layer across separated networks.”
BackgroundDing, T., & Hirose, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-36711-4_34 · Cited by 2
A Collaborative Multi-modality Selection Method Based on Data Utility Assessment
Xiao, Y., Gu, Y., Wang, J., & Wu, T. (2019) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/smartworld-uic-atc-scalcom-iop-sci.2019.00120 · Cited by 3
“These applications also involve various kind of modalities, such as images, RGB-D video, audio, mocap, text, myoelectric, radar, GPS, etc [1].”
BackgroundA Survey of Multimodal Human-Machine Interface
Ding, W., Wang, C., Fang, B., Sun, F., & Shan, J. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-2336-3_35 · Cited by 2
Robust Symmetry Prediction with Multi-Modal Feature Fusion for Partial Shapes
Xi, J., Zheng, K., Zhong, Y., Li, L., Cai, Z., & Chen, J. (2022) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.030298 · Cited by 2
Hybrid piezoelectric-magnetic neurons: a proposal for energy-efficient machine learning
Scott, W., Jeffrey, J., Heard, B., Nikonov, D. E., Young, I. A., Manipatruni, S., Naeemi, A., & Iraei, R. M. (2018) · doi:10.1145/3190645.3190688 · Cited by 2
Kim, C., Bae, J., Baek, I., Jeong, J., Lee, Y. J., Park, K., Shim, S. H., & Kim, S. B. (2023) · IEEE Access · doi:10.1109/access.2023.3271282 · Cited by 3
“In recent multimodal deep learning methods, a deep learning structure suitable for each modality is used to extract feature vectors and concatenate them to solve problems [22].”
MethodMultiModal Data Challenge in Metaverse Technology
El-Din, D. M., Hassanein, A. E., & Darwish, A. (2023) · Studies in big data · doi:10.1007/978-3-031-29132-6_11 · Cited by 2
On combining image features and word embeddings for image captioning
Bartosiewicz, M., Iwanowski, M., Wiszniewska, M., Frączak, K., & Leśnowolski, P. (2023) · Annals of Computer Science and Information Systems · doi:10.15439/2023f997 · Cited by 2
“Image captioning methods combining text and visual data belong to the multi-modal machine-learning approaches [22], [40], [59].”
MethodExcavating Multimodal Correlation for Movie Recommendation System
Pati, U. S., Swain, S., Kumari, R., Mishra, V., & Bandyopadhyay, A. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10306669 · Cited by 2
“As a result, multimodal deep learning is opening the door for the extraction of improved representations from unorganised data from many types of sources [16].”
BackgroundDamasceno, L. P., Rexhepi, E., Shafer, A., Whitehouse, I., Japkowicz, N., Cavalcante, C. C., Corizzo, R., & Boukouvalas, Z. (2023) · Machine Learning · doi:10.1007/s10994-023-06424-8 · Cited by 4
Leveraging Bioclimatic Context for Supervised and Self-supervised Land Cover Classification
Leonhardt, J., Drees, L., Gall, J., & Roscher, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54605-1_15 · Cited by 2
“Improving neural networks by learning from multiple data modalities at once is an intensely studied topic in the field of deep learning [31].”
BackgroundHyperparameter Tuning by Evolutionary Algorithm for Object Detection on Multimodal Satellite Imagery
Gordienko, N., Gordienko, Y. G., Rokovyi, O., Alienin, O., & Stirenko, S. (2024) · Algorithms for intelligent systems · doi:10.1007/978-981-99-8438-1_25 · Cited by 2
Variational Gaussian process for multisensor classification problems
Rohani, N., Ruiz, P., Molina, R., & Katsaggelos, A. K. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.08.035 · Cited by 2
Deep Learning for Reintegrating Biology
Müller, R., Han, J., Chandrasekaran, S., & Bogdan, P. (2021) · Integrative and Comparative Biology · doi:10.1093/icb/icab015 · Cited by 2
“Of particular interest in this context would be deep multimodal learning paradigms [23] that could be a good fit to the multi-modal nature of many biological data sets.”
BackgroundCross-Modal Object Detection Based on a Knowledge Update
Gao, Y., Zhou, H., Chen, L., Shen, Y.-T., Guo, C., & Zhang, X. (2022) · Sensors · doi:10.3390/s22041338 · Cited by 2
“The similarity between cross-modal learning and multimodal fusion is that the data of both come from all modalities, but the difference is that the data of the former are only available [1] in a certain modal, while the data of”
BackgroundEgger, J., Pepe, A., Gsaxner, C., Jin, Y., Li, J., & Kern, R. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.773 · Cited by 52
Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation
Chawla, H., Varma, A., Arani, E., & Zonooz, B. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561441 · Cited by 27
“As discussed earlier in Sec III-B, this can be explained by the disentangling of the intended higherlevel abstractions from the shortcut features [16], [29], [30].”
BackgroundADL based Framework For Multimodal Data Fusion in Traffic Jam prediction
Godhbani, S., Elkosantini, S., Suh, W., & Lee, S. M. (2022) · International Conference on Software, Knowledge, Information Management and Applications (SKIMA) · doi:10.1109/skima57145.2022.10029488 · Cited by 6
“Therefore, addressing multi-source heterogeneous data fusion in the era of big data has become a research hotspot in recent years [1-5].”
BackgroundTowards Melanoma Detection Using Radar and Image Data
Mammadova, F., Onwuchekwa, D., & Obermaisser, R. (2023) · Mediterranean Conference on Embedded Computing (MECO) · doi:10.1109/meco58584.2023.10155072 · Cited by 2
“Early fusion allows fusing multiple data sources into one vector, which can be dissimilar before the classification process [52].”
BackgroundFeatsFlow: Traceable representation learning based on normalizing flows
Zhang, W., Pei, Z., & Wang, F. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107151 · Cited by 1
Multimodal Learning for Road Safety Using Vision Transformer ViT
Rhanizar, A., & Akkaoui, Z. E. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47366-1_10 · Cited by 1
Multimodal Depression Recognition Using Audio and Visual
Xu, X., Zhang, G., Mao, X., & Lu, Q. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0903-8_22 · Cited by 1
Deep Learning Based Multi-Modal Data Fusion for Car Crash Detection
Bhowmik, T. K., & Rateria, R. (2023) · International Conference on Computers and Devices for Communication · doi:10.1109/codec60112.2023.10466222 · Cited by 1
“Late fusion trains separate models for each modality, offering flexibility and independence, like audio-visual processing and sentiment analysis [3].”
BackgroundIntroduction to Multimodal and Tensor Data Analytics
Gaw, N. B., Gahrooei, M. R., & Pardalos, P. M. (2024) · Springer optimization and its applications · doi:10.1007/978-3-031-53092-0_1 · Cited by 1
Prerouting Timing Prediction Across Different Technology Nodes
Zhang, X., Zhu, B., Liu, F., Jiang, J., Wang, Z., Xu, P., Xu, H. L., & Yu, B. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3523426 · Cited by 1
“Multimodal learning [37] is one kind of ML strategy, which offers an innovative pathway to understanding data complexity.”
Hypergraph-Based Organizers’ Behavior Scheduling Optimization: High-Order Relationships Discovery
Xu, Z., & Zhang, H. (2025) · IEEE Access · doi:10.1109/access.2025.3528651 · Cited by 1
Wang, Y., Jiang, W., Jiang, H., Dai, D., Ma, P., Wang, Y., & Wang, Z. (2025) · Geocarto International · doi:10.1080/10106049.2025.2459099 · Cited by 1
“Inspired by this study and considering intermediate fusion (Ramachandram and Taylor 2017), another shaded relief map generation method based on Pix2pixHD was designed.”
Simone, A. D., Gravina, M., & Sansone, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80507-3_10 · Cited by 1
A Comparison of Late-Fusion Training Strategies for Quad-Modal Joint Embeddings
Fürer, D. L., Bernstein, A., & Rossetto, L. (2024) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi62980.2024.10859218 · Cited by 1
Research on Chronic Disease Auxiliary Diagnosis System Based on Multimodal Deep Learning
Chang, H., Han, T., & Wang, Z. (2024) · International Conference on Computing, Robotics and System Sciences (ICRSS) · doi:10.1109/icrss65752.2024.00059 · Cited by 1
“The main concept is to enhance the model's comprehension and processing abilities of information by combining various data sources like images, text, audio, etc. [7].”
TDMER: A Task-Driven Method for Multimodal Emotion Recognition
Xu, Q., Gu, Y., Li, C., Zhang, H., Lin, H. X., & Liu, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889666 · Cited by 1
“Conventional MER methods generally fall into two categories: i) designing intricate fusion strategies [8]–[11] for multimodal features; ii) using cross-modal attention mechanisms [12]–[15] to enhance multimodal features.”
Veerabhadrappa, R., & Poornima, A. S. (2025) · Computer Systems Science and Engineering · doi:10.32604/csse.2025.061285 · Cited by 1
“Authors Barua et al. [23] and Ramachandram et al. [28] reviewed trends in representation learning and fusion techniques, emphasizing the difficulty of selecting appropriate methods for various applications and integrating modality-specific information.”
Multimodal Learning for Enhanced Detection in Oral Cancer Screening
Zhang, J., Tian, Y., Song, B., & Lee, E.-J. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10981235 · Cited by 3
“Ramachandram and Taylor highlight recent advances in deep multimodal learning, discussing its application in healthcare, where combining imaging data with electronic health records (EHRs) has led to more comprehensive diagnostic tools [12].”
IPGRN: An Integrated Progressive Gated Refinement Network for Breast Tumor Analysis
Dai, Z., Huang, G., Li, Y., He, J., Yuan, X., Pun, C., Zhong, G., Ling, B. W., Wu, Y., & Li, J. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3571865 · Cited by 1
“To integrate diverse data types into the decision-making process, recent advancements in multimodal learning have been extensively explored [16], [17], [25].”
Hu, Y., Wang, S., Hu, Q., & Li, J. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3576794 · Cited by 1
“In general, the methods of multimodal data fusion are categorized into three main types: input level fusion (early fusion), decision level fusion (late fusion), and intermediate fusion [20].”
Li, Z., Li, Z., Lam, S., Wang, X., Wang, P., Song, L., Lee, F. K.-H., Yip, C. W.-Y., Cai, J., & Li, T. (2025) · Cancers · doi:10.3390/cancers17142350 · Cited by 1
“Deep learning techniques like convolutional neural networks (CNNs) have been widely applied for classification tasks, providing potential solutions for medical diagnosis and analysis [16].”
Subašić, S., Banjanović-Mehmedović, L., Subašić, H., Karabegović, I., & Husak, E. (2025) · Mechanisms and machine science · doi:10.1007/978-3-032-02106-9_56 · Cited by 1
Wan, Y., & Zhou, W. (2025) · International Conference on Computer Science and Blockchain (CCSB) · doi:10.1109/ccsb66722.2025.11154191 · Cited by 1
“However, our investigation finds that when confronted with multi-source heterogeneous data, these advanced models predominantly rely on simple input-layer concatenation—a form of early fusion— and generally lack effective fusion mechanisms [12].”
Integrating Cross-Modal Semantic Learning with Generative Models for Gesture Recognition
Zhai, S., Dai, Z., Jin, Z., Qin, P., & Zeng, J. (2025) · Sensors · doi:10.3390/s25185783 · Cited by 1
“Multimodal learning integrates complementary information from multiple sensing sources to improve model performance [32].”
Progressive Dynamic Interaction Network With Audio Supplement for Video Moment Localization
Wu, G., Yang, Z., & Zhang, J. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3615180 · Cited by 1
“However, by not fusing the features of each modality early on, important interactions between modalities might be missed [27].”
Dino, H. I., & Hassan, M. M. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106399 · Cited by 3
Kunishima, A., Inaba, D., Iyoshi, S., Ikeda, Y., Goto, M., Muramatsu, R., Hashimoto, M., Yoshida, K., Mogi, K., Yoshihara, M., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-21128-w · Cited by 1
Identification of natural product-based drug combination (NPDC) using artificial intelligence
Niu, T., Zhu, Y., Mou, M., Fu, T., Yang, H., Sun, H., Liu, Y., Zhu, F., Zhang, Y., & Liu, Y. (2025) · Chinese Journal of Natural Medicines · doi:10.1016/s1875-5364(25)60942-3 · Cited by 6
Prajwalasimha, S. N., Sumathi, D., Shelke, N. M., Pimpalkar, A. P., Saini, D. K. J. B., & Balla, R. (2025) · International Conference on Intelligent Cyber Physical Systems and Internet of Things (ICoICI) · doi:10.1109/icoici65217.2025.11254079 · Cited by 1
“Attention-based sequence-aware models with residual layers have also shown better performance on benchmarking datasets such as the TON_IoT and CICIDS2017 [19].”
Pancreas Segmentation With Multi-Phase Feature Aggregation and Modality Adaptive Transformer
Tan, L., Sheng, W., Zhang, J., Zou, W., Jia, D., Chen, Q., Sheng, J., Qian, Y., Feng, Q., Liao, Z., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3628709 · Cited by 2
Robustness over complexity: decision-level fusion for fine-grained volcanic lithology identification
Jin, X., Yu, T., Wang, P., Nie, M., & Lu, S. (2025) · Earth Science Informatics · doi:10.1007/s12145-025-02060-z · Cited by 1
Deng, X., Yang, C., Qin, L.-H., Lin, X.-F., Zhong, F., & Liao, J.-Y. (2026) · Cancer Imaging · doi:10.1186/s40644-025-00987-6 · Cited by 1
“Among these, early (data-level) fusion and feature-level fusion are widely favored because they impose lower requirements on the data and the models [36, 37].”
Liu, Y., Hu, B., Hao, X., Huang, J., & Huang, J. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109527 · Cited by 1
Combining multimodal fatigue fracture surface images for analysis with a CNN
Jones, K., Shade, P. A., John, R., Musinski, W. D., Holm, E. A., & Rollett, A. (2026) · Scientific Reports · doi:10.1038/s41598-025-97184-z · Cited by 1
Wang, H., Zhang, B., Xie, M., Sun, Y., Ye, W., Dong, C., Yang, Z., & Wang, X. (2026) · Minerals · doi:10.3390/min16030318 · Cited by 1
“Compared to using data from a single modality, this approach can significantly enhance model performance, leading to more accurate predictive results [19,20].”
Zhang, J., Luo, Z., Zhao, Y., & Gao, M. (2026) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2026.3676190 · Cited by 1
“Compared to decision-level fusion, feature-level fusion o ff ers a more flexible multidomain data fusion approach [40].”
Li, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-44473-w · Cited by 3
“More sophisticated fusion strategies, including attention mechanisms and cross-modal transformations, have shown improved performance in speech recognition and video understanding 11 .”
Theodorakopoulos, L., & Theodoropoulou, A. (2026) · Electronics · doi:10.3390/electronics15071405 · Cited by 1
Gupta, S., Ray, K., Kaiser, S., Hossain, S., & Faubert, J. (2026) · Algorithms · doi:10.3390/a19040279 · Cited by 2
Eacuello, S. S., Prasad, R., & Sodhi, M. S. (2026) · Sensors · doi:10.3390/s26082405 · Cited by 1
Depression and anxiety characterization and detection with multimodal deep learning
Lu, T., Cho, L., Qiu, Z., Wang, Y., Niu, W., Leung, Y. W., Wang, Y., Zhang, H., Chang, W. C., Tanveer, M., et al. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00632-6 · Cited by 4
“At the same time, fusion techniques (early, late and intermediate fusion) and cross-modal learning (for example, attention mechanisms, shared representations) were proposed to enhance the integration between modalities, addressing the aforementioned issue 50,51 .”
Lv, Y., Wang, X., Liu, Y., & Lu, M. (2026) · Measurement · doi:10.1016/j.measurement.2026.121637 · Cited by 1
Ran, R., Wang, L., Li, G., Yuan, L., Lei, L., & Wei, Z. (2026) · Electronics · doi:10.3390/electronics15091910 · Cited by 1
An Explainable Fusion-Based Multimodal Learning Model for Melanoma Detection
Beeda, S., Rajasekar, M., & Gopalan, A. (2026) · Contemporary Computing Innovations Conference (CCIC) · doi:10.1109/ccic68129.2026.11486214 · Cited by 1
“Techniques such as Grad-CAM [12] offer visual interpretability by high-lighting the most influential lesion regions contributing to a model’s decision.”
CSI-Tuples-Based 3-D Channel Fingerprints Construction Assisted by Multimodal Learning
Xie, C., Li, Y., Chen, R., He, G., & Gao, X. (2026) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2026.3693681 · Cited by 1
“In conventional multimodal learning tasks, raw multimedia data cannot be directly processed by machines, so it is necessary to conduct the unified description and processing, known as the MultiModal Representation (MMR) [60], [61].”
A survey on multimodal federated learning
Grosso, S., Benarba, N., Bouchenak, S., Esposito, R., & Polato, M. (2026) · Neural Computing and Applications · doi:10.1007/s00521-025-11815-0 · Cited by 1
“Additionally, when one modality is significantly more informative than the others, early fusion may cause that modality to dominate the learning process, resulting in suboptimal performance [106].”
AN OVERVIEW OF MULTIMODAL LEARNING: CONCEPTS, CHALLENGES, APPLICATIONS AND DATASETS
Ateş, A. H., & Yurtsever, U. (2026) · Konya Journal of Engineering Sciences · doi:10.36306/konjes.1716813 · Cited by 1
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.