Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 601–700 of 1,002
Chen, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-31341-2 · Cited by 10
“Technical challenges include the complexity of synchronizing heterogeneous data streams with different sampling rates and structural characteristics from classroom environments 35 .”
MMFSeg: Multi-Structure Multi-Feature Fusion for Segmentation of Road Potholes
Feng, Z., Guo, Y., Fan, R., & Sun, Y. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3613629 · Cited by 4
“However, it is sensitive to noise and may lead to information loss [35], [36].”
Deep learning-based astronomical multimodal data fusion: A comprehensive review
Shao, W., Fan, D., Cui, C., Xu, Y., Wei, S., & Lyu, X. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.104103 · Cited by 10
“…implementations of decision-level fusion can be broadly categorized into two types: rule-driven methods (e.g., voting, weighted averaging) and model-driven methods (e.g., bagging, boosting, MLPs) (Yang et al., 2025; Ramachandram and Taylor, 2017; Baltrušaitis et al., 2018; Zhao et al., 2024).”
InfluentialLee, H., & Li, W. (2026) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.12.023 · Cited by 5
Lima, L. D. B., Teixeira, S. S., Bordalo, V., Lacoste, S., Guimond, S., Sousa, D. L., Pinheiro, D. N., Moreira, R., & Teles, A. S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124879 · Cited by 5
Identifying chaotic dynamics in noisy time series through multimodal deep neural networks
Giuseppi, A., Menegatti, D., & Pietrabissa, A. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad7190 · Cited by 5
“In this direction, multimodal DNN [30] are a class of DNN specialized to deal with heterogeneous input data, that is, data of various structures/ modalities (e.g. the combination of images, signal readings and text).”
Multimodal representation learning for medical analytics - a systematic literature review
Hansen, E. R., Sagi, T., & Hose, K. (2024) · Health Informatics Journal · doi:10.1177/14604582241290474 · Cited by 8
“Additional surveys contextualize MRL in the general challenge of multimodal ML, 6 with a focus on fusion-based MRL 11 or its mathematical-theoretical foundations, 12 disregarding the application domain.”
Wang, Z., Wang, P., Liu, Y., Liu, Y., & Wang, T. (2024) · Materials & Design · doi:10.1016/j.matdes.2024.113504 · Cited by 7
Deep multimodal learning for time series analysis in social computing: a survey
Yang, C., Chen, Y., Li, Z., Wang, X., Shi, K., Yao, L., Xu, G., & Guo, Z. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00363-x · Cited by 7
“Selecting the appropriate modality to serve as supplemental information across different application scenarios can be challenging[37].”
Zhang, Z., Zhang, Y., Liu, W., Liu, C., & Liu, T. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3616251 · Cited by 6
Multi-modal texture fusion network for detecting AI-generated images
Yu, H., & Xu, B. H. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1663292 · Cited by 9
“Crop yield prediction utilizing multimodal deep learning”
Jácome-Galarza, L.-R. (2021) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti52073.2021.9476420 · Cited by 5
Deep Graph Fusion Based Multimodal Evoked Expressions From Large-Scale Videos
Ho, N.-H., Yang, H.-J., Kim, S.-H., Lee, G.-S., & Yoo, S. B. (2021) · IEEE Access · doi:10.1109/access.2021.3107548 · Cited by 5
“Input modalities can be merged in various ways, including concatenation, pooling and through employing a gated unit [41], [42].”
BackgroundMayr, A., Scheffler, F., Fuder, R., Raffin, T., Kißkalt, D., & Franke, J. E. (2023) · Procedia CIRP · doi:10.1016/j.procir.2023.06.150 · Cited by 4
Kandoi, R., Dixit, D., Tyagi, M., & Yadav, R. S. (2024) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2024.58787 · Cited by 4
Vellenga, K., Steinhauer, H. J., Falkman, G., & Björklund, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00726 · Cited by 13
“Commonly, fusion strategies are categorized as early, intermediate and late fusion [33].”
BackgroundUnleashing Network Potentials for Semantic Scene Completion
Wang, F., Sun, Q., Zhang, D., & Tang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00982 · Cited by 4
“The crux lies in the inadequate harnessing of all modal information by most joint training methods [28].”
Tariq, S., Rehman, A. U., Abubakar, M., Iqbal, W., AlSagri, H. S., Alduraywish, Y. A., & Alhakbani, H. A. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.055787 · Cited by 5
Li, J., Li, X., Chen, Y., Wang, Y., Wang, B., Zhang, X., & Zhang, N. (2024) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-024-03253-2 · Cited by 5
Singh, H., Déniz, Ó., Ruiz-Santaquiteria, J., Muñoz, J. D., & Bueno, G. (2025) · Applied Sciences · doi:10.3390/app15115830 · Cited by 4
“To integrate these feature vectors, we employed an early fusion [49] mechanism that creates a unified representation, combining the information extracted from both the hand patches and HPE.”
Multimodal Learning Data Fusion and Analysis Based on Self-Attention Mechanism
Yue, Y. (2025) · IEEE International Conference on Electronic Technology, Communication and Information (ICETCI) · doi:10.1109/icetci64844.2025.11084003 · Cited by 4
“For example, some methods only focus on intra-modal dependencies, ignoring inter-modal correlations; others only focus on inter-modal correlations, neglecting intra-modal contextual information [14].”
Statistical NLG for Generating the Content and Form of Referring Expressions
Li, X., van Deemter, K., & Lin, C. (2018) · International Conference on Natural Language Generation · doi:10.18653/v1/w18-6561 · Cited by 5
“…specifically on Lexical Choice and the generation of SUMTIME-style sentences such as “MAINLY W-NW 10 OR LESS” (a brief weather prediction about the Vector-based Knowledge Representation (KR) is often used in deep learning-based and other Machine Learning approaches (Ramachandram and Taylor, 2017).”
MethodSpeaker-adapted neural-network-based fusion for multimodal reference resolution
Kleingarn, D., Nabizadeh, N., Heckmann, M., & Kolossa, D. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5925 · Cited by 5
“The first is at which level of processing the fusion happens and the second how the fusion is performed (see Atrey et al. (2010); Ramachandram and Taylor (2017) for a comprehensive overview).”
BackgroundFulton, C., Bustillo, M., McGuinness, C., Guérin, S., & Browne, A. (2020) · New Review of Academic Librarianship · doi:10.1080/13614533.2020.1765818 · Cited by 4
“Research has shown that multimodal ways of learning, such as those applied in this project, result in deeper learning experiences for students (Ramachandram and Taylor, 2017).”
BackgroundWhat should we pay attention to when classifying violent videos?
Teixeira, M. V. A., & Avila, S. (2021) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3465481.3470059 · Cited by 4
“This fusion scheme is more flexible than the early fusion approach by exploiting the decisions of different classifiers and being feature independent [45].”
Efficient Calibration Tool for Thermistor
(2015) · Sensors and Materials · doi:10.18494/sam.2015.1094 · Cited by 4
Nan, C. (2022) · Mobile Information Systems · doi:10.1155/2022/1653452 · Cited by 5
“put forward that in the process of carrying out multidimensional psychological reform of English Teaching in institution of higher learning Based on employment orientation, active cooperation is required from all aspects, such as the importance attached to it by the leaders of the University, the strong support provid…”
BackgroundEncrypted Traffic Detection: Beyond the Port Number Era
Doroud, H., Alaswad, A., & Dressler, F. (2022) · Conference on Local Computer Networks · doi:10.1109/lcn53696.2022.9843432 · Cited by 6
“Aceto et al. [6] proposed DISTILLER as a multimodal multitask deep learning architecture [14] to classify encrypted traffic.”
BackgroundPerceiver Hopfield Pooling for Dynamic Multi-modal and Multi-instance Fusion
Rößle, D., Cremers, D., & Schön, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15919-0_50 · Cited by 4
A hybrid fusion precipitation bias correction approach for Yin‐He global spectral model
Hu, Y., Yin, F., Zhang, W., & Deng, K. (2022) · Meteorological Applications · doi:10.1002/met.2097 · Cited by 4
“Such flexible fusion strategies allow the model to utilize the complementary information between different variables efficiently (Ramachandram & Taylor, 2017).”
BackgroundMethodInfluentialMayr, A., Bauer, J., & Franke, J. E. (2022) · Procedia CIRP · doi:10.1016/j.procir.2022.10.073 · Cited by 4
Nakach, F.-Z., Zerouaoui, H., & Idri, A. (2023) · Data Technologies and Applications · doi:10.1108/dta-08-2022-0330 · Cited by 4
End-to-End Chinese Lip-Reading Recognition Based on Multi-modal Fusion
Liu, Y., Lin, C., Wang, M., Liang, S., Chen, Z., & Chen, L. (2022) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic57696.2022.10075247 · Cited by 5
“As multimedia data is often a medium for the transmission of multiple information, for example, a video can contain textual, visual, and auditory information at the same time, multi-modal learning [1] has become an essential tool for the analysis and understanding of multimedia content.”
BackgroundA transformer-based network for pollen particle classification
Xu, D., & Li, J. (2022) · International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI) · doi:10.1109/mlbdbi58171.2022.00025 · Cited by 4
Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding
Chen, K., Zhang, R., Mensah, S., & Mao, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.586 · Cited by 10
“The image features are extracted by a Faster R-CNN (Ren et al., 2015) pre-trained on the Visual Genome (Krishna et al., 2017) dataset with a ResNet-101 (He et al., 2016) backbone.”
MethodAction class relation detection and classification across multiple video datasets
Yoshikawa, Y., Shigeto, Y., Shimbo, M., & Takeuchi, A. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.08.002 · Cited by 5
“In our preliminary experiment, we have attempted an intermediate fusion strategy [34].”
MethodZhou, H., Wang, S., Jiang, C., Zhang, X., Guo, L., & Yuan, Y. (2023) · ACM International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing (MobiHoc) · doi:10.1145/3565287.3610277 · Cited by 5
“Future MEC is anticipated to support increasingly complex applications with high reliability and robustness, which necessitates the use of data with multiple modalities [23].”
BackgroundNguyen, C.-V. T., Nguyen, H.-S., Le, D.-T., & Ha, Q.-T. (2023) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse59128.2023.10299456 · Cited by 5
“Multimodal learning has attracted significant research attention due to its wide-ranging applications [2], [5], [10].”
BackgroundMAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers
Shaikh, M. B., Chai, D., Islam, S. M. S., & Akhtar, N. (2023) · European Workshop on Visual Information Processing · doi:10.1109/euvip58404.2023.10323051 · Cited by 6
“The field of audiovisual multimodal learning has a long and diverse history, both preceding and during the deep learning era [19].”
BackgroundEntity Linking Supported Multimodal Data: Fusing Text and Image features for Higher Accuracy
Chen, Y., Wu, P., Zhao, X., & Dai, Y. (2023) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml60161.2023.10424744 · Cited by 4
“Common methods include recurrent neural networks (RNN) [11], long short-term memory networks (LSTM) [18], and transformers [19].”
BackgroundMulti-view Deep Neural Networks for multiclass skin lesion diagnosis
Pérez, E., & Ventura, S. (2022) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins54846.2022.9854997 · Cited by 5
“Early fusion joins raw or preprocessed features from different modalities before passing them to a predictive model, which will obtain a single prediction [8].”
BackgroundMultimedia Signal Processing: A history of the Multimedia Signal Processing Technical Committee
Bajić, I. V., Mrak, M., Frederic Dufaux, A., Magli, E., & Chen, T. (2023) · IEEE Signal Processing Magazine · doi:10.1109/msp.2023.3260989 · Cited by 4
“This is in part due to the ability of deep neural networks to effectively model relationships in multimodal data [25].”
BackgroundCV.eDNA: A hybrid approach to invertebrate biomonitoring using computer vision and DNA metabarcodingPreprint
Blair, J. D., Weiser, M. D., Siler, C. D., Kaspari, M. E., Smith, S. N., McLaughlin, J. F., & Marshall, K. E. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.02.610558 · Cited by 5
“In deep learning, multimodal models can receive data from multiple modalities as input to inform their classifications (Ramachandram and Taylor, 2017).”
Bihler, M., Roming, L., Cibiraite-Lukenskiene, D., Aderhold, J., Keil, A., Schlüter, F., Gruna, R., & Heizmann, M. (2026) · Scientific Data · doi:10.1038/s41597-026-07053-1 · Cited by 4
“Each model was trained independently using data from four individual sensor modalities (RGB, NIR, IR, and THz) as well as early-fused (EF) data 26 , where all sensor modalities are combined into a single input by channel-wise concatenation.”
Chen, Y., Gunraj, H., Zeng, E. Z., Meyer, R., Gilles, M., & Wong, A. M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00012 · Cited by 6
“Multimodal Data Fusion: Multimodal learning [22, 17, 23, 8, 14] has been rigorously studied.”
BackgroundCascaded cross-modal transformer for audio–textual classification
Ristea, N.-C., Anghel, A., & Ionescu, R. T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10869-1 · Cited by 7
“Taming the complexity of real-world classification tasks through the combination of various modalities is not an easy challenge, requiring the development of robust and efficient methods to effectively fuse multiple data sources (Ramachandram and Taylor 2017; Gao et al. 2020; Stahlschmidt et al.…”
Raman spectral unmixing via multimodal time-frequency transformations and deep learning
Shang, L., Ding, C., Ji, X., Zhu, J., Wu, J., Xiong, S., Wang, H., & Yin, J. (2025) · Optics Express · doi:10.1364/oe.555722 · Cited by 3
Ullah, R., Zhang, S., Asif, M., & Wahab, F. E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110082 · Cited by 6
Sensing Technologies for Hand Gesture Recognition in Human–Robot Interaction: A Review
Liang, B., Li, N., Li, G., Huang, J., Yu, Z., & Zhao, X. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3635622 · Cited by 6
“This approach allows us to cover a wide and representative range of algorithms in a structured and logical manner [102], [103], [104]. a) CNNs model: CNN is a predominant deep learning technique in HGR, designed to hierarchically extract spatial features through learnable kernels and pooling…”
Analyzing the impact of shifts in encrypted mobile-app traffic on multimodal few-shot learning
Monda, D. D., Bovenzi, G., Montieri, A., Persico, V., & Pescapè, A. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111935 · Cited by 7
Altinses, D., & Schwung, A. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131270 · Cited by 4
“The integration of information from multiple modalities plays a pivotal role in multimodal learning frameworks, where the goal is to consolidate diverse data into a unified, compact representation that can be used for effective analysis and decision-making [17].”
Popello, V., Zucco, C., Milano, M., & Cannataro, M. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356777 · Cited by 3
“…fusion, which combines raw or pre-processed features into a single input vector before training, allowing joint learning and capturing interactions between modalities from the outset; hybrid fusion, which integrates elements of both approaches to leverage their respective advantages [27] [28].”
Ativi, W. X., Chen, W., Kwao, L., Ayivi, W., Sam, F., Alqahtani, A., Gu, Y. H., & Al–antari, M. A. (2026) · Scientific Reports · doi:10.1038/s41598-025-32794-1 · Cited by 5
“Early fusion concatenates raw features before processing but suffers from dimensional explosion and fails to preserve modality-specific characteristics 16 .”
Explainable deep learning approaches and clinical insights for cancer biomarker identification
Srivastava, K., & Srivastava, R. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1810793 · Cited by 3
“DL offers an effective solution by handling high-dimensional, heterogeneous, and unstructured data and by uncovering nonlinear and multimodal relationships that are dif fi - cult to detect using conventional analytical methods (60, 61).”
Ding, W., Hu, Y., Lu, Y., Zhai, Z., Li, K., & He, H. (2026) · Green Chemistry · doi:10.1039/d6gc02374f · Cited by 3
End-to-end driving model based on deep learning and attention mechanism
Zhu, W., Lu, Y., Zhang, Y., Wei, X., & Wei, Z. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-211206 · Cited by 4
“Multimodal fusion [24] is an important research direction in the field of multimodal research, which makes full use of the complementary information in multimodality.”
Liang, Y., Li, S., You, X., Guo, Y., & Tang, J. (2023) · Current Bioinformatics · doi:10.2174/0115748936272040231117114252 · Cited by 3
Bo, W., Sullivan, S. S., Zhang, X., Gao, M., & Xu, W. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3419059 · Cited by 3
Person-Centric Sensing in Indoor Environments
Strohmayer, J., Lumetzberger, J., Heitzinger, T., & Kampel, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-59531-8_11 · Cited by 4
Voluntary forward-looking disclosures and default risk pricing
Chen, C., Wei, M., Zhang, H., & Yan, J. (2024) · Accounting and Business Research · doi:10.1080/00014788.2024.2381507 · Cited by 3
“Research in Artificial Intelligence (AI) in recent years has made rapid advances in the fusion of multi-modal information and its applications (e.g., Ramachandram & Taylor, 2017; Baltrusaitis et al., 2019).”
A multi-modal approach for mixed-frequency time series forecasting
Filho, L. L., Werneck, R. D. O., Castro, M., Júnior, P. R. M., Lustosa, A., Zampieri, M., Linares, O. C., Moura, R., Morais, E., Amaral, M., et al. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10305-z · Cited by 4
“Furthermore, multimodal learning models can be divided into early and late fusion approaches [43].”
Dietz, S., Altstidl, T., Zanca, D., Eskofier, B. M., & Nguyen, A. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650421 · Cited by 4
“Traditional early fusion approaches struggle with MTTS since there usually does not exist a matching sample from one modality for each sample of the other.”
Deep metric loss for multimodal learning
Moon, S., & Lee, H. (2025) · Machine Learning · doi:10.1007/s10994-024-06709-6 · Cited by 4
“Multimodal learning functions by combining data from different modalities to enhance the accuracy, robustness, and generalizability of machine learning models.”
InfluentialMultimodal Natural Disaster Scene Recognition with Integrated Large Model and Mamba
Shao, Y., & Xu, L. (2025) · Applied Sciences · doi:10.3390/app15031149 · Cited by 5
“Multimodal learning (MML) [29] is a general approach to building AI models that extract and correlate information from diverse modalities.”
Relevance of advanced imaging analysis units in radiology departments: a narrative review
Martín‐Noguerol, T., Paulano‐Godino, F., Úbeda, P. L., Riascos, R., & Luna, A. (2025) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-025-03529-4 · Cited by 3
Li, K., Qiu, Z., Li, J., Deng, F., Zou, K., Xu, Y., Huang, C., Wang, R., Yu, Z., Chen, Y., et al. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1701545 · Cited by 3
Deep Multimodal Sparse Representation-Based Classification
Abavisani, M., & Patel, V. M. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191317 · Cited by 5
“, late fusion), or intermediately [12, 13, 14].”
BackgroundPhoto- and Radioluminescence of ZnO-Precipitated Glass-Ceramics
Masai, H., Yanagida, T., & Fujiwara, T. (2015) · Sensors and Materials · doi:10.18494/sam.2015.1061 · Cited by 3
On the Fusion of Multiple Audio Representations for Music Genre Classification
Silva, D. F., da Silva, M. V., Filho, R. S., & da Silva, A. C. M. (2021) · Simpósio Brasileiro de Computação Musical (SBCM) · doi:10.5753/sbcm.2021.19423 · Cited by 4
Evaluating multimodal strategies for multi-label movie genre classification
Paulino, M. A. D., Costa, Y. M. G., & Feltrim, V. D. (2022) · International Conference on Systems, Signals and Image Processing · doi:10.1109/iwssip55020.2022.9854451 · Cited by 5
Off-Screen Sound Separation Based on Audio-visual Pre-training Using Binaural Audio
Yoshida, M., Togo, R., Ogawa, T., & Haseyama, M. (2023) · Sensors · doi:10.3390/s23094540 · Cited by 4
“This research field is rapidly growing, and many methods for advanced video recognition have been proposed [14].”
MethodMazzola, C., Romeo, M., Rea, F., Sciutti, A., & Cangelosi, A. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191452 · Cited by 4
“This was inspired by a gradual fusion of modalities at an intermediate level of the network, which has been demonstrated beneficial [28], [29], and by the training on joint representations of temporal sequences as, for instance, in Nakisa et al.”
MethodResultLi, Z., Sang, Y., Deng, X., & Tian, H. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-5834-4_25 · Cited by 3
Cascaded Cross-Modal Transformer for Request and Complaint Detection
Ristea, N.-C., & Ionescu, R. T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612846 · Cited by 4
“Tackling the complexity of real-world data through the combination of multiple modalities is a challenging task, requiring the development of a robust and efficient method for aggregating all sources of information [8, 20, 23].”
MethodMAAL: Multimodality-Aware Autoencoder-based Affordance Learning for 3D Articulated Objects
Liang, Y., Wang, X., Zhu, L., & Yang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00027 · Cited by 5
“However, the advantages of late fusion in multi-modal tasks are insignificant [36, 12, 41] compared with early fusion.”
BackgroundCross-Modality Calibration in Multi-Input Network for Axillary Lymph Node Metastasis Evaluation
Gravina, M., Santucci, D., Cordelli, E., Soda, P., & Sansone, C. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3397246 · Cited by 3
“There are different combining strategies such as majority voting, averaged fusion, Bayes’ rule, or those exploiting the use of a metamodel [4].”
BackgroundMethodInfluentialOpenPSS: An Open Page Stream Segmentation Benchmark
Heusden, R. V., Kamps, J., & Marx, M. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72437-4_24 · Cited by 3
“In early ensembling, the final layers of two models are combined before the final classification is made, and a prediction is obtained by adding a final classification layer on top of this combined output to obtain the final prediction [20].”
Han, Y., He, J., Du, X., Han, X., & Liu, Y. (2024) · Urban forestry & urban greening · doi:10.1016/j.ufug.2024.128582 · Cited by 4
The iCub Multisensor Datasets for Robot and Computer Vision Applications
Kırtay, M., Albanese, U., Vannucci, L., Schillaci, G., Laschi, C., & Falotico, E. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418847 · Cited by 5
“Multisensory integration is an essential skill that biological agents (e.g., humans) can perform with apparent ease to achieve a considerable number of vision-based tasks (Lahat et al. , 2015; Ramachandram & Taylor, 2017).”
The application of artificial intelligence to drug sensitivity prediction
Li, X., Wu, X., Wan, X., Zhong, F., Cui, C., Chen, Y., Chen, L., Chen, K., Jiang, H., & Zheng, M. (2020) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/tb-2020-0557 · Cited by 6
Geng, L. (2021) · Complexity · doi:10.1155/2021/9963246 · Cited by 3
“Knee and ankle joints are excluded from the pose features due to their poor localization accuracy caused by occlusion [19]. orientation of the joints is represented using quadratic representation. essence of recall-based action recognition is a query process in which the object of the query is the…”
BackgroundRetractedForming robot trust in heterogeneous agents during a multimodal interactive game
Kırtay, M., Öztop, E., Kuhlen, A. K., Asada, M., & Hafner, V. V. (2022) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl53763.2022.9962212 · Cited by 3
“We emphasize that, in this work, the term modality refers to sensory readings that are associated with different aspects (e.g., audio and visual) of the experimental tasks [22], [23].”
BackgroundNitsch, J., Nieto, J. I., Siegwart, R., Schmidt, M. T., & Cadena, C. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793628 · Cited by 3
“This finding is in line with suggestions in [23] and [1] that there is no clear preferable strategy yet when to fuse modalities.”
ResultMulti-view learning review: understanding methods and their application
배강일, 이영섭, & 임창원 (2019) · 응용통계연구 = The Korean journal of applied statistics · doi:10.5351/kjas.2019.32.1.041 · Cited by 4
Fusing Visual and Textual Information to Determine Content Safety
Leonardo, R., Hu, A., Uzair, M., Lu, Q., Fu, I., Nishiyama, K., Subrahmannian, S. M., & Ravichandran, D. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00324 · Cited by 4
“1) Dimension reduction: When working with features of many dimensions, it is beneficial to perform dimension reduction and feature selection techniques to remove noise from unnecessary features [17], [18].”
BackgroundMulti-model data fusion for cervical precancerous lesions detection
Song, H., Yan, L., Yang, J., Shen, X., Guo, Y., & Ren, P. (2020) · Chinese Automation Congress · doi:10.1109/cac51589.2020.9327402 · Cited by 3
“1) Early fusion [9], [10]: First merge multiple layers of features, and then train the predictor on the fused features.”
BackgroundWeatherNet: Nowcasting Net Radiation at the Edge
Nueve, E., Jackson, R., Sankaran, R., Ferrier, N., & Collis, S. (2021) · IEEE Conference on Technologies for Sustainability · doi:10.1109/sustech51236.2021.9467444 · Cited by 3
Yu, H. Q., O’Neill, S. T., & Kermanizadeh, A. (2023) · Bioengineering · doi:10.3390/bioengineering10101134 · Cited by 4
“While humans can seamlessly inte-604 grate visual, auditory, and other sensory data to accomplish tasks, machines struggle to synthesize multiple data types [46–48].”
BackgroundEgger, J., Pepe, A., Gsaxner, C., & Li, J. (2020) · arXiv (Cornell University) · Cited by 3
Social Determinants of Health: Insights from Location Big DataPreprint
Macha, M., Li, B., & Foutz, N. Z. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3923956 · Cited by 3
Self-supervised learning for early detection of neurodegenerative diseases with small data
Jiang, H. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/166402 · Cited by 9
Khan, N., & Ahmad, Z. (2023) · doi:10.32920/22734332 · Cited by 3
Dual Variational Knowledge Attention for Class Incremental Vision Transformer
Duan, H., Sun, R. Q., Ojha, V., Shah, T., Huang, Z., Ouyang, Z., Huang, Y., Long, Y., & Ranjan, R. K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650317 · Cited by 6
“By adopting the strategy of independent classifiers [42] in combination with sigmoid activation and binary cross-entropy loss L cls , we eliminate the possibility of late fusion [43] of task embeddings.”
MethodTCE-DBF: textual context enhanced dynamic bimodal fusion for hate video detection
Xiong, H., Jiao, W., & Cai, Y. (2024) · Data Technologies and Applications · doi:10.1108/dta-02-2024-0211 · Cited by 2
Multi-task classification network for few-shot learning
Ji, Z., Liu, Y., Wang, X. J., Liu, J., Cao, J., & Yu, Y. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00354-y · Cited by 2
Multi-Sensor Multi-Stream Data Fusion Using Deep Learning for Hyperspectral Point Cloud Segmentation
Afifi, A. J., Rizaldy, A., Arbash, E., Lorenz, S., Gloaguen, R., & Heizmann, M. (2024) · Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing · doi:10.1109/whispers65427.2024.10876491 · Cited by 2
“Fusion models for multimodal data are commonly implemented through early, late, or intermediate fusion techniques [6, 9].”
Xu, J., Yuan, X., Zheng, L., & Lin, D. (2025) · IEEE Access · doi:10.1109/access.2025.3571523 · Cited by 3
Wen, Q., Wei, P., Li, F., Hu, Q., Zeng, B., & Feng, G. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733355 · Cited by 3
“…Self-MM[36]), which bridge the modality gap and learn effective representations by adaptively aligning potential information across modalities [25]; and (b) reconstruction-based methods (DMD [17], EMT-DLFR [28]), which achieve efficient fusion of unaligned multimodal data by combining…”
Statistical machine learning tools for probabilistic closures of turbulence models
Lemos, J. D., & dos Santos, F. P. (2025) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0267896 · Cited by 2
Semantic-Guided Prototype Refinement for Few-Shot Apple Leaf Disease Recognition
Li, Y., He, J., Gao, Y., Zhang, Y., Ma, S., & Liu, H. (2025) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc65474.2025.11091063 · Cited by 2
“Multimodal learning focuses on integrating information from diverse data sources, such as text, images, and audio, to leverage the complementarity and synergy between modalities [21].”
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.