Deep Multimodal Learning: A Survey on Recent Advances and Trends
The DOI matches its Crossref record; 109 of the 113 references it cites were found in a registry or an index.
Abstract
The success of deep learning has been a catalyst to solving increasingly complex machine-learning problems, which often involve multiple data modalities. We review recent advances in deep multimodal learning and highlight the state-of the art, as well as gaps and challenges in this active research field. We first classify deep multimodal learning architectures and then discuss methods to fuse learned multimodal representations in deep-learning architectures. We highlight two areas of research-regularization strategies and methods that learn or optimize multimodal fusion structures-as exciting areas for future work.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, early fusion strategies are sensitive to different sampling rates of modalities [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“However, naively concatenating features from different modalities might result in worse performances(Ramachandram &Taylor, 2017).”
From A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction (2022) · cites this paper as Ramachandram &Taylor, 2017
In the citing paper
“Largely, fusion strategies can be categorized according to the state of the input to the fusion layers into early, intermediate and late fusion [2] (blue layer in Figure 2).”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Additionally, the sub-models’ errors can be uncorrelated and thus have complementary effects [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
In the citing paper
“Thus, fusing modalities gradually rather than all in one layer can be beneficial [2].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [2]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Cited by
Items 801–900 of 1,002
Wan, T., Jiang, Y., (309588), X. H., Zheng, Y., & Ma, W. (2026) · Case Studies in Construction Materials · doi:10.1016/j.cscm.2026.e06230 · Cited by 2
Intelligent prediction model for geoeffectiveness of coronal mass ejections
Tang, X., Yang, X., Xu, L., & Huang, X. (2026) · Frontiers in Astronomy and Space Sciences · doi:10.3389/fspas.2026.1839173 · Cited by 1
Ganji, Z., Nikparast, F., Shoeibi, N., Shoeibi, A., Hoseinkhani, M., Hashemi, S. M., & Zare, H. (2026) · Photodiagnosis and Photodynamic Therapy · doi:10.1016/j.pdpdt.2026.105563 · Cited by 1
Nondestructive identification of maize seed varieties using interpretable RGB-spectral fusion
Xie, H., Ji, H., Bi, X., Wang, M., & Bi, C. (2026) · Journal of Stored Products Research · doi:10.1016/j.jspr.2026.103199 · Cited by 1
Han, X., Yue, C., Zhang, X., Ji, X., Lin, S., Chen, M., Wang, X., Zhang, Y., Xu, W., Liu, Y., et al. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1906621 · Cited by 1
“Evidence suggests that individuals with depression exhibit cognitive biases characterized by negative interpretations of neutral or even positive stimuli (25, 26).”
SDFA-Net: Urban Manhole Cover Detection via Fisheye Image and LiDAR Point Cloud Fusion
Lan, Q., Hu, S., Li, J., Huang, Y., Li, Z., & Fan, Q. (2026) · Remote Sensing · doi:10.3390/rs18183206 · Cited by 1
Sensory Data Assisted Downlink Channel Prediction for Massive MIMO
Yang, Y., Gao, F., Xing, C., An, J., & Alkhateeb, A. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500427 · Cited by 1
“Multimodal learning aims to build models that can fully exploit the constructive and complementary information lying in multimodal data, thus gaining performance advantages over methods that only use data of a single modality [10].”
BackgroundMethodInfluentialTIPS: A Framework for Text Summarising with Illustrative Pictures
Golec, J., Hachaj, T., & Sokal, G. (2021) · Entropy · doi:10.3390/e23121614 · Cited by 1
Bayrak, Ş., & Yücel, E. (2022) · Studies in computational intelligence · doi:10.1007/978-981-19-2416-3_1 · Cited by 1
Gain Pattern Reconstruction of GPS Satellite Antennas Using a Global Receiver Network
Allende-Alba, G., Thoelert, S., & Caizzone, S. (2022) · NAVIGATION Journal of the Institute of Navigation · doi:10.33012/navi.530 · Cited by 1
Multi-encoder attention-based architectures for sound recognition with partial visual assistance
Boes, W., & hamme, H. V. (2022) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-022-00252-9 · Cited by 3
“This technique can be utilized in conjunction with deep learning models, such as generative adversarial networks [9], and currently occupies most of the literary space.”
BackgroundGraph Kernel Learning for Predictive Toxicity Models
Xu, Y., Chou, C.-H., Han, N., Pei, J., & Lai, L. (2023) · Computational methods in engineering & the sciences · doi:10.1007/978-3-031-20730-3_6 · Cited by 1
Targeted deep learning: Framework, methods, and applications
Huang, S., & Lederer, J. (2023) · Stat · doi:10.1002/sta4.556 · Cited by 1
“Another framework is multimodal learning , where data from different sources is used (Ramachandram & Taylor, 2017).”
BackgroundMethodClosed-Form Study of Undetected Range Errors Induced by Ionospheric Anomalies for GAST-D GBAS
Li, W., & Jiang, Y. (2023) · NAVIGATION Journal of the Institute of Navigation · doi:10.33012/navi.603 · Cited by 1
Work-in-Progress: Multimodal Odour and Image Recognition on Synthetic Dataset
Monori, F., & Tisan, A. (2023) · IEEE International Symposium on Industrial Electronics (ISIE) · doi:10.1109/isie51358.2023.10228134 · Cited by 2
“There are three modality fusion approaches that are usually considered: early fusion, intermediate (or mid-) fusion and late fusion [11].”
Multimodadl Graph Signal Denoising With Simultaneous Graph Learning using Deep Algorithm Unrolling
Takanami, K., Bandoh, Y., Takamura, S., & Tanaka, Y. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222480 · Cited by 1
Evaluation of Multi-Modal Learning for Predicting Coolant Pump Failures in Heavy Duty Vehicles
Fan, Y., Atoui, M. A., Nowaczyk, S., & Rognvaldsson, T. S. (2023) · PHM Society Asia-Pacific Conference · doi:10.36001/phmap.2023.v4i1.3716 · Cited by 1
“The fusion of multi-modal data can be performed in different stages, namely early, intermediate and late fusion (Ramachandram & Taylor, 2017).”
BackgroundA Parallel Multimodal Integration Framework and Application for Cake Shopping
Fang, H., Weng, D., & Tian, Z. (2023) · Applied Sciences · doi:10.3390/app14010299 · Cited by 1
Gordienko, N., Gordienko, Y. G., Rokovyi, O., Alienin, O., Polukhin, A., & Stirenko, S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386393 · Cited by 1
“Researchers are actively devising innovative fusion strategies within multimodal deep learning neural network models to effectively amalgamate data from diverse modalities [12].”
BackgroundA Review of RGB-D Image Classification Methods
Yadav, N. K., & Saraswat, M. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-9531-8_2 · Cited by 1
Image Aesthetics Assessment Based on Visual Perception and Textual Semantic Understanding
Liu, Y., Wen, Z., Li, S., Fan, D., & Zhai, G. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-3626-3_4 · Cited by 1
Distributed Estimation by Two Agents with Different Feature Spaces
Raghavan, A., & Johansson, K. H. (2024) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc60939.2024.10644868 · Cited by 1
“As information is received sequentially, the knowledge about the observed phenomenon evolves in the KS.”
BackgroundZhang, W., Xie, R., Li, J., Wang, L., Li, X., & Peng, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125483 · Cited by 1
Estimation of soil properties for sustainable crop production using multisource data fusion
Tsakiridis, N. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-24139-0.00010-2 · Cited by 1
Motor Fault Diagnosis Method Based on Multimodal and Spatial Attention Mechanism
Zhan, T., Yang, Y., He, Y., & Feng, K. (2024) · Global Reliability and Prognostics and Health Management Conference (PHM) · doi:10.1109/phm-beijing63284.2024.10874512 · Cited by 1
Obayemi, A., & Nguyen, K. A. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00244 · Cited by 1
3D medical image segmentation via sequential 2D slice processing
Liew, Y. Z., Tan, A. H. P., Majeed, A. P. P. A., Nguyen, A., Paoletti, P., & Chen, W. (2025) · ICT Express · doi:10.1016/j.icte.2025.07.008 · Cited by 2
PIST-Net: A Multimodal Deep Learning Framework for Red Giant Stellar Parameter Estimation
Cui, Y., Wu, J., He, Y., Lv, C., Chang, J., Jiang, B., & Zhang, Y. (2025) · Publications of the Astronomical Society of the Pacific · doi:10.1088/1538-3873/adfd89 · Cited by 1
Supervised Multimodal Fission Learning
Mao, L., Wang, Q., Su, Y., Lure, F. Y. M., Chong, C. D., Schwedt, T. J., & Li, J. (2026) · INFORMS Journal on Data Science · doi:10.1287/ijds.2024.0059 · Cited by 3
Multimodal Sensory Representation for Object Classification via Neocortically-inspired Algorithm
Kırtay, M., Vannucci, L., Albanese, U., Falotico, E., & Laschi, C. (2018) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2018.8761024 · Cited by 1
“Unlike multimodal learning studies that were introduced in the literature of machine learning and robotics [1], [2], we”
BackgroundMethodInfluentialMaterializing Architecture for Processing Multimodal Signals for a Humanoid Robot Control System
Akikawa, M., & Yamamura, M. (2021) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2021.p0335 · Cited by 1
Collaborative Image Understanding
Bibas, K., Shalom, O. S., & Jannach, D. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557260 · Cited by 1
“Notable examples are image and text classification, see Baltrušaitis et al. [1] and Ramachandram and Taylor [43] for related surveys on multi-modal learning.”
BackgroundAFSFusion: An Adjacent Feature Shuffle Combination Network for Infrared and Visible Image Fusion
Hu, Y., Xu, S., Cheng, X., Zhou, C., & Xiong, M. (2023) · Applied Sciences · doi:10.3390/app13095640 · Cited by 2
Evaluating the Quality of Soft Computing in a Fog Computing Environment
Zhukova, N. A., & Subbotin, A. (2023) · Seminar on Information Systems Theory and Practice (ISTP) · doi:10.1109/istp60767.2023.10427099 · Cited by 1
“The problem is to quantitatively compare the results of the work of various methods, in the case when data processing is performed using different algorithms, on technical means that differ in architecture, in different execution environments [14], [15].”
BackgroundResultFu, D., Weber, C., Guochun, Y., Kerzel, M., Nan, W., Barros, P. V. A., Wu, H., Liu, X., & Wermter, S. (2019) · doi:10.31234/osf.io/s3u9j · Cited by 1
“, 2010; Shinn-Cunningham, 2008; Lee and Choo, 2013), most of them only focus either on a single modality or on general crossmodal processing (Lahat et al., 2015; Ramachandram and Taylor, 2017).”
BackgroundHuman Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial SensorsPreprint
Ahmad, Z., & Khan, N. (2023) · doi:10.32920/22734263.v1 · Cited by 1
A Dataset and Benchmarks for Multimedia Social AnalysisPreprint
Xue, B., Chan, D. M., & Canny, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08335 · Cited by 1
“This area of research has been relatively under-explored [20], and we hope that making access to a large-scale multi-modal dataset simple and accessible will help to spur research in this area.”
BackgroundHuman Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial SensorsPreprint
Ahmad, Z., & Khan, N. (2023) · doi:10.32920/22734263 · Cited by 1
Optimizing OCTA layer fusion option for deep learning classification of diabetic retinopathyPreprint
Yao, X., Ebrahimi, B., Le, D., Abtahi, M., Dadzie, A. K., Lim, J. I., & Chan, R. V. P. (2023) · doi:10.1364/opticaopen.23639007.v1 · Cited by 1
CMI-Net: a Unified Framework for Physiological Time Series Classification with Incomplete ModalitiesPreprint
Shen, Q. (2023) · doi:10.36227/techrxiv.24099339.v1 · Cited by 1
Ding, J., Wang, B., Qiao, Y., & Zhang, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4597961 · Cited by 1
Gómez-Martínez, V., Chushig-Muzo, D., Veierød, M. B., Granja, C., & Soguero-Ruíz, C. (2023) · Research Square · doi:10.21203/rs.3.rs-3781517/v1 · Cited by 2
Borredá, P. F. (2024) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/203192 · Cited by 2
iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokensin Large Multimodal ModelsPreprint
Hu, L., Shang, F., Wan, L., & Feng, W. (2024) · Qeios · doi:10.32388/69kflk · Cited by 1
Representation, information extraction, and summarization for automatic multimedia understanding
Harrando, I. (2022) · theses.fr (ABES) · doi:10.70675/702b8ba3z3acaz4ce3za9b9zf2cb1609e328 · Cited by 2
Audiovisual speech representation learning applied to emotion recognition
Sadok, S. (2024) · theses.fr (ABES) · doi:10.70675/0f15a36fzecf8z4cffz94e7z120e7d8fd39d · Cited by 4
[Research progress on electronic health records multimodal data fusion based on deep learning].
(2024) · PubMed Central · doi:10.7507/1001-5515.202310011 · Cited by 1
Camastra, C., Pelagi, A., Quattrone, A., & Sarica, A. (2026) · Brain Sciences · doi:10.3390/brainsci16040405 · Cited by 1
“…deep learning–based multimodal fusion have introduced increasingly expressive architectures—such as attention-based models, variational subspace learning, and graph neural networks (GNN)—that are specifically designed to address modality heterogeneity and complex cross-modal interactions [6,10,11].”
Deep Neural Networks with Auxiliary-Model Regulated Gating for Resilient Multi-Modal Sensor FusionPreprint
Zhao, C., Shim, M. S., Li, Y., Zhang, X., & Li, P. (2019) · arXiv (Cornell University)
Multimodal deep learning based approach for cells state classification
Silva, P. R. (2020) · ACM Symposium on Applied Computing · doi:10.1145/3341105.3374228
Assessing Modality Selection Heuristics to Improve Multimodal Deep Learning for Malware Detection
Bhuiyan, F. A., Brown, K., & Sharif, B. (2020) · Proceedings of Student Research and Creative Inquiry Day
Deep Multimodal Transfer-Learned Regression in Data-Poor DomainsPreprint
McClenny, L. D., Haile, M., Attari, V., Sadler, B. M., Braga-Neto, U., & Arróyave, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09310 · Cited by 2
“Multimodal Learning The concept of combining different domains of input into a single estimator for the desired learning task has been of distinct interest in the last decade [1,2].”
BackgroundA Multisensory Learning Architecture for Rotation-invariant Object RecognitionPreprint
Kırtay, M., Schillaci, G., & Hafner, V. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06292
“arning studies was reported in the fields of machine learning and robotics. Here, we merely introduce representative studies; however, we point out that more detailed information can be found in [9], [16], [4]. Since the employed dataset is recently constructed, and there are no benchmarked studies yet, we present the…”
BackgroundMemory based fusion for multi-modal deep learningPreprint
Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08076
“Different modalities including video, multi-spectral data and sensor data have been used in the above applications [23, 24, 25].”
Khan, N., & Ahmad, Z. (2023) · doi:10.32920/22734332.v1
Chen, Q., Keenan, T. D. L., Allot, A., Peng, Y., Agrón, E., Domalpally, A., Klaver, C. C. W., Luttikhuizen, D. T., Colyer, M. H., Cukras, C. A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05142
“This is not effective, especially when the modalities are very different [31 32].”
Attentional Multi-layer Feature Fusion Convolution Network for Audio-visual Speech Enhancement.Preprint
Xu, X., Wang, Y., Xu, D., Zhang, C., Peng, Y., Jia, J., & Chen, B. (2021) · arXiv (Cornell University)
MULTIMODAL ANALYSIS: Informed content estimation and audio source separationPreprint
Meseguer-Brocal, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13276
Deep Neural Network Based Student Response Modeling With Uncertainty, Multimodality and Attention
Ding, X. (2020) · SMU Scholar (Southern Methodist University)
Multimodal Data Evaluation for Classification Problems
Moctezuma, D., Muñiz-Sánchez, V., & García, J. (2021) · doi:10.5121/csit.2021.112105
“Once obtained the representation for each modality, they are combined generally with early or late fusion techniques [21], [22].”
MethodA Multimodal Dynamic Neural Network for Call for Help Recognition in Elevators
Ju, R., Chu, H., Wang, Y., Deng, Q., Cheng, M., & Li, M. (2021) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3461615.3491112
“One is the joint representation, also known as early fusion [D’mello and Kory 2015; Nojavanasghari et al. 2016; Ramachandram and Taylor 2017], which aims at fusing multimodal features in the middle parts of neural networks.”
BackgroundLearning Reaching Tasks Using an Arm Robot Equipped with MACMSA
Akikawa, M., & Yamamura, M. (2022) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2224/1/012008
“In recent years, techniques to control robots using deep learning have been developed to increase the accuracy of recognition [4,5].”
BackgroundOrgan preservation in rectal cancer: Capita Selecta
Haak, H. E. (2022) · Research Publications (Maastricht University) · doi:10.26481/dis.20220607hh
TripletMultiDTI: Multimodal Representation Learning in Drug-Target Interaction PredictionPreprint
Dehghan, A., Razzaghi, P., Abbasi, K., & Gharaghani, S. (2022) · Research Square · doi:10.21203/rs.3.rs-1881506/v1
Refcount Field Identification for Linux Kernel Based on Deep Learning
Tan, X., Yang, X., Cao, J., & Zhang, Y. (2022) · International Journal of Software and Informatics · doi:10.21655/ijsi.1673-7288.00288
2017 Index IEEE Signal Processing Magazine Vol. 34
(2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2773299
Tianxing, M., Osipov, V. Y., Zhukova, N. A., & Subbotin, A. (2022) · Research Square · doi:10.21203/rs.3.rs-2330139/v1
Matabane, M. E., & Seo, B. (2022) · Alternation Interdisciplinary Journal for the Study of the Arts and Humanities in Southern Africa · doi:10.29086/2519-5476/2022/v29n1a9
Multimodal reinforcement learning for partner specific adaptation in robot-multi-robot interaction
Kırtay, M., Hafner, V. V., Asada, M., Kuhlen, A. K., & Öztop, E. (2022) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids53995.2022.10000205
“We stress that the term modality refers to sensory data associated with different aspects of the interactive task, such as processing audio and visual data [13], [14].”
BackgroundScaling Subspace-Driven Approaches Using Information Fusion
Ghanem, S., & Krim, H. (2023) · Artificial intelligence · doi:10.5772/intechopen.109946
Hierarchical Transformer for Multi-Label Trailer Genre Classification
Cai, Z.-H., Ding, H., Wu, X., Xu, M., & Cui, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095502 · Cited by 1
“The frames and audio from a trailer are two different modalities, and there is usually information complementarity between different modalities [24].”
BackgroundA multimodal dynamical variational autoencoder for audiovisual speech representation learningPreprint
Sadok, S., Leglaive, S., Girin, L., Alameda-Pineda, X., & Séguier, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03582 · Cited by 1
REVIEW OF VECTOR EMBEDDINGS FUSION METHODS
Shaptala, R. (2022) · Telecommunication and information technologies · doi:10.31673/2412-4338.2022.048489
“У роботі [2] автори аналізують дані методи у площині практичних застосувань та наводять перелік наборів даних та завдань, які вирішуються ними.”
Successful Youtube video identification using multimodal deep learning
Rodrigues, L. D. S., Sakiyama, K. M., de Souza, L. F., Matsubara, E. T., & Nogueira, B. M. (2022) · Anais do Symposium on Knowledge Discovery, Mining and Learning (KDMiLe) · doi:10.5753/kdmile.2022.227792
“Our training protocol follows parts of [Ramachandram and Taylor 2017], which utilizes late fusion with the aggregation of decisions from multiple classifiers, each trained on separate modalities.”
BackgroundMethodInfluentialImproving information fusion in deep learning
Don, D. P. M. K. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.241291
Mutli-Modality Federate Learning with Multi-Source Data for Forest Fire Prediction
Mutakabbir, A. (2023) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2023-15634 · Cited by 1
Mejoras a la capacidad de generalización de la inteligencia artificial
Fumanal-Idocin, J. (2023) · Academica-e (Universidad Pública de Navarra) · doi:10.48035/tesis/2454/45885
A Real-World Clinical Data Mining of Post COVID-19 Patients
Gharibian, A., Mizera, J., Shrestha, B., Mikulková, Z., Genzor, S., Sova, M., Kudělka, M., & Kriegová, E. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-40971-4_41 · Cited by 1
Deep Learning-Based University-Assisted Management Solution for Public Health Emergencies
Liu, J., Zhang, L., & Wang, C. (2023) · Atlantis highlights in social sciences, education and humanities/Atlantis Highlights in Social Sciences, Education and Humanities · doi:10.2991/978-94-6463-230-9_32
Factorization in molecular modeling and belief propagation algorithms
Du, B., & Tian, P. (2023) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2023935
A Mix Fusion Spatial-Temporal Network for Facial Expression Recognition
Shu, C., & Feng, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8469-5_25
Interplay Between Neural Computational Energy and Multimodal Processing in Robot-Robot Interaction
Kırtay, M., Hafner, V. V., Asada, M., & Öztop, E. (2023) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/icdl55364.2023.10364527 · Cited by 1
“First, most of the work focused on multimodal pattern recognition by performing machine learning methods disregarding the computational cost of recognition/recalling operations [10].”
BackgroundRosales, R., Munoz, P., & Paulitsch, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00475
“Modality and point of view diversity [37] is an approach to address the failure modes of sensors such as cameras, radar, and lidar.”
BackgroundRepresenting Health Data and Medical Knowledge for Deep Learning
Hansen, E. R. (2023) · VBN Forskningsportal (Aalborg Universitet) · doi:10.54337/aau621651364
A Multimodal Target Speech Extraction Algorithm Based on Long Short Term Attention Mechanism
Li, Y., Li, R., & Li, M. (2023) · IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC) · doi:10.1109/icspcc59353.2023.10400336
“Intermediate fusion [11] combines early and late fusion approaches.”
BackgroundZhang, P., Zhao, H., Li, M., Chen, Y., Zhang, J., Wang, F., & Wu, X. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394129
“According to different fusion periods, feature fusion methods can be divided into early fusion, intermediate fusion and late fusion [15].”
BackgroundQi, J. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/30/20230096
Encoding and decoding multimodal brain images for disease biomarker discovery
Chan, Y. H. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/173338
Multimodal Sensor Fusion Frameworks With Application to Human Action Recognition
Ahmad, Z. (2024) · doi:10.32920/25234624.v1 · Cited by 1
Multimodal Deep Learning Analysis for Biomedical Data Fusion
Sinha, D., Rao, B. J., Basha, D. K., Kumar, P. P., Shilpa, N., & Sharma, S. (2024) · Wiley eBooks · doi:10.1002/9781394197705.ch4 · Cited by 1
One-vs-Many Scheme Based Multi-View Classification with Twin SVMPreprint
Liu, B., Su, S., Xiao, Y., Liu, Z., Li, W., Liu, W., & Jiang, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4761231
A Semantic Segmentation Network Exploiting Multiple Cues and Channel Exchanging
Hao, Y., Deng, W., Chen, X., Tang, J., Lu, H., & Zhou, Z. (2023) · China Automation Congress (CAC) · doi:10.1109/cac59555.2023.10450711
“It is recently specified as the deep learning-based method leveraging convolutional neural networks (CNN) [10].”
MethodWang, K., Gu, T., & Du, X. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-237691
RetractedYao, Y., Du, Z., Wang, X., & Wang, Q. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805668
Image classification with symbolic hints using limited resources
Jørgensen, M. G., Tětková, L., & Hansen, L. K. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301360
SEMIGRUPOS DE CLASE C0 EN L2 ([−π,π])
Ayala, Y. S. S. (2024) · doi:10.22533/at.ed.7242428057
Reales, L. A. N., Mendoza, E. L. B., Ortiz, J. M. V., & Nieto, F. V. (2024) · doi:10.22533/at.ed.7242428055
Metadata
- Authors
- Ramachandram, D., & Taylor, G. W.
- Affiliations
- Universiti Sains Malaysia (MY)
- University of Guelph (CA)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2017-11
- Volume
- 34
- Issue
- 6
- Pages
- 96-108
- ISSN
- 1053-5888 (ISSN-L)
- OpenAlex
- W2767290858
- Cited by
- 1,139
- References
- 113
- Citations by year
- 2026: 194 · 2025: 182 · 2024: 192 · 2023: 183 · 2022: 141 · 2021: 111 · 2020: 67 · 2019: 47 · 2018: 19 · 2017: 1
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal representation learning, multimodal architectures, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.