A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
Method
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Deep learning in cancer diagnosis, prognosis and treatment selection
Tran, K. A., Kondrashova, O., Bradley, A., Williams, E. D., Pearson, J. V., & Waddell, N. (2021) · Genome Medicine · doi:10.1186/s13073-021-00968-x · Cited by 932
“CNNs, RNNs, deep belief networks and autoencoders (AE) [21]; score-level fusion [22]; or multimodal data fusion [23].”
BackgroundZhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480
“Compared to the existing surveys [2, 4, 45, 46, 121, 128, 227, 235, 243] that usually focus on one single task (such as multimodal object recognition) with one specific combination of two modalities (such as RGB+depth data), this survey owns a broader scope covering various modalities and their…”
BackgroundA Systematic Literature Review on Crop Yield Prediction with Deep Learning and Remote Sensing
Muruganantham, P., Wibowo, S., Grandhi, S., Samrat, N. H., & Islam, N. (2022) · Remote Sensing · doi:10.3390/rs14091990 · Cited by 339
“The ANN approach was not efficient to capture the variability in micro-topography, which affected the prediction and had large model errors for fields with high yield variability [75].”
BackgroundVideo Summarization Using Deep Neural Networks: A Survey
Apostolidis, E., Adamantidou, E., Metsai, A. I., Mezaris, V., & Patras, I. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3117472 · Cited by 350
Deep Gait Recognition: A Survey
Sepas-Moghaddam, A., & Etemad, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3151865 · Cited by 307
“ods [253], [254], [255], [256], [257], [258], while multi-modal deep learning methods [259], [260], notably based on fusion [261], joint learning [233], and attention [262] net-”
BackgroundAttention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
Deep learning based multimodal biomedical data fusion: an overview and comparative review
Duan, J., Xiong, J., Li, Y., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102536 · Cited by 284
Task-Oriented Multi-User Semantic Communications for VQA
Xie, H., Qin, Z., & Li, G. Y. (2022) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2021.3136045 · Cited by 243
“The recent successful approaches for multimodal data fusion are mostly based on neural networks, and representative techniques include Deep Belief Net (DBN), Stacked Autoencoder (SAE), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) [11].”
BackgroundDeep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review
Cui, C., Yang, H., Wang, Y., Zhao, S., Asad, Z., Coburn, L. A., Wilson, K. T., Landman, B. A., & Huo, Y. (2023) · Progress in Biomedical Engineering · doi:10.1088/2516-1091/acc2fe · Cited by 218
“Deep learning includes high-level abstraction of complex phenomenon within high-dimensional data, which tends to benefit multimodal fusion in extracting and modeling the complex relationships of different modalities and outcomes [6, 7].”
Review of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235
Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214
A Review on Methods and Applications in Multimodal Deep Learning
Jabeen, S., Li, X., Amin, M. S., Bourahla, O., Li, S., & Jabbar, A. (2023) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3545572 · Cited by 196
“[41] 2020 NC(MIT) Not Available Mogadala et al.”
BackgroundShaban, A., & Yousefi, S. (2024) · Springer Optimization and Its Applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 189
Artificial Intelligence and Biosensors in Healthcare and Its Clinical Relevance: A Review
(2023) · doi:10.1109/access.2023.3285596 · Cited by 205
“The main goal of methods used to combine multimodal data is to combine the data with values from various scales and distributions into a global feature space, where the data may be represented more consistently [76].”
MethodEffective Techniques for Multimodal Data Fusion: A Comparative Analysis
Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023) · Sensors · doi:10.3390/s23052381 · Cited by 180
“A few issues should be taken into account [4] when it comes to fusing several modalities:”
BackgroundMethodInfluentialUse of Multi-Modal Data and Machine Learning to Improve Cardiovascular Disease Care
Amal, S., Safarnejad, L., Omiye, J. A., Ghanzouri, I., Cabot, J. H., & Ross, E. G. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.840262 · Cited by 172
“Methods used to fuse multimodal data fundamentally aim to integrate the data with values of different scales and distributions into a global feature space (i.e., database) in which data can be represented in a more uniform manner (9).”
MethodMachine learning for next-generation intelligent transportation systems: A survey
Yuan, T., Da Rocha Neto, W., Rothenberg, C. E., Obraczka, K., Barakat, C., & Turletti, T. (2022) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.4427 · Cited by 179
“Emerging technologies, such as multimodal data fusion [257] and deep reasoning [258], are promising avenues for deep sensing in ITS.”
Bayoudh, K. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 159
A systematic review of data fusion techniques for optimized structural health monitoring
Hassani, S., Dackermann, U., Mousavi, M., & Li, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102136 · Cited by 182
Artificial Intelligence for Alzheimer’s Disease: Promise or Challenge?
Fabrizio, C., Termine, A., Caltagirone, C., & Sancesario, G. (2021) · Diagnostics · doi:10.3390/diagnostics11081473 · Cited by 146
“In order to overcome this limitation and extract multimodal feature representations, DL architectures can be used, which do not need feature engineering, due to their ability to non-linearly transform input variables [78].”
BackgroundSayyad, S., Kumar, S., Bongale, A. M., Kamat, P. V., Patil, S., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3101284 · Cited by 175
A review on EEG-based multimodal learning for emotion recognition
Pillalamarri, R., & Shanmugam, U. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11126-9 · Cited by 183
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126
“…simulation and DL [256] Open-set recognition [257] Domain adaptation [255] Multimodal DL Multimodal learning models [258] Data fusion strategies [259] Omics applications [260] Efficient DL Parallelization and distribution [261] Compression and acceleration [262] Biomedical imaging applications…”
BackgroundMethodMultimodal Classification: Current Landscape, Taxonomy and Future Directions
Sleeman, W. C., Kapoor, R., & Ghosh, P. (2023) · ACM Computing Surveys · doi:10.1145/3543848 · Cited by 124
“This cross-modality fusion architecture is often also used with deep belief network (DBN) or autoencoder-style networks [25, 32, 59].”
BackgroundInfluentialApplications of deep learning in congestion detection, prediction and alleviation: A survey
Kumar, N., & Raubal, M. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103432 · Cited by 147
Kabir, H., Tham, M., & Chang, Y. C. (2023) · Digital Communications and Networks · doi:10.1016/j.dcan.2023.05.006 · Cited by 143
A Comprehensive Survey Analysis for Present Solutions of Medical Image Fusion and Future Directions
Faragallah, O. S., El‐Hoseny, H. M., El‐Shafai, W., El‐Rahman, W. A., El‐sayed, H. S., El‐Rabaie, E. M., El‐Samie, F. E. A., & Geweid, G. G. N. (2020) · IEEE Access · doi:10.1109/access.2020.3048315 · Cited by 137
A survey of multimodal deep generative models
Suzuki, M., & Matsuo, Y. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2035253 · Cited by 109
“There have already been many surveys on multimodal (or multi-view) machine learning [3, 5, 17, 45, 87, 108, 117, 122].”
Barua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 152
“References [20], [21], and [22] are three recent survey works of multimodal fusion, and they classify approaches including different modalities.”
BackgroundA Comprehensive Review on Deep Learning-Based Data Fusion
Hussain, M., O’Nils, M., Lundgren, J., & Mousavirad, S. J. (2024) · IEEE Access · doi:10.1109/access.2024.3508271 · Cited by 140
Diagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review
Shoeibi, A., Khodatars, M., Jafari, M., Ghassemi, N., Moridian, P., Alizadehsani, R., Ling, S. H., Khosravi, A., Alinejad‐Rokny, H., Lam, H., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.12.010 · Cited by 121
Multimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101
“A handful of review papers have also been written on this topic (Ramachandram and Taylor, 2017; Gao et al. , 2020).”
BackgroundEnsemble Multifeatured Deep Learning Models and Applications: A Survey
Abimannan, S., El-Alfy, E.-S. M., Chang, Y., Hussain, S., Shukla, S., & Satheesh, D. (2023) · IEEE Access · doi:10.1109/access.2023.3320042 · Cited by 119
“…used in a broader sense and involves the amalgamation of information from multiple sources (which could be raw data, features, modalities, views, algorithms or models) and can occur at different levels (sensor-level, feature-level, score-level, decision-level, or rank-level) [17], [18], [19], [20].”
BackgroundA comprehensive review of robot intelligent grasping based on tactile perception
Li, T., Yan, Y., Yu, C., An, J., Wang, Y., & Chen, G. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102792 · Cited by 126
A systematic review of intermediate fusion in multimodal deep learning for biomedical applications
Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105509 · Cited by 95
Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking
Wang, R., Zhu, J., Wang, S., Wang, T., Huang, J., & Zhu, X. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-024-00347-3 · Cited by 136
“For instance, early fusion strategies begin integrating information from different modalities in the initial stages of data processing, such as using LSTM networks to link features of each modality for key sequence information extraction [5, 6].”
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90
Multimodal image fusion: A systematic review
Kalamkar, S. P., & Amalanathan, G. M. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100327 · Cited by 112
Deep Learning for Technical Document Classification
Jiang, S., Hu, J., Magee, C. L., & Luo, J. (2022) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2022.3152216 · Cited by 111
“It is important to note that, multimodal data fusion would naturally require extra computation and training time than unimodal learning [97].”
BackgroundRecent Advances and Trends in Multimodal Deep Learning: A ReviewPreprint
Jabeen, S., Li, X., Amin, M. S., Li, S., & Abdul, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11087 · Cited by 140
“The fundamentals of MMDL fusion methods and their challenges are discussed by Gao et al. [40].”
MethodLearning from data with structured missingness
Mitra, R., McGough, S. F., Chakraborti, T., Holmes, C. C., Copping, R., Hagenbuch, N., Biedermann, S., Noonan, J., Lehmann, B., Shenvi, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00596-z · Cited by 139
“These issues are particularly acute in areas that require extensive data fusion, multi-view learning, and/or data linkage that draw information from multiple sources and studies (8; 9; 10).”
A review of cancer data fusion methods based on deep learning
Zhao, Y., Li, X., Zhou, C., Peng, H., Zheng, Z., Chen, J., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102361 · Cited by 95
Multimodal semantic communication accelerated bidirectional caching for 6G MEC
Wang, C., Yu, X., Xu, L., Wang, Z., & Wang, W. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.10.036 · Cited by 94
Rousset, G., Despinoy, M., Schindler, K., & Mangeas, M. (2021) · Remote Sensing · doi:10.3390/rs13122257 · Cited by 93
“This variation was considered negligible for this study, as Deep Learning normally copes well with multimodal data distributions [27].”
MethodMultimodal brain tumor detection using multimodal deep transfer learning
Razzaghi, P., Abbasi, K., Shirazi, M., & Rashidi, S. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109631 · Cited by 84
Neurosymbolic AI for Reasoning Over Knowledge Graphs: A Survey
DeLong, L. N., Mir, R. F., & Fleuriot, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3420218 · Cited by 134
“For example, a news article might contain both text and images to describe a story [46], [84].”
Danilevicz, M. F., Bayer, P. E., Boussaid, F., Bennamoun, M., & Edwards, D. (2021) · Remote Sensing · doi:10.3390/rs13193976 · Cited by 75
“In most multimodal deep learning models, each deep learning module specializes in a single data type depicting one aspect of the phenomenon, which are then concatenated and used to inform the prediction [25,26].”
MethodMultimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 128
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 186
Self-citationFederated Learning on Multimodal Data: A Comprehensive Survey
Lin, Y., Gao, Y., Gong, M., Zhang, S., Zhang, Y., & Li, Z. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1398-0 · Cited by 96
“For example, Bayoudh et al.[26] focused on computer vision-related fields such as vision and language, summarizing six perspectives in the current literature on deep multimodal learning.”
BackgroundMDMN: Multi-task and Domain Adaptation based Multi-modal Network for early rumor detection
Zhou, H., Ma, T., Rong, H., Qian, Y., Tian, Y., & Al-Nabhan, N. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116517 · Cited by 71
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 81
Multimodal Machine Learning Guides Low Carbon Aeration Strategies in Urban Wastewater Treatment
Wang, H., Wang, Y., Wang, X., Yin, W.-X., Yu, T., Xue, C.-H., & Wang, A. (2024) · Engineering · doi:10.1016/j.eng.2023.11.020 · Cited by 69
Fusion of tactile and visual information in deep learning models for object recognition
Babadian, R. P., Faez, K., Amiri, M., & Falotico, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.11.032 · Cited by 67
Deep Learning in Diverse Intelligent Sensor Based Systems
Zhu, Y., Wang, M., Yin, X., Zhang, J., Meijering, E. H. W., & Hu, J. (2022) · Sensors · doi:10.3390/s23010062 · Cited by 84
Self-citationLiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification
Yang, J. X., Zhou, J. W., Wang, J., Tian, H., & Liew, A. W. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3389651 · Cited by 102
“D profiles to measure the distance between the sensor and the surface of the Earth, offering precise topographic and structural information [3].”
BackgroundFew-Shot Image Classification: Current Status and Research Trends
Liu, Y., Zhang, H., Zhang, W., Lu, G., Tian, Q. C., & Ling, N. (2022) · Electronics · doi:10.3390/electronics11111752 · Cited by 80
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, D., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 63
Multimodality Fusion Aspects of Medical Diagnosis: A Comprehensive Review
Kumar, S., Rani, S., Sharma, S., & Min, H. (2024) · Bioengineering · doi:10.3390/bioengineering11121233 · Cited by 85
Chen, Z., Zhang, S., Shi, N., Li, F., Wang, Y., & Cui, J. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109615 · Cited by 63
Self-citationAttention based hybrid deep learning model for wearable based stress recognition
Tanwar, R., Phukan, O. C., Singh, G., Pal, P. K., & Tiwari, S. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107391 · Cited by 69
Livestock feeding behaviour: A review on automated systems for ruminant monitoring
Chelotti, J. O., Martinez-Rau, L. S., Ferrero, M., Vignolo, L. D., Galli, J. R., Planisich, A. M., Rufiner, H. L., & Giovanini, L. L. (2024) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2024.08.003 · Cited by 76
“Integrating complementary data sources ( multimodal data fusion ) can lead to better recognition performances than algorithms using individual sources (Gao, Li, Chen, & Zhang, 2020).”
BackgroundDeep learning for low-data drug discovery: Hurdles and opportunities
van Tilborg, D., Brinkmann, H., Criscuolo, E., Rossen, L., Özçelik, R., & Grisoni, F. (2024) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2024.102818 · Cited by 61
Digital twin systems for musculoskeletal applications: A current concepts review
Diniz, P., Grimm, B., Garcia, F., Fayad, J., Ley, C., Mouton, C., Oeding, J. F., Hirschmann, M. T., Samuelsson, K., & Seil, R. (2025) · Knee Surgery, Sports Traumatology, Arthroscopy · doi:10.1002/ksa.12627 · Cited by 100
CNN‐Transformer for visual‐tactile fusion applied in road recognition of autonomous vehicles
Shi, R., Yang, S., Chen, Y., Wang, R., Zhang, M., Lu, J., & Cao, Y. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.11.023 · Cited by 71
Surface-Enhanced Raman Scattering-Based Multimodal Techniques: Advances and Perspectives
Tan, E. X., Zhong, Q., Chen, J. R. T., Leong, Y. X., Leon, G. K., Tran, C. T., Phang, I. Y., & Ling, X. Y. (2024) · ACS Nano · doi:10.1021/acsnano.4c12996 · Cited by 64
Khader, F., Kather, J. N., Müller-Franzes, G., Wang, T., Han, T., Tayebi Arasteh, S., Hamesch, K., Bressem, K., Haarburger, C., Stegmaier, J., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-37835-1 · Cited by 57
S.Rajarajeshwari, G. K. P., & Selvi, G. C. (2024) · IEEE Access · doi:10.1109/access.2024.3494840 · Cited by 68
“There are several key considerations to keep in mind when fusing different modalities [121], [122]: Intermodality: This refers to the integration of various modalities, which can enhance model predictions and improve their robustness [123].”
DeepMNF: Deep Multimodal Neuroimaging Framework for Diagnosing Autism Spectrum Disorder
Abbas, S. Q., Chi, L., & Chen, Y. P. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102475 · Cited by 64
Attallah, O. (2023) · Chemosensors · doi:10.3390/chemosensors11070364 · Cited by 61
“According to the multimodal machine learning paradigm, multimodal fusion techniques can be model-based or model-agnostic [59].”
BackgroundLarge scale foundation models for intelligent manufacturing applications: a survey
Zhang, H., Semujju, S. D., Wang, Z., Lv, X., Xu, K., Wu, L., Jia, Y., Wu, J., Liang, W., Zhuang, R., et al. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-024-02536-7 · Cited by 83
Deng, Z., Zheng, Y., Lan, T., Zhang, L., Yun, Y., & Song, W. (2025) · Food Chemistry · doi:10.1016/j.foodchem.2025.142930 · Cited by 54
Li, G., Yang, L., Li, S., Luo, X., Qu, X., & Green, P. (2021) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2021.3085986 · Cited by 53
Precipitable water vapor fusion of MODIS and ERA5 based on convolutional neural network
Lu, C., Zhang, Y., Zheng, Y., Wu, Z., & Wang, Q. (2022) · GPS Solutions · doi:10.1007/s10291-022-01357-6 · Cited by 52
“Deep learning has been undergoing major development, exhibiting strong feature extraction capabilities in processing massive data due to its data-driven essence (Chen et al. 2017; Gao et al. 2020; Liu et al. 2020).”
BackgroundGuo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 69
“Image-text multimodal models [108, 169, 170, 171, 172, 173, 174, 175]are a product of artificial intelligence that can process and understand various types of information, such as text and images, simultaneously.”
BackgroundFoundations and Innovations in Data Fusion and Ensemble Learning for Effective Consensus
Du, K.-L., Zhang, R., Jiang, B., Zeng, J., & Lu, J. (2025) · Mathematics · doi:10.3390/math13040587 · Cited by 63
Ten quick tips for biomarker discovery and validation analyses using machine learning
Díaz‐Uriarte, R., de Lope, E. G., Giugno, R., Fröhlich, H., Nazarov, P. V., Nepomuceno-Chamorro, I. A., Rauschenberger, A., & Glaab, E. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010357 · Cited by 67
Formalizing Multimedia Recommendation through Multimodal Deep Learning
Malitesta, D., Cornacchia, G., Pomo, C., Merra, F. A., Noia, T. D., & Sciascio, E. D. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3662738 · Cited by 81
“After that, the recent years have seen a growing interest in systematically reviewing and schematizing techniques for multimodal fusion [32], spanning different application domains such as medicine [97], conversational artificial intelligence [92], and visual content syntesis [129], up to…”
BackgroundGenerative Deep Learning for de Novo Drug Design─A Chemical Space Odyssey
Özçelik, R., Brinkmann, H., Criscuolo, E., & Grisoni, F. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00641 · Cited by 71
Image and Text fusion for UPMC Food-101 using BERT and CNNs
Gallo, I., Ria, G., Landro, N., & Grassa, R. L. (2020) · International Conference on Image and Vision Computing New Zealand (IVCNZ) · doi:10.1109/ivcnz51579.2020.9290622 · Cited by 63
“On e-commerce websites, for each item on sale, a user can select a product based on a text and image that shows characteristics, colours and other features of the product [1] and in general in the literature exists different approaches to multimodal classification [2] [3].”
Pohtongkam, S., & Srinonchat, J. (2021) · Sensors · doi:10.3390/s21186024 · Cited by 46
“In this experiment, multimodal learning [81,82] was applied to increase the effectiveness of the recognition rate.”
MethodA Hybrid Deep Learning Framework with Decision-Level Fusion for Breast Cancer Survival Prediction
Othman, N. A., Abdel-Fattah, M. A., & Ali, A. T. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7010050 · Cited by 52
Multi-Source Information Fusion Technology and Its Application in Smart Distribution Power System
He, X., Dong, H., Yang, W., & Li, W. (2023) · Sustainability · doi:10.3390/su15076170 · Cited by 50
Zhang, Y., Zhao, T., Gao, S., & Raubal, M. (2023) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2023.2234959 · Cited by 44
“The emergence of attention techniques provides an opportunity to combine traffic speed with multimodal contexts by dynamically weighting them before being fused together (Liu et al. 2018, Gao et al. 2020).”
BackgroundSeo, S., Kim, Y., Han, H., Son, W., Hong, Z.-Y., Sohn, I., Shim, J., & Hwang, C. (2021) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.670670 · Cited by 52
“Classi fi cation with multimodal data often occurs in many machine learning applications (Baltru ˇ saitis et al., 2019; Gao et al., 2020).”
BackgroundLi, H., Ju, W., Song, Y., Cao, Y., Yang, W., & Li, M. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108561 · Cited by 43
Multimodal fusion techniques: Review, data representation, information fusion, and application areas
Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 59
Di, X., Cui, K., & Wang, R.-F. (2025) · Remote Sensing · doi:10.3390/rs17132235 · Cited by 51
Sending or not? A multimodal framework for Danmaku comment prediction
Xi, D., Xu, W. X., Chen, R., Zhou, Y., & Yang, Z. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102687 · Cited by 48
Kumar, A., Manikandan, R., Kose, U., Gupta, D., & Satapathy, S. C. (2021) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3457187 · Cited by 39
Politikos, D. V., Adamopoulou, A., Petasis, G. C., & Galgani, F. (2022) · Ocean & Coastal Management · doi:10.1016/j.ocecoaman.2022.106466 · Cited by 48
Towards intelligent cooperative robotics in additive manufacturing: Past, present, and future
Rescsanski, S., Hebert, R. J., Haghighi, A., Tang, J., & Imani, F. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102925 · Cited by 47
Context in object detection: a systematic literature review
Jamali, M., Davidsson, P., Khoshkangini, R., Ljungqvist, M. G., & Mihailescu, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11186-x · Cited by 53
Cohesive Multi-Modality Feature Learning and Fusion for COVID-19 Patient Severity Prediction
Zhou, J., Zhang, X., Zhu, Z., Lan, X., Fu, L., Wang, H., & Wen, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3063952 · Cited by 38
“Multi-modality data fusion and classification aim to integrate the data of different distributions, sources, and types into a global space in which both inter-modality and cross-modality can be represented in a unified manner [11].”
BackgroundA heterogeneous multi-modal medical data fusion framework supporting hybrid data exploration
Zhang, Y., Sheng, M., Liu, X., Wang, R., Lin, W., Ren, P., Wang, X., Zhao, E., & Song, W. (2022) · Health Information Science and Systems · doi:10.1007/s13755-022-00183-x · Cited by 39
“Artificial intelligence techniques are also implemented in recent studies [6–8], including deep neural networks for feature extraction and image classification [6, 7, 9], and ensemble learning for classification of imbalanced streaming data [10].”
MethodA Review on Data Fusion of Multidimensional Medical and Biomedical Data
Azam, K. S. F., Ryabchykov, O., & Bocklitz, T. W. (2022) · Molecules · doi:10.3390/molecules27217448 · Cited by 45
“[57] provides a good summary of the deep learning data fusion models.”
BackgroundUtilizing multimodal artificial intelligence to advance cardiovascular diseases
Yang, X.-Y., Li, Y., Wang, J., Jia, Y., Zhang, Y., & Chen, M. (2025) · Precision Clinical Medicine · doi:10.1093/pcmedi/pbaf016 · Cited by 50
Liu, D., Zhen, H., Kong, D., Chen, X., Zhang, L., Yuan, M., & Wang, H. (2021) · Scientific Programming · doi:10.1155/2021/6699313 · Cited by 41
Pre-gating and contextual attention gate — A new fusion method for multi-modal data tasks
Zhang, D., Nayak, R., & Bashar, M. A. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106553 · Cited by 47
MDF-FND: A dynamic fusion model for multimodal fake news detection
Lv, H., Yang, W., Yin, Y., Wei, F., Peng, J., & Geng, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113417 · Cited by 48
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.