A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
Method
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Items 301–400 of 837
Real-time rainfall estimation using deep learning: Influence of background and rainfall intensity
Qin, X., Zhu, Q., Shen, J., Chen, H., & Gao, X. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106496 · Cited by 7
FedVLP: Visual-aware latent prompt generation for Multimodal Federated Learning
Pan, H., Zhao, X., Jiang, Y., He, L., Wang, B., & Shu, Y. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104442 · Cited by 7
Zhou, J., Niu, X., Liu, Z., Ma, Q., Li, J., Hu, B., & Wu, Z. Y. (2025) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-025-10324-6 · Cited by 7
“Multimodal data fusion and explainable AI technolo - gies provide more comprehensive information and greater decision transparency for wafer map analysis, making the application of ML models more practical and reliable [ 188].”
MOTIF: A text-aware and disentangled Mamba-based architecture for multimodal sentiment analysis
Hu, B., & Li, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114651 · Cited by 9
Xu, C., Zhao, L., Wen, H., Zhang, Y., Wang, L., & Zhang, L. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.111443 · Cited by 9
Kas, M., & Ruichek, Y. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109252 · Cited by 8
A model for recognizing farming behaviors of plantation workers
Xu, J., Zhao, W., Wei, C., Hu, X., & Li, X. (2022) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2022.107395 · Cited by 7
Third Generation Neural Nets and Their Applications in Multi-modal Deep Learning: A Survey
Sayad, I. A. E., Pannu, M., Gourde, J., & Nakshabandi, M. A. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47448-4_3 · Cited by 7
Alfeo, A. L., Cimino, M. G. C. A., & Gagliardi, G. (2024) · SN Computer Science · doi:10.1007/s42979-024-02635-5 · Cited by 7
“The intermediate information representations between two layers (i.e. the so-called latent space), can also be used as features in the following recognition task [22].”
A Comprehensive Analysis of Techniques and Applications in Multimodal Deep Learning
Tomar, V., Sharma, S., Arora, S., & Tyagi, A. D. (2024) · International Conference on Computing, Sciences and Communications (ICCSC) · doi:10.1109/iccsc62048.2024.10830379 · Cited by 7
Luo, Q., Huang, N., Bartles, D. L., Simpson, T. W., & Beese, A. M. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-025-02576-7 · Cited by 9
“Key challenges in using multi-source frameworks include identifying the optimal feature representative for each single-source input modality and restoring the characteristics from each modality representative after data fusion (Gao et al., 2020; Lahat et al., 2015; Y. Li et al., 2019).”
Audio–visual self-supervised representation learning: A survey
Alsuwat, M. S., Al-Shareef, S., & Alghamdi, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129750 · Cited by 10
Udayakumar, P., & Subhashini, R. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.105200 · Cited by 11
Fall risk assessment through a synergistic multi-source DNN learning model
Andreeva, O., Ding, W. Y., Leveille, S. G., Cai, Y., & Chen, P. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102280 · Cited by 7
MMRCA: MultiModal Root Cause Analysis
White, G., Diuwe, J., Fonseca, E., & O’Brien, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14135-5_14 · Cited by 9
AMFFNet: Asymmetric Multiscale Feature Fusion Network of RGB-NIR for Solid Waste Detection
Cai, Z., Fang, H., Jiang, F., Yang, J., Ji, T., Hu, Y., & Wang, X. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3300445 · Cited by 8
“Recently, more and more researchers have been working on multi-modal fusion [28], [29].”
BackgroundForest Fire Prediction Using Multi-Source Deep Learning
Mutakabbir, A., Lung, C., Ajila, S. A., Zaman, M., Naik, K., Purcell, R., & Sampalli, S. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-52265-9_9 · Cited by 7
Concise and Effective Network for 3D Human Modeling From Orthogonal Silhouettes
Liu, B., Liu, X., Yang, Z., & Wang, C. C. L. (2022) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4054001 · Cited by 7
“Based on the results obtained from experiment, we believe that our architecture can be used as a basic network, which can derive different variants by modifying the structures of BlockA and BlockB according to the strategy of multi-modal fusion learning [53].”
MethodComparative assessment of text-image fusion models for medical diagnostics
Gusarova, N., Lobantsev, A., Vatian, A. S., Kapitonov, A., & Shalyto, A. A. (2020) · Информационно-управляющие системы · doi:10.31799/1684-8853-2020-5-70-79 · Cited by 8
“The analysis of reviews [6, 17, 20, 21, 27, 28] devoted to the problem of multimodal fusion allows us to highlight the main aspects that determine specific technological solutions for the fusion of images and texts in medicine: — features the fusion is based on; —”
BackgroundIoT-based critical infrastructure enabled radar information fusion
Yang, J., Zhang, Z., Mao, W., Sun, Y., Bao, Y., & Song, H. H. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.107723 · Cited by 7
Similarity-based multimodal regression
Chen, A. A., Weinstein, S. M., Adebimpe, A., Gur, R. C., Gur, R. E., Merikangas, K. R., Satterthwaite, T. D., Shinohara, R. T., & Shou, H. (2024) · Biostatistics · doi:10.1093/biostatistics/kxad033 · Cited by 6
Qu, Y., Sun, J., Yu, J., Tian, Y., & Dong, H. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3424556 · Cited by 6
“Deep-learning (DL) methods, as typical data-based techniques, have been developed to extract and fuse multiscale features of different types of data [25].”
MethodMultimodal representation learning for medical analytics - a systematic literature review
Hansen, E. R., Sagi, T., & Hose, K. (2024) · Health Informatics Journal · doi:10.1177/14604582241290474 · Cited by 8
“28 Multiple layers of fully connected perceptrons are used to ultimately fuse uni-modal representations in either early, intermediate, or late layers of the network structure (Figure 7(a)).”
How to price a dataset: a deep learning framework for data monetization with alternative data
Hao, J., Deng, Z., Li, J., & Li, J. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06016-y · Cited by 6
“Incorporating alternative data, such as textual information, in addition to conventional features, may be an effective solution for describing data characteristics and enhancing the accuracy of data pricing (Gao et al. 2020).”
Rodrigues, L. D. S., Filho, E. C., Sakiyama, K. M., Santos, M. F., Jank, L., Carromeu, C., Silveira, E., Matsubara, E. T., Marcato, J., & Gonçalves, W. N. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.107957 · Cited by 6
Patel, B., Orlichenko, A., Patel, A., Qu, G., Wilson, T. W., Stephen, J. M., Calhoun, V. D., & Wang, Y. (2024) · Applied Sciences · doi:10.3390/app14104144 · Cited by 8
Urban Functional Zone Classification via Advanced Multi-Modal Data Fusion
Liu, T., Chen, H., Ren, J., Zhang, L., Chen, H., Hong, R., Li, C., Cui, W., Guo, W., & Wen, C. (2024) · Sustainability · doi:10.3390/su162411145 · Cited by 8
Deep learning for multi-modal data fusion in IoT applications
Saghir, A., Akbar, A., Zafar, A., & Hassan, A. (2025) · Mehran University Research Journal of Engineering and Technology · doi:10.22581/muet1982.3171 · Cited by 7
“Various mathematical theories and probability-based approaches exist for data fusion but with very complex models [13-20].”
Deep-Learning Based Multi-Modalities Fusion for the Detection of Brain-Related Diseases: A Review
Imran, S. M. A., Arif, M., Jaffar, A., Khushi, H. M. T., & Hussain, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-77620-5_11 · Cited by 8
Learning a Data Center Model for Efficient Demand Response
Clark, Q., Acun, F., Paschalidis, I. C., & Coskun, A. K. (2024) · ACM SIGEnergy Energy Informatics Review · doi:10.1145/3727200.3727215 · Cited by 8
“We use aggregate fusion, where the feature vectors are concatenated, but more sophisticated approaches exist [52] [13].”
Escudero-Arnanz, Ó., Martínez-Agüero, S., Martín-Palomeque, P., Marques, A. G., Mora-Jiménez, I., Álvarez-Rodríguez, J., & Soguero-Ruíz, C. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00351-9 · Cited by 10
“The concept of “data fusion” or “multimodality” refers to integrating data from multiple sources to derive complementary and comprehensive insights [37].”
Unsupervised Deep Learning: Taxonomy and algorithms
Chefrour, A., & Souici‐Meslati, L. (2022) · Informatica · doi:10.31449/inf.v46i2.3820 · Cited by 6
Hypergraph Variational Autoencoder for Multimodal Semi-supervised Representation Learning
Liu, J., Du, X., Li, Y., & Hu, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15937-4_33 · Cited by 9
Guidelines for conducting biofeedback-enhanced QoE studies in mulsemedia-enhanced virtual reality
da Silveira, A. C., Raisamo, R., Spyridonis, F., Covaci, A., Ghinea, G., & Santos, C. A. S. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3617023.3617029 · Cited by 6
“Although it is an active area of research [14, 19] how this is undertaken in the case of multiple sensory modalities and mulsemedia data including, for instance, EEG, EDA or ECG signals, as well as eye-tracking data is not well understood.”
BackgroundIMAD: Image-Augmented Multi-Modal Dialogue
Moskvoretskii, V., Frolov, A., & Kuznetsov, D. (2024) · Journal of Mathematical Sciences · doi:10.1007/s10958-024-07434-0 · Cited by 9
“These problems are solved with multi-modal deep learning models, that are increasingly important today due to the exponential growth of multimedia data in various domains [19, 20, 21, 22].”
Prediction of Public Trust in Politicians Using a Multimodal Fusion Approach
Syed, M. S. S., Pirogova, E., & Lech, M. (2021) · Electronics · doi:10.3390/electronics10111259 · Cited by 7
“The authors in [46] provided a review of methods for multimodal data fusion.”
BackgroundPointBoost: LiDAR-Enhanced Semantic Segmentation of Remote Sensing Imagery
Zhang, Y., Wang, Y., Yao, Y., Zhou, W., & Zhang, B. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3286912 · Cited by 6
“Different types of remote sensing data are suitable for individual application, with optical imagery and Light Detection and Ranging (LiDAR), which are prominent examples [5], [6], [7].”
BackgroundMa, R., Lou, J., Li, P., & Gao, J. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/8868781 · Cited by 6
Self-citationJony, R. I. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.238058 · Cited by 9
XAI-based Data Visualization in Multimodal Medical DataPreprint
Sharma, S., Singh, M., Mcdaid, L. J., & Bhattacharyya, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.11.664302 · Cited by 16
Online Energy Efficient Multimodal Probabilistic Semantic Communication
Dai, J., Li, J., Zhao, Z., Yang, Z., Ye, J., Yang, Q., Huang, C., & Zhang, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3572209 · Cited by 5
“Representative methods include deep belief networks (DBN), stacked auto-encoders (SAE), convolutional neural networks (CNN), and recurrent neural networks (RNN) [10].”
Tan, L., Ma, L., Chen, C., Lu, G., & Lü, H. (2025) · Sustainable Cities and Society · doi:10.1016/j.scs.2025.106527 · Cited by 5
Li, X., Li, Q., Zhang, J., Sun, J., Li, R., & Li, J. (2025) · Journal of Energy Storage · doi:10.1016/j.est.2025.118684 · Cited by 6
Self-citationShi, G., Guan, H., Wang, Y., Ma, N., 崔敏超, & Luo, M. (2025) · Optics & Laser Technology · doi:10.1016/j.optlastec.2025.113990 · Cited by 5
Reconstruction of multimodal aesthetic critical discourse analysis framework
Dong, M. (2022) · Applied Mathematics and Nonlinear Sciences · doi:10.2478/amns.2021.2.00165 · Cited by 5
Zhang, P., Zhang, W., Sun, W., Xu, J., Hu, H., Wang, L., & Wong, L. (2024) · BMC Genomics · doi:10.1186/s12864-024-09967-9 · Cited by 5
“Integrating multiple types of networks allows for the combination of multi-dimensional information, compensating for the limitations of a single network [29, 30].”
BackgroundJin, Y., Liu, J., Ren, K., Wang, X., Deng, K., Fan, Z., Deng, C., & Yue, Y. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3435957 · Cited by 9
“Different modalities reflect unique physical mechanisms and observational characteristics of TCs [44], [45].”
BackgroundDual Optimization of Deep CNN for Motor Imagery EEG Tasks Classification
Al-Saegh, A., Daood, A. I., & Ismail, M. H. (2024) · Diyala Journal of Engineering Sciences · doi:10.24237/djes.2024.17405 · Cited by 10
“A substantial amount of training samples is required to tune the learnable parameters; otherwise, the underfitting problem may occur [19].”
Big data fusion with knowledge graph: a comprehensive overview
Liu, J., Lan, R., Du, Y., Yuan, X., Xu, H., Li, T., Huang, W. X., & Zhang, P. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06549-4 · Cited by 6
“Datafusionmakes consistent, accurateanduseful representa-tionofdatasetsfromdifferentsources,typesanddistributions to reduce uncertainty, improving reliability and correlation [34, 91, 150].”
Advancing Pain Recognition Through Statistical Correlation-Driven Multimodal Fusion
Gu, X., Wang, Z., Jin, I., & Wu, Z. (2024) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw63320.2024.00056 · Cited by 8
“However, the challenge of effectively integrating and processing this heterogeneous data persists, and overcoming this challenge is essential for advancing multimodal data fusion and maximizing the efficacy of analytical methods for pain recognition [10].”
UniC: a dataset for emotion analysis of videos with multimodal and unimodal labels
Du, Q., Labat, S., Demeester, T., & Hoste, V. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09837-0 · Cited by 7
“Therefore, multimodal emotion recognition (Gao et al., 2020) was introduced to capture this rich emotion information by jointly modelling text, speech, and facial expressions (Mittal et al., 2020).”
COST: Contrastive one-stage transformer for vision-language small object tracking
Zhang, C., Liu, L., Gao, J., Xin, S., Wen, H., Zhou, X., Ge, S., & Wang, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103604 · Cited by 10
Self-citationHierarchical multi-modal fusion architecture search for stock market forecasting
Lee, K., Kim, Y. J., Kim, M., & Kim, H. Y. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113581 · Cited by 9
Multimodal Emotion Analysis Based on Visual, Acoustic and Linguistic Features
Koren, L., Stipanĉić, T., Ričko, A., & Orsag, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05061-9_23 · Cited by 5
Wen, J., Abeel, T., & Weerdt, M. M. D. (2023) · Frontiers in Plant Science · doi:10.3389/fpls.2023.1160645 · Cited by 5
Multimodal Machine Learning in Prognostics and Health Management of Manufacturing Systems
Jose, S., Nguyen, K. T. P., & Medjaher, K. (2023) · Springer series in reliability engineering · doi:10.1007/978-3-031-30510-8_9 · Cited by 7
BizGraphQA: A Dataset for Image-based Inference over Graph-structured Diagrams from Business Domains
Babkin, P., Watson, W., Ma, Z., Cecchi, L., Raman, N., Nourbakhsh, A., & Shah, S. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591875 · Cited by 9
Nutrient Deficiency Classification using AlexNet Architecture
Sabri, N., Zulkifli, Z. A., Shari, A. A., Halim, F. H., Zaini, H., & Ismail, K. N. (2023) · IEEE Control and System Graduate Research Colloquium (ICSGRC) · doi:10.1109/icsgrc57744.2023.10215422 · Cited by 5
“Convolutional Neural Network (CNN) is the most commonly used type of deep learning for large-scale image classification and recognition [3].”
BackgroundDefining and Researching “Dynamic Systems of Systems”
Adler, R., Elberzhager, F., Falcão, R., & Siebert, J. (2024) · Software · doi:10.3390/software3020009 · Cited by 8
“Recent advances in deep learning have improved multimodal data fusion techniques, but this research and its applications are still in a preliminary stage [60].”
BackgroundAsbach, J. C., Singh, A. K., Iovoli, A. J., Farrugia, M. K., & Lê, A. (2025) · Medical Physics · doi:10.1002/mp.17672 · Cited by 5
Dictionary-Based Fusion of Contact and Acoustic Microphones for Wind Noise Reduction
Tammen, M., Li, X.-L., Doclo, S., & Theverapperuma, L. (2022) · International Workshop on Acoustic Signal Enhancement · doi:10.1109/iwaenc53105.2022.9914710 · Cited by 6
“, by training deep neural networks (DNNs) [1], [5], [6].”
BackgroundZhou, H., Zhou, Z., & Bai, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892296 · Cited by 6
Towards a multi-modal Deep Learning Architecture for User Modeling
Tato, A. A. N., & Nkambou, R. (2023) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.36.133328 · Cited by 5
“Multimodal learning involves relating information from multiple sources (Ngiam et al. 2011) (Jing et al. 2020).”
BackgroundImage Annotation via Reconstitution Graph Learning Model
Chen, S., Wang, M., & Chen, X. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8818616 · Cited by 5
Multimodal-Multisensory ExperimentsPreprint
Razavi, M., Yamauchi, T., Janfaza, V., Leontyev, A., Longmire-Monford, S., & Orr, J. M. (2020) · Preprints.org · doi:10.20944/preprints202008.0614.v1 · Cited by 5
Fighting the disagreement in Explainable Machine Learning with consensusPreprint
Banegas‐Luna, A. J., Martínez-Cortés, C., & Pérez‐Sánchez, H. (2024) · Research Square · doi:10.21203/rs.3.rs-4106083/v1 · Cited by 7
Advanced Neural Networks for Multimodal Data Fusion in Interdisciplinary Research
Dhivya, K., Kumar, S. N., Victoria, D. R. S., Sherly, S. I., & Durgadevi, G. (2024) · Advances in IT Standards and Standardization Research · doi:10.4018/979-8-3693-4759-1.ch008 · Cited by 4
Cui, W., Si, J., Zhang, L., Han, L., & Chen, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3585737 · Cited by 4
“2) Multimodal Fusion: To fully use the complementary property of multimodal data and enhance representation ability of precipitation characteristics [47], [48], [49], we fused the outputs of two branches as follows : where I Fused denotes the fusion feature from multimodal branches.”
Driver Interaction Intent Prediction With Dynamic Scene Semantic Fusion for Intelligent Cockpit
Hu, H., Qiao, X., Wei, Z., Peng, D., Tian, C., Zhao, R., & Gao, F. (2026) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2025.3642533 · Cited by 4
“3) Late Fusion (AVG-Late, GMU-Late) [35]: processes each modality with independent encoders, then perform fusion via AVG, or GMU, respectively, followed by direct decoding to predictions.”
Wali, R., Xu, H., Cheruiyot, C., Saleem, H. N., Janshoff, A., Habeck, M., & Ebert, A. (2024) · Biological Chemistry · doi:10.1515/hsz-2024-0023 · Cited by 6
Liu, H., Zhu, H., Liu, F., Deng, L., Wu, G., Han, Z., & Zhao, L. (2024) · Plants · doi:10.3390/plants13091177 · Cited by 4
“Gao et al. [45] proposed some pioneering deep learning models to integrate these big data and stimulate new deep learning multimodal data fusion techniques.”
BackgroundEnhancing trabecular CT scans based on deep learning with multi-strategy fusion
Ge, P., Li, S., Liang, Y., Zhang, S., Zhang, L., Hu, Y., Yao, L., & Wong, P. K. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102410 · Cited by 4
Cascaded cross-modal transformer for audio–textual classification
Ristea, N.-C., Anghel, A., & Ionescu, R. T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10869-1 · Cited by 7
“…the complexity of real-world classification tasks through the combination of various modalities is not an easy challenge, requiring the development of robust and efficient methods to effectively fuse multiple data sources (Ramachandram and Taylor 2017; Gao et al. 2020; Stahlschmidt et al. 2022).”
Pratyush, P., & KC, D. B. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4213-9_8 · Cited by 4
He, Y., Ji, F. Q., Sun, B., Wang, F., Zhang, L., & Jiang, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-82058-7 · Cited by 5
Transformer Networks and Loss with Punishment for Optimized Management of Urban Water Supply System
Wang, Y., Wang, H., Chen, J.-J., Yin, W.-X., Li, J., Zhang, Z., Li, H.-W., Wang, M.-F., & Wang, A.-J. (2025) · ACS ES&T Water · doi:10.1021/acsestwater.4c00881 · Cited by 8
Fairness-Aware Graph Neural Networks for ICU Length of Stay Prediction in IoT-Enabled Environments
Maroudis, A.-C., Karathanasopoulou, K. N., Stylianides, C., Dimitrakopoulos, G. J., & Panayides, A. S. (2025) · IEEE Access · doi:10.1109/access.2025.3560180 · Cited by 4
“In this case the overall predictive logic flow can be expressed as, where f is a fusion layer (e.g., concatenation, weighted summation), and W h represents the parameters of the final predictive layer [22].”
A multi-scenario model fusion and verification method for digital twin machine tool
Li, H., Yan, P., Zhou, H., Pei, J., & Wang, B. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100859 · Cited by 8
Robust Anomaly Detection of Multivariate Time Series Data via Adversarial Graph Attention BiGRU
Xing, Y., Tan, J., Zhang, R., & Wan, J. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9050122 · Cited by 4
Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification
Awan, M., Nadeem, A., Awan, M. J., Mustafa, A., & Husain, S. S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93806-1_15 · Cited by 5
Wang, Z., Liu, X., Han, K., Lei, L., Shi, C., Liu, W., & Guo, Q. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf142 · Cited by 5
Chen, H., & Tang, D. M. (2025) · International Conference on Artificial Intelligence and Smart Manufacturing · doi:10.1145/3756423.3756555 · Cited by 8
Wang, R., & Liu, F. (2025) · Applied Sciences · doi:10.3390/app152011259 · Cited by 4
Chen, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-22173-1 · Cited by 4
Asali, E., Li, G., Saeidifar, M., Liu, T. S., Oso, O. M., Mandiga, A., Bodempudi, V. U. C., & Kota, S. A. R. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111305 · Cited by 5
Improving image-retrieval performance of foundation models in gastrointestinal endoscopic images
Kim, K., Park, J., Kim, S. H., & Hwang, Y. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1727884 · Cited by 4
“The loss is formally defined in Equation (3) as where ( ) f x As illustrated in Figure 2, we adopted a parameter-efficient fine-tuning (PEFT) strategy using low-rank adaptation (LoRA) to minimize computational demands and the risk of overfitting (19–21).”
Weigelt, L., Wengert, M., Wachendorf, M., & Wijesingha, J. (2025) · Precision Agriculture · doi:10.1007/s11119-025-10301-w · Cited by 4
“In contrast, more complex fusion strategies, such as model-level late fusion and deep learning-based fusion, typically require greater computational resources and larger datasets, resulting in models that are more difficult to interpret (Gao et al., 2020).”
Ativi, W. X., Chen, W., Kwao, L., Ayivi, W., Sam, F., Alqahtani, A., Gu, Y. H., & Al–antari, M. A. (2026) · Scientific Reports · doi:10.1038/s41598-025-32794-1 · Cited by 5
“Traditional fusion strategies can be categorized into three paradigms 15 .”
Decoding Multimodal Text Analytics: Tasks, Datasets, Fusion Models, and Future Frontiers
Nath, T., Gupta, V., Gupta, M., & Sharma, R. R. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70083 · Cited by 4
“Multimodal data can be defined as data generated from heterogeneous networks, which have high volume, high variety, high velocity, and high veracity (Gao et al. 2020).”
Li, K., Han, S., Yang, L., Sun, Z., Zhan, Y., Xu, H., Ma, L., Gao, J., & Jiang, H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3327755 · Cited by 5
“Traditional deep learning methods can only handle single-modal information like medical imaging and fail to meet thehighaccuracyrequirementsofBI-RADSgrading[15].”
BackgroundSelf-citationIntegrated noninvasive diagnostics for prediction of survival in immunotherapy
Yeghaian, M., Bodalal, Z., Bucho, T. M. T., Kurilova, I., Blank, C. U., Smit, E. F., van der Heijden, M. S., Nguyen‐Kim, T. D. L., van den Broek, D. A., Beets‐Tan, R. G. H., et al. (2024) · Immuno-Oncology Technology · doi:10.1016/j.iotech.2024.100723 · Cited by 8
“9-11 Based on these early successes, integrative approaches were also applied to the fi eld of medical AI, 9 particularly in oncology.”
Wang, Y., Cai, H., Chen, Z., Hu, P., Han, Y., & Xu, B. (2024) · IEEE International Conference on e-Business Engineering (ICEBE) · doi:10.1109/icebe62490.2024.00046 · Cited by 5
“Multimodal learning involves enabling machines to learn, understand, and reason across multiple modalities [15].”
Self-citationEfficient Solvers for Wyner Common Information With Application to Multi-Modal Clustering
Huang, T.-H., & Gamal, H. E. (2025) · IEEE Transactions on Information Theory · doi:10.1109/tit.2025.3532280 · Cited by 7
“I N MULTI-MODAL learning, the idea of extracting “good” features that are shared between multiple modalities of data for improved performance over single modal counterparts has attracted great interest [1], [2], [3].”
Chen, Y., Zhou, B., Xiaopeng, C., Ma, C., Cui, L., Feng, L., Han, X., Chen, L., Wu, S., & Ye, D. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1448669 · Cited by 4
“However, when in unstructured environments, many researchers fusing on multiple data sources (Gao et al., 2020) in nowadays.”
Introduction to Multimodal Generative AI
Brindha, R., Pongiannan, R. K., Bharath, A., & Sanjeevi, V. K. S. M. (2025) · Springer eBooks · doi:10.1007/978-981-96-2355-6_1 · Cited by 4
Adaptive Multi-Resolution Feature Fusion for Fine-Grained Visual Classification
Yang, Y., Chang, D., Du, R., Song, Y.-Z., & Ma, Z. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3546951 · Cited by 5
“Multimodal data includes different forms of information such as images, text, and audio [18], [37].”
Data Fusion for Deep Learning on Transport Mode Detection: A Case Study
Moreau, H., Vassilev, A., & Chen, L. (2021) · Proceedings of the international neural networks society · doi:10.1007/978-3-030-80568-5_12 · Cited by 4
The Benefits of Depth Information for Head-Mounted Gaze Estimation
Stojanov, S., Talathi, S. S., & Sharma, A. (2022) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3517031.3529638 · Cited by 4
“Just like gaze-estimation, there is an abundance of prior research on fusion methods, which is impossible to cover due to space limitations, hence, we refer the reader to [Gao et al. 2020] for a comprehensive review of deep-learning based fusion approaches.”
MethodTowards Exploiting Stakeholder Resources for Data-Driven Resilient Hazard Management
Fu, X., Yuan, T., Wang, W., Wolk, F., & Strunz, G. (2022) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw56584.2022.00055 · Cited by 5
“Multimodal data fusion methods [31] are presented to explore the complementary and cross-modality information.”
BackgroundMarinoni, A., Chlaily, S., Khachatrian, E., Eltoft, T., Selvakumaran, S., Girolami, M., & Jutten, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03682 · Cited by 5
“Transfer learning and ensemble learning are the new trends for solving the problems that arise from training data and test data having different distributions [3], [11], [17]–[19].”
BackgroundMethodInfluential
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.