A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
Method
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Items 201–300 of 837
A geometric framework for multiclass ensemble classifiers
Wu, S., Li, J., & Ding, W. (2023) · Machine Learning · doi:10.1007/s10994-023-06406-w · Cited by 21
“Another research topic is multi-model data fusion (Gao et al., 2020).”
BackgroundSotubadi, S. V., Pallissery, S. S., & Nguyen, V. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110141 · Cited by 21
Sensing in Smart Cities: A Multimodal Machine Learning Perspective
Sadiq, T., & Omlin, C. W. P. (2025) · Smart Cities · doi:10.3390/smartcities9010003 · Cited by 24
“These fusion strategies are often implemented through deep learning architectures, including CNNs [47], RNNs [48], and transformers [25], which are tailored to specific types of data and integration approaches.”
An improved real time detection of data poisoning attacks in deep learning vision systems
Raghavan, V., Mazzuchi, T. A., & Sarkani, S. (2022) · Discover Artificial Intelligence · doi:10.1007/s44163-022-00035-3 · Cited by 20
“The diversity and the high throughput of the data sources have made current deep learning models multimodal [22] and more accurate.”
BackgroundBhatt, S., Ghazanfar, M. A., & Amirhosseini, M. H. (2023) · Machine Learning and Applications An International Journal · doi:10.5121/mlaij.2023.10301 · Cited by 18
“It is often used to analyse the attitudes, opinions, and emotions expressed in text data to better understand how people feel about a particular topic or subject [6].”
BackgroundA Review of Machine Learning Approaches for Non-Invasive Cognitive Impairment Detection
Alsuhaibani, M., Fard, A. P., Sun, J., Poor, F. F., Pressman, P., & Mahoor, M. H. (2025) · IEEE Access · doi:10.1109/access.2025.3555176 · Cited by 28
“In general, merging two different types of features in deep learning models helps boost the performance [124]. a) Data preprocessing and feature extraction: The pre-processing steps and features extraction methods have been introduced previously in Sec.”
Identifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion
Hu, C., Yin, M., Liu, B., Li, X., & Ye, Y. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3472713 · Cited by 21
“With labeled data (ground-truth for machine learning), we have built a brand new drug-dealer identification system by leveraging the latest advances in deep learning including Bidirectional Encoder Representations from Transformers (BERT) [10] based text classification, ResNet-based [28] image classification, as well…”
MethodDeep multiview learning to identify imaging-driven subtypes in mild cognitive impairment
Feng, Y., Kim, M., Yao, X., Liu, K., Long, Q., Shen, L., & Initiative, A. D. N. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04946-x · Cited by 19
“…have been studies that proposed data fusion methods to fuse information from multimodal data and reduce dimensionality, such as multi-kernel support vector machine (SVM) [2], multimodal random forest [9], and deep learning [10, 11], but are often used for supervised tasks such as classification.”
BackgroundInvestigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning
Yu, Y., Lee, H. J., Kim, B. C., Kim, J. U., & Ro, Y. M. (2020) · arXiv · doi:10.48550/arxiv.2005.10987 · Cited by 15
“When a number of data types are being engaged in the processing of information, it is called multimodal data fusion models [12, 6].”
BackgroundPrivacy-preserving activity recognition using multimodal sensors in smart office
Zhang, X., Zheng, P., Peng, T., Li, D., Zhang, X., & Tang, R. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.05.023 · Cited by 15
Cho, H. S., Koo, W., & Kim, H. (2023) · IEEE Transactions on Semiconductor Manufacturing · doi:10.1109/tsm.2023.3283101 · Cited by 21
“Multimodal fusion aims to leverage the complementarity of multimodal data, which contain multiple data modalities, to provide better prediction performance [30].”
BackgroundRizaldy, A., Afifi, A. J., Ghamisi, P., & Gloaguen, R. (2024) · Remote Sensing · doi:10.3390/rs16132336 · Cited by 19
El-Din, D. M., Hassanein, A. E., & Hassanien, E. E. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02150-2 · Cited by 19
Multi-modal data clustering using deep learning: A systematic review
Raya, S. P., Orabi, M., Afyouni, I., & Aghbari, Z. A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128348 · Cited by 16
Generative models for tabular data: A review
Kim, D.-K., Ryu, D., Lee, Y., & Choi, D.-H. (2024) · Journal of Mechanical Science and Technology · doi:10.1007/s12206-024-0835-0 · Cited by 20
Integrating Multimodal Affective Signals for Stress Detection from Audio-Visual Data
Ghose, D., Gitelson, O., & Scassellati, B. M. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685717 · Cited by 19
“The main objective of multimodal fusion is to take advantage of the unique strengths of every modality to produce more accurate predictions than what can be achieved with a single modality [22].”
Time series analysis in compressor-based machines: a survey
Forbicini, F., Vago, N. O. P., & Fraternali, P. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11065-0 · Cited by 16
“Methods and techniques supporting the analysis of multi-modal data are reviewed, e.g. in [191, 192].”
Katharria, A., Pant, M., Velásquez, J. D., Snášel, V., Rajwar, K., & Deep, K. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.104040 · Cited by 19
Artificial Intelligence for Earth System Predictability (AI4ESP) (2021 Workshop Report)
Hickmon, N. L., Varadharajan, C., Hoffman, F. M., Wainwright, H. M., & Collis, S. (2022) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1888810 · Cited by 19
Hao, X., Wang, Y., Li, K., Zhu, T., & Herasevich, V. (2025) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00136-z · Cited by 20
“According to the level at which the features are fused, there are four strategies: feature level, decision level, hybrid (with feature and decision level), and algo-rithm level [97].”
Fink, O., Nejjar, I., Sharma, V., Niresi, K. F., Sun, H., Dong, H., Xu, C., Wei, A., Bizzi, A. A. C., Theiler, R., et al. (2026) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2026.112376 · Cited by 26
Meng, X., & Tan, Y. (2023) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2023.101458 · Cited by 14
Singh, S., Kumar, R., Panda, S. S., & Hegde, R. S. (2024) · Digital Discovery · doi:10.1039/d4dd00107a · Cited by 16
FedDAF: Federated deep attention fusion for dangerous driving behavior detection
Liu, J., Yang, N., Lee, Y.-L., Huang, W. X., Du, Y., Li, T., & Zhang, P. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102584 · Cited by 15
Challenges and opportunities in Machine learning for bioenergy crop yield Prediction: A review
Dayil, J. L., Akande, O., Mahmoud, A. E. D., Kimera, R., & Omole, O. (2024) · Sustainable Energy Technologies and Assessments · doi:10.1016/j.seta.2024.104057 · Cited by 22
Bearing Fault Diagnosis Grounded in the Multi-Modal Fusion and Attention Mechanism
Yang, J., Han, H., Dong, X. T., Wang, G., & Zhang, S. (2025) · Applied Sciences · doi:10.3390/app15031531 · Cited by 14
Sheng, M., Wang, S., Zhang, Y., Wang, K., Wang, J., Luo, Y., & Hao, R. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104101 · Cited by 19
Hải, N. N., Nguyen, T.-T., & Ngo, Q. T. (2025) · Journal of risk and financial management · doi:10.3390/jrfm18080417 · Cited by 17
OccMamba: Semantic Occupancy Prediction with State Space Models
Li, H., Hou, Y., Xing, X., Ma, Y., Sun, X., & Zhang, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01116 · Cited by 49
“Multi-modal fusion aims to make the strength of different input modalities, thus making more accurate and robust perception [7, 15, 17, 18, 25, 27, 43].”
Matsuzaka, Y., & Uesawa, Y. (2020) · Current Issues in Molecular Biology · doi:10.21775/cimb.042.455 · Cited by 15
“Moreover, in deep learning, it was demonstrated that the relationship between the data size and the generalization performance is proportionally related, thereby enabling high accuracy modeling from big data (Gao et al., 2020; Lee and Chen, 2020).”
BackgroundRotating Machinery Fault Diagnosis Based on Spatial-Temporal GCN
Li, C., Mo, L., & Yan, R. (2021) · International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD) · doi:10.1109/icsmd53520.2021.9670851 · Cited by 15
Interpretability for reliable, efficient, and self-cognitive DNNs: From theories to applications
Kang, X., Guo, J., Song, B., Cai, B., Sun, H., & Zhang, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126267 · Cited by 16
Hu, X., Jiang, Y., Xia, X., Chen, C., Liu, W., Wan, P., Bin, K., & Zhong, P. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104586 · Cited by 27
Fouché, A., & Zinovyev, A. Y. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1191961 · Cited by 16
“Data integration of biological data is tightly related to several machine learning topics such as domain adaptation (Pan et al., 2010; You et al., 2019; Farahani et al., 2021), data fusion (Castanedo, 2013; Gao et al., 2020) and manifold alignment (Wang et al., 2011).”
BackgroundFashionERN: Enhance-and-Refine Network for Composed Fashion Image Retrieval
Chen, Y., Zhong, H., He, X., Peng, Y., Zhou, J., & Cheng, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27885 · Cited by 26
“Multimodal fusion methods are designed to handle inputs from diverse modalities (Gao et al. 2020; Han et al. 2022b; Chen et al. 2023b) and have been widely applied in down-stream tasks such as image captioning (Yang et al. 2023; Brooks, Holynski, and Efros 2023), VQA (Ye, You, and Ma 2022; Dou et…”
BackgroundWav2NeRF: Audio-driven realistic talking head generation via wavelet-based NeRF
Shin, A.-H., Lee, J. H., Hwang, J., Kim, Y., & Park, G.-M. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105104 · Cited by 13
Niu, H., Fan, R., Chen, J., Xu, Z., & Feng, R. (2025) · Science of Remote Sensing · doi:10.1016/j.srs.2025.100208 · Cited by 13
Jeong, H., Choi, J.-S., Moon, J., Yoon, Y., & Cho, K. H. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.145999 · Cited by 15
Papadopoulos, G. T., Leonidis, A., Antona, M., & Stephanidis, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05409-9_40 · Cited by 19
Luo, S. (2021) · International Conference on Innovation in Artificial Intelligence (ICIAI) · doi:10.1145/3461353.3461388 · Cited by 16
“Especially in recent years, with the rapid development of deep learning in computer vision, NLP, etc., many technologies based on multimodal deep learning have emerged, such as multimodal data fusion [22] and multimodal representation learning [87].”
BackgroundLGMMFusion: A LiDAR-guided multi-modal fusion framework for enhanced 3D object detection
Cheng, H., Liu, C., Gu, W., Wu, Y., Zhao, M., Liu, W., & Wang, N. Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0331195 · Cited by 13
Wu, P., Wu, K., Li, Z., Liu, H., Yang, K., Zhou, R., Zhou, Z., Xing, N., & Wu, S. (2023) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-22-679 · Cited by 14
“Integrative analysis of multimodal data can aid in identifying new features of complex diseases, such as cancer (1).”
BackgroundDeep clustering framework review using multicriteria evaluation
Ros, F., Riad, R., & Guillaume, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111315 · Cited by 16
Complementary knowledge augmented multimodal learning method for yarn quality soft sensing
Xu, C., Xu, L., Zhao, S., Yu, L., & Zhang, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108057 · Cited by 16
Lin, Z., Yan, H., & Zhao, L. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12970 · Cited by 14
Yu, Z., Du, P., Yi, L., Luo, W., Li, D., Zhao, B., Li, L., Zhang, Z., Zhang, J., Zhang, J., et al. (2024) · Fundamental Research · doi:10.1016/j.fmre.2024.06.003 · Cited by 14
“The state-of-art AI technologies, such as Physics Informed Neural Networks [ 22 ], Multimodal Deep Learning [ 23 ], and conversational Large Language Models (LLM) [ 24 ], are not only advantageous in improving data quality but also in exploring natural mechanisms [ 25 ] and integrating the…”
Self-citationA comparative study on noise filtering of imbalanced data sets
Szeghalmy, S., & Fazekas, A. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112236 · Cited by 12
Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding
Lohner, A., Compagno, F., Francis, J., & Oltramari, A. (2024) · IEEE International Automated Vehicle Validation Conference (IAVVC) · doi:10.1109/iavvc63304.2024.10786395 · Cited by 19
“…strategies for fusing the outputs of a multimodal model before the downstream task at hand, such as early and late stage fusion involving concatenation [Sleeman et al., 2022, Pawłowski et al., 2023], merging [Sleeman et al., 2022], or sampling from a shared embedding space [Gao et al., 2020].”
Graph-Based Methods for Multimodal Indoor Activity Recognition: A Comprehensive Survey
Javadi, S., Riboni, D., Borzì, L., & Zolfaghari, S. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3523240 · Cited by 16
“I N recent years, the landscape of social computing has experienced a significant shift, marked by increasing variety of data modalities ranging from text and images to audio and videos [1], [2], [3].”
Balhorn, L. S., Degens, K., & Schweidtmann, A. M. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109121 · Cited by 18
Daly, K., & Olukoya, O. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107959 · Cited by 19
Bhatt, S., Ghazanfar, M. A., & Amirhosseini, M. H. (2023) · International Conference on Machine Learning & Applications (CMLA) · doi:10.5121/csit.2023.131001 · Cited by 12
“It is often used to analyse the attitudes, opinions, and emotions expressed in text data to better understand how people feel about a particular topic or subject [6].”
BackgroundBridging Multimedia Modalities: Enhanced Multimodal AI Understanding and Intelligent Agents
Gautam, S. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614225 · Cited by 21
“Gao et al. [7] delve into pioneering models that fuse diverse data types, ofering fundamental insights specifcally in the context of multimodal big data.”
BackgroundTraining Multimedia Event Extraction With Generated Images and Captions
Du, Z., Li, Y., Guo, X., Sun, Y., & Li, B. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612526 · Cited by 33
“As digital media quickly evolve, news reports today frequently present information with a combination of text and image, providing a more comprehensive view of events than text alone [11, 52].”
BackgroundEnhancing additive manufacturing with computer vision: a comprehensive review
Yu, H., Li, W., Li, D., Wang, L., & Wang, Y. (2024) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-13689-3 · Cited by 18
“With the development of techniques related to the consistent expression of cross-media data, using multi-modal information as input could provide richer data sources and more comprehensive information expression for the specific applications of CV in the multiple stages of AM [88].”
A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical ApplicationsPreprint
Guarrasi, V., Aksu, F., Caruso, C. M., Feola, F. D., Rofena, A., Ruffini, F., & Soda, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4952813 · Cited by 21
Deep Learning in Otolaryngology: A Narrative Review
Junior, S. L. N., Navarathna, N., D’Cruz, M., Brooks, J., Maron, B. A., & Isaiah, A. (2025) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2025.3911 · Cited by 13
Cao, J., Chen, J., Wang, X., Huang, W., Chen, D., Zhao, T., Tu, W., & Li, Q. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.11.012 · Cited by 16
Classification of traffic accidents’ factors using TrafficRiskClassifier
Sun, W., Abdullah, L. N., Khalid, F. B., & Sulaiman, P. S. (2024) · International Journal of Transportation Science and Technology · doi:10.1016/j.ijtst.2024.05.002 · Cited by 16
Upadhya, J., Poudel, K. N., & Ranganathan, J. (2024) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/3632634.3655878 · Cited by 11
“Multimodal deep learning combines various data sources and metadata to form a complete approach that enables the model to recognize complex patterns of stress and uncover hidden associations that unimodal methods fail to detect [11, 23, 28, 33, 36].”
BackgroundCompact bilinear pooling and multi-loss network for social media multimodal classification
Li, Y., Zheng, X., Zhu, M. L., Mei, J., Chen, Z., & Tao, Y. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03482-w · Cited by 15
“Despite progress in multimodal classification, three spe-cific challenges remain [12].”
Self-citationAlthenayan, A. S., Alsalamah, S., Aly, S. F., Nouh, T. A., & Mirza, A. A. (2022) · Applied Sciences · doi:10.3390/app122010535 · Cited by 11
Akins, A., Kultgen, D., & Heifetz, A. (2023) · Energies · doi:10.3390/en16134965 · Cited by 11
BIOMEX-DB: A Cognitive Audiovisual Dataset for Unimodal and Multimodal Biometric Systems
Moreno-Rodríguez, J. C., Atenco-Vázquez, J. C., Ramírez-Cortés, J. M., Arechiga, R. O., Gómez‐Gil, P., & Fonseca-Delgado, R. (2021) · IEEE Access · doi:10.1109/access.2021.3100035 · Cited by 14
“The first group of experiments is based on Deep Learning models (DL), which have been proven to provide very good results in a variety of applications in the fields of artificial intelligence, machine learning, and pattern recognition [25], and specifically in data fusion [26].”
MethodData modalities, consumer attributes and recommendation performance in the fashion industry
Sysko-Romańczuk, S., Zaborek, P., Wróblewska, A., Dąbrowski, J., & Tkachuk, S. (2022) · Electronic Markets · doi:10.1007/s12525-022-00579-3 · Cited by 13
“It seems evident in past research that adding different modalities of data should increase the performance of recommendation algorithms (Gao et al., 2020; Lahat et al., 2015; Liu et al., 2018).”
BackgroundFusion of energy sensors with missing values
Buonanno, A., Gennaro, G. D., Graditi, G., Nogarotto, A., Palmieri, F., & Valenti, M. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04752-9 · Cited by 10
“In this way it is possible to represent the observed data in a lower space that can be used for downstream tasks such as clustering, classification, forecasting, etc. Fusion models based on Autoencoders have been investigated in different works [15–18], also in the energy context [19–22].”
Asset pricing via fused deep learning with visual clues
Tan, J., Deveci, M., Li, J., & Zhong, K. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102049 · Cited by 10
Computer vision-based video signal fusion using deep learning architectures
Ghyabi, M., & Lattanzi, D. (2025) · Journal of Civil Structural Health Monitoring · doi:10.1007/s13349-024-00891-w · Cited by 12
“The network hyperparameters are determined based on previous research and empirical testing [38, 43].”
Zhong, L., Fan, Y., Wu, Y., Yangyang, G., Gao, Z., Zhou, A., Shao, Q., & Zhang, A. (2025) · Journal of Food Composition and Analysis · doi:10.1016/j.jfca.2025.107403 · Cited by 10
Shirshahi, A., Moshiri, B., & Aliyari-Shoorehdeli, M. (2025) · Process Safety and Environmental Protection · doi:10.1016/j.psep.2025.107097 · Cited by 10
Goswami, N., Dogra, A., Bakshi, S., & Goyal, B. (2025) · The Open Neuroimaging Journal · doi:10.2174/0118744400417835251022042920 · Cited by 13
Multi-Frequency RF Sensor Data Adaptation for Motion Recognition with Multi-Modal Deep Learning
Rahman, M. M., & Gürbüz, S. Z. (2021) · IEEE Radar Conference (RadarConf) · doi:10.1109/radarconf2147009.2021.9455204 · Cited by 10
“…1) a single sensor case, where adversarial domain adaptation [23] is used to transform the data from one RF sensor to resemble that of another, and 2) a multi-sensor case, where multi-modal deep learning [24], [25] is designed for joint target recognition using measurements from all sensors.”
MethodMultimodal Data Fusion to Detect Preknowledge Test-Taking Behavior Using Machine Learning
Man, K. (2023) · Educational and Psychological Measurement · doi:10.1177/00131644231193625 · Cited by 12
“Therefore, it unveils significant information hidden in the data set, which could enhance detection of aberrant test-taking behaviors (Gao et al., 2020; Gaw et al., 2022; Lahat et al., 2015).”
Zhang, H., Li, X., Sun, Y., Fu, X., Qiu, C., & Carroll, J. M. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00054 · Cited by 14
“There is evidence that multi-modal data fusion for emotion recognition is an important approach to address the limitations of each single-channel analysis mentioned above [24].”
BackgroundRevolutionizing Utility of Big Data Analytics in Personalized Cardiovascular Healthcare
Sharma, P., Sharma, P., Sharma, K. H., Varma, V., Patel, V., Sarvaiya, J. N., Tavethia, J., Mehta, S., Bhadania, A., Patel, I., et al. (2025) · Bioengineering · doi:10.3390/bioengineering12050463 · Cited by 14
“Their capability to individualize treatment based on genetic profiles and improve the treat-ment outcomes for patients with FH represents another landmark towards personalized medicine [49].”
Powerset Fusion Network for Target Classification in Unattended Ground Sensors
Wang, Y., Cheng, X., Xiang, L., Li, B., & Yuan, X. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3067648 · Cited by 10
“Recently, convolutional neural networks (CNNs) are proposed for multimodal fusion [11].”
BackgrounddtwParallel: A Python package to efficiently compute dynamic time warping between time series
Escudero-Arnanz, Ó., Marques, A. G., Soguero-Ruíz, C., Mora-Jiménez, I., & Robles, G. (2023) · SoftwareX · doi:10.1016/j.softx.2023.101364 · Cited by 12
Multimodal Deep Learning for Crop Yield Prediction
Jácome-Galarza, L.-R. (2022) · Communications in Computer and Information Science · doi:10.1007/978-3-031-18347-8_9 · Cited by 9
DT-Transformer: A Text-Tactile Fusion Network for Object Recognition
Qiu, S., Li, B., Wang, X., Wang, H., & Ye, H. (2024) · IEEE Transactions on Haptics · doi:10.1109/toh.2024.3506791 · Cited by 10
“Multimodal fusion methods integrate complementary information from different modal data, (e.g., image, audio, text, haptic, etc.) to improve the performance of task-specific pattern recognition models [23].”
Traffic flow prediction based on depthwise separable convolution fusion network
Yu, Y., Sun, W., Liu, J., & Zhang, C. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00637-9 · Cited by 9
“Multimodal data fusion [28] refers to various forms of combination of two or more modal data.”
BackgroundDetecting Heart Failure Through Voice Analysis using Self-Supervised Mode-Based Memory Fusion
Priyasad, D., Partovi, A., Sridharan, S., Kashefpoor, M., Fernando, T., Denman, S., Fookes, C., Tang, J., & Kaye, D. M. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-643 · Cited by 9
“Fusion of different modes is proven to improve the performance of a neural network, if carefully designed [21, 22, 23].”
BackgroundLiang, B., Han, S., Li, W., Huang, G., & He, R. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103123 · Cited by 9
DeepGraviLens: a multi-modal architecture for classifying gravitational lensing data
Vago, N. O. P., & Fraternali, P. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08766-9 · Cited by 11
“The survey [17] overviews DL methods for multi-modal data fusion in general whereas [57] focuses on biomedical data fusion.”
Huang, X., Peng, L., Mukherjee, S., Hamilton, C., Shi, X., Srinivasan, V., Davis, E., & Deng, Y. (2024) · Research in Nondestructive Evaluation · doi:10.1080/09349847.2024.2319765 · Cited by 9
Multi-Modal Gaze Following in Conversational Scenarios
Hou, Y., Zhang, Z., Horányi, N., Moon, J., Cheng, Y., & Chang, H. J. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00122 · Cited by 17
“Combining audio and vision modalities can provide richer and complementary information compared to unimodal approaches [19].”
BackgroundA New Residual Dense Network for Dance Action Recognition From Heterogeneous View Perception
Yang, X., Lyu, Y., Sun, Y., & Zhang, C. G. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.698779 · Cited by 9
Streamlining Multimodal Data Fusion in Wireless Communication and Sensor Networks
Bocus, M. J., Wang, X., & Piechocki, R. J. (2023) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2023.3322983 · Cited by 11
“However, obtaining a unified representation from multiple modalities can be challenging as they are often of different data types, having different distributions, levels of sparsity, and dimensions [2].”
BackgroundOchege, F. U., Yuan, X., Ezekwe, I. C., Ling, Q., Nzabarinda, V., Kayiranga, A., Xie, M., Shi, H., & Luo, G. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131115 · Cited by 8
Wei, S., Tian, X., Zuo, E., Zhang, X., Lü, C., Gu, J., Chen, C., Lv, X., & Chen, C. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103053 · Cited by 8
Multi-modal contrastive causal consistency fusion for anomaly detection in additive manufacturing
Gu, J., Wang, Y., Chen, J., Zhang, M., Wang, Z., & Chen, J. (2025) · Additive manufacturing · doi:10.1016/j.addma.2025.104816 · Cited by 12
High-performance fire detection framework based on feature enhancement and multimodal fusion
Zhou, Z., Zhao, H., Li, X., Liu, Y., Jiang, T., Jing, J., & Guo, Y. (2025) · Journal of Safety Science and Resilience · doi:10.1016/j.jnlssr.2025.03.004 · Cited by 12
Yang, X., Wang, H., Zhang, R., Sun, S., & Zhang, M. (2026) · Ocean Engineering · doi:10.1016/j.oceaneng.2026.124517 · Cited by 8
Fusion of heterogeneous industrial data using polygon generation & deep learning
Elhefnawy, M., Ouali, M.-S., Ragab, A., & Amazouz, M. (2023) · Results in Engineering · doi:10.1016/j.rineng.2023.101234 · Cited by 9
A Review of Biosensors and Artificial Intelligence in Healthcare and their Clinical Significance
Tariq, M. (2024) · Psychology & Psychological Research International Journal · doi:10.23880/pprij-16000392 · Cited by 9
Space-aware Socioeconomic Indicator Inference with Heterogeneous Graphs
Zou, X., Huang, J., Hao, X., Yang, Y., Wen, H., Yan, Y., Huang, C., Chen, C., & Liang, Y. (2025) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3748636.3762727 · Cited by 15
“Multimodal learning integrates data from various modalities to enhance model performance across tasks [1, 5, 8].”
HMDFF: A Heterogeneous Medical Data Fusion Framework Supporting Multimodal Query
Ren, P., Lin, W., Liang, Y., Wang, R., Liu, X., Zuo, B., Chen, T., Li, X., Sheng, M., & Zhang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90885-0_23 · Cited by 8
Crop Yield Prediction Using Spatio Temporal CNN and Multimodal Remote Sensing
Arunarani, A. R., Selvanayaki, S., Ansari, M. S. A., Walid, M. A. A., Devireddy, N., & Keerthi, M. M. (2023) · International Conference on Edge Computing and Applications (ICECAA) · doi:10.1109/icecaa58104.2023.10212267 · Cited by 8
“Deep learning is a subset of machine learning that employs organizational frameworks to connect the various levels, and it differs from other classic machine learning techniques due to the fact that it can analyze both data without labels and structures [20].”
BackgroundXu, C., Ding, J., Wang, B., Qiao, Y., Zhang, L., & Zhang, Y. F. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100748 · Cited by 14
Low-Rank Tensor Fusion for Enhanced Deep Learning-Based Multimodal Brain Age Estimation
Liu, X., Zheng, G., Beheshti, I., Ji, S., Gou, Z., & Cui, W.-K. (2024) · Brain Sciences · doi:10.3390/brainsci14121252 · Cited by 7
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.