A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
Method
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Items 401–500 of 837
Scalable multimodal assessment of the micro-neighborhood using orthogonal visual inputs
Despotović, M., & Brunauer, W. (2024) · Journal of Housing and the Built Environment · doi:10.1007/s10901-024-10153-2 · Cited by 5
“While visual information has proven to be a rich unimodal source for many different tasks, it can also be incorporated into the modeling process simultaneously with other data, such as textual and structured data (Gao et al., 2020).”
Multisource Heterogeneous Data Fusion Methods Driven by Digital Twin on Basis of Prophet Algorithm
Li, M. (2025) · IET Software · doi:10.1049/sfw2/5041019 · Cited by 3
“With the rapid development of science and technology, ef fi - cient fusion of heterogeneous data from multiple sources has become an urgent problem [1, 2].”
Small parallel residual convolutional neural network and traffic congestion detection
Jiang, S., & Feng, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-97942-z · Cited by 3
“This involves the integration and analysis of multi-source information 22,23 .”
Quantum-inspired multimodal fusion with Lindblad master equation for sentiment analysis
Yan, K., Lai, P., Yang, Y., Ren, Y., Badarch, T., Chen, Y., & Zheng, X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130710 · Cited by 7
DED bead geometry and profile prediction with multimodal spatio-temporal neural networks
Zhang, W., Wang, Y., Chen, Y., Deng, X., Wang, Y., Duan, M., Hu, P., & Tang, K. (2025) · Additive manufacturing · doi:10.1016/j.addma.2025.104952 · Cited by 3
Generative Deep Learning for Advanced Battery Materials
Rajagopal, D., Cierpka, A., Nestler, B., & Koeppe, A. (2025) · Batteries & Supercaps · doi:10.1002/batt.202500494 · Cited by 5
Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical World
Zheng, Y. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768625 · Cited by 6
“Models discussed in [15, 44, 63, 64] can be regarded as instances of this paradigm.”
InfluentialLuo, Q., Huang, N., Bartles, D. L., & Beese, A. M. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-025-02693-3 · Cited by 4
“In data fusion, it is critical that the feature representatives from multiple sources, here, referred to as “multimodal” data, capture critical information from each data type (Gao et al., 2020; Tan et al., 2018).”
Chen, K., Chen, G., Lin, J., Gao, X., & Zhao, X. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ae1312 · Cited by 3
“Given the outstanding performance of multimodal fusion technologies in related fields [26], researchers have progressively explored combining tactile and visual information to enable robots to comprehensively understand grasping environments [27].”
Minyilu, Y., Yimer, M. A., & Meshesha, M. (2025) · JMIR AI · doi:10.2196/67584 · Cited by 5
“Technically, this process of integrating multiple modalities of data (possibly taken from different sources) is called MMDF [13,14].”
Domain-aware graph neural networks for source code vulnerability detection
Foulefack, R. Z. L., & Marchetto, A. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108104 · Cited by 3
Liu, Y., Cong, S., & Gao, Y. (2026) · Journal of environmental chemical engineering · doi:10.1016/j.jece.2026.122269 · Cited by 3
Liu, J., Ding, X., & Gan, Y. (2026) · Clinical Psychology Review · doi:10.1016/j.cpr.2026.102728 · Cited by 5
Generative adversarial fault-tolerance for robust multimodal sensor fusion in industrial systems
Altinses, D., & Schwung, A. (2026) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2026.114225 · Cited by 3
Early prediction of litchi downy blight via spatiotemporal graph-based multimodal data fusion
Lu, J., Tong, H., Zhou, Z., Yubin, L. A. N., Wang, T., & Wang, W. (2026) · Agricultural Systems · doi:10.1016/j.agsy.2026.104747 · Cited by 3
Semisupervised Deep Embedded Clustering with Adaptive Labels
Chen, Z., Li, C., Gao, J., Zhang, J., & Li, P. (2021) · Scientific Programming · doi:10.1155/2021/6613452 · Cited by 3
“,e deep code network aims to learn latent features of data in a low-dimensional space on the basis of the encoder-decoder network [41].”
BackgroundSelf-citationDesign of English Teaching Sharing System Combining Internet of Things and Memory Mechanism
Guo, J. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/2712199 · Cited by 3
“Therefore, the fi elds or records in the data table of the 7 Wireless Communications and Mobile Computing database are represented by a pretrained word embedding vector and word embedding vector, so that each fi eld is transformed into text vector of di ff erent granularity [30].”
MethodLearning RNA structure prediction from crowd-designed RNAs
(2022) · Nature Methods · doi:10.1038/s41592-022-01607-y · Cited by 3
Towards Ensuring Software Interoperability Between Deep Learning Frameworks
Lee, Y. K., Park, S. H., Lim, M. Y., Lee, S.-H., & Jeong, J. (2023) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2023-0016 · Cited by 3
An emotion-driven and topic-aware dialogue framework for human–robot interaction
Lu, I.-C., Huang, J.-Y., & Lee, W. (2023) · Advanced Robotics · doi:10.1080/01691864.2023.2297902 · Cited by 4
“Following the above two strategies, hybrid approaches have been proposed to combine different fusion methods to further improve recognition accuracy [16,17].”
BackgroundLiu, S., Bu, J., Ye, H., Chen, J., Jiang, S., Tao, M., Guo, L., Wang, C., & Li, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642346 · Cited by 5
“With the deployment of heterogeneous networks, an unprecedented volume of data is being generated and collected at an unprecedented rate [37], giving rise to numerous corresponding visualization systems.”
BackgroundAutoAMS: Automated attention-based multi-modal graph learning architecture search
Al-Sabri, R., Gao, J., Chen, J., Olouladé, B. M., & Wu, Z. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106427 · Cited by 5
Self-citationAn empirical cryptocurrency price forecasting model
Kadhim, A. H., Farttoosi, D. S. A., Shakir, H. S., & Almagtome, A. H. H. (2024) · International Journal of Information Technology · doi:10.1007/s41870-024-01944-5 · Cited by 3
Person-Centric Sensing in Indoor Environments
Strohmayer, J., Lumetzberger, J., Heitzinger, T., & Kampel, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-59531-8_11 · Cited by 4
A multi-modal approach for mixed-frequency time series forecasting
Filho, L. L., Werneck, R. D. O., Castro, M., Júnior, P. R. M., Lustosa, A., Zampieri, M., Linares, O. C., Moura, R., Morais, E., Amaral, M., et al. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10305-z · Cited by 4
“…our research focuses on the development and validation of novel multimodal architectures for time series forecasting in the oil industry, and such architectures refer to models whose topologies are designed to simultaneously deal with different types of data, such as image, audio, and video [5].”
Li, T. (2025) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001425520019 · Cited by 4
Qi, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3553560 · Cited by 4
“Gao et al. [46] present methodologies for deep learning-based multimodal data fusion, demonstrating improved model performance but highlighting the need for more sophisticated fusion mechanisms.”
An Interpretable SAR Image Filtering Algorithm
Nurmamat, P., Wan, H., Chen, J., Huang, Z., Yang, L., Li, M., Yang, W., Zeng, H., Chen, J., & Diniz, P. S. R. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3557380 · Cited by 3
“Tedious raw data need to be transformed into a form that can be processed by the model to accelerate convergence [35].”
Yue, Z., Xu, J., Li, Z., Xing, H., & Cheng, X. (2025) · International Journal of Remote Sensing · doi:10.1080/01431161.2025.2496532 · Cited by 5
“Multimodal remote sensing images are a crucial product of remote sensing technology, offering rich geographic information that helps people better understand and manage the Earth (Farah et al. 2018; Jing et al. 2020; Shahrzad and Azam 2022; Tin 2022).”
Zhao, J., Yin, Y.-L., Shao, M., Wang, Q., Feng, W., & Lu, L. (2025) · Measurement · doi:10.1016/j.measurement.2025.117942 · Cited by 5
Olfactory biosensor for smart agriculture
Chen, X., Ma, L., Wan, Z., Zhang, R., Yin, M., Yang, Z., & Xiao, X. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108611 · Cited by 3
Houssein, E. H., Ibrahim, I. A., Mahdy, M. A., Kayed, M., Albarrak, A. M., & Mohamed, W. M. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01293-5 · Cited by 4
“…capture technologies, inertial readings from wearable sensors, depth data from time-of-flight cameras, and RGB video frames provide a nuanced representation of human motion, capturing spatial, temporal and contextual insights that individual modalities may not fully express in isolation [28].”
ZSIF: A zero-shot solar irradiance forecasting model based on satellite images and numerical series
Dong, C.-R., Tang, J., Zhang, F., Hua, Q., & Lim, B.-H. (2025) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2025.102044 · Cited by 5
Hierarchical Signal Fusion Network for Pulsar Detection with Phase-Correlation and Signal Attentions
Wu, H., & Chi, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747382 · Cited by 3
Qu, C., Zhou, Z., Liu, Z., Jia, S., Ma, L., & Sheela, L. M. I. (2022) · Advances in Materials Science and Engineering · doi:10.1155/2022/4632137 · Cited by 3
RetractedIntegration of lncRNAs, Protein-Coding Genes and Pathology Images for Detecting Metastatic Melanoma
Liu, S., Fan, Y., Li, K., Zhang, H., Wang, X., Ju, R., Huang, L., Duan, M., & Zhou, F. (2022) · Genes · doi:10.3390/genes13101916 · Cited by 4
“Both image and OMIC data provide complementary information about melanoma, and the integrated analysis of multimodal data is important to fully utilize these different data modalities [35,36].”
Wang, Q., & Xu, R. (2022) · PubMed Central · Cited by 6
Ji, P., Shan, F., Li, F., Sun, H., Wang, M., & Shan, D. (2023) · Security and Communication Networks · doi:10.1155/2023/5627246 · Cited by 3
Cascaded Cross-Modal Transformer for Request and Complaint Detection
Ristea, N.-C., & Ionescu, R. T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612846 · Cited by 4
“Tackling the complexity of real-world data through the combination of multiple modalities is a challenging task, requiring the development of a robust and efficient method for aggregating all sources of information [8, 20, 23].”
MethodReal-Time Air Pollution Monitoring System Employing IoT
Ghose, B., & Rehena, Z. (2024) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot59759.2024.10467734 · Cited by 5
“The authors in [14] discussed various aspects of deploying deep learning architectures in multimodal data fusion tasks.”
BackgroundOccupation Prediction with Multimodal Learning from Tweet Messages and Google Street View Images
Liu, X., Peng, B., Wu, M., Wang, M., Cai, H., & Huang, Q. (2024) · AGILE GIScience Series · doi:10.5194/agile-giss-5-36-2024 · Cited by 5
Self-Supervised, Multi-View, Semantics-Aware Anchor Clustering
Wei, K., Li, H., Liu, Q., & Zhang, X. (2024) · Electronics · doi:10.3390/electronics13234782 · Cited by 3
Quantify production planning efficiency through predictive modeling in manufacturing systems
Monaco, S., Apiletti, D., Francica, A., & Cerquitelli, T. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.110919 · Cited by 7
“In contrast, modern deep learning architectures, such as trans-formers (Vaswani, 2017), have demonstrated remarkable success in analyzing text and integrating multiple modalities, including text and images or images and audio (Gao, Li, Chen, & Zhang, 2020).”
AI and Personal Data in the Metaverse: A Risk Assessment
Simoes, S., del Rincón, J. M., & McWilliams, G. (2024) · IEEE Smart World Congress (SWC) · doi:10.1109/swc62898.2024.00340 · Cited by 3
“Data obtained from multiple sensors in smartphones and wearable devices are combined using multimodal data fusion techniques [51] to perceive user activities and interactions in the Metaverse [32].”
Li, Z., Zou, F., Liu, H., Liu, T., Zhou, G., Peng, X., Wang, L., Duan, S., & Jia, Y. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3571483 · Cited by 3
Sanchez, I. M., Mitra, S., Piplai, A., & Mittal, S. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229181 · Cited by 3
“Knowledge distillation is particularly advantageous over conventional feature-level fusion strategies—such as concatenation, attention-based fusion, or weighted averaging [21], [22]—because it explicitly transfers predictive capabilities through soft labels.”
Proceedings of the 22nd Engineering Applications of Neural Networks Conference: EANN 2021
Charalampos, S., Nousi, P., Tosidis, P., Tsampazis, K., Passalis, N., Tefas, A., & Nikolaidis, N. (2021) · Proceedings of the international neural networks society · doi:10.1007/978-3-030-80568-5 · Cited by 3
A Multimodal Deep Learning Approach to Emotion Detection and Identification
Ayyalasomayajula, S. C., Ionescu, B. E., Trifan, M., & Ionescu, D. (2022) · IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI) · doi:10.1109/saci55618.2022.9919496 · Cited by 3
“MobileNetV2 uses depth-wise separable convolution as the building blocks [29] and comprises features like linear bottlenecks between the layers and shortcut connections between the bottle-necks [30].”
Two heads are better than one: Dual systems obtain better performance in facial comparison
Li, Z., Xie, L., & Song, H. (2023) · Forensic Science International · doi:10.1016/j.forsciint.2023.111879 · Cited by 3
A Deep Fusion Gaussian Mixture Model for Multiview Land Data Clustering
Li, P., Chen, Z., Gao, J., Zhang, J., Jin, S., Zhao, W., Xia, F., & Wang, L. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8880430 · Cited by 3
Self-citationKatharria, A., Pant, M., Velasquez, J. D., Snåšel, V., Rajwar, K., & Deep, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5400616 · Cited by 5
Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models
Feng, S. Y., Lu, K., Tao, Z., Alikhani, M., Mitamura, T., Hovy, E. H., & Gangal, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21306 · Cited by 8
Zhi-xiu, Z. (2024) · Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS) · doi:10.1109/acctcs61748.2024.00024 · Cited by 2
“Gao et al. [3] conducted a survey on the application of deep learning for multimodal data fusion.”
BackgroundKdouri, L., Hmamouche, Y., Seghrouchni, A. E. F., & Chaminade, T. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9040031 · Cited by 2
“In recent years, deep learning has been successfully applied to process multimodal data, as well as learn correlations and joint representations across different modalities [17,18].”
Xu, J., Yuan, X., Zheng, L., & Lin, D. (2025) · IEEE Access · doi:10.1109/access.2025.3571523 · Cited by 3
Fluorescence sensing of nitro explosives based on deep learning
Xiong, J., Zhang, J.-Y., Zheng, X., Hu, T., Xiang, H., Li, Y., Xiao, B., Wang, X., & Yang, R. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102690 · Cited by 2
Self-citationQuantum-inspired complex spiking neural network for multi-source data joint classification
Liu, Y., Li, Y., Xu, H., & Lin, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103526 · Cited by 3
Cheng, C., Chen, P.-W., Jen, H.-Y., & Wu, Y.-T. (2025) · Mathematics · doi:10.3390/math13172804 · Cited by 3
3D point cloud lithology identification based on stratigraphically constrained continuous clustering
Gan, B., Jing, R., Shao, Y., Liu, Y., Duan, X., Li, P., & Li, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-18946-3 · Cited by 3
Self-citationIto, Y., Ishizawa, C., & Kageyama, Y. (2025) · Displays · doi:10.1016/j.displa.2025.103267 · Cited by 2
Yan, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-22059-2 · Cited by 2
Priyadharsini, M., Tyagi, B., Naga Priyadarsini, R., & Mohankumar, B. (2025) · Scientific Reports · doi:10.1038/s41598-025-23062-3 · Cited by 6
Liu, S., Huang, M., Yang, G., & Zhou, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-24135-z · Cited by 2
“Multimodal deep learning refers to the computational approaches that can process and relate information from multiple modalities, such as text, images, audio, and spatiotemporal data, to perform various tasks including classification, detection, and relationship extraction 21 .”
Cheng, C. (2025) · Informatica · doi:10.31449/inf.v49i14.11191 · Cited by 2
Soybean phenological stage identification based on multimodal data and a dynamic gating fusion model
Liu, Q., Jing, H., Wen, X., Wu, X., Yan, L., Yang, Q., Jia, S., Guo, S., fan, F., & Jin, X. (2026) · Smart Agricultural Technology · doi:10.1016/j.atech.2026.101827 · Cited by 2
Huang, J., Yang, X., Liang, X., Chen, S., Sun, Y., Mok, G. S. P., Li, S., Wang, Y., & Tan, T. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.103956 · Cited by 4
Narejo, K. R., Zan, H., Mostafa, S. M., Karim, F. K., Mehmood, F., & Yaseen, A. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00447-y · Cited by 2
Xia, T., Wang, X., Yang, Z., & Zhao, R. (2026) · iScience · doi:10.1016/j.isci.2026.115335 · Cited by 6
“164,165 Limitations of 2D models for studying 3D mechanosensing Although two-dimensional (2D) culture systems have been widely used to investigate cellular mechanosensing mechanisms, fundamental differences exist between 2D and three-dimensional (3D) microenvironments.”
Depression and anxiety characterization and detection with multimodal deep learning
Lu, T., Cho, L., Qiu, Z., Wang, Y., Niu, W., Leung, Y. W., Wang, Y., Zhang, H., Chang, W. C., Tanveer, M., et al. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00632-6 · Cited by 4
“Multimodal deep learning integrates data from diverse modalities to enhance model performance 49 .”
Learning Human–Environment Interactions via Wearable AI Interfaces
Wen, F., Wang, S., & Zhang, T. (2026) · ACS Nano · doi:10.1021/acsnano.6c05494 · Cited by 3
Multimodal machine learning in real estate appraisal: models, data, and trends
Huang, C., Liang, B., Li, Z., Wang, J., & Chen, F. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01204-8 · Cited by 3
“…surveys highlight that multi-level fusion mitigates the limitations of shallow concatenation by modelling non-linear cross-modal dependencies [78, 79] Hybrid fusion is frequently instantiated via attention-based or Transformer-based cross-modal blocks interleaved with modality-specific processing…”
A dilated causal convolutional model for surgical skill assessment using optical neuroimaging
Subedi, A., Rahul, F. N. U., Dutta, A., Norfleet, J., Makled, B., Intes, X., & De, S. (2022) · Clinical and Translational Neurophotonics · doi:10.1117/12.2610190 · Cited by 3
“Most common deep learning architecture used in fNIRS studies are Convolutional Neural Nets (CNN) [14,15,16,17], Long Short Term Memory (LSTM) variant of Recurrent Neural Nets [18,19,20,21], and Multilayer Perceptrons (MLP) [8,22].”
BackgroundSource Code Vulnerability Detection Based On SAR-GIN
Xia, X., Wang, Y., & Yang, Y. (2021) · 2021 2nd International Conference on Electronics, Communications and Information Technology (CECIT) · doi:10.1109/cecit53797.2021.00202 · Cited by 2
Singh, V., & Singh, G. (2023) · doi:10.1109/icaia57370.2023.10169451 · Cited by 2
Machine Learning for Data Fusion: A Fuzzy AHP Approach for Open Issues
Kukreja, V., Abraham, A., Kalaiselvi, K., Thilak, K. D., Hariharan, S., & Chen, S.-Y. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.045136 · Cited by 3
Person Identification with Voice and Ear-print in Degraded Conditions for Smart Home Access
Boujnah, S., Ferjaoui, R., & Khalifa, A. B. (2023) · International Conference on Cyberworlds (CW) · doi:10.1109/cw58918.2023.00071 · Cited by 3
“…parts, Diagnosis and detection of machine defects) and different other types ( Multimodal Biometry, Pedestrian detection and tracking, Facial recognition of emotions, Recognition of human actions, Regulation and management of transport systems, Location of the speaker) [10], [11], [12], [13].”
BackgroundA Novel Approach to Trajectory Situation Awareness Using Multi-modal Deep Learning Models
Dai, X., Ying, C., & Dai, L. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0885-7_19 · Cited by 2
Research on deep learning-based land cover extraction method using multi-source mixed samples
Lan, Y., Hu, Q., Wang, S., Li, J., Zhao, P., & Ai, M. (2024) · International Symposium on Multispectral Image Processing and Pattern Recognition (MIPPR) · doi:10.1117/12.2689788 · Cited by 2
“They play an increasingly important role in national security and industrial production 2-4 .”
BackgroundSpeech recognition using an english multimodal corpus with integrated image and depth information
Wang, B. (2024) · Scientific Reports · doi:10.1038/s41598-024-78557-2 · Cited by 3
Personalized decision-making for agents in face-to-face interaction in virtual reality
Dongye, X., Weng, D., Jiang, H., Tian, Z., Bao, Y., & Chen, P. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01591-7 · Cited by 2
“Another approach is to consolidate multimodal information into a unified tensor and fuse feature semantics through tensor computations [28].”
A Novel Moisture Measurement Method for the Sintering Mixture Based on Multivariate Feature Fusion
Gong, J., Pan, D., Jiang, Z., Yu, H., Gui, G., & Gui, W. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3551563 · Cited by 2
“Overall, the combination of multiple modalities has the potential to provide more diverse information about the same object, typically achieving higher accuracy compared to single-source approaches [26], [27].”
Vision Transformer-Driven LiDAR Data Fusion for Enhanced Hyperspectral Image Classification
Bhavanam, S. R. (2024) · IEEE India Geoscience and Remote Sensing Symposium (InGARSS) · doi:10.1109/ingarss61818.2024.10983980 · Cited by 2
“…HSI with other modalities leverages their complementary strengths, addressing challenges like spectral aliasing in HSI, for example, by using LiDAR’s precise elevation data to distinguish between objects with identical spectral signatures but different heights, such as roads and roofs [3].”
He, H., Xie, H., Shen, G., Fu, B., You, H., & Sanchez, V. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227447 · Cited by 2
“In other words, the RSEmbed module is a data augmentation mechanism that enhances the 4S-Classifier’s generalization capability by aligning the representations of rare species across different data types [31].”
Timing Is Everything: Finding the Optimal Fusion Points in Multimodal Medical Imaging
Guarrasi, V., Mogensen, K., Tassinari, S., Qvarlander, S., & Soda, P. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227201 · Cited by 4
Chen, B., Yu, J., Feng, H., Rungsardthong, V., & Zhang, X. (2025) · Food and Bioprocess Technology · doi:10.1007/s11947-025-04044-9 · Cited by 2
“…mechanism of the human brain, deep learning leverages neural networks to automatically analyze and identify patterns and features within large datasets, enabling the solution of complex computational problems and the extraction of valuable information from diverse data sources (Gao et al., 2020).”
Lee, H. J., & Ro, Y. M. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506748 · Cited by 4
Fan, Y., Wang, J., Shu, G., Xiao, S., Eb, Y. J., & Sopian, K. (2025) · Journal of Physics Conference Series · doi:10.1088/1742-6596/3001/1/012033 · Cited by 2
He, Z., Nidhi, M. H., Guo, Z., Li, L., Guo, C., Hou, J., Wang, X., & Liu, K. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.101321 · Cited by 3
Recreating Lunar Environments by Fusion of Multimodal Data Using Machine Learning Models
Castillo, A. C., Marroquin-Escobedo, J. A., Gonzalez-Irigoyen, S., Martinez-Santoyo, M., Villalpando‐Hernandez, R., & Vargas‐Rosales, C. (2022) · International Electronic Conference on Sensors and Applications (ECSA) · doi:10.3390/ecsa-9-13326 · Cited by 2
Kagalkar, R. M. (2022) · International Journal of Image Graphics and Signal Processing · doi:10.5815/ijigsp.2022.04.05 · Cited by 2
“This paper [18] addresses and presents some pioneering deep learning models to fuse these multimodal big data.”
BackgroundData and Decision Fusion with Uncertainty Quantification for ML-based Healthcare Decision Systems
Bezirganyan, G. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3616004 · Cited by 4
“Hence, while traditional machine learning and deep learning techniques are designed for uni-modal inputs, a multi-modal approach is necessary to leverage all the information from the different modalities and capture cross-modal correlations [4].”
BackgroundMulti-View Representation Learning via View-Aware Modulation
Wang, R., Sun, H., Nie, X., Lin, Y., Xi, X., & Yin, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612494 · Cited by 4
“More taxonomies can be found in the recent surveys [6, 14, 16].”
BackgroundA Novel Deep Learning-Based Model for Classification of Wheat Gene Expression
Ismail, A., Hamdy, W., Al‐Zoghby, A. M., Awad, W. A., Ebada, A. I., Nam, Y., Kang, B.-G., & Abouhawwash, M. (2024) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.038192 · Cited by 2
Zhang, B., Liu, X., Xie, X. N., Tong, X., Jia, Y., Shi, T., & Qu, W. (2024) · ACM Transactions on Sensor Networks · doi:10.1145/3651618 · Cited by 3
Zhang, C., & He, D. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/7917021 · Cited by 2
“We also adopt IoU (Intersection-Over-Union) to illustrate the effectiveness of the proposed method [35, 36].”
MethodInformation Exchange and Fusion in Smart Healthcare
Ding, S., Wu, D. D., Zhao, L., & Li, X. (2022) · AI for risks · doi:10.1007/978-981-19-2560-3_4 · Cited by 2
Multimodal-Multisensory Experiments: Design and ImplementationPreprint
Razavi, M., Yamauchi, T., Janfaza, V., Leontyev, A., Longmire-Monford, S., & Orr, J. M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.01.405795 · Cited by 2
“2020 conducted a survey study on using deep learning techniques for multimodal data 692 fusion and how they can help find new interpretations of the data [28].”
BackgroundUltralow-Dose Pediatric Total-Body PET/CT Imaging Using an Artificial Intelligence TechniquePreprint
Zhang, Q., Xiao, Z., Zhang, X., Hu, Y., Zhao, Y., Wang, J., Feng, J., Li, C., Zhou, Y., Yang, Y., et al. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4000475 · Cited by 2
Minyilu, Y., Yimer, M. A., & Meshesha, M. (2024) · medRxiv · doi:10.1101/2024.01.07.24300957 · Cited by 3
Benani, A., Ohayon, S., Laleye, F., Bauvin, P., Messas, E., Bodard, S., & Tannier, X. (2025) · medRxiv · doi:10.1101/2025.03.12.25322656 · Cited by 15
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.