A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
Method
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Items 501–600 of 837
Rag: The Revolution of Retrieval-Augmented Text GenerationPreprint
zhang, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5252835 · Cited by 3
Face tracking by fusing convolutional neural networks and particle filtering
Zhang, Z., Lin, S., & Xu, H. (2023) · IEEE International Conference on Power, Electronics and Computer Applications (ICPECA) · doi:10.1109/icpeca56706.2023.10075786 · Cited by 1
“In recent years, deep learning [4-5] has become a hot topic in the field of machine learning, and deep convolutional neural networks [6][7] have achieved unprecedented success in the field of image recognition, and the excellent feature extraction and classification capabilities of convolutional…”
BackgroundSensor Data Fusion for Improving Out of Domain Performance on SAR Image Classification
Sobien, D., Kauffman, J. A., Higgins, E., Freeman, L., & Pitt, J. S. (2024) · AIAA SciTech Forum · doi:10.2514/6.2024-2056 · Cited by 1
Autofusion: Fusing Multi-Modalities with Interactions
Nguyen, T.-T., Deligianni, F., & Nguyen, H. D. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.522 · Cited by 1
Kang, Z., Nah, F. F., & Siau, K. L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76821-7_6 · Cited by 1
Hypergraph-Based Organizers’ Behavior Scheduling Optimization: High-Order Relationships Discovery
Xu, Z., & Zhang, H. (2025) · IEEE Access · doi:10.1109/access.2025.3528651 · Cited by 1
Improved Defect Sizing in Adhesive Joints Through Feature-Based Data Fusion
Bhat, G. A., Smagulova, D., & Jasiūnienė, E. (2025) · Journal of Nondestructive Evaluation · doi:10.1007/s10921-024-01146-w · Cited by 2
Li, Y., Chen, M., Jiang, X., Wang, Y., Li, Y., & Wei, S. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105040 · Cited by 1
Hybrid Data Fusion for Elegant Emotion Recognition in Cloud-Based Big Data Systems
Jin, H., Sun, X., Jin, Z., & Kim, Y.-G. (2025) · International Conference on Multi-Agent Systems for Collaborative Intelligence (ICMSCI) · doi:10.1109/icmsci62561.2025.10894446 · Cited by 1
SV-data2vec: Guiding Video Representation Learning with Latent Skeleton Targets
Doždor, Z., Hrkać, T., & Kalafatić, Z. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00677 · Cited by 1
“We will also explore more advanced fusion techniques [10] to boost multimodal fine-tuning performance and apply SV-data2vec to other tasks and datasets to assess its robustness and adaptability.”
Two-Stage Dynamic Fusion Framework for Multimodal Classification Tasks
Ge, S., & Chen, Y. H. (2025) · INFORMS journal on computing · doi:10.1287/ijoc.2023.0448 · Cited by 2
Revisiting Intelligent Settlement and Nutritional Estimation of Small-bowl Dishes via Deep Learning
Fan, X., Zou, J., Fan, S., & Liu, Y. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3769748.3773350 · Cited by 2
“Furthermore, it is equipped with a meticulously constructed nutritional composition database and empirically measured density data for all 44 dishes, providing comprehensive resource support for end-to-end intelligent catering solutions [5, 12].”
Cao, A., Tan, S., Xiao, P., Li, Q., Li, W., & Zhang, Z. (2025) · Ultrasonics · doi:10.1016/j.ultras.2025.107921 · Cited by 1
Jeong, S., Choi, S., Oh, H., Kim, H., Chang, H.-S., Song, J., Kim, H. J., Kim, D.-J., & Chung, H. (2025) · Microchemical Journal · doi:10.1016/j.microc.2025.116661 · Cited by 1
Chen, Y., Wang, P., Zeng, J., Tan, M., Shan, K., 聂立波, Xue, Y., & Wang, T. (2026) · Talanta · doi:10.1016/j.talanta.2025.129345 · Cited by 1
CAFFT: Cross-Attention Feature Fusion Transformer for Risky Driving Behavior Recognition
Wang, S., Liang, J., Yu, X., Wang, W., Pan, C., & Chen, L. (2026) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2026.3651971 · Cited by 3
“Multimodal data fusion is the key technology to address this issue, aiming to integrate data from di ff erent modalities and leverage their complementarity and redundancy to enhance the overall system performance [18].”
Wang, Y., & Yao, J. (2026) · Measurement · doi:10.1016/j.measurement.2026.120505 · Cited by 2
Zheng, S., Yan, P., Ni, S., & Wang, D. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114097 · Cited by 1
FPZNet: Fuzzy Perception Zero-Shot Learning for Infrared and Visible Image Fusion
Wang, S., Han, B., Yang, L., Chen, K., Sun, G., & Xu, F. X. F. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3665200 · Cited by 1
“This observation challenges the need for high-dimensional feature spaces in multimodal fusion and suggests that most cross-modal interactions can be captured by a relatively small number of principal components [58].”
Multimodal Fault Diagnosis of Rolling Bearings Based on GRU–ResNet–CBAM
Xu, K., Zhang, J., Liu, D., Wang, C., Wang, R., & Zhou, F. (2026) · Machines · doi:10.3390/machines14030318 · Cited by 1
Self-citationTheodorakopoulos, L., & Theodoropoulou, A. (2026) · Electronics · doi:10.3390/electronics15071405 · Cited by 1
Ran, R., Wang, L., Li, G., Yuan, L., Lei, L., & Wei, Z. (2026) · Electronics · doi:10.3390/electronics15091910 · Cited by 1
Huang, L., Miao, J., Chen, H., Mei, X., Wu, Z., Wang, F., & Zhang, Y. (2026) · Journal of Marine Science and Engineering · doi:10.3390/jmse14110993 · Cited by 1
Liu, F., Lai, T., Xu, W., & Li, G. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349393 · Cited by 2
Concept drift handling in security applications: A survey across unimodal to multimodal data
Li, C., Wang, X., & Song, L. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134490 · Cited by 1
A Methodology to Manage Structured and Semi-structured Data in Knowledge Oriented Graph
Bellandi, V., Ceravolo, P., D’Andrea, G., Maghool, S., & Siccardi, S. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-08223-8_18 · Cited by 1
Ho, A. (2023) · Oxford University Press eBooks · doi:10.1093/med/9780197556269.003.0005 · Cited by 1
Sparse Maximum Margin Learning from Multimodal Human Behavioral Patterns
Zheng, E., Yu, Q., & Zheng, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25676 · Cited by 1
“In addition, DNNs may not be easily interpretable (Gao et al. 2020).”
BackgroundZhu, M., Chen, Y., He, J., & Yi, R. (2023) · Journal of Raman Spectroscopy · doi:10.1002/jrs.6590 · Cited by 1
T2KG: Transforming Multimodal Document to Knowledge Graph
Galiano, S., Muñoz, R., Gutiérrez, Y., Montoyo, A., & Ureña, L. A. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_043 · Cited by 2
“Several approaches have been proposed for integrating text and image data, including deep neural networks for image and text (Gao et al., 2020; Zhang et al., 2020a).”
BackgroundLeveraging Machine Learning for Effective Data Management
Sellami, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-662-70140-9_1 · Cited by 1
Unraveling the Mysteries of Alzheimer's Disease Using Artificial Intelligence
Tripathi, S., Sharma, Y., & Kumar, D. (2024) · Reviews on Recent Clinical Trials · doi:10.2174/0115748871330861241030143321 · Cited by 2
Multi-modal Machine Learning in Gastrointestinal Endoscopy: A Review
Chan, I. N., Wong, P. K., Yan, T., Hu, Y., & Chan, C. I. (2024) · International Conference on Biomedical and Bioinformatics Engineering (ICBBE) · doi:10.1145/3707127.3707129 · Cited by 2
“Task-specific models are developed using a variety of architectures like stacked autoencoders, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Trans-formers [16, 42, 56], whereas current large models are usually built with Transformers.”
Research on Chronic Disease Auxiliary Diagnosis System Based on Multimodal Deep Learning
Chang, H., Han, T., & Wang, Z. (2024) · International Conference on Computing, Robotics and System Sciences (ICRSS) · doi:10.1109/icrss65752.2024.00059 · Cited by 1
“[5].”
SUMMR: A Unified Multimodal Representation Framework for Songs
Ye, L., Shen, B., Su, Y., Chen, X., Gong, Y., Zhou, Y., & Lu, J. (2025) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001425580017 · Cited by 1
A neural approach to the Turing Test: The role of emotions
Pizzi, R., Quan, H., Matteucci, M., Mentasti, S., & Sassi, R. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107362 · Cited by 1
Piazza, T. D., Lazarus, C., Nempont, O., & Boussel, L. (2025) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.3046901 · Cited by 2
A Lightweight Infrared and Visible Image Fusion Method for Object Detection
Zhang, C., & Ji, W. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4276-2_19 · Cited by 1
Single-Source Dual-Stream Representation Learning for DNA Sequence Classification
Zhou, J., Zhang, Z., Wang, M., Zhou, W., & Li, H. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733433 · Cited by 1
“For now, multi-modal approaches have emerged as a potent solu-tion for enriching information representation, demonstrating robust capabilities across a wide range of tasks [8, 17, 44].”
A novel estimation of scientific and technological innovation index using multi-modal data
Qiu, Y., Chen, Z., Zou, C., & You, X. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2534104 · Cited by 1
“Currently, numerous studies focus on multimodal data fusion to integrate POIs and remote sensing imagery (Gao et al. 2020; Lu et al. 2022; Xu et al. 2020).”
Self-citationLearn from the best: A universal self-distillation approach with historical logits
Shi, L.-D., Giunchiglia, F., Zhang, H.-D., Shi, D., Song, R., Li, J., Diao, X., Zhao, A., & Xu, H. (2026) · Expert systems with applications · doi:10.1016/j.eswa.2025.129340 · Cited by 2
Lee, M. J., & Zhang, R. (2025) · Kalpa publications in computing · doi:10.29007/txp9 · Cited by 1
Tang, Z., Sun, B. T., He, X., Yuan, H., Yu, D., Liu, Z., Li, M., Chen, B., & Zhao, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04927-8_35 · Cited by 2
Yavari, M., Passmore, E., Rutz, E., Thomason, P., & Ackland, D. C. (2025) · Gait & Posture · doi:10.1016/j.gaitpost.2025.110010 · Cited by 1
Han, L., & Wang, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130389 · Cited by 2
ML-LQI: A multi-modal learning method for low-quality imbalanced modality data with interpretability
Zhao, F., Cao, X., Duan, Y. G., Yang, X., Zhang, X., Zhang, X., & Zhang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114991 · Cited by 1
Freshwater modeling in Aotearoa New Zealand: Current practice and future directions
Dost, K., Muraoka, K., Ausseil, A.-G., Benavidez, R., Blue, B., Conland, N., Daughney, C., Semádeni-Davies, A. F., Hoang, L. H., Hooper, A., et al. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106820 · Cited by 1
Khanal, P., & Khanal, A. (2025) · Crimson Journal of Science and Technology · doi:10.3126/cjost.v1i1.88577 · Cited by 1
“Finally, multimodal fusion networks aggregate heterogeneous QFT-sensor outputs (e.g., magnetometry, photonics, optomechanics) via late-or cross-attention fusion, improving robustness and calibration under missing-modality and drift conditions(J. Gao et al., 2020).”
Yang, F., & Gong, F. (2026) · Scientific Reports · doi:10.1038/s41598-026-36023-1 · Cited by 2
FedFusionNet: Advancing oral cancer recurrence prediction through federated fusion modeling
Aurnob, A. R., Tanim, S. A., Shrestha, T. E., Mridha, M. F., & Mistry, D. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104205 · Cited by 1
Two-phase CNN for model data fusion: Predicting 3D chlorophyll-a in the Mediterranean Sea
Tonelli, T., Cossarini, G., Manzoni, L., & Pietropolli, G. (2026) · Ocean Modelling · doi:10.1016/j.ocemod.2026.102707 · Cited by 1
Combining multimodal fatigue fracture surface images for analysis with a CNN
Jones, K., Shade, P. A., John, R., Musinski, W. D., Holm, E. A., & Rollett, A. (2026) · Scientific Reports · doi:10.1038/s41598-025-97184-z · Cited by 1
Wang, B., Guo, D., Yang, F., Zhang, H., Teng, P., Tavares, A., & Xu, H. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115672 · Cited by 2
Bhagyalatha, U., Sahoo, B. K., & Muppidi, S. (2026) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2026.112204 · Cited by 1
Fitting Bayesian Item Response Theory Models Using Deep Learning Computational Frameworks
Luo, N., Han, Y., Jinbo, H. E., Zhang, X., & Ji, F. (2026) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986261439301 · Cited by 2
Yang, J., ZHANG, Q.-D., Liu, Y., Li, J.-Y., Chen, P., Xu, Y.-P., Liu, D., & Chu, X. (2026) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2026.118864 · Cited by 1
MSSPE: Multi-Source Stellar Parameter Estimation under Incomplete Photometric Conditions
Wu, J., He, Y., Jiang, B., & Zhang, Y. (2026) · The Astronomical Journal · doi:10.3847/1538-3881/ae592f · Cited by 1
“Consequently, with the rapid advancements in multimodal deep learning ( J. Gao et al. 2020; Y.-L. Lee et al. 2023; S. Zhang et al. 2023; Q. Zhang et al. 2024; X. Zhang et al. 2024; Y. Zhu et al. 2024 ) , astronomical data analysis has increasingly adopted multisource integration approaches ( J.-F.…”
Li, C. S., Feng, S., Wang, L., Hou, J., Wan, X., Zhang, F., & Hu, B. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020054 · Cited by 3
AN OVERVIEW OF MULTIMODAL LEARNING: CONCEPTS, CHALLENGES, APPLICATIONS AND DATASETS
Ateş, A. H., & Yurtsever, U. (2026) · Konya Journal of Engineering Sciences · doi:10.36306/konjes.1716813 · Cited by 1
Mabasha, S., Gupta, S., & Babu, K. K. (2026) · International Journal of Computers and Applications · doi:10.1080/1206212x.2026.2690510 · Cited by 1
“Early surveys, the 2020 survey by Gao et al. [97] broad discusses DL approaches for the fusion of multimodal data in various fields, including healthcare, and groups them at three levels of fusion: pixel, feature, and decision.”
Pan, J., Chang, T., Lyu, M., Meng, W., Wang, Q., Ma, Y., Chen, Z., Li, X., Sun, C., Liang, R., et al. (2026) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-026-00248-6 · Cited by 1
“Many primarily focus on studies published before 2020 [8, 9], offering perspectives that no longer reflect recent methodological advances.”
Self-citationA Review of TinyML for Human Activity Recognition on Edge Devices
Lamaakal, I., Yahyati, C., Maleh, Y., Makkaoui, K. E., & Ouahbi, I. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8090271 · Cited by 1
Research on Multi-agency Data Fusion Mode Under Regional Medical Integration
Wang, Y., Fang, W., Zhu, W. H., & Ding, J. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-82565-2_22 · Cited by 1
TIPS: A Framework for Text Summarising with Illustrative Pictures
Golec, J., Hachaj, T., & Sokal, G. (2021) · Entropy · doi:10.3390/e23121614 · Cited by 1
Bayrak, Ş., & Yücel, E. (2022) · Studies in computational intelligence · doi:10.1007/978-981-19-2416-3_1 · Cited by 1
Unger, M., Li, P., Sen, S., & Tuzhilin, A. (2023) · ACM Transactions on Management Information Systems · doi:10.1145/3578710 · Cited by 3
“• Multi-Modal Embedding: we provide product recommendations based on the composite customer embeddings generated through multi-modal ML techniques (Gao et al. 2020) that fuse the inputs from multiple domains.”
MethodSelf-citationHo, A. (2023) · Oxford University Press eBooks · doi:10.1093/med/9780197556269.003.0001 · Cited by 1
Towards heart infarction detection via image-based dataset and three-stream fusion framework
Zhong, C., Yang, D., Wang, S., & Zhang, L. (2023) · Computer Communications · doi:10.1016/j.comcom.2023.12.021 · Cited by 1
Cross-Source Data Fusion in Industrial Automation
Yang, T., Chen, L., Liang, J., Wang, Q., & Zhang, W. (2023) · International Conference on Mobile Networks and Wireless Communications (ICMNWC) · doi:10.1109/icmnwc60182.2023.10435824 · Cited by 1
Deep learning for retina structural biomarker classification using OCT images
Xu, C., Zheng, H., Liu, K., Chen, Y., Chen, Y., Niu, C., Jin, S., Li, Y., Gao, H., Hu, J., et al. (2024) · International Workshop on Advanced Imaging Technology (IWAIT) · doi:10.1117/12.3026739 · Cited by 3
“…integrates diverse data—varying in distribution, source, and type—into a unified space, representing inter-modality and cross-modality uniformly, 10 which allows models to integrate image and text information, enabling more precise diagnoses in the medical field by combining OCT images with…”
BackgroundMiddleware Support for Edge Data Analytics over Heterogeneous Scenarios
Hu, B., & Hu, W. (2023) · ACM/IEEE Symposium on Edge Computing (SEC) · doi:10.1145/3583740.3626613 · Cited by 1
“…analytics, existing applications tend to be monolithic and purpose-built, ranging from research prototypes intended to explore new applications [39, 56, 76] or extract more insight from multimodal data [75], to industry solutions for individual use cases (e.g., Siemens Industrial Edge for…”
Explainable Stacking Models based on Complementary Traffic Embeddings
Gioacchini, L., Santos, W., Lopes, B., Drago, I., Mellia, M., Almeida, J. M., & Gonçalves, M. A. (2024) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw61312.2024.00035 · Cited by 5
“…we focus on a particular type of model ensemble – stacking [14] – which consists of learning how to combine multiple models trained on different sources and fusing their predictions (e.g. through a meta-model in case of Meta-learning) to achieve a more comprehensive view [10] of a given network.”
BackgroundExploring Semantic Understanding and Generative Modeling in Speech-Text Multimodal Data Fusion
Yu, H., Wang, X., Sun, Y., Yang, Y., & Sun, Y. (2024) · Applied Mathematics and Nonlinear Sciences · doi:10.2478/amns-2024-3156 · Cited by 1
Sunspot Prediction Based on Multi-Layer LSTM Stacking and Multimodal Information Fusion
Yang, H., Yang, W., Wang, Z., & Xu, Y. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743994 · Cited by 1
Complementary effects of multimodal interaction on sensor data fusion
Niu, R. (2024) · AIP conference proceedings · doi:10.1063/5.0223089 · Cited by 1
Text-Based Audio Correction Using Deep Learning Framework
Kusuma, S., Krishnan, S. G., Vikram, R., Sufiya, U. D., Vikas Reddy, R., & Shashidhar Reddy, G. (2024) · International Conference on Emerging Research in Computational Science (ICERCS) · doi:10.1109/icercs63125.2024.10895944 · Cited by 1
“Understanding the larger context of deep learning in this subject may be gained from a survey of deep learning techniques for audio and speech processing [6].”
Emotion-Driven AI Collaboration and Multimodal Learning in Vocational Education
Zhang, H., & Leong, W. Y. (2024) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir62538.2024.10959747 · Cited by 1
Tang, J., Zhang, S., & Chen, Q. (2025) · IEEE International Radar Conference · doi:10.1109/radar52380.2025.11031764 · Cited by 1
“Unlike traditional fusion techniques—such as signal averaging or basic feature fusion via summation or concatenation, which often treat modalities equally [17]—our approach further exploits the distinct advantages of each modality in different scenarios.”
Gautam, S., Riegler, M. A., & Halvorsen, P. (2025) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms65348.2025.00090 · Cited by 2
“Multimodal learning approaches aim to combine these data sources to build richer models [9].”
A Multimodal Deep Learning Approach for Skin Allergy Diagnosis Using Image and Symptom Data
Pradhan, S., & Mishra, B. K. (2025) · Cureus Journal of Computer Science. · doi:10.7759/s44389-025-06071-1 · Cited by 1
Nie, X., & Ahmad, F. (2025) · Future Technology · doi:10.55670/fpll.futech.4.4.2 · Cited by 1
Multimodal Prediction of Public Trust in Politicians from Speech and Text
Syed, M. S. S., Pirogova, E., & Lech, M. (2020) · International Conference on Signal Processing and Communication Systems · doi:10.1109/icspcs50536.2020.9310046 · Cited by 1
“A review of multimodal data fusion can be found in [37].”
BackgroundAn Approach to Spatiotemporal Trajectory Clustering Based on Community Detection
Wang, X., Niu, X., Zhu, J., & Liu, Z. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/5582341 · Cited by 1
“, audio [1], video [2], text [3], and image [4]), and can be analyzed jointly or separately [5, 6].”
BackgroundBousarhane, B., & Bouzidi, D. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-07969-6_2 · Cited by 2
Multi-view Representation Learning from Malware to Defend Against Adversarial Variants
Hu, J. L., Ebrahimi, M. R., Li, W., Li, X., & Chen, H. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00066 · Cited by 3
“Concatenation is the most basic fusion mechanism and simply combines views additively without further processing [23].”
BackgroundOn the Design of the MITLL Trimodal Dataset for Identity Verification
Singer, E., Borgström, B. J., Alperin, K., Nguyen, T., Dagli, C. K., Dale, M. R., & Ross, A. A. (2023) · International Workshop on Biometrics and Forensics (IWBF) · doi:10.1109/iwbf57495.2023.10157658 · Cited by 1
“The multimodal data fusion architecture in the survey by Gao et al. [11] references many studies that likewise employ multimodal datasets as the bases for the study of multimodal data fusion deep learning models.”
BackgroundSoulier, T., Colliot, O., Ayache, N., & Rohaut, B. (2023) · Anaesthesia Critical Care & Pain Medicine · doi:10.1016/j.accpm.2023.101301 · Cited by 3
SPLIT: Stance and Persuasion Prediction with Multi-modal on Image and Textual Information
Zhang, J., Yu, S., Li, X., Geng, J., Zheng, Z., & Ho, J. C. (2023) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/2023.argmining-1.19 · Cited by 1
“Multimodal fusion methods are techniques used to combine and integrate information from multiple modalities (e.g., text, images, audio) into a unified representation for analysis or decision-making (Gao et al., 2020).”
MethodSelf-citationEnhancing Classification with Multimodality: A Hierarchical Approach to Fashion Item Categorization
Ngom, W., Azib, M., & Ayadi, M. F. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-73110-5_13 · Cited by 2
Deep Multimodal Transfer-Learned Regression in Data-Poor DomainsPreprint
McClenny, L. D., Haile, M., Attari, V., Sadler, B. M., Braga-Neto, U., & Arróyave, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09310 · Cited by 2
“Multimodal Learning The concept of combining different domains of input into a single estimator for the desired learning task has been of distinct interest in the last decade [1,2].”
BackgroundWu, P., Wu, K. C., Li, Z., Liu, H., Yang, K., Zhou, R., Zhou, Z., Xing, N., & Wu, S. (2021) · Research Square · doi:10.21203/rs.3.rs-628673/v1 · Cited by 1
Multimodal Investigation of Bladder Cancer Data Based on CT, Whole Slide Image and TranscriptomePreprint
Wu, P., Wu, K., Li, Z., Liu, H., Yang, K., Zhou, R., Zhou, Z., Xing, N., & Wu, S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3943141 · Cited by 1
Road Ahead for Multi‐modal Intelligent Sensing in the Deep Learning Era
Zoha, A., Ramzan, N., Jamshed, M. A., & Rehman, M. U. (2024) · Wiley eBooks · doi:10.1002/9781394257744.ch12 · Cited by 1
Multi-modal and multi-region distance model for neuroimaging: Application to ABCD studyPreprint
Zhang, X., Vandekar, S., Chen, A. F., Kang, K., Seidlitz, J., Alexander‐Bloch, A. F., & Liu, J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.21.700689 · Cited by 1
Ambroise, C. (2024) · theses.fr (ABES) · doi:10.70675/d2529d35z93e6z4d60zb2c5z752d5f0edda3 · Cited by 2
The Evolutionary Deep Learning Model for Electrical Load Forecasting
Peng, F., Li, D., An, T., Wang, H., Tian, C., & Chen, Z. (2020) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics50389.2020.00147
“Deep learning has many classifications [19], such as fully connected neural network, Stacked Auto Encoder (SAE) [20], CNN [21], RNN [22], and LSTM network [23].”
BackgroundSelf-citationPatent Data for Engineering Design: A Review
Jiang, S., Sarica, S., Song, B., Hu, J., & Luo, J. (2022) · Proceedings of the Design Society · doi:10.1017/pds.2022.74 · Cited by 1
“…patent documents normally contain multimodal information on design, we can take advantage of multimodal deep learning techniques to develop more intelligent design applications [71], including multimodal knowledge graph construction, cross-modal idea generation, and cross-modal design retrieval.”
BackgroundAn Implementation of the “Guess Who?” Game Using CLIP
Sarri, A. M., & Rodríguez-Fernández, V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91608-4_41
“…to learn at the same time from different data modalities (image, audio, text, tabular data...) is a trending topic in the field of machine learning in general, and deep learning in particular, with many domains of application such as self-driving cars, healthcare and the Internet of Things [5].”
BackgroundVarshney, P. K. (2021) · Springer eBooks · doi:10.1007/978-3-030-63416-2_301
Varshney, P. K. (2020) · Springer eBooks · doi:10.1007/978-3-030-03243-2_301-1
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.