A Survey on Deep Learning for Multimodal Data Fusion
The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
With the wide deployments of heterogeneous networks, huge amounts of data with characteristics of high volume, high variety, high velocity, and high veracity are generated. These data, referred to multimodal big data, contain abundant intermodality and cross-modality information and pose vast challenges on traditional data fusion methods. In this review, we present some pioneering deep learning models to fuse these multimodal big data. With the increasing exploration of the multimodal big data, there are still some challenges to be addressed. Thus, this review presents a survey on deep learning for multimodal data fusion to provide readers, regardless of their original community, with the fundamentals of multimodal deep learning fusion method and to motivate new multimodal data fusion techniques of deep learning. Specifically, representative architectures that are widely used are summarized as fundamental to the understanding of multimodal deep learning. Then the current pioneering multimodal data fusion deep learning models are summarized. Finally, some challenges and future topics of multimodal data fusion deep learning models are described.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For more generally applicable models, several deep learning frameworks have been extended and enable prediction using multimodal data (Gao et al. 2020).”
From Similarity-based multimodal regression (2024) · cites this paper as Gao et al. 2020
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“ResNet (He et al., 2016) A shortcut connection is used to cross several layers to back propagate the network loss to previous layers.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 6×
“The image subnetwork is a representative deep convolutional neural network, such as Alexnet and Inception, which effectively encodes the image input into a concise representation.”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“LSTM (Hochreiter & Schmidhuber, 1997) The memory block is introduced to model the long-time dependency well.”
BackgroundInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×
“Then the gradients of the weight, the visible bias, and the hidden bias are computed in the following forms: Unfortunately, in those gradient-computing equations, the probability x P ( x ) P ( h i = 1 | x ) is difficult to compute (Hinton, Osindero, & Teh, 2006).”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Similar to the fully connected architecture, the CNN is also trained to fit the training data, using the same algorithm (LeCun et al., 1989; LeCun, Bot-tou, Bengio, & Haffner, 1998; Zeiler & Fergus, 2014).”
Method
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial nets. In Z.Ghahramani, M.Welling, C.Cortes, N. D.Lawrence, & K. Q.Weinberger (Eds.), Advances in neural information processing systems, 27 (pp. 2672–2680). Red Hook, NY: Curran.
Cited by
Items 101–200 of 837
Edge AI for Smart Cities: Foundations, Challenges, and Opportunities
Velaga, K. S., Guo, Y., & Yu, W. (2025) · Smart Cities · doi:10.3390/smartcities8060211 · Cited by 44
Plant Physiological Analysis to Overcome Limitations to Plant Phenotyping
Haworth, M., Marino, G., Atzori, G., Fabbri, A., Daccache, A., Killi, D., Carli, A., Montesano, V., Conte, A., Balestrini, R., et al. (2023) · Plants · doi:10.3390/plants12234015 · Cited by 47
“Though studies in plant phenotyping have spontaneously learnt on several input sources, as in Ewald et al. [292], the reader may further explore the issues associated in extensive data fusion surveys [293], especially for complex learning models [294].”
BackgroundApplication of Multimodal Fusion Deep Learning Model in Disease Recognition
Liu, X., Qiu, H., Li, M., Zhou, Y., Yang, Y., & Yan, Y. (2024) · IEEE International Conference on Sensors, Electronics and Computer Engineering (ICSECE) · doi:10.1109/icsece61636.2024.10729504 · Cited by 43
“This design not only preserves image information at multiple scales, but also improves the fusion effect by accurately locating key areas and features through attention allocation at the spatial and channel levels[23].”
BackgroundWord2Pix: Word to Pixel Cross-Attention Transformer in Visual Grounding
Zhao, H., Zhou, J. T., & Ong, Y.-S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3183827 · Cited by 50
Zhang, Y., Zhang, H., Xiao, L., Bai, Y., Calhoun, V. D., & Wang, Y.-P. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3161828 · Cited by 41
“These data fusion methods are based on feature learning, but have not fully explored both the structure and cross-modality relationship among multi-modal data [8].”
MethodLi, D. X., Pang, Z., Chen, Y., Yang, K., Shao, J., Luo, Y., Zeng, Y., He, C., & Gao, Y. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.113226 · Cited by 51
Massafra, R., Comes, M. C., Bove, S., Didonna, V., Gatta, G., Giotta, F., Fanizzi, A., Forgia, D. L., Latorre, A., Pastena, M. I., et al. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12060953 · Cited by 40
“The fusion of the information from different sources, and especially from different kinds of images, through AI models has already been proven as promising [46].”
BackgroundZhao, X., Liao, Y., Tang, Z., Xu, Y., Tao, X., Wang, D., Wang, G., & Lu, H. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1107284 · Cited by 42
“In recent years, researchers have employed computational techniques such as machine learning and deep learning methods (Gao et al., 2020; Liang et al., 2021; Wang and Deng, 2021; Yan et al., 2021; Ye et al., 2021) to model and measure human personality from the first impression behavior data, which is called personali…”
BackgroundA revisit of the amygdala theory of autism: Twenty years after
Wang, S., & Li, X. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108519 · Cited by 44
Guo, Y., Ge, H., & Jin-hong, L. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1159063 · Cited by 40
“Thus, those data consisting of two or more modalities are called multimodal data (Gao et al., 2020).”
BackgroundClustered Multi-Criteria Routing Algorithm for Mega Low Earth Orbit Satellite Constellations
Jiao, J., Yang, P., Du, Z., Wang, Y., & Zhang, Q. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3396350 · Cited by 39
“In additional, consider the presence of multimodal data services [8], [9] in mega LEO satellite constellations, such as mission critical push to talk (MCPTT) and mission critical data (MCD) services [10], [11], it is crucial to design a routing algorithm for the multimodal data services coexistence…”
BackgroundHe, K., Zhang, X., Xu, D., Gong, J., & Xie, L. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3185887 · Cited by 35
“By analyzing the aggregation of tracer factors in different imaging regions, the activity and metabolism information of human body can be captured by functional imaging technology [3].”
MethodMMHFNet: Multi-modal and multi-layer hybrid fusion network for voice pathology detection
Mohammed, H. M. A., Ömeroğlu, A. N., & Oral, E. A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119790 · Cited by 37
Wang, S., Shibghatullah, A. S., Iqbal, T. J., & Keoy, K. H. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10371-3 · Cited by 40
“Combining deep learning with semantic fusion strategies [150] offers a novel approach for multimodal emotion recognition, with promising implications.”
A Hybrid Multimodal Data Fusion-Based Method for Identifying Gambling Websites
Wang, C., Zhang, M., Shi, F., Xue, P., & Li, Y. (2022) · Electronics · doi:10.3390/electronics11162489 · Cited by 39
A review and critical analysis of multimodal datasets for emotional AI
Al-Azani, S., & El-Alfy, E.-S. M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11271-1 · Cited by 40
One-class learning for fake news detection through multimodal variational autoencoders
Gôlo, M. P. S., de Souza, M. C., Rossi, R. G., Rezende, S. O., Nogueira, B. M., & Marcacini, R. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106088 · Cited by 40
Real-time multimodal interaction in virtual reality - a case study with a large virtual interface
Cao, L., Zhang, H., Peng, C. Y., & Hansberger, J. T. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14381-6 · Cited by 41
“Deep learning models were proposed for fusing temporal multimodal information with the praised capability to learn from a large number of data [10, 12, 17, 27, 28].”
BackgroundArtificial Intelligence Techniques for the effective diagnosis of Alzheimer’s Disease: A Review
Shastry, K. A., & Sanjay, H. A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16928-z · Cited by 36
“Owing to their capacity to “non-linearly” change input parameters, DL architectures could be utilized to bypass this barrier as well as derive heterogeneous attribute representations without the requirement for attribute extraction [115].”
A survey of multimodal event detection based on data fusion
Mondal, M., Khayati, M., Sandlin, H.-Â., & Cudré-Mauroux, P. (2024) · The VLDB Journal · doi:10.1007/s00778-024-00878-5 · Cited by 41
“[43],datafusionaimsto“integratethe data of different distributions, sources, and types into a global space in which both intermodality and cross-modality can be represented in a uniform manner”.”
Integrating multi-omics and machine learning for disease resistance prediction in legumes
Mohamedikbal, S., Al-Mamun, H. A., Bestry, M., Batley, J., & Edwards, D. (2025) · Theoretical and Applied Genetics · doi:10.1007/s00122-025-04948-2 · Cited by 41
“Multimodal deep learning enables the fusion of data from different sources and types including images, text and high-dimensional data into a single model (Gao et al. 2020).”
An Ensemble Deep Learning Approach Combining Phenotypic Data and fMRI for ADHD Diagnosis
Qin, Y., Lou, Y., Huang, Y., Chen, R., & Yue, W. (2022) · Journal of Signal Processing Systems · doi:10.1007/s11265-022-01812-0 · Cited by 37
“A large number of reviews and surveys [6, 10–12] have been discussed around the topic of multimodal data learning and analysis [13–15].”
BackgroundMethodWang, K., Yu, L., Xu, J., Zhang, S., & Qin, J. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108720 · Cited by 35
Liu, C., Wang, Y., Fang, Y., Yang, C., & Gui, W. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3359416 · Cited by 33
“The widespread use of intelligent devices has led to a wealth of multimodal data that can be utilized for various downstream tasks [12], [13].”
BackgroundLu, L., d’Avigneau, A. M., Pan, Y., Sun, Z., Luo, P., & Brilakis, I. K. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106134 · Cited by 39
Advancements in Medical Radiology Through Multimodal Machine Learning: A Comprehensive Overview
Haq, I. U., Mhamed, M., Alharbi, M., Osman, H., Hamd, Z. Y., & Liu, Z. (2025) · Bioengineering · doi:10.3390/bioengineering12050477 · Cited by 40
“…have addressed the application of ag-nostic have highlighted the signi fi cance of fusion in have examined models and architectures, focusing [13], and particularly in the healthcare sector the prospective utilization of MMML, frequently modality within the multimodal framework [20– on…”
Zia, K., Chiumento, A., & Havinga, P. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3215731 · Cited by 39
“Similarly, inputs from other layers can be represented with abstractions which can be combined to extract cross modalities as it is done in multi-modal learning [72].”
BackgroundMan, T., Osipov, V. Y., Zhukova, N. A., Subbotin, A., & Ignatov, D. I. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102427 · Cited by 34
A deep multimodal network for multi-task trajectory prediction
Lei, D., Xu, M., & Wang, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102597 · Cited by 32
Shamsuddin, D., Danilevicz, M. F., Al-Mamun, H. A., Bennamoun, M., & Edwards, D. (2024) · Remote Sensing · doi:10.3390/rs16214043 · Cited by 34
“To address this challenge, machine learning approaches such as deep learning and incorporating data from different sources have been used as powerful tools to improve the accuracy of prediction models [16,17].”
InfluentialWang, K., Song, Y., Sheng, H., Xu, J., Zhang, S., & Qin, J. (2022) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2022.05.010 · Cited by 32
Patent Data for Engineering Design: A Critical Review and Future Directions
Jiang, S., Sarica, S., Song, B., Hu, J., & Luo, J. (2022) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4054802 · Cited by 32
“Provided that patent documents normally contain multimodal information on design, we can take advantage of multimodal deep learning techniques to develop more systematic and informative design representations[112], such as multimodal knowledge graphs and resultant graph embeddings.”
BackgroundApplications of deep learning methods in digital biomarker research using noninvasive sensing data
Jeong, H., Jeong, Y. W., Park, Y., Kim, K., Park, J. H., & Kang, D. R. (2022) · Digital Health · doi:10.1177/20552076221136642 · Cited by 36
“8,9 In this review, we introduce several kinds of digital biomarkers used in recent times, report their clinical applications, and review deep learning techniques used for the supervized learning and dimensionality reduction.”
BackgroundA survey of multimodal federated learning: background, applications, and perspectives
Pan, H., Zhao, X., He, L., Shi, Y., & Lin, X. S. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01422-9 · Cited by 33
“…a significant role in enhancing the performance of multimodal models [36], the representation and fusion of multimodal data in deep learning are akin to data mining, where researchers predominantly rely on four pioneering architectures that DBN [37], SAE [38], CNN, and RNN, for improvements [22].”
Rethinking Satellite Data Merging: From Averaging to SNR Optimization
Kim, S., Sharma, A., Liu, Y., & Young, S. I. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3107028 · Cited by 31
“While data merging using deep learning-based techniques have become more popular of late [14], [15], weighted averaging continues to be extremely useful for its simplicity and applicability when data are limited, and interpretability of results important in decision- and policy-making [6], [16].”
BackgroundUniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning
Zou, H., Shen, M., Chen, C., Hu, Y., Rajan, D., & Chng, E. S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.41 · Cited by 44
Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
Hao, C., & Chen, D. (2021) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas51828.2021.9458577 · Cited by 31
Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview
Feng, X., Shu, W., Li, M., Li, J., Xu, J., & He, M. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-04915-3 · Cited by 35
“One of prevailing multimodal learning applications is based on Autoen-coder (AE) models [60], providing DL models with capabilities to integrate different data modalities into a single end-to-end optimized model [61].”
MethodModeling Both Intra- and Inter-Modality Uncertainty for Multimodal Fake News Detection
Wei, L., Hu, D., Zhou, W. J., & Hu, S. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3229966 · Cited by 34
“Although previous approaches obtain promising performance, we argue that they still suffer from the following issues: 1) Generally, the multimodal data comes from heterogeneous sources [17], is low quality and contains noise [18].”
BackgroundA deep learning model for multi-modal spatio-temporal irradiance forecast
Shan, S., Li, C., Wang, Y., Fang, S., Zhang, K., & Wei, H. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122925 · Cited by 29
Reis, M. J. C. S. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9050039 · Cited by 33
“Traffic Forecasting: The framework uses LSTM models to predict future congestion: Traffic Forecasting: The framework uses LSTM models to predict future congestion: where W represents the model parameters.”
Wang, W., Han, R., Zhang, M., Wang, Y., Wang, T., Wang, Y., Shang, X., & Peng, J. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab459 · Cited by 27
Bimodal variational autoencoder for audiovisual speech recognition
Sayed, H. M., ElDeeb, H. E., & Taie, S. A. (2021) · Machine Learning · doi:10.1007/s10994-021-06112-5 · Cited by 29
“Furthermore, deep learning models have played an essential role in capturing the intermo-dality features across different modalities and generating a unified representation of them (Gao et al., 2020).”
BackgroundDeep Multimodal Guidance for Medical Image Classification
Mallya, M., & Hamarneh, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_29 · Cited by 43
Wang, K., Song, Y., Huang, Z., Sun, Y., Xu, J., & Zhang, S. (2022) · Measurement · doi:10.1016/j.measurement.2022.111215 · Cited by 29
A Framework to Evaluate Fusion Methods for Multimodal Emotion Recognition
Peña, D., Aguilera, A., Dongo, I., Heredia, J., & Cardinale, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3240420 · Cited by 34
“In such scenarios, the fusion process should aggregate these different data sources in a specific way that captures the relationships between them [18], and, if possible, find a way to rely more on certain modalities when the quality of the sample from another one proves to be particularly poor…”
BackgroundRashid, M. T., Wei, N., & Wang, D. (2023) · Collective Intelligence · doi:10.1177/26339137231170825 · Cited by 38
“2018f, 2019d), and feature concatenation (Gao et al. 2020).”
MethodLearning-based needle tip tracking in 2D ultrasound by fusing visual tracking and motion prediction
Yan, W., Ding, Q., Chen, J., Yan, K., Tang, R. S. Y., & Cheng, S. S. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102847 · Cited by 32
Liang, Y., Chen, G., Li, S., & Xu, Z. (2023) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-5460 · Cited by 26
Zhang, Q., Hu, Y., Zhou, C., Zhao, Y., Zhang, N., Zhou, Y., Yang, Y., Zheng, H., Fan, W., Liang, D., et al. (2024) · EJNMMI Physics · doi:10.1186/s40658-023-00605-z · Cited by 31
Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics
Cwiek, A. P., Rajtmajer, S. M., Wyble, B., Honavar, V., Grossner, E. C., & Hillary, F. G. (2022) · Network Neuroscience · doi:10.1162/netn_a_00212 · Cited by 30
“…series observations – may push the limits of traditional modeling, ML approaches can capitalize on the complexity of multi-modal datasets (Baltrušaitis et al., 2019; Gao et al., 2020; Guo et al., 2019) and provide opportunity to examine interactions amongst variables otherwise impossible to test.”
BackgroundLee, T., Teisseyre, P., & Lee, J. (2023) · IEEE Access · doi:10.1109/access.2023.3240422 · Cited by 29
“Deep neural networks (DNNs) have recently demonstrated excellent prediction performance [9], [10] owing to their information fusion capability that helps capture nonlinear relationships between various financial data sources [11].”
BackgroundMultimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models
Ye, J., Hai, J., Song, J., & Wang, Z. (2023) · doi:10.1101/2023.08.24.23294597 · Cited by 24
Li, F., & Xu, J. (2025) · IEEE Access · doi:10.1109/access.2025.3552039 · Cited by 43
“Information systems (IS) are the technologies, people, and processes that work together to collect, assess, store, and dispense data due to make better decisions and run institutions [1], [2].”
Lv, Y., Hao, J., Tang, M., & Wu, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111767 · Cited by 29
A multimodal time-series method for gifting prediction in live streaming platforms
Xi, D., Tang, L., Chen, R., & Xu, W. X. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103254 · Cited by 33
Multimodal sensor fusion in the latent representation space
Piechocki, R. J., Wang, X., & Bocus, M. J. (2023) · Scientific Reports · doi:10.1038/s41598-022-24754-w · Cited by 30
Comparison of multimodal RGB-thermal fusion techniques for exterior wall multi-defect detection
Yang, X., Guo, R., & Li, H. (2023) · Journal of Infrastructure Intelligence and Resilience · doi:10.1016/j.iintel.2023.100029 · Cited by 49
Multimodal data fusion for geo-hazard prediction in underground mining operation
Liang, R., Zhang, C., Huang, C., Li, B., Saydam, S., Canbulat, I., & Munsamy, L. (2024) · Computers & Industrial Engineering · doi:10.1016/j.cie.2024.110268 · Cited by 25
Rashid, M. B., Rahaman, M. S., & Rivas, P. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030074 · Cited by 29
“This surge in data spans several modalities, encompassing visual elements in photographs, the semantic aspects of text, and auditory signals, thus offering a holistic view of the environment [3,4].”
BackgroundZhang, K., Li, P., & Wang, J. (2024) · Remote Sensing · doi:10.3390/rs16214113 · Cited by 26
Self-citationMultimodal malware classification using proposed ensemble deep neural network framework
Nazim, S., Alam, M. M., Rizvi, S. S. A., Mustapha, J. C., Hussain, S. S., & Su’ud, M. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-96203-3 · Cited by 27
“DL frameworks are predominantly appropriate for intermediate fusion as they allow for the smooth incorporation of marginal representations through a shared layer, while efficiently mapping hierarchical illustrations to real-world scenarios 30,31 .”
Multimodal AI systems for enhanced laying hen welfare assessment and productivity optimization
Essien, D., & Neethirajan, S. R. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.101564 · Cited by 28
OpenSync: An open-source platform for synchronizing multiple measures in neuroscience experiments
Razavi, M., Janfaza, V., Yamauchi, T., Leontyev, A., Longmire-Monford, S., & Orr, J. M. (2021) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2021.109458 · Cited by 25
Shen, W., Tang, H.-W., Li, J., Li, X., & Chen, S. (2023) · Journal of Cheminformatics · doi:10.1186/s13321-022-00675-8 · Cited by 25
Research on multimodal based learning evaluation method in smart classroom
Qianyi, Z., & Zhiqiang, L. (2023) · Learning and Motivation · doi:10.1016/j.lmot.2023.101943 · Cited by 25
Multi-modal lifelog data fusion for improved human activity recognition: A hybrid approach
Oh, Y., & Kim, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102464 · Cited by 25
Ferreira, R. E. P., & Dórea, J. R. R. (2025) · Journal of Dairy Science · doi:10.3168/jds.2024-25650 · Cited by 25
Applications of Machine Learning in Analysis and Design of Aerospace Composite Structures
You, C., Cai, Y., Wu, W., Zhang, H., Gao, X., & Song, Y. (2025) · Thin-Walled Structures · doi:10.1016/j.tws.2025.113914 · Cited by 29
SLAFusion: Attention fusion based on SAX and LSTM for dangerous driving behavior detection
Liu, J., Huang, W. X., Li, H., Ji, S., Du, Y., & Li, T. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119063 · Cited by 22
Wu, X., Mao, J., Xie, H., & Li, G. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102977 · Cited by 28
Naku, W., Nambisan, A. K., Roman, M., Zhu, C., Gerald, R. E., & Huang, J. (2023) · ACS Omega · doi:10.1021/acsomega.2c05451 · Cited by 23
“It is also possible to use ideas from sensor fusion and data fusion 48,49 to explore more ways of combining the information in the response signals from the three different surfaces.”
MethodTicino: A multi-modal remote sensing dataset for semantic segmentation
Barbato, M. P., Piccoli, F., & Napoletano, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123600 · Cited by 26
A power quality disturbances classification method based on multi-modal parallel feature extraction
Tong, Z., Zhong, J., Li, J., Wu, J., & Li, Z. (2023) · Scientific Reports · doi:10.1038/s41598-023-44399-7 · Cited by 20
“Multimodal data fusion aims to combine different distributed and different types of data in a single space, including images, audio, and measurement signals 29 .”
BackgroundMoving From Narrative to Interactive Multi-Modal Sentiment Analysis: A Survey
Ma, J., Rong, L., Zhang, Y., & Tiwari, P. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3610288 · Cited by 24
“To exploit the advantages of both feature and decision fusion strategies, hybrid fusion that is the combination of feature fusion and decision fusion, is designed with high complexity [31].”
COVID-19 Hierarchical Classification Using a Deep Learning Multi-Modal
Althenayan, A. S., Alsalamah, S., Aly, S. F., Nouh, T. A., Mahboub, B. H., Salameh, L. I. J., Alkubeyyer, M., & Mirza, A. A. (2024) · Sensors · doi:10.3390/s24082641 · Cited by 21
“In several fields, especially in diagnosis by medical assistants, deep learning approaches have accomplished significant advances in multi-modal structures by learning features from different sources of data [14].”
BackgroundRecent advances in continuous nanomanufacturing: focus on machine learning-driven process control
Venkatesan, S., Cullinan, M. A., & Bâldea, M. (2024) · Reviews in Chemical Engineering · doi:10.1515/revce-2024-0029 · Cited by 24
Restrepo, D. S., Wu, C., Cajas, S. A., Nakayama, L. F., Celi, L. A., & López, D. M. (2024) · medRxiv · doi:10.1101/2024.06.03.24308401 · Cited by 34
“This integration, known as multimodal data fusion, leverages mainly deep learning techniques [4, 5] and is critical for building systems that can interpret complex data in a manner akin to human cognition, thereby enhancing decision-making processes in clinical applications [6–13]: with fundus…”
BackgroundCombining piano performance dimensions for score difficulty classification
Ramoneda, P., Jeong, D., Eremenko, V. E., Tamer, N. C., Miron, M., & Serra, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121776 · Cited by 19
Zheng, F., Yin, H., Zhang, J., Duan, H., & Gupta, H. V. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131706 · Cited by 19
Self-citationBaek, A. M. C., Kim, T., Seong, M., Lee, S., Kang, H., Park, E., Jung, I. D., & Kim, N. (2025) · Virtual and Physical Prototyping · doi:10.1080/17452759.2025.2474532 · Cited by 23
“The above methods for handling combined data represent alternative multimodal fusion approaches documented in the literature [103], but they were not selected as the primary methodology in this paper.”
Transformers for Multi-Modal Image Analysis in Healthcare
Sagheer, S. V. M., Meghana, K. H., Ameer, P. M., Parayangat, M., & Abbas, M. A. E. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.063726 · Cited by 30
Ortiz-Pérez, D., Benavent-Lledó, M., Rodriguez-Juan, J., García‐Rodríguez, J., & Tomás, D. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114264 · Cited by 29
“Fusion strategies are categorized into different groups including early, late, or joint among others [50], [51].”
Yin, T., Peng, Y., Chao, K., & Li, Y. (2025) · npj Science of Food · doi:10.1038/s41538-025-00393-z · Cited by 25
“Additionally,multimodaldatafusion,alongwiththeautomationand standardizationofexperimentalprocedures,couldreducevariabilityindata collectionandpreprocessing,therebyenhancingreproducibilityinveter-inarydrugdetectionapplications 227 – 229 .”
Multi-Frequency RF Sensor Fusion for Word-Level Fluent ASL Recognition
Gürbüz, S. Z., Rahman, M. M., Kurtoğlu, E., Malaia, E., Gürbüz, A. C., Griffin, D. J., & Crawford, C. S. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3078339 · Cited by 19
Krishnamoorthy, U., & Balasubramani, S. (2024) · Advanced Theory and Simulations · doi:10.1002/adts.202400479 · Cited by 22
Artificial Intelligence in Gerontology: Data-Driven Health Management and Precision Medicine
Zhang, S., Wu, L., Zhao, Z., Fernández, J. R., & Chen, M. (2024) · Advances in Gerontology · doi:10.1134/s2079057024600691 · Cited by 24
“Effectively integrating heterogeneous, multimodal data remains a significant challenge in AI applications for gerontology, but advancements in multimodal learning models offer potential solutions for comprehensive health assessments and personalized interventions [61, 62].”
Jeong, H., Yun, B., Na, S., Son, M., Chae, S. H., Kim, C., & Cho, K. H. (2024) · Journal of Membrane Science · doi:10.1016/j.memsci.2024.123105 · Cited by 18
Combining 2D image and point cloud deep learning to predict wheat above ground biomass
Zhu, S., Zhang, W., Yang, T., Wu, F. L., Jiang, Y., Yang, G., Zain, M., Zhao, Y., Yao, Z., Liu, T., et al. (2024) · Precision Agriculture · doi:10.1007/s11119-024-10186-1 · Cited by 18
“Early fusion yields comprehensive multimodal information, facilitating more in-depth analysis and decision-making (Gao et al., 2020).”
Shuffled ECA-Net for stress detection from multimodal wearable sensor data
Kim, N., Lee, S., Kim, J., Choi, S. Y., & Park, S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109217 · Cited by 23
Semisupervised charting for spectral multimodal manifold learning and alignment
Pournemat, A., Adibi, P., & Chanussot, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107645 · Cited by 18
A spasticity assessment method for voluntary movement using data fusion and machine learning
Chen, Y., Yu, S., Cai, Q., Huang, S., Ma, K., Zheng, H., & Xie, L. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.102353 · Cited by 19
Deep Feature Fusion for Rumor Detection on Twitter
Luo, Z., Li, Q., & Zheng, J. J. (2021) · IEEE Access · doi:10.1109/access.2021.3111790 · Cited by 21
“Withthewide deployment of heterogeneous neural networks, multimodal data fusion via multi-input deep models becomes a popular research topic which integrates the data of different distributions, sources, and types into a global space [34].”
BackgroundThyroid cancer under the scope of emerging technologies
Tarabichi, M., Demetter, P., Craciun, L., Maenhaut, C., & Detours, V. (2021) · Molecular and Cellular Endocrinology · doi:10.1016/j.mce.2021.111491 · Cited by 19
Less is more: A closer look at semantic-based few-shot learning
Zhou, C., Yu, Z., Yuan, X., Zhou, S., Bu, J., & Wang, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102672 · Cited by 22
Deep learning-based geological map generation using geological routes
Li, C., Li, F., Liu, C., Tang, Z., Fu, S., Lin, M., Lv, X., Liu, S., & Liu, Y. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114214 · Cited by 20
Exploring Inner Speech Recognition via Cross-Perception Approach in EEG and fMRI
Qin, J., Zong, L., & Liu, F. (2024) · Applied Sciences · doi:10.3390/app14177720 · Cited by 26
Chen, X., Xu, H., Yu, S., Hu, W., Zhang, Z., Wang, X., Yuan, Y., Wang, M., Chen, L., Lin, X., et al. (2025) · Biology · doi:10.3390/biology14060651 · Cited by 23
“Furthermore, spatiotemporal alignment remains critical for single-cell spatial transcriptomics (e.g., 10x Visium) to avoid cross-modal representation mismatches with imaging or textual data [123–126].”
A New Kinect-Based Posture Recognition Method in Physical Sports Training Based on Urban Data
He, D., & Li, L. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8817419 · Cited by 17
“The scoring module firstly extracts the user’s motion information by using the Kinect module and then extracts the angle features and position information of the data [21, 22].”
Method
Metadata
- Authors
- Gao, J., Li, P., Chen, Z., & Zhang, J.
- Affiliations
- Dalian University of Technology (CN)
- Corresponding author
- Gao, Jing
- Li, Peng
- Chen, Zhikui
- Zhang, Jianing
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2020-05-01
- Volume
- 32
- Issue
- 5
- Pages
- 829-864
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W3011727199
- Cited by
- 821
- References
- 99
- Citations by year
- 2026: 152 · 2025: 213 · 2024: 161 · 2023: 129 · 2022: 90 · 2021: 57 · 2020: 19
- FWCI
- 5.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, multimodal deep learning, heterogeneous networks, deep learning architectures, fusion model
- MeSH terms
- Deep Learning, Algorithms, Artificial Intelligence, Surveys and Questionnaires, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.