Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 201–300 of 1,001
Sensitivity Analysis of Sensors in a Hydraulic Condition Monitoring System Using CNN Models
König, C., & Helmi, A. M. (2020) · Sensors · doi:10.3390/s20113307 · Cited by 38
“The combination of information from different sensors at the data level is referred to as early information fusion of multimodal data [28].”
BackgroundLehnertz, K., Bröhl, T., & Rings, T. (2020) · Frontiers in Physiology · doi:10.3389/fphys.2020.598694 · Cited by 40
“Similarly, appropriate concepts and analysis techniques need to be developed to match these time scales and to allow multimodal data fusion (Lahat et al., 2015).”
BackgroundA Survey of Computer Vision Technologies in Urban and Controlled-environment Agriculture
Luo, J., Li, B., & Leung, C. (2023) · ACM Computing Surveys · doi:10.1145/3626186 · Cited by 50
Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications
Mena, F., Arenas, D., Nuske, M., & Dengel, A. R. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3361556 · Cited by 40
“MV does not constrain that views must be complementary, represent different physical quantities [14], or be from different data (e.g., images, signals, and metadata).”
Graph-based multimodal fusion with metric learning for multimodal classification
Angelou, M., Solachidis, V., Vretos, N., & Daras, P. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.06.013 · Cited by 41
A Comprehensive Survey on Multimodal Data Representation and Information Fusion Algorithms
Gaonkar, A., Chukkapalli, Y., Raman, P. J., Srikanth, S., & Gurugopinath, S. (2021) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit51480.2021.9498415 · Cited by 42
Torku, A., Chan, A. P. C., Yung, E. H. K., & Seo, J. (2022) · Building and Environment · doi:10.1016/j.buildenv.2022.109533 · Cited by 41
“A complete setup of the framework making use of multiple modalities for each data set to interact and inform each other is termed multimodal [52,65].”
BackgroundMethodXue, Y., Zhu, J., Huang, X., Xu, X., Li, X., Zheng, Y., Zhu, Z., Jin, K., Ye, J., Gong, W., et al. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104233 · Cited by 46
Tensor methods for multisensor signal processing
Miron, S., Zniyed, Y., Boyer, R., de Almeida, A. L. F., Favier, G., Brie, D., & Comon, P. (2020) · IET Signal Processing · doi:10.1049/iet-spr.2020.0373 · Cited by 36
“Moreover, it is likely that the joint exploitation of multi-diversity/modality sensing technologies for data fusion [59]–[61] further increases the data tensor order.”
BackgroundAlternating diffusion maps for multimodal data fusion
Katz, O., Talmon, R., Lo, Y., & Wu, H. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.01.007 · Cited by 40
Parallel Vehicular Networks: A CPSS-Based Approach via Multimodal Big Data in IoV
Han, S., Wang, X., Zhang, J. J., Cao, D., & Wang, F. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2867039 · Cited by 35
“for realizing rapid content dissemination in VNs has been attracted much more attentions [26], [34]–[36].”
BackgroundA Reliable Mixed-Norm-Based Multiresolution Change Detector in Heterogeneous Remote Sensing Images
Touati, R., Mignotte, M., & Dahmane, M. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2934602 · Cited by 32
“two different imaging modalities may be complementary (as it is especially the case of SAR and optical or multispectral sensors), and this complementarity could be exploited (not only in geoscience imaging [13]) for further improving the CD and analysis of complex land cover types or for sensors operating in extreme c…”
BackgroundFu, D., Weber, C., Yang, G., Kerzel, M., Nan, W., Barros, P. V. A., Wu, H., Liu, X., & Wermter, S. (2020) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2020.00010 · Cited by 42
“…articles on selective attention in the field of both psychology and computer science (Shinn-Cunningham, 2008; Frintrop et al., 2010; Lee and Choo, 2013), most of them only focus either on a single modality or on general crossmodal processing (Lahat et al., 2015; Ramachandram and Taylor, 2017).”
BackgroundMultimodal Navigation Systems for Users with Visual Impairments—A Review and Analysis
Kuriakose, B., Shrestha, R., & Sandnes, F. E. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4040073 · Cited by 42
“This review does not go into detail about the technical aspects of multimodal fusion and fission, but the complexity involved in the implementation is high [102].”
BackgroundAlternating Diffusion Map Based Fusion of Multimodal Brain Connectivity Networks for IQ Prediction
Xiao, L., Stephen, J. M., Wilson, T. W., Calhoun, V. D., & Wang, Y. (2018) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2018.2884129 · Cited by 33
“σ = C · max j min i,i =j ( d(2)(Ri ,Rj ) ) , (20) where C is typically set in the range [2, 3].”
BackgroundNumerical Optimization-Based Algorithms for Data Fusion
Vervliet, N., & Lathauwer, L. D. (2019) · Data handling in science and technology · doi:10.1016/b978-0-444-63984-4.00004-1 · Cited by 31
Fusion of Heterogeneous Earth Observation Data for the Classification of Local Climate Zones
Zhang, G., Ghamisi, P., & Zhu, X. X. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2914967 · Cited by 34
“Due to the rich characteristics of natural processes and environments, it is rare for a single acquisition method to provide a complete understanding of certain phenomenon [13]–[15].”
BackgroundOutlier-Robust Multi-Aspect Streaming Tensor Completion and Factorization
Najafi, M., He, L., & Yu, P. S. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/442 · Cited by 31
“, 2010], image processing [Lahat et al., 2015], social network analysis [Ermiş et al.”
MethodSimulation and sensor data fusion for machine learning application
Saadallah, A., Finkeldey, F., Buß, J., Morik, K. J., Wiederkehr, P., & Rhode, W. (2022) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101600 · Cited by 35
An Ensemble Deep Learning Approach Combining Phenotypic Data and fMRI for ADHD Diagnosis
Qin, Y., Lou, Y., Huang, Y., Chen, R., & Yue, W. (2022) · Journal of Signal Processing Systems · doi:10.1007/s11265-022-01812-0 · Cited by 37
“A large number of reviews and surveys [6, 10–12] have been discussed around the topic of multimodal data learning and analysis [13–15].”
BackgroundCausal reasoning in typical computer vision tasks
Zhang, K., Sun, Q., Zhao, C., & Tang, Y. S. (2023) · Science China Technological Sciences · doi:10.1007/s11431-023-2502-9 · Cited by 45
“As a result, selecting an analytical model that properly depicts the link between modalities and provides a meaningful combination of them is critical.[102].”
Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications
Cajas, S. A., Samanta, J., Suárez‐Cetrulo, A. L., & Carbajo, R. S. (2025) · Future Internet · doi:10.3390/fi17090417 · Cited by 56
“Multimodal fusion creates unified representations of diverse modalities, allowing events to be interpreted from multiple perspectives and data sources in next-generation networks [265].”
Multi-modal classifier fusion with feature cooperation for glaucoma diagnosis
Benzebouchi, N. E., Azizi, N., Ashour, A. S., Dey, N., & Sherratt, R. S. (2019) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2019.1653383 · Cited by 31
“Multimodal data classification has become a very active area of research in recent years and has been used in many applications of practical interest [46], including the use of ensemble learning techniques [13].”
BackgroundMultimodal Biometric Template Protection Based on a Cancelable SoftmaxOut Fusion Network
Jihyeon, K. I. M., Jung, Y. G., & Teoh, A. B. J. (2022) · Applied Sciences · doi:10.3390/app12042023 · Cited by 34
Min, J. H., Ham, S. W., Kim, D., & Lee, E. H. (2022) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221130026 · Cited by 31
“The concatenated output of each model (Outputs 1 and 2 in Figure 3) is used as the input vector for Model 3 ( 28 ).”
MethodGuo, Y., Ge, H., & Jin-hong, L. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1159063 · Cited by 35
“The information obtained from different modal forms naturally differs, so the fusion of information from different modalities allows the establishment of complementary relationships between modalities to eliminate redundant features and make the features more representative (Lahat et al., 2015).”
BackgroundZhang, L., Yang, Q., & Zhu, Z. (2024) · Foods · doi:10.3390/foods13121936 · Cited by 34
“At its core lies the fusion model structure [47], which comprises a feature extraction fusion stage and a perception scoring stage.”
BackgroundPerformance of methods that separate common and distinct variation in multiple data blocks
Måge, I., Smilde, A. K., & van der Kloet, F. M. (2018) · Journal of Chemometrics · doi:10.1002/cem.3085 · Cited by 29
“X2 1⁄4 T C123 ð Þ 2 P C123 ð Þ′ 2 þ T C12 ð Þ 2 P C12 ð Þ′ 2 þ T C23 ð Þ 2 P C23 ð Þ′ 2 þ T D2 ð Þ 2 P D2 ð Þ′ 2 þ E2; (4)”
An optimized data fusion strategy for structural damage assessment using electromechanical impedance
Singh, S. K., Sikdar, S., & Malinowski, P. (2021) · Smart Materials and Structures · doi:10.1088/1361-665x/abdc07 · Cited by 30
Zhou, Q., Cui, Y., Pedrycz, W., & Deng, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102083 · Cited by 29
How to Integrate Data from Multiple Biological Layers in Mental Health?
Silva, R. F., & Plis, S. (2019) · Springer eBooks · doi:10.1007/978-3-030-03553-2_8 · Cited by 28
Cai, M., Zhao, L., Hou, G., Zhang, Y., Wu, W., Jia, L., Zhao, J., Wang, L., & Qiang, Y. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106812 · Cited by 31
MDF-Net for abnormality detection by fusing X-rays with clinical data
Hsieh, C., Nobre, I. B., Sousa, S. C., Ouyang, C., Brereton, M. F., Nascimento, J. C., Jorge, J. A., & Moreira, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-41463-0 · Cited by 33
CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities
Kontras, K., Chatzichristos, C., Phan, H., Suykens, J. A. K., & Vos, M. D. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3354388 · Cited by 40
“They potentially contain complementary information for the identification of the underlying data distribution [24], [49].”
BackgroundFerreira, D. D. S., Pereira, F. M. V., Olivieri, A. C., & Pereira‐Filho, E. R. (2024) · Analytica Chimica Acta · doi:10.1016/j.aca.2024.342522 · Cited by 28
Imaging 3D chemistry at 1 nm resolution with fused multi-modal electron tomography
Schwartz, J., Di, Z. W., Jiang, Y., Manassa, J., Pietryga, J., Qian, Y., Cho, M. G., Rowell, J. L., Zheng, H., Robinson, R. D., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-47558-0 · Cited by 33
“To illustrate, in Figure 2, matching the resolution of fused multi-modal chemical tomography using traditional methods would require 45 EELS maps—a five-fold dose increase.”
MethodA review on enhancing agricultural intelligence with large language models
Li, H.-X., Wu, H., Li, Q., & Zhao, C. (2025) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2025.05.006 · Cited by 69
Student engagement assessment using multimodal deep learning
Yan, L., Wu, X., & Wang, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325377 · Cited by 38
“A key property of multi-modality is complementarity, in the sense that each modality brings to the whole some type of added value that cannot be deduced or obtained from any of the other modalities in the setup [24].”
What Is a Multi-Modal Knowledge Graph: A Survey
Peng, J., Hu, X., Huang, W., & Yang, J. K. (2023) · Big Data Research · doi:10.1016/j.bdr.2023.100380 · Cited by 42
Pham, D.-H., Meignen, S., Dia, N., Fontecave-Jallon, J., & Rivet, B. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2865253 · Cited by 30
“The key idea behind denoising techniques based on NMF lies in that the information about a single natural phenomenon can be acquired using different devices, called modalities [19], [20].”
BackgroundWang, L., Wang, C., Li, C., Murai, T., Bai, Y., Song, Z., Zhang, S., Zhang, Q., Huang, Y., Bi, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01933-3 · Cited by 33
“Whilelatefusionrequireslessdataandallowsfor easierintegrationofmultiplesensingstrategiesandalgorithms,itmaynot fullyleveragethecomplementarystrengthsofdifferentmodalities 115 .”
He, X., Fang, Y., Xiang, N., Li, Z., Wang, Q., Yang, C., Wang, H. H., & Pan, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129532 · Cited by 39
“Each sample in CMU-MOSEI is labeled with sentiment intensity within the range of [-3, 3], alongside classification into 6 basic emotions.”
Multimodal destination image and user engagement: a sequential research design
Tan, J., Cheng, M., Chen, J., Zhu, J., Yu, Q., & Chen, S. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105209 · Cited by 34
Flexible Sensor‐Based Human–Machine Interfaces with AI Integration for Medical Robotics
Wang, Y., Jiang, Z., Kwon, S. H., Ibrahim, M., Dang, A., & Dong, L. (2025) · Advanced Robotics Research · doi:10.1002/adrr.202500027 · Cited by 41
Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis
Verma, S., Wang, J., Ge, Z., Shen, R., Jin, F., Wang, Y., Chen, F., & Liu, W. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00065 · Cited by 29
“view of the underlying phenomenon and thus enhances the generalization performance for sentiment prediction [1], [2].”
BackgroundA New View of Multisensor Data Fusion: Research on Generalized Fusion
Guo, C., Liu, Z., Yu, G., & Liang, J. (2021) · Mathematical Problems in Engineering · doi:10.1155/2021/5471242 · Cited by 39
Imaging atomic-scale chemistry from fused multi-modal electron microscopy
Schwartz, J., Di, Z. W., Jiang, Y., Fielitz, A. J., Ha, D., Perera, S. D., Baggari, I. E., Robinson, R. D., Fessler, J. A., Ophus, C., et al. (2022) · npj Computational Materials · doi:10.1038/s41524-021-00692-5 · Cited by 29
Real-time multimodal interaction in virtual reality - a case study with a large virtual interface
Cao, L., Zhang, H., Peng, C. Y., & Hansberger, J. T. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14381-6 · Cited by 41
“Tensor decomposition models [16, 21, 22] have also been used for multimodal fusion, in which data is factorized to the product of matrices capturing shared features across modalities.”
MethodSørensen, M., Eeghem, F. V., & Lathauwer, L. D. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2706183 · Cited by 33
Collaborative Multi-Robot Systems for Search and Rescue: Coordination and Perception
Queralta, J. P., Taipalmaa, J., Pullinen, B. C., Sarker, V. K., Gia, T. N., Tenhunen, H., Gabbouj, M., Raitoharju, J., & Westerlund, T. (2020) · arXiv · doi:10.48550/arxiv.2008.12610 · Cited by 27
“…parallel data integration with high-level decision making, (b) sequential processing of modalities when different modalities have difference conficende or quality levels, (c) true fusion with high-level features or with multivariate features, and (d) true fusion with minimal reduction [207], [208].”
BackgroundHamed, S. K., Aziz, M. J. A., & Yaakub, M. R. (2024) · IEEE Access · doi:10.1109/access.2024.3443092 · Cited by 35
“While late fusion enhances performance by handling errors from multiple models independently, improvements are realized only when models complement each other [34], [35], [36].”
BackgroundA Flexible Optimization Framework for Regularized Matrix-Tensor Factorizations With Linear Couplings
Schenker, C., Cohen, J. E., & Acar, E. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3045848 · Cited by 27
“Nevertheless, data fusion poses many challenges, and there is an emerging need for data fusion methods that can take into account different characteristics of data from multiple sources in many disciplines [1]–[4].”
BackgroundA feature level image fusion for IR and visible image using mNMRA based segmentation
Singh, S., Mittal, N., & Singh, H. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06900-7 · Cited by 26
“This fused image provides information which cannot be obtained by independently examining several pictures [7–10].”
BackgroundWang, K., Song, Y., Huang, Z., Sun, Y., Xu, J., & Zhang, S. (2022) · Measurement · doi:10.1016/j.measurement.2022.111215 · Cited by 29
Yang, W., Zhou, Y., Cao, Y., Zhang, H., Zhang, Q., & Wang, H. (2017) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2017.2672521 · Cited by 26
“Bayesian inference is a typical and important fusion method [14], [15].”
MethodTop-k multi-class SVM using multiple features
Yan, C., Luo, M., Liu, H., Li, Z., & Zheng, Q. (2017) · Information Sciences · doi:10.1016/j.ins.2017.08.004 · Cited by 30
An Improved Hector SLAM Algorithm based on Information Fusion for Mobile Robot
Yu, N., & Zhang, B. (2018) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis.2018.8691198 · Cited by 26
“2 Information fusion based on extended Kalman Filtering Information fusion can be defined as use of computer technology to analyze the observational information of several sensors under certain criteria, optimize the information process which has completed the required decision making and estimation tasks [14].”
MethodTopic-guided abstractive multimodal summarization with multimodal output
Rafi, S. M., & Das, R. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08821-5 · Cited by 30
“The necessity of a multi-modal summarization has gained importance from previous research by fusing [40] two or more modality information into a piece of multimodal information.”
The first-kind flexible tensor SVD: innovations in multi-sensor data fusion processing
Huang, J., Zhang, F., Coombs, T. A., & Chu, F. (2024) · Nonlinear Dynamics · doi:10.1007/s11071-024-10489-y · Cited by 25
“This is a data fusion methodology [17].”
Supervised independent vector analysis through pilot dependent components
Nesta, F., & Koldovský, Z. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952213 · Cited by 25
Embedded Emotion Recognition within Cyber-Physical Systems using Physiological Signals
Miranda, J., Marino, R., Lanza-Gutiérrez, J. M., Riesgo, T., García-Valderas, M., & López-Ongil, C. (2018) · Conference on Design of Circuits and Integrated Systems · doi:10.1109/dcis.2018.8681496 · Cited by 29
“Reviewing works within the emotion recognition field using physiological signals, we check that many systems include sensing devices, such as ElectroEncephaloGrams (EEGs), which are difficult to use in real-world applications due to the prize, size, or conspicuous appearance [2], [4], [5].”
BackgroundIn search of multimodal brain alterations in Alzheimer's and Binswanger's disease
Fu, Z., Iraji, A., Caprihan, A., Adair, J. C., Sui, J., Rosenberg, G. A., & Calhoun, V. D. (2019) · NeuroImage Clinical · doi:10.1016/j.nicl.2019.101937 · Cited by 27
Thakuriah, P. V., Siła-Nowicka, K., Hong, J., Boididou, C., Osborne, M., Lido, C. M., & McHugh, A. (2019) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2019.101427 · Cited by 27
Ahn, C. R., Ham, Y., Kim, J., & Kim, J. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.206 · Cited by 29
“The ranked score does not have a universal unit as every entity is given a unitless number, and thus that is only effective to compare with others relatively, not deterministically quantify the value [44].”
BackgroundWang, Y., Yang, Y., Sun, H., Dai, J., Zhao, M.-X., Teng, C., Ke, Z.-H., Yang, M., Zhong, L., & Zhu, W. (2020) · Vibrational Spectroscopy · doi:10.1016/j.vibspec.2020.103057 · Cited by 25
Liu, K., & El-Gohary, N. (2020) · Journal of Computing in Civil Engineering · doi:10.1061/(asce)cp.1943-5487.0000921 · Cited by 28
Deep multimodal learning for municipal solid waste sorting
Lu, G., Wang, Y., Xu, H., Yang, H., & Zou, J. T. (2021) · Science China Technological Sciences · doi:10.1007/s11431-021-1927-9 · Cited by 27
Zhan, P., Man, K., Wind, S. A., & Malone, J. (2022) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986221111085 · Cited by 25
“Multimodal data refer to complementary and diverse data obtained by multiple instruments, measurement devices, or acquisition techniques for the same description object (Lahat et al., 2015).”
BackgroundEfficient Fusion and Reconstruction for Communication and Sensing Signals in Green IoT Networks
Jing, Z., Mu, J., Li, X., Zhou, Q., & Tian, Q. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3222513 · Cited by 26
“However, the difference is that decision-level fusion may cause more information loss by fusing raw data into several symbols, which is not conducive to subsequent data recovery and reconstruction [9].”
BackgroundLiang, Y., Chen, G., Li, S., & Xu, Z. (2023) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-5460 · Cited by 26
Application of Data Mining to “Big Data” Acquired in Audiology: Principles and Potential
Mellor, J. C., Stone, M. A., & Keane, J. A. (2018) · Trends in Hearing · doi:10.1177/2331216518776817 · Cited by 27
“The grouping of dimensions/features available from hearing aid data logging can be referred to as a modality (refer to the Glossary for a more rigid definition; Lahat, Adali, & Jutten, 2015).”
BackgroundNamed Data Networking’s Intrinsic Cyber-Resilience for Vehicular CPS
Bouk, S. H., Ahmed, S. H., Hussain, R., & Eun, Y. (2018) · IEEE Access · doi:10.1109/access.2018.2875890 · Cited by 24
“Let, an auto emission monitoring application running on a consumer requests the current CO2 information from all the vehicles within a particular area or road segment, then all the vehicles with produce the emission information (multimodal emission data) [130].”
BackgroundMulti-Modal Neuroimaging Analysis and Visualization Tool (MMVT)Preprint
Felsenstein, O., Peled, N., Hahn, E., Rockhill, A. P., Frank, D., Libster, A. M., Nossenson, Y., Folsom, L., Gholipour, T., Macadams, K., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10079 · Cited by 34
“While multimodal approaches offer the possibility of minimizing the limitations of each modality, there are several challenges that arise from both data collection and data analysis (Lahat et al., 2015).”
Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models
Ye, J., Hai, J., Song, J., & Wang, Z. (2023) · doi:10.1101/2023.08.24.23294597 · Cited by 24
“In multimodal analysis, there are two common modality fusion strategies: early fusion and late fusion.[14, 15] Early fusion, a data-level fusion method, amalgamates data from diverse modalities prior to model input.”
MethodSaggu, S., Daneshvar, H., Samavi, R., Pires, P., Sassi, R. B., Doyle, T. E., Zhao, J., Mauluddin, A., & Duncan, L. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02450-1 · Cited by 30
“Data fusion comes with its own challenges [39] but being able to identify children/youth more likely to visit the ED has clinical utility.”
BackgroundUser-generated photos in hotel demand forecasting
Xu, J. B., Zhang, W., Li, H., Xiang, Z., & Zhang, J. (2024) · Annals of Tourism Research · doi:10.1016/j.annals.2024.103820 · Cited by 30
Multimodality in tourism and hospitality: a critical and narrative review
Yu, X., & Cheng, M. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105245 · Cited by 24
Correlated coupled matrix tensor factorization method for simultaneous EEG-fMRI data fusion
Mosayebi, R., & Hossein‐Zadeh, G. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.102071 · Cited by 26
Coupled tensor decompositions for data fusion
Chatzichristos, C., Eyndhoven, S. V., Kofidis, E., & Huffel, S. V. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-824447-0.00016-9 · Cited by 30
Chemometric and sensometric techniques in enological data analysis
Mafata, M., Brand, J., Medvedovici, A. V., & Buica, A. (2022) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2022.2089624 · Cited by 23
“Data fusion is classified under low, medium, and high level according to increasing levels of complexity (Lahat, Adali, and Jutten 2015; Cocchi 2019a; Borràs et al. 2015), taking both statistical (Cocchi 2019a) and strategic approach (Lahat, Adali, and Jutten 2015).”
BackgroundMethodInfluentialComprehensive Review of Multimodal Medical Data Analysis: Open Issues and Future Research Directions
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2022) · Acta Informatica Pragensia · doi:10.18267/j.aip.202 · Cited by 27
Guelib, B., Zarour, K., Hermessi, H., Bounab, R., & Khlifa, N. (2023) · IEEE Access · doi:10.1109/access.2023.3276722 · Cited by 27
“Furthermore, a complex phenomenon like disease, characterized by its rich nature, is rarely understood by a single modality alone [3].”
A depression detection model based on multimodal graph neural network
Xia, Y., Liu, L., Dong, T., Chen, J., Cheng, Y., & Tang, L. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18079-7 · Cited by 25
Orthogonally constrained independent component extraction: Blind MPDR beamforming
Koldovský, Z., Tichavský, P., & Kautský, V. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081389 · Cited by 24
“ICA, MICA and BSE are bases of further extensions as is Independent Vector Analysis (IVA) [8] or Joint BSS (JBSS) [9]; see also [10] for further extensions and perspectives.”
MethodMultidataset Independent Subspace Analysis With Application to Multimodal Fusion
Silva, R. F., Plis, S., Adalı, T., Pattichis, M. S., & Calhoun, V. D. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3028452 · Cited by 24
“In multimodal fusion of heterogeneous data [25], [26], robust identification of flexible joint features (yk) originating from all data modalities (xm) can yield one-of-a-kind views into a system’s properties.”
BackgroundStock price crash prediction based on multimodal data machine learning models
Sheng, Y., Qu, Y., & Ma, D. (2024) · Finance research letters · doi:10.1016/j.frl.2024.105195 · Cited by 32
DaFF: Dual Attentive Feature Fusion for Multispectral Pedestrian Detection
Althoupety, A. S., Wang, L., Feng, W., & Rekabdar, B. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00305 · Cited by 41
Comprehensive Analysis on Machine Learning Approaches for Interpretable and Stable Soft Sensors
Cao, L., Wang, J., Su, J., Luo, Y., Cao, Y., Braatz, R. D., & Gopaluni, R. B. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3556830 · Cited by 63
“Research into multimodal data fusion [102], deep representation learning [103], and transfer learning [77] can facilitate the development of models capable of handling complex data types and extracting meaningful features across different modalities.”
A review of multi-modal deep learning towards agentic smart manufacturing
Leng, J., Wang, J., Zhou, L., Zhao, R., Chen, C., Zhang, D., Zheng, S., Liu, Q., & Shen, W. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.104117 · Cited by 31
Fusion of EEG and fMRI via Soft Coupled Tensor Decompositions
Chatzichristos, C., Davies, M. E., Escudero, J., Kofidis, E., & Theodoridis, S. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553077 · Cited by 23
“Different types of fusion can be realized [1, 7, 8].”
BackgroundMethodInfluentialSequence neural network for recommendation with multi-feature fusion
Gu, X. Q., Hai-ping, Z., & Jian, L. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118459 · Cited by 25
Hou, B., Zhou, Y., Li, S., Wei, Y., & Song, J. (2024) · Earth Planets and Space · doi:10.1186/s40623-024-02005-8 · Cited by 26
“…learning (MMML) could construct models that can process and relate information from multiple heterogeneous modalities (Baltrušaitis et al. 2018; Lahat et al. 2015), having been paying attention to a variant of multi-disciplinary fields, e.g., computer vision (Luo et al. 2022; Radford et al.…”
BackgroundA Deep Neuro-Fuzzy Method for ECG Big Data Analysis via Exploring Multimodal Feature Fusion
Lyu, X., Rani, S., Manimurugan, S., & Feng, Y. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3416217 · Cited by 29
An event based multi-sensor fusion algorithm with deadzone like measurements
Wang, G., Li, N., & Zhang, Y. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.10.004 · Cited by 23
Thersleff, T., Budnyk, S., Drangai, L., & Slabon, A. (2020) · Ultramicroscopy · doi:10.1016/j.ultramic.2020.113116 · Cited by 24
Research on multimodal based learning evaluation method in smart classroom
Qianyi, Z., & Zhiqiang, L. (2023) · Learning and Motivation · doi:10.1016/j.lmot.2023.101943 · Cited by 23
Enhancing Online Learning: A Multimodal Approach for Cognitive Load Assessment
Xue, Y., Wang, K., & Qiu, Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2327198 · Cited by 31
“(Lahat et al., 2015) There are three common strategies for multimodal data fusion: data-level fusion, feature-level fusion, and decision-level fusion.”
Multi-modal lifelog data fusion for improved human activity recognition: A hybrid approach
Oh, Y., & Kim, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102464 · Cited by 25
Multimodal data fusion for geo-hazard prediction in underground mining operation
Liang, R., Zhang, C., Huang, C., Li, B., Saydam, S., Canbulat, I., & Munsamy, L. (2024) · Computers & Industrial Engineering · doi:10.1016/j.cie.2024.110268 · Cited by 25
An Integrated BIM-IoT Framework for Real-Time Quality Monitoring in Construction Site
Wang, J., Luo, H., Pena-Mora, F. A., Zhou, W., & Fang, W. (2024) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-14984 · Cited by 27
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.