Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 501–600 of 1,001
Integrating Dynamical Systems Modeling with Spatiotemporal scRNA-Seq Data Analysis
Zhang, Z., Sun, Y., Peng, Q., Li, T., & Zhou, P. (2025) · Entropy · doi:10.3390/e27050453 · Cited by 26
Dia, N., Fontecave-Jallon, J., Guméry, P.-Y., & Rivet, B. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553294 · Cited by 9
“It is now common to record physiological signals with different types of sensors leading to multimodal recordings [9][10][11].”
BackgroundMultimodal Data Fusion based on the Global Workspace Theory
Bao, C., Fountas, Z., Olugbade, T., & Bianchi‐Berthouze, N. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418849 · Cited by 11
Dimitrievski, A., Zdravevski, E., Lameski, P., Villasana, M. V., Pires, I. M., García, N. M., Flórez‐Revuelta, F., & Trajkovik, V. (2021) · Sensors · doi:10.3390/s21093030 · Cited by 9
“These diverse data streams need to be fused, which poses another set of challenges, as eloquently described in [42].”
BackgroundMultimodal Representations for Synchronized Speech and Real-Time MRI Video Processing
Köse, Ö. D., & Saraçlar, M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3084099 · Cited by 9
“In the case of heterogeneous data fusion, one goal is to handle the process such that the drawbacks coming from one modality are inhibited by the other [3].”
BackgroundPATCH: A Plug-in Framework of Non-blocking Inference for Distributed Multimodal System
Wang, J., Wang, G., Zhang, X., Liu, L., Zeng, H., Xiao, L., Cao, Z., Gu, L., & Li, T. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610885 · Cited by 18
“Figure 2 shows the three main strategies (early fusion, model-level fusion, and late fusion) in existing multimodal systems [23, 43, 47, 88].”
BackgroundPerformance bounds for coupled models
Ren, C., Farias, R. C., Amblard, P., & Comon, P. (2016) · IEEE Sensor Array and Multichannel Signal Processing Workshop (SAM) · doi:10.1109/sam.2016.7569651 · Cited by 9
“It generally consists in extracting useful information from a collection of datasets acquired by multiple measurement devices [4].”
BackgroundAthreya, A. P., Banerjee, S. S., Neavin, D. R., Kaddurah‐Daouk, R., Rush, A. J., Frye, M. A., Wang, L., Weinshilboum, R. M., Bobo, W. V., & Iyer, R. K. (2017) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb.2017.8058559 · Cited by 9
“Several other researchers have proposed the combination other modalities of biological data, but not together with clinical data that includes a significant amount of a patient responses to symptom questionnaire [12]–[14].”
Collecting public RGB-D datasets for human daily activity recognition
Wu, H., Ma, X., Zhang, Z., Wang, H., & Li, Y. (2017) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881417709079 · Cited by 9
Wei, Z., Zhao, H., Zhao, L., & Yan, H. (2019) · Information Sciences · doi:10.1016/j.ins.2019.06.044 · Cited by 9
Person Recognition with HGR Maximal Correlation on Multimodal Data
Liang, Y., Ma, F., Li, Y., & Huang, S. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413164 · Cited by 9
“Therefore, multimodal person recognition is believed to have a more comprehensive view of real world [12], similar to how humans recognize others.”
BackgroundMethodInfluentialAlgorithmic Behavioral Science: Machine Learning as a Tool for Scientific DiscoveryPreprint
Ludwig, J., & Mullainathan, S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4164272 · Cited by 17
Lim, M., Heo, K.-S., Kim, J.-M., Kang, B., Lin, W., Zhang, H., Shen, D. G., & Kam, T. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3355966 · Cited by 8
“Nevertheless, many recent studies have focused on feature-level fusion strategies [32], [33] to ensure that the information from each modality can be more actively exchanged in earlier stages.”
BackgroundPersonalized Coupled Tensor Decomposition for Multimodal Data Fusion: Uniqueness and Algorithms
Borsoi, R. A., Usevich, K., Brie, D., & Adalı, T. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3510680 · Cited by 8
Lee, M. J., & Zhang, R. (2024) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6249 · Cited by 13
SIN-Seg: A joint spatial-spectral information fusion model for medical image segmentation
Dai, S., Ye, K., Zhan, C., Tang, H., & Zhan, L. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.02.024 · Cited by 9
“Multimodal learning fields often meet such problems [43,44].”
Liu, K., Zhang, Y., & Lu, S. (2025) · Electronics · doi:10.3390/electronics14051052 · Cited by 8
A self-supervised multimodal framework for 1D physiological data fusion in remote health monitoring
Cañellas, M. L., Casado, C. Á., Nguyen, L. N., & López, M. B. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103397 · Cited by 13
Enhancing emotion recognition through multi-modal data fusion and graph neural networks
Devarajan, K., Ponnan, S., & Perumal, S. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100291 · Cited by 11
Large language models for battery prognostics
Zhao, J., Wang, Y., Cai, M., Xie, L., & Wang, H. (2025) · Journal of Energy Chemistry · doi:10.1016/j.jechem.2025.11.021 · Cited by 9
Lahat, D., & Christian Jutten, B. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472501 · Cited by 8
“For data fusion, there is particular interest in such results, as a large number of questions regarding the combination of various modalities and data sets is still largely open; see, e.g., [13, Section V.”
BackgroundSelf-citationAutomated Alignment of Multi-Modal Plant Images Using Integrative Phase Correlation Approach
Michael, H., Junker, A., Neumann, K., Altmann, T., & Gladilin, E. (2018) · Frontiers in Plant Science · doi:10.3389/fpls.2018.01519 · Cited by 8
“A broad spectrum of methods for image registration has been previously developed in context of biomedical and geographic imaging (Zitova and Flusser, 2003; Xiong and Zhang, 2010; Lahat et al., 2015; Brock et al., 2017; Goshtasby, 2017).”
MethodJoint-cross-loading multimodal cognitive diagnostic modeling incorporating visual fixation counts
Peida, Z. (2022) · Acta Psychologica Sinica · doi:10.3724/sp.j.1041.2022.01416 · Cited by 8
Reinforcement learning architecture for cyber–physical–social AI: state-of-the-art and perspectives
Li, X., Wang, P., Jin, X., Jiang, Q., Zhou, W., & Yao, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10450-2 · Cited by 13
“…social systems are introduced in CPSS, the data sources in CPSS have posed new features, such as large scale, multi source, heterogeneous network, cross domains, cross media, cross language, dynamic evolution and universality (Zeng et al. 2016; Wang 2010; Wang et al. 2021; Lahat et al. 2015) .”
BackgroundLópez, H. M. C., Cabada, R. Z., Estrada, M. L. B., & Mitre-Hernández, H. A. (2023) · Soft Computing · doi:10.1007/s00500-023-08076-1 · Cited by 9
“The suitable representation of the data is significant for the correct training of a multimodal classification system (Lahat et al. 2015).”
MethodChen, Z.-Y., Fan, Z., & Sun, M. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3267858 · Cited by 10
“Moreover, deep learning methods have been widely used in learning from heterogeneous data, e.g., text, video, and audio, or from multiple different sources of homogeneous data, e.g., multiple images via different imaging tools [35], [36].”
MethodAutomated Scoring of Asynchronous Interview Videos Based on Multi-Modal Window-Consistency Fusion
Lv, J., Chen, C., & Liang, Z. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3294335 · Cited by 17
“Multi-modal fusion is a hot research topic in this field [31] [32] [33], which focuses on exploring the complementary relationship between various modalities to improve the performance of systems [34].”
BackgroundPakravan, M., & Shamsollahi, M. B. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2840085 · Cited by 9
HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities
Chen, J., & Zhang, A. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482262 · Cited by 17
“However, the problem is that many tasks may not have a complete set of all available modalities due to data corruptness or data missingness [Lahat et al., 2015].”
BackgroundInfluentialRuan, H., Lei, T., Chen, K., & Lu, J. Z. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3176534 · Cited by 8
Sensor Fusion and Multimodal Learning for Robotic Grasp Verification Using Neural Networks
Gohil, P., Thoduka, S., & Plöger, P. G. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9955646 · Cited by 9
“Combining multiple modalities at different levels such as data level, feature level or decision level is called data or sensor fusion [7].”
BackgroundGAFNet: A Global Fourier Self Attention Based Novel Network for multi-modal downstream tasks
Susladkar, O., Deshmukh, G., Makwana, D., Mittal, S., Teja, R. S. C., & Singhal, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00521 · Cited by 8
“Further, in AI-based decision-making systems, multiple modalities can improve predictive performance [23, 18, 5].”
BackgroundGong, X., Lin, Q., Debals, O., Vervliet, N., & Lathauwer, L. D. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472138 · Cited by 8
“Third-order tensors with mode-1 and mode-2 rank equal to L 1 and L 2 , without rank constraint in the third mode, are said to have multilinear rank-( L 1 , L 2 , · ).”
BackgroundZhu, X., Yao, J., Xiao, G., Xie, Y., Rodriguez-Canales, J., Parra, E. R., Behrens, C., Wistuba, I. I., & Huang, J. (2016) · 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822559 · Cited by 8
“For computer-assisted diagnosis with multi-modal data, one of the open challenges is how to integrate multi-modal data for clinical outcomes prediction [1], [2].”
BackgroundMethodInfluentialThe similarity-consensus regularized multi-view learning for dimension reduction
Meng, X., Wang, H., & Feng, L. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105835 · Cited by 7
Chen, Z., Zhao, L., Li, Q., Song, X., & Zhang, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-87049-2_3 · Cited by 8
Robust CP Tensor Factorization With Skew Noise
Huang, X., Xu, S., Zhang, C., & Zhang, J. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.2991581 · Cited by 8
“Essentially, many problems can be cast into low-rank models including multimodal data fusion [2], image denoising [3], representation learning [4] and etc. Recently, low-rank tensor factorization (LRTF) has emerged as a generalization of low-rank matrix factorization (LRMF) [1] because lots of data…”
BackgroundBethel, E. W., & Greenwald, M. (2016) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1525145 · Cited by 13
Multimodal U-Net: A Novel Approach for 2-D Inversion of Magnetotelluric Data
Li, X., Dong, Y., He, J., Yang, X., Zhou, W., & Hu, X. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3547157 · Cited by 10
“Conventional multimodal fusion techniques often rely on matrix and tensor decompositions [42], which can be categorized into feature-level fusion [early fusion, Fig.”
TGFN-SD: A text-guided multimodal fusion network for swine disease diagnosis
Yang, G., Li, Q., Zhao, C., Wang, C., Yan, H., Meng, R., Liu, Y., & Yu, L. (2025) · Artificial Intelligence in Agriculture · doi:10.1016/j.aiia.2025.03.002 · Cited by 16
Remote Sensing for Monitoring Macroplastics in Rivers: A Review
Marye, A. T., Caramiello, C., Nardi, D. D., Miglino, D., Proietti, G., Saddi, K. C., Biscarini, C., Manfreda, S., Poggi, M., & Tauro, F. (2025) · Wiley Interdisciplinary Reviews Water · doi:10.1002/wat2.70020 · Cited by 13
“Such factors can complicate the integration process, potentially affecting the quality and reliability of the fused data (Lahat et al. 2015; Ghamisi et al. 2019; Li et al. 2022).”
Adaptive Fusion of LiDAR Features for 3D Object Detection in Autonomous Driving
Wang, M., Li, D., Casas, J. R., & Ruiz‐Hidalgo, J. (2025) · Sensors · doi:10.3390/s25133865 · Cited by 11
“Current cooperative perception fusion methods among vehicles are mainly categorized into three types: early fusion, late fusion, and intermediate fusion [10].”
MOTIF: A text-aware and disentangled Mamba-based architecture for multimodal sentiment analysis
Hu, B., & Li, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114651 · Cited by 9
Xu, P. J., Kan, S. X., Jin, J., Zhang, Z. J., Gu, Y. X., Zhang, B., & Zhou, Y. L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131817 · Cited by 9
Kim, H. G., Jun, B.-M., Jeong, H., Yoon, Y., & Cho, K. H. (2025) · Journal of Water Process Engineering · doi:10.1016/j.jwpe.2025.109105 · Cited by 8
A comparative evaluation of handling missing data points and modalities in electronic health records
Bhandari, A., & Tyagi, S. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106302 · Cited by 10
Khachatrian, E., Chlaily, S., Eltoft, T., & Marinoni, A. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3124308 · Cited by 7
An Entity-Matching System Based on Multimodal Data for Two Major E-Commerce Stores in Mexico
Estrada-Valenciano, R., Muñiz-Sánchez, V., & De-la-Torre-Gutiérrez, H. (2022) · Mathematics · doi:10.3390/math10152564 · Cited by 8
Robust Multimodal Depth Estimation using Transformer based Generative Adversarial Networks
Khan, M. F. F., Devulapally, A., Advani, S., & Narayanan, V. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548418 · Cited by 10
“When one of the sensor modalities undergoes some technical, physical or environmental hindrance, the whole model performance becomes vulnerable [10].”
BackgroundGuo, A., Chen, Z., Ma, Y., Lv, Y., Yan, H., Li, F., Xing, Y., Luo, Q., & Zheng, H. (2024) · Artificial Intelligence Chemistry · doi:10.1016/j.aichem.2024.100058 · Cited by 9
Multimodal data fusion techniques in smart healthcare
Karbout, K., Ghazouani, M. E., Lachgar, M., & Hrimech, H. (2024) · International Conference on Global Aeronautical Engineering and Satellite Technology (GAST) · doi:10.1109/gast60528.2024.10520803 · Cited by 9
“Three techniques are employed for the fusion of multimodal data [12] [13].”
MethodOn Integrating Human Decisions with Physical Sensors for Binary Decision Making
Wimalajeewa, T., Varshney, P. K., & Rangaswamy, M. (2018) · International Conference on Information Fusion (FUSION) · doi:10.23919/icif.2018.8455639 · Cited by 7
“INTRODUCTION The problem of data/decision fusion with physics-based sensor data (hard data) has been a research topic for decades in the signal processing community [1]–[3].”
BackgroundA method to compare the discriminatory power of data-driven methods: Application to ICA and IVA
Levin‐Schwartz, Y., Calhoun, V. D., & Adalı, T. (2018) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2018.10.008 · Cited by 8
“Additionally, each method exploits different properties of the signal (Lahat, Adali, & Jutten, 2015) and such comparisons are inherently subjective, since they rely on visual interpretation.”
BackgroundMethodDutta, A., Breloff, S. P., Dai, F., Sinsel, E. W., Carey, R. E., Warren, C. M., & Wu, J. Z. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103322 · Cited by 9
“To integrate multidimensional data, it is essential to learn the relationships between those dimensions to understand the multidimensional nature of a phenomenon [12].”
BackgroundMuliukov, A. R., Rodriguez, L., Miramond, B., Khacef, L., Schmidt, J., Berthet, Q., & Upegui, A. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.825879 · Cited by 8
“Due to the rich characteristics of natural phenomena, it is rare that a single modality provides a complete representation of the phenomenon of interest (Lahat et al., 2015).”
Attack-resilient Fusion of Sensor Data with Uncertain Delays
Chen, Y., Zhang, T., Kong, F., Zhang, L., & Deng, Q. (2022) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3532181 · Cited by 7
“Thus, the technique can considerably increase the system’s robustness to disturbances by reducing its dependence on a specific sensor [26, 29].”
BackgroundBelyaeva, I., Gabrielson, B., Wang, Y., Wilson, T. W., Calhoun, V. D., Stephen, J. M., & Adalı, T. (2022) · Neuroinformatics · doi:10.1007/s12021-022-09599-y · Cited by 6
“The multi-task joint learning enables the use of the complementary information (Lahat et al., 2015; Acar et al., 2013) from multiple stimuli and thus could result in the latent components with a higher discriminative power.”
BackgroundFusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction in Tuberculosis
D’Souza, N. S., Wang, H., Giovannini, A. A., Foncubierta–Rodríguez, A., Beck, K. L., Boyko, O. B., & Syeda-Mahmood, T. F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16449-1_28 · Cited by 8
“Existing approaches often infer matrix or tensor encodings from individual modalities [13] combined with early, late, or intermediate fusion [24,2,26] of the individual representations.”
BackgroundUrban classification by multi-feature fusion of hyperspectral image and LiDAR data
Cao, Q., Ma, A., Zhong, Y., Zhao, J. J., Zhao, B., & Zhang, L. (2019) · National Remote Sensing Bulletin · doi:10.11834/jrs.20197512 · Cited by 8
Visual–tactile fusion object classification method based on adaptive feature weighting
Zhang, P., Bai, L., Shan, D., Wang, X., Li, S., Zou, W., & Chen, Z. (2023) · International Journal of Advanced Robotic Systems · doi:10.1177/17298806231191947 · Cited by 7
“Lahat et al.(14) found that each modality in multimodal learning contributed to the overall process and that these gains could not be deduced or calculated from a single modality.”
BackgroundA Review of Biosensors and Artificial Intelligence in Healthcare and their Clinical Significance
Tariq, M. (2024) · Psychology & Psychological Research International Journal · doi:10.23880/pprij-16000392 · Cited by 9
Research on Online Review Information Classification Based on Multimodal Deep Learning
Liu, J., Sun, Y., Zhang, Y., & Lu, C. (2024) · Applied Sciences · doi:10.3390/app14093801 · Cited by 8
Impact of noise correlation on multimodality
Chlaily, S., Amblard, P., Michel, O., & Christian Jutten, B. (2016) · European Signal Processing Conference (EUSIPCO) · doi:10.1109/eusipco.2016.7760237 · Cited by 7
“Noise is one of the challenges rarely addressed since most studies ignore it or consider independent noises [1].”
BackgroundMutual acoustic identification in the swarm of e-puck robots
Nemec, D., Janota, A., Hruboš, M., Gregor, M., & Pirník, R. (2017) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881417710794 · Cited by 7
Physiological Signals Fusion Oriented to Diagnosis - A Review
Uribe, Y. F., Álvarez-Uribe, K. C., Peluffo-Ordóñez, D. H., & Becerra, M. A. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-98998-3_1 · Cited by 8
Knowledge Transfer in a Pair of Uniformly Modelled Bayesian Filters
Jirsa, L., Pavelková, L., & Quinn, A. (2019) · International Conference on Informatics in Control, Automation and Robotics (ICINCO) · doi:10.5220/0007854104990506 · Cited by 7
“(Lahat et al., 2015) provides an overview of the main challenges in multimodal data fusion across various disciplines, while the survey in (Gravina et al., 2017) discusses clear motivations and advantages of multi-sensor data fusion.”
BackgroundDuque, A. F., Wolf, G., & Moon, K. R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30047-9_10 · Cited by 13
Detecting Anomalous Behaviour Using Heterogeneous Data
Ali, A. M., Angelov, P. P., & Gu, X. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-46562-3_17 · Cited by 10
“According to [24], there are three challenges in data fusion: a) data is produced from very complex systems such as biological, environmental, sociological and psychological systems; b) increased diversity, the number, the type, and scope of the data; c) working with heterogeneous data sets means that the respective a…”
BackgroundNeuralIO: Indoor?Outdoor Detection via Multimodal Sensor Data Fusion on Smartphones
Wang, L., Sommer, L., Zhou, Y., Huang, Y., Wang, J., Riedel, T., & Beigl, M. (2020) · Sensors and Materials · doi:10.18494/sam.2020.2586 · Cited by 7
Quantifying the advantage of multimodal data fusion for survival prediction in cancer patientsPreprint
Νικολάου, Ν., Salazar, D., RaviPrakash, H., Gonçalves, M., Mulla, R., Burlutskiy, N., Markuzon, N., & Jacob, E. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.08.574756 · Cited by 7
Similarity-based multimodal regression
Chen, A. A., Weinstein, S. M., Adebimpe, A., Gur, R. C., Gur, R. E., Merikangas, K. R., Satterthwaite, T. D., Shinohara, R. T., & Shou, H. (2024) · Biostatistics · doi:10.1093/biostatistics/kxad033 · Cited by 6
Deciphering the myths of engagement with destination short video: the perspective of modality
Guo, S., Wang, D., & Deng, N. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105286 · Cited by 9
Nichols, E., Keller, K. P., Chang, H. H., Chiang, Y., Gross, A. L., Hayes‐Larson, E., Henn, B. C., Kezios, K., Meijer, E., Shih, R. A., et al. (2025) · Alzheimer s & Dementia · doi:10.1002/alz.70928 · Cited by 6
“17,18 Combining larger quantities of high-dimensional data for measuring exposures can potentially improve measurement but also raises challenges in efforts to reduce dimensionality while retaining important signals in all available data.”
Prévost, C., Usevich, K., Haardt, M., Comon, P., & Brie, D. (2022) · Signal Processing · doi:10.1016/j.sigpro.2022.108573 · Cited by 6
Baciu, M., O'Sullivan, L., Torlay, L., & Banjac, S. (2023) · Revue Neurologique · doi:10.1016/j.neurol.2023.02.067 · Cited by 9
Capturing the Concept Projection in Metaphorical Memes for Downstream Learning Tasks
Acharya, S., Das, B., & Sudarshan, T. S. B. (2023) · IEEE Access · doi:10.1109/access.2023.3347988 · Cited by 8
“…learning can be categorized as follows : Fusion: Several strategies have been proposed that rely on variants of concatenation methods such as [22], [23], [24], and [25], early fusion, late fusion, hybrid fusion, tensor fusion, hierarchical fusion etc. depending on how the different…”
BackgroundDeep learning for multisource medical information processing
Gezimati, M., & Singh, G. K. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-313233-9.00009-6 · Cited by 9
Luo, Q., Huang, N., Bartles, D. L., Simpson, T. W., & Beese, A. M. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-025-02576-7 · Cited by 7
“Key challenges in using multi-source frameworks include identifying the optimal feature representative for each single-source input modality and restoring the characteristics from each modality representative after data fusion (Gao et al., 2020; Lahat et al., 2015; Y. Li et al., 2019).”
Dong, J., Hao, M., Ding, F., Chen, S., Wu, J., Zhuo, J., & Jiang, D. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9030063 · Cited by 11
Bai, J., Wang, S., Xu, Q., Lai, K., Xu, S., Zhang, J., Ju, Y., & He, Z. (2025) · Bulletin of Engineering Geology and the Environment · doi:10.1007/s10064-025-04152-y · Cited by 6
“(2) Complexity of Multimodal Data Fusion: Multimodal AI requires integrating data from various sensors, which differ not only in form (e.g., 1D signals, 2D images) but also in source and the physical meaning (Bai et al. 2025; Dai et al. 2023a, b; Li et al. 2022a, b; Lahat et al. 2015).”
On the Optimum Linear Soft Fusion of Classifiers
Vergara, L., & Salazar, A. (2025) · Applied Sciences · doi:10.3390/app15095038 · Cited by 8
Cao, Y., Ju, X., Chen, X., & Gong, L. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00543-3 · Cited by 12
“Another challenge is the computational overhead introduced by incorporating multiple modalities (Lahat et al. 2015), which can make model training and optimization more resource-intensive.”
Flexible large-scale fMRI analysis: A survey
Kim, S.-J., Calhoun, V. D., & Adalı, T. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953372 · Cited by 7
Du, J., Han, M., Yan, H., Chen, Y., & Lin, H. (2019) · Electronics · doi:10.3390/electronics8050550 · Cited by 6
“Assuming N = 3 and A ( 3 ) = I I 3 for the third-order tensor T ∈ C I 1 × I 2 × I 3 , we have: This model is called Tucker-2 model or Tucker-(2, 3) model, and is widely applied in data analysis and parameter estimation [4].”
BackgroundMethodNeuralIO: Indoor Outdoor Detection via Multimodal Sensor Data Fusion on Smartphones
Wang, L., Sommer, L., Riedel, T., Beigl, M., Zhou, Y., & Huang, Y. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-51005-3_13 · Cited by 10
Dutta, A., Breloff, S. P., Dai, F., Sinsel, E. W., Warren, C. M., Carey, R. E., & Wu, J. Z. (2021) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0002106 · Cited by 6
Jia, C., Akhonda, M. A. B. S., Levin‐Schwartz, Y., Long, Q., Calhoun, V. D., & Adalı, T. (2021) · Applied Sciences · doi:10.3390/app11188382 · Cited by 6
Multi-Sensor Data Fusion Approach for Kinematic Quantities
D’Arco, M., & Guerritore, M. (2022) · Energies · doi:10.3390/en15082916 · Cited by 6
MMRCA: MultiModal Root Cause Analysis
White, G., Diuwe, J., Fonseca, E., & O’Brien, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14135-5_14 · Cited by 9
Anomaly Detection Using Ensemble Classification and Evidence Theory
Arévalo, F., Ibrahim, M. T., Alison, M. P. C., & Schwung, A. (2023) · IEEE Access · doi:10.1109/access.2023.3280048 · Cited by 6
“The use of heteregeneous sources improves the results in information fusion, in this specific case the use of diverse classifiers [8].”
BackgroundDespotović, M., Koch, D., Stumpe, E., Brunauer, W., & Zeppelzauer, M. (2023) · Journal of European real estate research · doi:10.1108/jerer-11-2022-0036 · Cited by 9
End-to-End Learning with Multiple Modalities for System-Optimised Renewables Nowcasting
Vohra, R., Rajaei, A., & Cremer, J. L. (2023) · IEEE PowerTech Conference · doi:10.1109/powertech55446.2023.10202904 · Cited by 7
“Three different ’fusion’ methods to combine these modalities are early, late, and intermediate fusion [8].”
BackgroundA Feature Selection Method for Multimodal Multispectral LiDAR Sensing
Han, Y., Salido-Monzú, D., Butt, J. A., Schweizer, S., & Wieser, A. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.04.022 · Cited by 6
Timmerman, M. E., Kiers, H. A. L., & Ceulemans, E. (2016) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2016.05.001 · Cited by 6
“Boosted by technological advances, multiple data sources are available easier than ever, in many different fi elds of research [2].”
BackgroundDescription Logics and Rules for Multimodal Situational Awareness in Healthcare
Μεδίτσκος, Γ., Vrochidis, S., & Kompatsiaris, I. Y. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-51811-4_58 · Cited by 6
“using video analysis and speech recognition), lifestyle and environmental sensors [15].”
Methodda Costa, J. P. C. L., Miranda, R. K., & Zanatta, M. D. R. (2017) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner.2017.8008438 · Cited by 6
“For instance, multi-way analysis is unique under very mild conditions without the need of additional constraints [14], [17], and allows exploiting the diversity of multimodal datasets [16].”
BackgroundA Jacobi Generalized Orthogonal Joint Diagonalization Algorithm for Joint Blind Source Separation
Gong, X., Mao, L., Liu, Y.-L., & Lin, Q. (2018) · IEEE Access · doi:10.1109/access.2018.2850784 · Cited by 6
“For an overview of J-BSS we refer to [28].”
MethodVelodyne LiDAR and monocular camera data fusion for depth map and 3D reconstruction
Akhtar, R., Qin, H., & Chen, G. (2019) · International Conference on Digital Image Processing · doi:10.1117/12.2539863 · Cited by 6
Comparison of feature point detectors for multimodal image registration in plant phenotyping
Michael, H., Junker, A., Neumann, K., Altmann, T., & Gladilin, E. (2019) · PLoS ONE · doi:10.1371/journal.pone.0221203 · Cited by 6
Warakagoda, N. D., Dirdal, J. A., & Faxvaag, E. (2019) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion43075.2019.9011341 · Cited by 8
“,ym, where m is the number of sensors [26].”
Background
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.