Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 601–700 of 1,001
Modeling Data Curation to Scientific Inquiry: A Case Study for Multimodal Data Integration
Esteva, M., Xu, W., Simone, N., Gupta, A. K., & Jah, M. K. (2020) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3383583.3398539 · Cited by 9
“However, traditional data curation practices do not yet address the challenges of multimodal data management and integration [1].”
BackgroundRobust Sensor Fusion Algorithms Against Voice Command Attacks in Autonomous Vehicles
Guan, J., Zheng, X., Wang, C., Zhou, Y., & Jolfaei, A. (2021) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom53373.2021.00126 · Cited by 8
Menegotto, A. B., & Cazella, S. C. (2021) · Studies in fuzziness and soft computing · doi:10.1007/978-3-030-70111-6_13 · Cited by 7
Variational Fusion of Hyperspectral Data by Non-Local Filtering
Mifdal, J., Coll, B., Froment, J., & Duran, J. (2021) · Mathematics · doi:10.3390/math9111265 · Cited by 6
“Image fusion has been an active field of research due to the growing availability of data and the need of gathering information from different imaging sources [1,2].”
BackgroundMulti-Task Learning with High-Dimensional Noisy Images
Ma, X., Kundu, S., & Initiative, A. D. N. (2022) · Journal of the American Statistical Association · doi:10.1080/01621459.2022.2140052 · Cited by 6
Scholz, P., Ulbricht, A., Joshi, Y., Gollwitzer, C., & Weidner, S. M. (2019) · International Journal of Materials Research (formerly Zeitschrift fuer Metallkunde) · doi:10.3139/146.111817 · Cited by 6
Eyndhoven, S. V., Vervliet, N., Lathauwer, L. D., & Huffel, S. V. (2019) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2019.8902954 · Cited by 6
“posed simultaneously, and the cost function is a linear combination of fit terms similar to those in (2) [1], [4], [14].”
BackgroundResultOn Cramér-Rao Lower Bounds with Random Equality Constraints
Prévost, C., Chaumette, É., Usevich, K., Brie, D., & Comon, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054031 · Cited by 6
“This problem lies in the framework of multimodal data fusion [25] between heterogeneous datasets.”
BackgroundEarly soft and flexible fusion of EEG and fMRI via tensor decompositionsPreprint
Chatzichristos, C., Kofidis, E., Lathauwer, L. D., Theodoridis, S., & Huffel, S. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07134 · Cited by 6
“These include metabolomics [9], array processing [10], sentiment analysis [11], multidimensional harmonic retrieval [12, 13], link prediction [14] and, of course, biomedical applications [1, 5, 7, 15, 2, 16] among many others.”
BackgroundMethodInfluentialLiang, C., Silva, R. F., Adalı, T., Jiang, R., Zhang, D., Qi, S., & Calhoun, V. D. (2025) · IEEE Signal Processing Magazine · doi:10.1109/msp.2025.3604570 · Cited by 6
“…and implementations of different combination strategies makes it essential to clarify the terminology commonly used in this context, particularly the terms “data fusion” and “data integration,” which are often used inter-changeably despite referring to distinct methodological paradigms [7], [8].”
A central multimodal fusion framework for outdoor scene image segmentation
Zhang, Y., Morel, O., Seulin, R., Mériaudeau, F., & Sidibé, D. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10357-y · Cited by 5
de Lucena, P. G. C., Carregosa, J. D. C., Eberlin, M. N., Wisniewski, A., & Santos, J. M. (2024) · Fuel · doi:10.1016/j.fuel.2024.133894 · Cited by 5
Selvakumaran, S., Rolland, I., Cullen, L., Davis, R., Macabuag, J., Chakra, C. A., Karageozian, N., Gilani, A., Geiß, C., Bravo-Haro, M., et al. (2025) · Progress in Disaster Science · doi:10.1016/j.pdisas.2025.100404 · Cited by 5
Gu, T., Liu, S., Pu, Q., Wang, J., Wang, B., Hu, X., Sun, P., Li, Q., Zhu, L., Liu, F., et al. (2025) · Sensors and Actuators B Chemical · doi:10.1016/j.snb.2025.138053 · Cited by 5
Liu, D., Li, D., Ding, H., Cao, Y., & Gao, K. (2025) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2025.105270 · Cited by 9
Li, H., Xing, W., Zhu, W., Li, C., Lyu, B., Liu, Z., & Heffernan, N. T. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99264-3_39 · Cited by 7
From imaging to omics: deep learning is bridging MRI and liquid biopsy in bone tumor diagnosis
Wang, F., Chen, J., Zheng, L., & Huang, X. (2026) · Journal of bone oncology · doi:10.1016/j.jbo.2026.100753 · Cited by 6
“One of the most pressing issues in multimodal modeling is the heterogeneity of the input data [222,230,231].”
A Survey of Advances in Multimodal Federated Learning with Applications
Barry, G., Konyar, E., Harvill, B., & Johnstone, C. (2024) · Springer optimization and its applications · doi:10.1007/978-3-031-53092-0_15 · Cited by 7
Multimodal Deep Learning for Cancer Survival Prediction: A Review
Zhang, G., Ma, C., Yan, C., Luo, H., Wang, J., Liang, W., & Luo, J. (2024) · Current Bioinformatics · doi:10.2174/0115748936289033240424071522 · Cited by 7
Urban Functional Zone Classification via Advanced Multi-Modal Data Fusion
Liu, T., Chen, H., Ren, J., Zhang, L., Chen, H., Hong, R., Li, C., Cui, W., Guo, W., & Wen, C. (2024) · Sustainability · doi:10.3390/su162411145 · Cited by 8
Wu, X., & He, A. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.12.006 · Cited by 9
Fatori, D., Shephard, E., Benette, D., Naspolini, N. F., Guzman, G. C., Wang, J. Y. T., Tótolo, P., Mafra, A. L., Isaias, C., Pereira-Santos, D., et al. (2025) · Development and Psychopathology · doi:10.1017/s0954579425000069 · Cited by 10
Frantzidιs, C. A. (2025) · Brain Sciences · doi:10.3390/brainsci15070667 · Cited by 6
“Another limitation is that the classification is based solely on textual input without the integration of multimodal sleep-related data (e.g., actigraphy, polysomnography summaries), which could enrich the context and improve diagnostic accuracy [46,47].”
Optimal 3D chemical imaging with multimodal electron tomography
Manassa, J., Millsaps, W., Schwartz, J., & Hovden, R. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01750-y · Cited by 5
Multi-modal constraint propagation via compatible conditional distribution reconstruction
Li, Y., & Lu, H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.09.067 · Cited by 4
An Event-B based Device Description Model in IoT with the Support of Multimodal System
Rath, C. K., Mandal, A. K., & Sarkar, A. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-16-4287-6_1 · Cited by 5
A multimodal IVA fusion approach to identify linked neuroimaging markers
Damaraju, E., Silva, R. F., Adalı, T., & Calhoun, V. D. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9631027 · Cited by 5
“Each modality provides rich, unique information about brain structure and/or function [1], [2].”
BackgroundMultimodal Machine Learning in Prognostics and Health Management of Manufacturing Systems
Jose, S., Nguyen, K. T. P., & Medjaher, K. (2023) · Springer series in reliability engineering · doi:10.1007/978-3-031-30510-8_9 · Cited by 7
Guidelines for conducting biofeedback-enhanced QoE studies in mulsemedia-enhanced virtual reality
da Silveira, A. C., Raisamo, R., Spyridonis, F., Covaci, A., Ghinea, G., & Santos, C. A. S. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3617023.3617029 · Cited by 6
“Although it is an active area of research [14, 19] how this is undertaken in the case of multiple sensory modalities and mulsemedia data including, for instance, EEG, EDA or ECG signals, as well as eye-tracking data is not well understood.”
BackgroundExplaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping
Ekim, B., & Schmitt, M. P. (2023) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss52108.2023.10283349 · Cited by 6
“While it is challenging for a single sensor modality to capture the entire rich characteristic of the surface [1], multimodal EO has long been found to be beneficial in many cases [2]: in multimodal settings, the individual modalities can interact The authors acknowledge the financial support from…”
A Multi-modal Approach to Lung Tumor Detection using Deep Learning
Zafar, A., Muneeb, S., Amir, M., Jamıl, A., & Hameed, A. A. (2023) · International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · doi:10.1109/aibthings58340.2023.10291022 · Cited by 5
“The main objective of fusing data is to efficiently amalgamate the distinctive attributes of each modality and produce a unified representation that captures the intricacies of the underlying phenomenon [12].”
BackgroundD’Souza, N. S., Wang, H., Giovannini, A. A., Foncubierta–Rodríguez, A., Beck, K. L., Boyko, O. B., & Syeda-Mahmood, T. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47679-2_11 · Cited by 8
Tire noise prediction based on transfer learning and multi-modal fusion
Liang, C., Hao, M., Shen, Y., Li, H., & Fan, J. (2024) · Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering · doi:10.1177/09544070241232606 · Cited by 5
“…networks and successively proposed many classical convolutional neural network models, such as the simple stacked convolutional structure model named VGGNet, 21 the net-in-network with multi-branch computation model named Inception Net, 22 and the residual structured network model named Resnet.”
Separate and Integrated Data Processing for the 3D Reconstruction of a Complex Architecture
Medici, M., Perda, G., Andrea, S., Farella, E. M., Settimo, S., & Fabio Remondino, A. (2024) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlviii-2-2024-249-2024 · Cited by 5
Yue, S. (2024) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-024-00591-0 · Cited by 12
“Due to the intrinsic similarity, the popularity of multimodality can be derived from the data fusion (Dalla Mura et al., 2015; Lahat et al., 2015), while multimodality is a natural fact that exists in our daily life, as Tadas et al. pointed out in their article (Baltruˇsaitis et al., 2018): ”Our…”
BackgroundLearning Trimodal Relation for Audio-Visual Question Answering with Missing Modality
Park, K. R., Lee, H. J., & Kim, J. U. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72633-0_3 · Cited by 14
Cloud-assisted hugtive robot for affective interaction
Zhou, P., Hao, Y., Yang, J., Li, W., Wang, L., Miao, Y., & Song, J. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3849-5 · Cited by 5
“As for fusion challenge for data [18], decision-level fusion is adopted, as shown in Fig.”
MethodMultimodal latent variable analysis
Papyan, V., & Talmon, R. (2017) · Signal Processing · doi:10.1016/j.sigpro.2017.07.016 · Cited by 5
“ng author. Email address: vardanp@campus.technion.ac.il (Vardan Papyan) Preprint submitted to Elsevier November 28, 2016 arXiv:1611.08472v1 [cs.CV] 25 Nov 2016 that enable its processing and analysis [1]. A prime example for the above is medical diagnosis based on collected bedside data, where one monitors a patient u…”
BackgroundInfluentialOptimal Measurement Times for Observing a Brownian Motion over a Finite Period Using a Kalman Filter
Aksenov, A., Amblard, P., Michel, O., & Christian Jutten, B. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-53547-0_48 · Cited by 5
“When a latent phenomenon is observed through different acquisition methods, more information can be acquired than from a single method, but making the most of these measurements is a challenge [5].”
BackgroundGolf video tracking based on recognition with HOG and spatial–temporal vector
Li, W., Lou, X., Mingli, D., & Lianqing, Z. (2017) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881417704544 · Cited by 5
Sensing architecture for automated/autonomous vehicles towards all-condition safety
Meng, H., & Zhang, W.-B. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995738 · Cited by 5
“Recently, the theoretical foundation of one of the possible deep fusion architectures has been established, which is called Multimodal Data Fusion [12].”
BackgroundLiao, W., Chanussot, J., & Philips, W. R. (2017) · Signals and communication technology · doi:10.1007/978-3-319-66330-2_6 · Cited by 6
An early fusion approach for multimodal emotion recognition using deep recurrent networks
Bucur, B., Somfelean, I., Ghiurutan, A., Lemnaru, C., & Dînșoreanu, M. (2018) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp.2018.8516437 · Cited by 5
“Missing values may occur in different scenarios, as specified in [8].”
BackgroundGe, W., Zhang, L., Li, X., Zhang, C., Sun, M., An, D. U., & Wu, J. (2021) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2021.08.013 · Cited by 6
Information Processing and Data Visualization in Networked Industrial Systems
Mulinka, P., Kalalas, C., Dzaferagic, M., Irene Macaluso, B., Rojas, D. G., Nardelli, P. H. J., & Marchetti, N. (2021) · IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) · doi:10.1109/pimrc50174.2021.9569603 · Cited by 5
“Such tasks rely on statistical analysis, machine learning (ML) techniques and knowledge rules [6].”
BackgroundPerception Enhancement and Improving Driving Context Recognition of an Autonomous Vehicle Using UAVs
Khezaz, A., Hina, M. D., & Ramdane-Chérif, A. (2022) · Journal of Sensor and Actuator Networks · doi:10.3390/jsan11040056 · Cited by 7
“Multimodal data fusion is defined by [41] as “the analysis of several data sets such that different data sets can interact and inform each other”, meaning that the information from different sources can be compared and cross-referenced to offer a better understanding of the situation where an…”
Trustworthiness assessment in multimodal human-robot interaction based on cognitive load
Kırtay, M., Öztop, E., Kuhlen, A. K., Asada, M., & Hafner, V. V. (2022) · IEEE RO-MAN · doi:10.1109/ro-man53752.2022.9900730 · Cited by 10
“In this work, the term modality refers to an employed sensor to record data that are associated with different aspects (e.g., audio and visual) of interactive experiments [26], [27].”
BackgroundPheSeq, a Bayesian deep learning model to enhance and interpret the gene-disease association studies
Yao, X., Ouyang, S., Lian, Y., Peng, Q., Zhou, X., Huang, F., Hu, X., Shi, F., & Xia, J. (2024) · Genome Medicine · doi:10.1186/s13073-024-01330-7 · Cited by 5
“There are three main types of data fusion strategies used in machine learning; early (data-level), intermediate (joint-level), and late (decision-level) [89, 90].”
BackgroundUss, M., Vozel, B., Лукин, В. В., & Chehdi, K. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2240895 · Cited by 5
Information–Estimation Relationship in Mismatched Gaussian Channels
Chlaily, S., Ren, C., Amblard, P., Michel, O., Comon, P., & Christian Jutten, B. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2687764 · Cited by 5
“This is particularly important when using several measurements coming from different sensors or devices, a situation referred to as multimodality [7], [8].”
BackgroundExtracting Highly Effective Features for Supervised Learning via Simultaneous Tensor Factorization
Verma, S., Liu, W., Wang, C., & Zhu, L. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11077 · Cited by 5
“These matrices can be represented as multiple labeled tensor (multidimensional arrays) sequences (Lahat, Adali, and Jutten 2015).”
BackgroundOn Multi-modal Fusion Learning in constraint propagation
Li, Y., & Lu, H. (2018) · Information Sciences · doi:10.1016/j.ins.2018.06.024 · Cited by 5
Learning Dynamic Stream Weights for Linear Dynamical Systems Using Natural Evolution Strategies
Schymura, C., & Kolossa, D. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682249 · Cited by 5
“An essential aspect of all multi-modal processing frameworks is the appropriate fusion of individual sensory modalities [13].”
BackgroundMulti-Modal Learning With Generalizable Nonlinear Dimensionality Reduction
Kaya, S., & Vural, E. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803196 · Cited by 5
“For instance, security systems produced in the recent years use face, voice, iris, and fingerprint data together for recognition purposes [1]; and facilities using LiDAR (Light Detection and Ranging) and SAR (Synthetic Aperture Radar) technologies can produce better topological information via different data types [2].”
MethodPerformance Bounds for Coupled CP Model in the Framework of Hyperspectral Super-Resolution
Prévost, C., Usevich, K., Comon, P., Haardt, M., & Brie, D. (2019) · IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing · doi:10.1109/camsap45676.2019.9022504 · Cited by 5
“This problem lies in the framework of multimodal data fusion [2] between heterogeneous datasets.”
BackgroundVADR: Discriminative Multimodal Explanations for Situational Understanding
Taylor, H., Hiley, L., Furby, J., Preece, A. D., & Braines, D. (2020) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion45008.2020.9190215 · Cited by 8
“…collaboration (HCC), knowledge representation and reasoning (KRR); multi-agent systems (MAS); machine learning (ML); natural language processing (NLP), vision and signal processing (VSP). data sources will commonly include multiple modalities such as imagery, acoustic and natural language data [3].”
BackgroundMultimodality for Rainfall Measurement
Messer, H. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-53547-0_32 · Cited by 5
Yang, H., Libedinsky, C., Guan, C., Ang, K. K., & So, R. Q. (2017) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2017.8037224 · Cited by 5
“, marjority voting) [8], [9], [10] which is operated on top of the initial base-training models from multiple sessions and days, namely, “boosting performance by classifier-level fusion (BP-ClsFu)”.”
BackgroundMethodA Generalization to Schur's Lemma with an Application to Joint Independent Subspace AnalysisPreprint
Lahat, D., & Christian Jutten, B. (2016) · HAL · Cited by 5
Self-citationChou, C.-A., Jin, X., Mueller, A. V., & Ostadabbas, S. (2019) · IEEE Sensors Letters · doi:10.1109/lsens.2018.2886544 · Cited by 5
“, [2]), the authors propose that forward progress requires establishing a common scientific framework for multimodal data fusion (MMDF) to enable apples-to-apples comparisons of solutions developed for applications as diverse as first-person indoor navigation [3], data-driven decision support systems for assisting med…”
BackgroundAdvanced Neural Networks for Multimodal Data Fusion in Interdisciplinary Research
Dhivya, K., Kumar, S. N., Victoria, D. R. S., Sherly, S. I., & Durgadevi, G. (2024) · Advances in IT Standards and Standardization Research · doi:10.4018/979-8-3693-4759-1.ch008 · Cited by 4
Prospects of machine learning applications in affective disorders
Mosolova, E., Alfimov, A. Е., Kostyukova, E. G., & Mosolov, S. N. (2025) · Digital Diagnostics · doi:10.17816/dd634885 · Cited by 6
Wei, Q., Zhang, X., Liang, X., Guo, Z., Zhang, Y., & Chen, Z. (2025) · Digital Health · doi:10.1177/20552076241310060 · Cited by 9
“53 The combination of clinical and molecular genetic features can further improve classi fi - cation accuracy and the development of personalized treat-ment plans.”
MSNet: Multimodal Self-attention Network for Depression Detection via Fusion of Eye Tracking and EEG
Zhu, F., Wu, B., Huo, Y., Dang, R., Hu, B., & Wang, Q. (2025) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3715669.3723106 · Cited by 4
LLM-CoSen: Revisiting Collaborative Sensing With Large Language Models (LLMs)
Feng, X., Sun, Z., Chen, Z., Luo, C., Zhou, Z., Leung, V. C. M., & Xu, W. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3583345 · Cited by 5
“[ 24], [25],[26]seriesstudiesconcatenatemulti-modaldatadirectlyat therawdatalevelalongdimensionstogeneratecomprehensive data.”
Leveraging Eye Movement for Instructing Robust Video-Based Facial Expression Recognition
Liu, Y., Wei, L., Liu, K., Chen, Z., Chen, Z., Tang, C., Chen, J., & Shan, S. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3599859 · Cited by 5
“Multi-level Cross-modal Fusion: Fusion is usually important for multimodal modeling [41], [42].”
Cognitive Internet of Things: A Review of Theory, Applications, and Recent Advances
Giuliano, A., McCafferty-Leroux, A., Yawney, J., & Gadsden, S. A. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3615461 · Cited by 15
“Data fusion techniques have also been proposed to fuse together time series data and imaging data in the biomedical context such as electroencephalograms and functional magnetic resonance images (fMRI) which provide complementary information about brain functions [179], [192].”
Wang, C., Zhu, M. S., & Zakaria, S. A. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-17656-0 · Cited by 4
“The alignment loss function typically incorporates both modality-specific reconstruction terms and cross-modal similarity constraints: Multimodal data fusion represents a critical component of cross-modal learning systems, generally categorized into early, late, and hybrid fusion approaches 12 .”
Multimodal learning of melt pool dynamics in laser powder bed fusion
Mojumder, S., Halder, P., & Tonge, T. (2026) · Progress in Additive Manufacturing · doi:10.1007/s40964-025-01496-5 · Cited by 5
Decoding Multimodal Text Analytics: Tasks, Datasets, Fusion Models, and Future Frontiers
Nath, T., Gupta, V., Gupta, M., & Sharma, R. R. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70083 · Cited by 4
“Modern data fusion originated in as early as the first half of the twentieth century (Lahat et al. 2015).”
Wang, Y., Fan, X., Wang, J., Ye, Q., Liu, J., Bai, Z., & Huan, H. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.112109 · Cited by 4
Tommaso, F. D., Noccaro, A., Formica, D., & Tagliamonte, N. L. (2024) · IEEE Access · doi:10.1109/access.2024.3487500 · Cited by 5
“The associate editor coordinating the review of this manuscript and approving it for publication was Yizhang Jiang . towards complex protocols and multi-modal data acquisition [1].”
BackgroundSepulveda, R. S., & Boman, M. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1369041 · Cited by 4
Sun, S., Zeng, Z., Li, Q., & Li, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113126 · Cited by 7
Empowering multimodal analysis with visualization: A survey
Wang, J., Deng, Z., Deng, D., Wang, X., Sheng, R., Cai, Y., & Qu, H. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100748 · Cited by 8
Deep multimodal learning for time series analysis in social computing: a survey
Yang, C., Chen, Y., Li, Z., Wang, X., Shi, K., Yao, L., Xu, G., & Guo, Z. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00363-x · Cited by 7
“…7] consider entirely distinct data forms like images, videos, and natural languages as separate modalities from time series data, while others [8, 9] regard data from different sources (such as various sensors) as distinct modalities, even if they exhibit typical time series characteristics,…”
Zhong, Q., Chan, C., Fu, L., & Zhou, R. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf254 · Cited by 6
“The successful integration of multisource data in this strategy offers a promising framework for tackling similar challenges encountered in imbalanced multimodal data fusion [80, 81].”
Chen, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-31341-2 · Cited by 10
An Effective Iterative Solution for Independent Vector Analysis With Convergence Guarantees
Cosserat, C., Gabrielson, B., Chouzenoux, É., Pesquet, J., & Adalı, T. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3620539 · Cited by 4
“This case presents itself when the datasets contain measures of a same phenomenon for different subjects [6], [9], different measurement methods [10], or more generally for various modalities [11].”
Coupled and Incomplete Tensors in Blind System Identification
Eeghem, F. V., Debals, O., Vervliet, N., & Lathauwer, L. D. (2018) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2873545 · Cited by 4
Robust Group Subspace Recovery: A New Approach for Multi-Modality Data Fusion
Ghanem, S., Panahi, A., Krim, H., & Kerekes, R. A. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2999461 · Cited by 4
“This application is similar to the emerging challenge of sensor networks and their role in advanced surveillance technology, as sensor networks can also use multimodal data fusion to obtain more information and insight from broad data sources [10].”
BackgroundIdentification of Subgroup Differences Using IVA: Application to fMRI Data Fusion
Luo, Z., Long, Q., Bhinge, S., Akhonda, M. A. B. S., & Adalı, T. (2020) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc44109.2020.9175837 · Cited by 4
“Especially, data-driven algorithms that are derived from independent principle have gained extensive attention both in the academic as well as industrial communities [1], [2], [4].”
BackgroundMethodInfluentialStrengthening Robustness Under Adversarial Attacks Using Brain Visual Codes
Rakhimberdina, Z., Liu, X., & Murata, T. (2022) · IEEE Access · doi:10.1109/access.2022.3204995 · Cited by 5
“Multi-modal classifiers are designed to 164 combine and process information from various modalities, 165 such as image, text, or audio data, to achieve higher perfor166 mance [12], [41].”
BackgroundA Multi-modal Biometric Approach Based on Score-level Fusion and Fine-tuning Deep Learning Features
Alghamdi, M. (2022) · IEEE International Conference on Intelligent Systems · doi:10.1109/is57118.2022.10019654 · Cited by 5
Cross-System Data Integration Based on Rule-Based NLP and Node2Vec
Xiao, J., Dut, W., Xu, Z., & Qian, Y. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00088 · Cited by 4
“Lahat et al. [18] presented the data fusion methods for multimodal data.”
MethodАлександрович, М. П. (2024) · Upravlenets · doi:10.29141/2218-5003-2023-14-6-1 · Cited by 5
An Efficient Analytic Solution for Joint Blind Source Separation
Gabrielson, B., Akhonda, M. A. B. S., Lehmann, I., & Adalı, T. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3394655 · Cited by 4
Alighting location estimation from public transit data: a case study of Shenzhen
Aslam, N. S., Barros, J., Han, L., Murcio, R., & Bei, H. (2024) · Transportation Planning and Technology · doi:10.1080/03081060.2024.2382247 · Cited by 4
“The second step focuses on determining boarding locations from SCD with the help of GPS-bus data required for only-entry systems when boarding location (Farzin 2008; Gordon et al. 2013; Wang, Attanucci, and Wilson 2011) or boarding location and time (Lahat, Adali, and Jutten 2015) are unavailable.”
Li, J., Li, X., Chen, Y., Wang, Y., Wang, B., Zhang, X., & Zhang, N. (2024) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-024-03253-2 · Cited by 5
Nian, Y., Su, X., Yue, H., Anwar, S., Li, J., Wang, W., Sheng, Y., Ma, Q., Liu, J., & Li, X. (2024) · Agriculture · doi:10.3390/agriculture14122258 · Cited by 4
Sathish, R., & Sheet, D. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109590 · Cited by 5
Hao, J., Zhang, Y., Song, L., Zhang, Z., & Shi, C. (2024) · Advanced Intelligent Systems · doi:10.1002/aisy.202400753 · Cited by 4
Survey on machine learning for MRI and PET fusion in alzheimer’s disease
Guelib, B., Bounab, R., Hermessi, H., Zarour, K., & Khlifa, N. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20631-6 · Cited by 5
Parameter Estimation from Heterogeneous/Multimodal Data Sets
Fijalkow, I., Heiman, E., & Messer, H. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2523886 · Cited by 4
Lahat, D., & Jutten, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93764-9_17 · Cited by 4
“These two types of links are sometimes referred to as “soft” versus “hard”, see [9] and references therein.”
BackgroundSelf-citationPomrehn, C., Klein, D., Kolb, A., Kaul, P. M., & Herpers, R. (2018) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2018.11.013 · Cited by 4
Cognitive Fusion for Target Tracking
Kyriakides, I. (2019) · Synthesis lectures on algorithms and software in engineering · doi:10.2200/s00951ed1v01y201908ase019 · Cited by 4
Data Fusion for Industry 4.0: General Concepts and Applications
Kondo, R. E., de Lima, E. D., Loures, E. D. F. R., Santos, E. A. P., & Deschamps, F. (2020) · Lecture notes on multidisciplinary industrial engineering · doi:10.1007/978-3-030-43616-2_38 · Cited by 5
Hierarchical multi-label propagation using speaking face graphs for multimodal person discovery
da Fonseca, G. B., Sargent, G., Sicre, R., Patrocínio, Z. K. G., Gravier, G., & Guimarães, S. J. F. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09692-x · Cited by 5
Fuzzy aggregation for multimodal remote sensing classification
Nock, K., & Gilmour, E. (2020) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz48607.2020.9177691 · Cited by 4
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.