Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 401–500 of 1,001
Multi-modal data clustering using deep learning: A systematic review
Raya, S. P., Orabi, M., Afyouni, I., & Aghbari, Z. A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128348 · Cited by 16
Shaffer, D. W., Wang, Y., & Ruis, A. R. (2025) · Journal of Learning Analytics · doi:10.18608/jla.2025.8423 · Cited by 21
Zhou, W. G., An, L., Han, R., & Li, G. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104179 · Cited by 14
Audio/video supervised independent vector analysis through multimodal pilot dependent components
Nesta, F., Mosayyebpour, S., Koldovský, Z., & Paleček, K. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081388 · Cited by 14
“Inspired by [11], in this work we further extend the model by considering posteriors derived from multimodal signals: • A neural network is trained using extensive prior acoustic data such that it produces a pilot signal that contains posteriors of the target source dominance in the observed…”
BackgroundMethodIdentifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion
Hu, C., Yin, M., Liu, B., Li, X., & Ye, Y. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3472713 · Cited by 21
Multimodal Data Visualization and Denoising with Integrated Diffusion
Kuchroo, M. R., Godavarthi, A., Tong, A., Wolf, G., & Krishnaswamy, S. (2021) · Machine learning for signal processing ... · doi:10.1109/mlsp52302.2021.9596214 · Cited by 14
“features, can offer viable solutions [1, 2].”
Das, B., & Sudarshan, T. S. B. (2023) · Online Social Networks and Media · doi:10.1016/j.osnem.2023.100247 · Cited by 20
Mellor, J., Stone, M. A., & Keane, J. A. (2018) · Trends in Hearing · doi:10.1177/2331216518773632 · Cited by 16
“Often these interrelated dimensions are acquired via the same sensor (Lahat, Adali, & Jutten, 2015).”
BackgroundVisualization of descriptive multiblock analysis
Skotare, T., Sjögren, R., Surowiec, I., Nilsson, D., & Trygg, J. (2018) · Journal of Chemometrics · doi:10.1002/cem.3071 · Cited by 14
A Tilt Sensor Node Embedding a Data-Fusion Algorithm for Vibration-Based SHM
Testoni, N., Zonzini, F., Marzani, A., Scarponi, V., & Marchi, L. D. (2019) · Electronics · doi:10.3390/electronics8010045 · Cited by 14
“Data-level fusion algorithms are encouraged by the widely shared opinion that reliability, resolution, availability, and accuracy are primary issues in every SHM process [29,30].”
BackgroundCoarse-to-Fine Registration of Airborne LiDAR Data and Optical Imagery on Urban Scenes
Nguyen, T. H., Daniel, S., Guériot, D., Sintès, C., & Caillec, J. L. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.2987305 · Cited by 14
“These errors, distortions and missing data from each of the two datasets induce incompleteness, imprecisions, and uncertainties within the registration and fusion processes of these data [22].”
BackgroundAn axiomatic distance methodology for aggregating multimodal evaluations
Escobedo, A. R., Moreno‐Centeno, E., & Yasmin, R. (2022) · Information Sciences · doi:10.1016/j.ins.2021.12.124 · Cited by 14
“While most existing data aggregation methods have been developed for a specific modality of data—continuous, ordinal, linguistic, etc.—there is a growing interest in developing new methodologies capable of integrating multiple modalities data (Lahat et al., 2015).”
MethodIntegrating multimodal data and machine learning for entrepreneurship research
Shrestha, Y. R., & He, V. F. (2025) · Strategic Entrepreneurship Journal · doi:10.1002/sej.1546 · Cited by 18
Xin, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22752-2 · Cited by 17
Integrating learning and reasoning services for explainable information fusion
Harborne, D., Willis, C. J., Tomsett, R., & Preece, A. D. (2018) · ORCA Online Research @Cardiff (Cardiff University) · Cited by 13
Singh, S. K., & Malinowski, P. (2022) · Composite Structures · doi:10.1016/j.compstruct.2022.115808 · Cited by 13
CRBSP:Prediction of CircRNA-RBP Binding Sites Based on Multimodal Intermediate Fusion
Liu, N., Zhang, Z., Wu, Y., Wang, Y., & Liang, Y. (2023) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2023.3272400 · Cited by 14
“Different modes have different information and expressions and must be fused to take account of cross-complementary phenomena and different information interactions [52].”
BackgroundTsinganos, N., Fouliras, P., Mavridis, I. K., & Gritzalis, D. A. (2024) · IEEE Access · doi:10.1109/access.2024.3359030 · Cited by 16
“In the context of our study, multimodal refers to the use of multiple modalities or sources of information to make predictions [64], [65], [66].”
BackgroundLin, H., Guo, R., Ma, D., Kuai, X., Yuan, Z., Du, Z., & He, B. (2024) · International Journal of Digital Earth · doi:10.1080/17538947.2024.2421950 · Cited by 13
“Emergency situations are complex and cannot be fully characterized by information from a single modality, which necessitates the integration of multiple modalities to gain a complete understanding of emergency situations (Lahat, Adali, and Jutten 2015).”
Hải, N. N., Nguyen, T.-T., & Ngo, Q. T. (2025) · Journal of risk and financial management · doi:10.3390/jrfm18080417 · Cited by 17
Online Learning for Multimodal Data Fusion With Application to Object Recognition
Shahrampour, S., Noshad, M., Ding, J., & Tarokh, V. (2017) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2017.2754141 · Cited by 13
“ANALYZING a system or task requires information from various measurements, experiments, and multiple resources, in that natural phenomena often have complex characteristics [1].”
BackgroundMulti-View Robust Feature Learning for Data Clustering
Zhao, L., Zhao, T., Sun, T., Liu, Z., & Chen, Z. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3026943 · Cited by 13
“To describe the same thing, different information from multiple channels or modalities co-aggregate to form multi-view data, showing the characteristics of the thing from different perspectives [1], [2].”
BackgroundPan, G., Xiao, L., Bai, Y., Wilson, T. W., Stephen, J. M., Calhoun, V. D., & Wang, Y. (2020) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2020.3048594 · Cited by 15
Mignotte, M. (2022) · AI Computer Science and Robotics Technology · doi:10.5772/acrt.02 · Cited by 13
Xiao, Z., Dai, B., Li, H., Wu, T., Xu, X., Zeng, Y., & Chen, T. (2017) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881417717058 · Cited by 13
Objective Pain Measurement based on Physiological Signals
Lin, Y., Wang, L., Xiao, Y., Urman, R. D., Dutton, R. W., & Ramsay, M. A. E. (2018) · Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · doi:10.1177/2327857918071056 · Cited by 17
Linked CP tensor decomposition algorithms for shared and individual feature extraction
Zdunek, R., Fonał, K., & Wołczowski, A. R. (2018) · Signal Processing Image Communication · doi:10.1016/j.image.2018.11.001 · Cited by 13
Embodied reports in paramedicine mixed reality learning
Aguayo, C., Dañobeitia, C., Cochrane, T., Aiello, S., Cook, S., & Cuevas, A. (2018) · Research in Learning Technology · doi:10.25304/rlt.v26.2150 · Cited by 20
“…of Cognition (Hallowell 2009; Luisi 2016), the Embodied Group from Chile (www.embodiedgroup.com) developed a multimodal data approach methodology (Lahat, Adali, and Jutten 2015), coined ‘embodied reports’, to assess emotional experiences via physiological and self-reported data (Dañobeitia 2017).”
BackgroundBraids of partitions for the hierarchical representation and segmentation of multimodal images
Tochon, G., Mura, M. D., Veganzones, M. A., Géraud, T., & Chanussot, J. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.05.029 · Cited by 13
Coupled Higher-Order Tensor Factorization for Hyperspectral and LiDAR Data Fusion and Classification
Xue, Z., Yang, S., Zhang, H., & Du, P. (2019) · Remote Sensing · doi:10.3390/rs11171959 · Cited by 16
“In this context, tensor decomposition can extract the shared components between data sources with those rank-one tensors, and the revealed structures of tensor decomposition may further contribute to interpretability, separability, robustness, and uniqueness in feature representation [45].”
BackgroundEnergy flexible manufacturing systems: A systematic literature review, trends and challenges
Xu, L., Liu, P., Wang, X. A., Lin, Y., Shi, Y., Huang, X., & Wang, X. V. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.010 · Cited by 12
Xu, W., Fu, Z., Tang, B., Xu, J., & Li, G. (2025) · Analytica Chimica Acta · doi:10.1016/j.aca.2025.344628 · Cited by 12
Human activity recognition for analyzing stress behavior based on Bi-LSTM
Sa-nguannarm, P., Elbasani, E., & Kim, J. (2023) · Technology and Health Care · doi:10.3233/thc-235002 · Cited by 16
“Moreover, it is pervasively applied to several fields [9].”
MARS: A Multiview Contrastive Approach to Human Activity Recognition From Accelerometer Sensor
Sharma, G., Dhall, A., & Subramanian, R. (2024) · IEEE Sensors Letters · doi:10.1109/lsens.2024.3357941 · Cited by 12
AI-Based Cameras to Detect a Leak of Chemicals or Gas in Real Time
Selvakumar, P., Sivaraja, M., Satishkumar, D., Vijayakumar, G., Kumar, K. S., & Navaneeth, B. S. (2024) · Advances in logistics, operations, and management science book series · doi:10.4018/979-8-3693-4276-3.ch009 · Cited by 12
Data Governance in Multimodal Behavioral Research
Jiang, Z., Zhu, Z., & Pan, S. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8070055 · Cited by 14
“To ensure data quality and consistency, the data governance framework would need to establish data standards and protocols for each modality, such as minimum sampling rates, calibration procedures, and data formatting requirements [6].”
BackgroundMethodInfluentialXia, Y., Li, T., Wang, Q., Yue, J., Peng, B., & Yi, X. (2024) · Physics of Fluids · doi:10.1063/5.0229049 · Cited by 12
Model Collaboration at Network Edge: Feature-Large Models for Real-Time IoT Communications
Yu, X., Zhang, S., Zhang, H., & Song, L. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3523953 · Cited by 16
“This growth will result in vast amounts of heterogeneous data, including sensor readings, images, audios, and videos [3].”
Data Fusion for MaaS: Opportunities and Challenges
Wu, J., Zhou, L., Chen, C., Shen, J., Lau, S. K., & Yong, J. (2018) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd.2018.8465224 · Cited by 13
“The link with Formula (1) is established via , n = Z, and Therefore, the constraint or assumption of statistical independence on the sources helps to achieve essential uniqueness or diversity in data-driven models [36].”
BackgroundVarghese, E. B., & Thampi, S. M. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104305 · Cited by 12
Akhonda, M. A. B. S., Levin‐Schwartz, Y., Calhoun, V. D., & Adalı, T. (2022) · Sensors · doi:10.3390/s22031224 · Cited by 15
“As such, IVA provides a fully multivariate approach for the analysis of multiple neuroimaging datasets [8,31].”
BackgroundSpeech and Language in Humanoid Robots
Cangelosi, A., & Ogata, T. (2017) · Springer eBooks · doi:10.1007/978-94-007-7194-9_135-1 · Cited by 13
Tensor maps for synchronizing heterogeneous shape collections
Huang, Q., Liang, Z., Wang, H., Zuo, S., & Bajaj, C. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322944 · Cited by 16
“Our approach is motivated from a recent trend that utilizes high-order tensor decompositions to solve challenging problems in machine learning and relevant fields [Anandkumar et al. 2014; Cichocki et al. 2015; Kolda and Bader 2009; Lahat et al. 2015; Lu et al. 2016; Sidiropoulos et al. 2017].”
MethodMulti-Task fMRI Data Fusion Using IVA and PARAFAC2
Lehmann, I., Acar, E., Hasija, T., Akhonda, M. A. B. S., Calhoun, V. D., Schreier, P. J., & Adalı, T. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747662 · Cited by 12
Random Tensor Theory for Tensor Decomposition
Ouerfelli, M., Tamaazousti, M., & Rivasseau, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20761 · Cited by 14
“…faces (Vasilescu and Terzopoulos 2002), Hyperspectral imagery (Lin and Bourennane 2013), DNN compression (Astrid and Lee 2017), chemical materials (Sun and Braatz 2020), multi-modal data fusion (Lahat, Adali, and Jutten 2015), data mining (Papalexakis, Faloutsos, and Sidiropoulos 2016), etc.).”
BackgroundAkar, Ö., & GÖRMÜŞ, E. T. (2019) · Geomatik · doi:10.29128/geomatik.476668 · Cited by 14
Wang, Q., Zhao, X., Huang, J., Feng, Y., Liu, Z., Su, J., Luo, Z., & Cheng, G. (2017) · Preprints.org · doi:10.20944/preprints201710.0076.v1 · Cited by 13
Wang, S., Song, X., Song, X., Gu, Y., Cong, Z., Shen, Y., & Yu, L. (2026) · Nano-Micro Letters · doi:10.1007/s40820-025-02042-2 · Cited by 21
“On the algorithmic side, residual noise persists despite hardware optimization, necessitating advanced signal processing techniques such as blind source separation, adaptive filtering, and deep learning models to extract meaningful neural data from contaminated signals [136].”
Applying Artificial Intelligence and machine learning in precision nutrition
Massara, P., Kirkland, J., Pagani, I., Huey, S. L., Hirsh, H., McDonald, D. T., Patel, L., Finkelstein, J. L., Gantz, M. G., Wang, F., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-75004-w · Cited by 16
Pajula, I., Joswig, N., Morrison, A. J., Sokolova, N., & Ruotsalainen, L. (2023) · IEEE Journal of Indoor and Seamless Positioning and Navigation · doi:10.1109/jispin.2023.3344077 · Cited by 15
“The late-fusion stage fuses the feature representations right before computing the output [26].”
BackgroundInfluentialZhang, Z., Ma, Y., & Liu, P. (2024) · Big Earth Data · doi:10.1080/20964471.2024.2358615 · Cited by 15
“The use of multimodal data takes advantage of the benefits of different types of remote sensing imagery (Knyaz, 2019; Lahat et al., 2015; Yang et al., 2019).”
PROSAC as a selection tool for SO-PLS regression: A strategy for multi-block data fusion
Diaz-Olivares, J. A., Bendoula, R., Saeys, W., Ryckewaert, M., Adriaens, I., Fu, X., Pastell, M., Roger, J., & Aernouts, B. (2024) · Analytica Chimica Acta · doi:10.1016/j.aca.2024.342965 · Cited by 11
Graph-Based Methods for Multimodal Indoor Activity Recognition: A Comprehensive Survey
Javadi, S., Riboni, D., Borzì, L., & Zolfaghari, S. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3523240 · Cited by 16
“The abundance of multimodal data presents both a challenge and an opportunity [4], [5], [6].”
Tuly, S. R., Ranjbari, S., Murat, E. A., & Arslanturk, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110108 · Cited by 13
Wei, T., Chen, H., Liu, W., Chen, L., & Wang, J. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3591926 · Cited by 19
“In contrast, synthetic aperture radar (SAR) images provide structural and scattering features, enhancing the diversity of representations [6], [7], [8], [9].”
An Edge-Cloud-Aided High-Order Possibilistic c-Means Algorithm for Big Data Clustering
Bu, F., Zhang, Q., Yang, L. T., & Yu, H. (2020) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2020.2992634 · Cited by 11
“A multimodal instance includesseveraldifferenttypesofobjects,eachofwhichiscalled amodality[5].”
Integration of Multimodal Data from Disparate Sources for Identifying Disease Subtypes
Zhou, K., Kottoori, B. S., Munj, S. A., Zhang, Z., Drăghici, S., & Arslanturk, S. (2022) · Biology · doi:10.3390/biology11030360 · Cited by 14
“It also leads to challenges such as overfitting due to increased dimensionality of data after concatenation, as well as data redundancies and dependencies [22].”
BackgroundFeng, B., Zhu, Q., Liu, M., Li, Y., Zhang, J., Fu, X., Zhou, Y. J., Li, M., He, H., & Yang, W. (2018) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi7090371 · Cited by 11
“Finally, a hybrid spatio-temporal index architecture is proposed based on the above discrepancy requirements.”
BackgroundJoint Independent Subspace Analysis: Uniqueness and Identifiability
Lahat, D., & Christian Jutten, B. (2018) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2880714 · Cited by 11
“, [13]), then, since pairs of multivariate random processes with different dimensions (i.”
BackgroundMethodInfluentialSelf-citationBrain-inspired Multimodal Learning Based on Neural Networks
Liu, C., Sun, F., & Zhang, B. (2018) · Brain Science Advances · doi:10.26599/bsa.2018.9050004 · Cited by 10
“The information obtained when observing phenomena is termed a “modality” [16].”
BackgroundUss, M., Vozel, B., Абрамов, С. К., & Chehdi, K. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2992597 · Cited by 11
“The challenging multimodal registration of RS images is important for the multisensor study of the earth’s surface where each modality (e.g., visible, infrared, thermal, radar, and LIDAR) provides complementary information on a study area [10].”
BackgroundGeometric Multimodal Learning Based on Local Signal Expansion for Joint Diagonalization
Behmanesh, M., Adibi, P., Chanussot, J., Christian Jutten, B., & Ehsani, M. S. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3053513 · Cited by 11
“Multimodal offers complementary information from different modalities [1].”
BackgroundData modalities, consumer attributes and recommendation performance in the fashion industry
Sysko-Romańczuk, S., Zaborek, P., Wróblewska, A., Dąbrowski, J., & Tkachuk, S. (2022) · Electronic Markets · doi:10.1007/s12525-022-00579-3 · Cited by 13
“It seems evident in past research that adding different modalities of data should increase the performance of recommendation algorithms (Gao et al., 2020; Lahat et al., 2015; Liu et al., 2018).”
BackgroundMULTI-MODAL DEEP LEARNING WITH SENTINEL-3 OBSERVATIONS FOR THE DETECTION OF OCEANIC INTERNAL WAVES
Drees, L., Kusche, J., & Roscher, R. (2020) · ISPRS annals of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-annals-v-2-2020-813-2020 · Cited by 12
“Apart from conventional data fusion methods (Lahat et al., 2015, Gupta, Cheng, 2006) in the field of Earth sciences, so far, only a few multi-modal deep learning approaches exist.”
MethodExplainable AI for Intelligence Augmentation in Multi-Domain OperationsPreprint
Preece, A. D., Braines, D., Cerutti, F., & Pham, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07563 · Cited by 25
“Moreover, the sources will commonly span multiple modalities, for example, imagery, acoustic and natural language data (Lahat, Adali, and Jutten 2015).”
BackgroundA Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical ApplicationsPreprint
Guarrasi, V., Aksu, F., Caruso, C. M., Feola, F. D., Rofena, A., Ruffini, F., & Soda, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4952813 · Cited by 21
A systematic review of multimodal learning analytics in computer-supported collaborative learning
Ouyang, F., & Bai, X. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105574 · Cited by 15
MISP: A Multimodal-based Intelligent Server Failure Prediction Model for Cloud Computing Systems
Lu, X., Wang, Y., Fu, Y., Sun, Q. L., Ma, X., Zheng, X., & Zhuo, C. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671568 · Cited by 13
“Such hurdles have restricted in-depth investigations into multimodal-based server failure prediction, often limiting the focus to single data type features and neglecting the rich potential of cross-modality features inherent in multimodal sequential data [18, 39, 49].”
Temperature correction of near-infrared spectra of raw milk
Diaz-Olivares, J. A., Grauwels, S., Fu, X., Adriaens, I., Saeys, W., Bendoula, R., Roger, J., & Aernouts, B. (2024) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2024.105251 · Cited by 10
Designing deep neural networks for driver intention recognition
Vellenga, K., Steinhauer, H. J., Karlsson, A., Falkman, G., Rhodin, A., & Koppisetty, A. C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109574 · Cited by 11
“A key aspect of fusing multiple modalities is that every modality adds information that cannot be observed by any of the other sensors in the setup (6).”
Gao, F., & Tang, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-06229-w · Cited by 15
“Mathematically, multimodal learning seeks to optimize a joint representation Z from multiple input modalities X 1 , X 2 , ..., X m : Z = f ( X 1 , X 2 , ..., X m ; θ ) (5) where f represents the fusion function with parameters θ 46 .”
RetractedLearning From Crowdsourced Noisy Labels: A signal processing perspective
Ibrahim, S., Traganitis, P. A., Fu, X., & Giannakis, G. B. (2025) · IEEE Signal Processing Magazine · doi:10.1109/msp.2025.3572636 · Cited by 16
“Various data fusion applications in remote sensing and healthcare that aggregate information from multiple sources of varying reliability also have strong connections to crowdsourcing techniques [8].”
Multi-Frequency RF Sensor Data Adaptation for Motion Recognition with Multi-Modal Deep Learning
Rahman, M. M., & Gürbüz, S. Z. (2021) · IEEE Radar Conference (RadarConf) · doi:10.1109/radarconf2147009.2021.9455204 · Cited by 10
“…1) a single sensor case, where adversarial domain adaptation [23] is used to transform the data from one RF sensor to resemble that of another, and 2) a multi-sensor case, where multi-modal deep learning [24], [25] is designed for joint target recognition using measurements from all sensors.”
MethodAlthenayan, A. S., Alsalamah, S., Aly, S. F., Nouh, T. A., & Mirza, A. A. (2022) · Applied Sciences · doi:10.3390/app122010535 · Cited by 11
Research on Mobile Network User Complaint Warning Method Based on Multimodal Data
Di, Z., Xiao, T., Li, B., Peng, Y., Li, J., Cheng, X., Xu, L., Zhu, X., & Zheng, H. (2023) · International Conference on Information and Communication Technologies for Disaster Management (ICT-DM) · doi:10.1109/ict-dm58371.2023.10286947 · Cited by 10
“The use of multiple sources and multi-modal data can improve the accuracy of user complaint warning [9-10].”
BackgroundMulti-block approach for the characterization and discrimination of Italian chickpeas landraces
Foschi, M., Biancolillo, A., Marini, F., Cosentino, F., Donato, F. D., & D’Archivio, A. A. (2023) · Food Control · doi:10.1016/j.foodcont.2023.110170 · Cited by 10
Lygerakis, F., Dave, V., & Rueckert, E. (2024) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur61395.2024.10597462 · Cited by 17
On the Spectral Property of Kernel-Based Sensor Fusion Algorithms of High Dimensional Data
Ding, X., & Wu, H. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.3026255 · Cited by 10
“This problem is usually understood as the sensor fusion problem [40].”
BackgroundTensor Network-Encrypted Physical Anti-counterfeiting Passport for Digital Twin Authentication
Li, J., Liu, R., Lin, H., Ye, S., Ye, M., Wang, X., & Zhu, X. (2021) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.1c18808 · Cited by 10
Attention deficit and hyperactivity disorder classification with EEG and machine learning
González, C. L. M., Ortiz, E. J. M., Escobar, J. J. M., & Rivera, J. A. D. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-820125-1.00034-8 · Cited by 12
Indirect-Reciprocity Data Fusion Game and Application to Cooperative Spectrum Sensing
Zhang, B., Chen, Y., Yu, J.-L., Chen, B., & Han, Z. (2017) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2017.2725836 · Cited by 10
“To jointly analyze information from multiple sensors and extract knowledge for various purposes, concepts of data fusion were proposed and corresponding techniques were introduced [1].”
MethodComparison and extension of three methods for automated registration of multimodal plant images
Michael, H., Junker, A., Neumann, K., Altmann, T., & Gladilin, E. (2019) · Plant Methods · doi:10.1186/s13007-019-0426-8 · Cited by 10
Munck, S., Swoger, J., Coll‐Lladó, M., Gritti, N., & Velde, G. V. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.05.003 · Cited by 10
Wang, Q., Zhao, X., Huang, J., Feng, Y., Su, J., & Luo, Z. (2017) · Preprints.org · doi:10.20944/preprints201710.0076.v2 · Cited by 9
“Regarding the variety of big data, it is best matched through multi-modalities, and its inherent heterogeneity makes it easier to gain a global view of phenomena[60].”
BackgrounddtwParallel: A Python package to efficiently compute dynamic time warping between time series
Escudero-Arnanz, Ó., Marques, A. G., Soguero-Ruíz, C., Mora-Jiménez, I., & Robles, G. (2023) · SoftwareX · doi:10.1016/j.softx.2023.101364 · Cited by 10
New Perspectives, Challenges, and Advances in Data Fusion in Neuroimaging
Sujata, P., Takale, D. G., Tyagi, S., Bhalerao, S., Tiwari, M., & Dhanraj, J. A. (2024) · Wiley eBooks · doi:10.1002/9781394197705.ch12 · Cited by 11
Multimodal Data Fusion based on the Global Workspace Theory
Bao, C., Fountas, Z., Olugbade, T., & Bianchi-Berthouze, N. (2020) · arXiv · doi:10.48550/arxiv.2001.09485 · Cited by 9
Ma, Y., Celepkolu, M., Boyer, K. E., Lynch, C. F., Wiebe, E., & Israel, M. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614127 · Cited by 14
“The main advantage of neuralnetwork-based fusion approaches over model-agnostic-based fusion approaches lies in their ability to exploit the underlying relationships and mutual information among modalities [2, 23].”
BackgroundData-driven visual model development and 3D visual analytics framework for underground mining
Liang, R., Zhang, C., Li, B., Saydam, S., Canbulat, I., & Munsamy, L. (2024) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2024.106054 · Cited by 14
Choi, B., & Hummel, M. A. (2024) · Environmental Research Letters · doi:10.1088/1748-9326/ada397 · Cited by 9
“However, multimodal data fusion is often a nontrivial task due to incompatible measurement units, different spatial/temporal resolutions, and varying uncertainties [23].”
A Dynamic Bayesian Network Based Framework for Multimodal Context-Aware Interactions
Han, V. Y., Wang, T., Cho, H., Todi, K., Fernandes, A. S., Levi, A., Zhang, Z., Grossman, T., Ion, A., & Jonker, T. R. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712070 · Cited by 13
A Hybrid Intelligent Model for Olympic Medal Prediction Based on Data-Intelligence Fusion
Li, N., Li, J., Fang, H., Wang, J., Yu, Q., & Shi, Y. (2025) · Technologies · doi:10.3390/technologies13060250 · Cited by 14
“DIF requires the organic fusion of data and intelligent methods, which is composed of a model, strategy, and algorithm [1].”
Intelligent soft wearable bioelectronics for neurological disorders
Kim, D., Bang, J., Jeong, J., Koo, D. L., & Ko, S. H. (2025) · Materials Horizons · doi:10.1039/d5mh00528k · Cited by 10
Triple-objective cross-view geolocalization of disaster-related VGI: the case of Hurricane Ian
Yin, W., Deuser, F., Liu, Z., Wei, J., Luo, X., Werner, M., Li, H., & Xue, Y. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2524857 · Cited by 9
“Decision-level fusion, on the other hand, processes text and image information separately before fusing their outputs (Lahat et al. 2015).”
Chen, W., Yue, W., Chang, K., Wang, H., Tan, K., Liu, X., & Cao, X. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131555 · Cited by 11
A Higher-Order Generalized Singular Value Decomposition for Rank-Deficient Matrices
Kempf, I., Goulart, P. J., & Duncan, S. (2023) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/21m1443881 · Cited by 9
Kautsar, A., Wahyuni, W. T., Syafitri, U. D., Muflihah, S., Mawadah, N., Rohaeti, E., Arif, Z., Prajogo, B., Amran, M. B., Rohman, A., et al. (2021) · Indonesian Journal of Chemistry · doi:10.22146/ijc.60321 · Cited by 9
Anchor-based sparse subspace incomplete multi-view clustering
Li, A., Feng, C., Wang, Z., Sun, Y., Wang, Z., & Sun, L. (2023) · Wireless Networks · doi:10.1007/s11276-023-03312-w · Cited by 9
Multimodal Data Fusion to Detect Preknowledge Test-Taking Behavior Using Machine Learning
Man, K. (2023) · Educational and Psychological Measurement · doi:10.1177/00131644231193625 · Cited by 12
“Therefore, it unveils significant information hidden in the data set, which could enhance detection of aberrant test-taking behaviors (Gao et al., 2020; Gaw et al., 2022; Lahat et al., 2015).”
Belyaeva, I., Gabrielson, B., Wang, Y., Wilson, T. W., Calhoun, V. D., Stephen, J. M., & Adalı, T. (2024) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2024.3364704 · Cited by 16
“…data into a two-dimensional (2D) representation does not fully leverage the multilinear structure of multisensor MEG/EEG, and (2) enforcing constraints such as independence or sparsity of components is typically required to ensure the uniqueness of the matrix decomposition [1], [7].”
Background
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.