Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 301–400 of 1,001
Multimodal fusion with relational learning for molecular property prediction
Zhou, Z., Li, Y., Hong, P., & Xu, H. (2025) · Communications Chemistry · doi:10.1038/s42004-025-01586-z · Cited by 36
CP-Robot: Cloud-Assisted Pillow Robot for Emotion Sensing and Interaction
Chen, M., Ma, Y., Hao, Y., Li, Y., Wu, D., Zhang⋆, Y., & Song, E. (2016) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-319-44350-8_9 · Cited by 22
“As for the integration problem of the data [32], we give out the integration on the basis of the decision-level, that is”
MethodCross-modal generative models for multi-modal plastic sorting
Neo, E. R. K., Low, J. S. C., Goodship, V., Coles, S. R., & Debattista, K. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.137919 · Cited by 25
“Multi-modal deep learning model is achieved by fusing the data from different modalities together, which allows the deep learning model to maximize the information learnt from different data modalities to improve classification accuracy (Lahat et al., 2015).”
Compressive Sensing-Based Detection With Multimodal Dependent Data
Wimalajeewa, T., & Varshney, P. K. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2770100 · Cited by 22
“ture based techniques for multimodal data fusion are discussed in [2].”
BackgroundMethodConvex Factorization Machine for Toxicogenomics Prediction
Yamada, M., Lian, W., Goyal, A., Chen, J., Wimalawarne, K., Khan, S. A., Samuel Kaski, A., Mamitsuka, H., & Chang, Y. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098103 · Cited by 25
“Such methodologies have been used in various prediction tasks such as toxicogenomics, bioinformatics, brain activity, chemometrics and temporal sales prediction [15, 21, 25].”
MethodCoupled matrix-tensor factorizations — The case of partially shared factors
Lathauwer, L. D., & Kofidis, E. (2017) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.2017.8335436 · Cited by 22
“Diversity is the key to data fusion [1], [16].”
BackgroundMultimodal Database for Human Activity Recognition and Fall Detection
Martínez-Villaseñor, L., Ponce-Espinosa, H., & Espinosa, R. (2018) · International Conference on Ubiquitous Computing and Ambient Intelligence (UCAmI) · doi:10.3390/proceedings2191237 · Cited by 24
“The main idea is that “Due to the rich characteristics of natural processes and environments, it is rare that a single acquisition method provides complete understanding thereof” [6].”
BackgroundJiao, C., Lian, B., Wang, Z., Song, Y., & Sun, T. (2020) · International Journal of Advanced Robotic Systems · doi:10.1177/1729881420948727 · Cited by 25
“42 The results from the visual and tactile recognition are fused on the decision-making layer as the final recognition conclusion.”
BackgroundMGKsite: Multi-Modal Knowledge-Driven Site Selection via Intra and Inter-Modal Graph Fusion
Liang, K., Meng, L., Li, H., Liu, M., Wang, S., Zhou, S., Liu, X., & He, K. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521742 · Cited by 42
“The early attempts integrate the features from various modalities with simple operations like summation [66], concatenation [67], and alignment learning [68], [69].”
Intelligent sensing devices and systems for personalized mental health
Xing, Y., Yang, Y., Yang, K., Lu, A., Xing, L., Mackie, K., & Guo, F. (2025) · Med-X · doi:10.1007/s44258-025-00057-3 · Cited by 28
“Data preprocessing and feature extraction are critical foundations for effective multimodal data fusion [190] (Fig.”
Han, X., Li, Z., Cao, H., & Hou, B. (2025) · Land · doi:10.3390/land14051069 · Cited by 32
Eyndhoven, S. V., Hunyadi, B., Lathauwer, L. D., & Huffel, S. V. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081162 · Cited by 21
“On the other end of the spectrum, data-driven blind source separation (BSS) techniques such as independent component analysis can be used to decompose both EEG and fMRI data into a set of constituting sources, in which case neural sources are found by imposing sometimes arbitrary restrictions [9], [10].”
MethodKondylakis, H., Axenie, C., Bastola, D. K., Katehakis, D. G., Kouroubali, A., Kurz, D., Larburu, N., Macía, I., Maguire, R., Maramis, C., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/22034 · Cited by 25
“Such an exploration unveils another challenge, namely multimodal data fusion [69], and the opportunity and potential that fusing extracted knowledge have toward a personalized therapeutic scheme [70].”
BackgroundBastida, J. O., Gallego, A., Rico-Juan, J. R., & Albarrán, P. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107693 · Cited by 27
Multimodal Data Fusion in High-Dimensional Heterogeneous Datasets Via Generative Models
Yılmaz, Y., Aktukmak, M., & Hero, A. O. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3109375 · Cited by 24
“Independent Component Analysis (ICA) and its extension Independent Vector Analysis (IVA), which is also a generalization of Canonical Correlation Analysis (CCA), are also used for multimodal data fusion [13], [14].”
Pamart, A., Abergel, V., Luca, L. D., & Veron, P. (2023) · Remote Sensing · doi:10.3390/rs15092408 · Cited by 21
Silva, R. F., Plis, S., Jing Sui (Beijing Normal University), my correct affiliation is beijing normal university, N. Q. U. O. S. A. T., Pattichis, M. S., Adalı, T., & Calhoun, V. D. (2016) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2016.2594945 · Cited by 25
“Multimodal data fusion applications are by and large viewed as MDU-type problems [174], [175].”
Cognitive computing for coalition situational understanding
Preece, A. D., Cerutti, F., Braines, D., Chakraborty, S., & Srivastava, M. B. (2017) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397426 · Cited by 21
“Moreover, the sources will commonly span multiple modalities (for example, imagery, acoustic and textual data [11]) requiring NLP and VSP in addition to ML.”
BackgroundHuang, X., Xie, K., Leng, S., Yuan, T., & Ma, M. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.02.046 · Cited by 25
Sharing and reusing cell image data
Zaritsky, A. (2018) · Molecular Biology of the Cell · doi:10.1091/mbc.e17-10-0606 · Cited by 19
Hazarika, A., Dutta, L., Barthakur, M., & Bhuyan, M. (2018) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2018.2866744 · Cited by 21
“Medical processes involve complicated measurements [5] similar to various industrial processes.”
BackgroundMulti-Modal Sensor Nodes in Experimental Scalable Agricultural IoT Application Scenarios
Loukatos, D., & Arvanitis, K. G. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-030-71172-6_5 · Cited by 22
Castro, J. P., Machado, R. C., Andrade, D. F., Babos, D. V., Pereira‐Filho, E. R., Garcia, J. A., Sperança, M. A., Gamela, R. R., & Costa, V. C. (2022) · Springer eBooks · doi:10.1007/978-3-031-14502-5_2 · Cited by 21
Spatiotemporal adaptive attention 3D multiobject tracking for autonomous driving
Zhang, X., Fan, Z., Tan, X., Liu, Q., & Shi, Y. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110442 · Cited by 24
Li, J., Lin, S., Zhang, L., Liu, Y., Peng, Y., & Hu, Q. (2023) · Frontiers of Environmental Science & Engineering · doi:10.1007/s11783-024-1791-x · Cited by 22
“This approach, which is reminiscent of the ability of the human brain to process diverse data types concurrently, has demonstrated multiple advantages in terms of efficient and accurate practical applications (Lahat et al., 2015).”
BackgroundAkhonda, M. A. B. S., Levin‐Schwartz, Y., Bhinge, S., Calhoun, V. D., & Adalı, T. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462031 · Cited by 20
“Therefore, reducing each modality to a feature, a lower dimensional multivariate representation of data, for each subject is an effective approach [4, 21].”
BackgroundEyndhoven, S. V., Dupont, P. J., Tousseyn, S., Vervliet, N., Paesschen, W. V., Huffel, S. V., & Hunyadi, B. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117652 · Cited by 22
Bayesian Joint Modeling of Multiple Brain Functional Networks
Lukemire, J., Kundu, S., Pagnoni, G., & Guo, Y. (2020) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1796357 · Cited by 29
Fusion-based methods for result diversification in web search
Wu, S., Huang, C., Li, L., & Crestani, F. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.01.006 · Cited by 21
“When the search results include text, image, and other types of media, then multimodal data fusion (Lahat et al., 2015) can be used to fuse multiple evidence from different types of media.”
MethodRecovering Hidden Components in Multimodal Data with Composite Diffusion Operators
Shnitzer, T., Ben‐Chen, M., Guibas, L. J., Talmon, R., & Wu, H. (2019) · SIAM Journal on Mathematics of Data Science · doi:10.1137/18m1218157 · Cited by 26
Neuroimaging-based variability in subtyping biomarkers for psychiatric heterogeneity
Wen, Z., Hammoud, M. Z., Siegel, C., LASKA, E. M., Amara, D. A., Etkin, A., Milad, M. R., & Marmar, C. R. (2024) · Molecular Psychiatry · doi:10.1038/s41380-024-02807-y · Cited by 28
“Advanced multimodal fusion algorithms from the machine learning fi eld [81] may offer advantages over traditional methods.”
Teles, A. S., de Moura, I. R., Silva, F., Roberts, A., & Stahl, D. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102981 · Cited by 27
Ferreira, R. E. P., & Dórea, J. R. R. (2025) · Journal of Dairy Science · doi:10.3168/jds.2024-25650 · Cited by 25
Dashtestani, H., Miguel, H. O., Condy, E. E., Zeytinoglu, S., Millerhagen, J., Debnath, R., Smith, E. G., Adalı, T., Fox, N. A., & Gandjbakhche, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-10942-1 · Cited by 21
Multi-Horizon Wind Power Forecasting Using Multi-Modal Spatio-Temporal Neural Networks
Miele, E. S., Ludwig, N., & Corsini, A. (2023) · Energies · doi:10.3390/en16083522 · Cited by 23
Wang, J., Yan, Y., & Zhao, G. (2023) · Computer Networks · doi:10.1016/j.comnet.2023.109796 · Cited by 19
Han, W., Chen, H., Kan, M., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.717 · Cited by 27
Albarracin, M., Hipólito, I., Tremblay, S. E., Fox, J. G., René, G., Friston, K. J., & Ramstead, M. J. D. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-47958-8_9 · Cited by 26
Yin, T., Peng, Y., Chao, K., & Li, Y. (2025) · npj Science of Food · doi:10.1038/s41538-025-00393-z · Cited by 25
“Additionally,multimodaldatafusion,alongwiththeautomationand standardizationofexperimentalprocedures,couldreducevariabilityindata collectionandpreprocessing,therebyenhancingreproducibilityinveter-inarydrugdetectionapplications 227 – 229 .”
Hunyadi, B., Paesschen, W. V., Vos, M. D., & Huffel, S. V. (2016) · European Signal Processing Conference (EUSIPCO) · doi:10.1109/eusipco.2016.7760246 · Cited by 21
“Integration may be performed using different approaches, among which data fusion offers true interaction between the modalities, by assigning symmetrical roles and simultaneously processing them [1].”
BackgroundComputer-Aided Hepatocarcinoma Diagnosis Using Multimodal Deep Learning
Menegotto, A. B., Becker, C. D. L., & Cazella, S. C. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-24097-4_1 · Cited by 19
Semisupervised charting for spectral multimodal manifold learning and alignment
Pournemat, A., Adibi, P., & Chanussot, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107645 · Cited by 18
Chatzichristos, C., Kofidis, E., Paesschen, W. V., Lathauwer, L. D., Theodoridis, S., & Huffel, S. V. (2021) · Human Brain Mapping · doi:10.1002/hbm.25717 · Cited by 34
“If “ phantom ” connections are imposed, the performance will deteriorate (Lahat et al., 2015).”
BackgroundInfluentialAssisted Learning: A Framework for Multi-Organization LearningPreprint
Xian, X., Wang, X., Ding, J., & Ghanadan, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00566 · Cited by 32
Multimodal Machine Learning for Mental Disorder Detection:A Scoping Review
Nguyen, T. T., Pham, V. H.-Q., Le, D.-T., Vu, X.-S., Deligianni, F., & Nguyen, H. D. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.10.134 · Cited by 25
Application of computer vision techniques to fermented foods: An overview
Song, Z., Li, Y., Zhao, H., Liu, X., Ding, H., Ding, Q., Ma, D., Liu, S., & Mao, J. (2025) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2025.104982 · Cited by 20
Ding, F., Zha, R., Pan, L., Chen, X. D., Martín, J. F. G., & Lan, W. (2025) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2025.105249 · Cited by 35
Characterization of Indicators for Adaptive Human-Swarm Teaming
Hussein, A., Ghignone, L., Nguyen, T., Salimi, N., Nguyen, H. T., Wang, M., & Abbass, H. A. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.745958 · Cited by 28
“When one data source becomes unavailable, reliance onmultimodality becomes a necessity (Lahat et al., 2015; Debie et al., 2019).”
BackgroundZhao, Y., Guo, S., Chen, Z., Shen, Q., Meng, Z., & Xu, H. (2022) · Applied Sciences · doi:10.3390/app12115408 · Cited by 20
Texture Analysis to Enhance Drone-Based Multi-Modal Inspection of Structures
Nooralishahi, P., Ramos, G., Pozzer, S., Ibarra‐Castanedo, C., López, F., & Maldague, X. (2022) · Drones · doi:10.3390/drones6120407 · Cited by 19
“Since the images are acquired from different sensors, the results usually differ in pixel size, perspective, and field of view [34].”
BackgroundDong, Y., Zhang, J., Lam, S., Zhang, X., Liu, A., Teng, X., Han, X., Cao, J. L., Li, H., Lee, F. K.-H., et al. (2023) · Cancers · doi:10.3390/cancers15072032 · Cited by 21
“Integration of complementary data from multiple types of datasets can lead to an intricate outcome than a simple summation of information [24].”
BackgroundLandslide Identification Method Based on the FKGRNet Model for Remote Sensing Images
Xu, B., Zhang, C., Liu, W., Huang, J., Su, Y., Yang, Y., Jiang, W., & Sun, W. (2023) · Remote Sensing · doi:10.3390/rs15133407 · Cited by 22
Yan, C., Su, Y., Liu, Y., Zhang, T., & Li, H. (2023) · Journal of Analytical Atomic Spectrometry · doi:10.1039/d3ja00214d · Cited by 18
Emotion Fusion-Sense (Emo Fu-Sense) – A novel multimodal emotion classification technique
Umair, M., Rashid, N., Khan, U. S., Hamza, A., & Iqbal, J. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106224 · Cited by 23
Wilhelm, Y., Schreier, U., Reimann, P., Mitschang, B., & Ziekow, H. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-64846-6_4 · Cited by 21
Spectroscopic fingerprinting and chemometrics for the discrimination of Italian Emmer landraces
Foschi, M., Biancolillo, A., Vellozzi, S., Marini, F., D’Archivio, A. A., & Boqué, R. (2021) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2021.104348 · Cited by 18
Attard, L., Debono, C. J., Valentino, G., & Castro, M. D. (2021) · Sensors · doi:10.3390/s21124040 · Cited by 18
“In multi-modal fusion, images coming from different sensors are combined, such as in [22,23].”
BackgroundHazarika, A., Barman, P., Talukdar, C., Dutta, L., Subaşı, A., & Bhuyan, M. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2914975 · Cited by 18
“[1] comprehensively explored the heterogeneous nature of signals obtained from various sensor modalities and necessity of the fusion technique in order to employ multiple signals of same or different modality and to extract latent information utilizing which reliable decision models can be made.”
BackgroundInfluentialAyena, J. C., Bouayed, A., Arous, M. B., Ouakrim, Y., Loulou, K., Ameyed, D., Savard, I., Kamel, L. E., & Mezghani, N. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1581285 · Cited by 26
Viswan, V., Shaffi, N., Malathy, E., Selvi, G. C., Kavitha, B. R., Abdesselam, A., Wang, S., Suganthan, P. N., Shezawi, I. A., & Mahmud, M. (2026) · Brain Informatics · doi:10.1186/s40708-025-00291-w · Cited by 24
“While the fusion of multimodal data with XAI techniques offers comprehensive insights and enhanced interpretability, it also introduces multiple practical and methodological challenges [82, 83].”
Akhonda, M. A. B. S., Gabrielson, B., Bhinge, S., Calhoun, V. D., & Adalı, T. (2021) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2021.109214 · Cited by 17
“Joint analysis, or fusion, of multiple datasets is becoming increasingly important in multiple fields and medical imaging is a very important one among those (Lahat et al., 2015; Mijović et al., 2014; Adalı et al., 2015a; Leeb et al., 2010).”
BackgroundRevisiting spatio-temporal layouts for compositional action recognition
Radevski, G., Moens, M., & Tuytelaars, T. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.278 · Cited by 32
Du, J., Han, M., Jin, L., Yan, H., & Li, X. (2020) · IEEE Access · doi:10.1109/access.2020.2973257 · Cited by 17
“In wireless communication systems, tensor modeling [15]–[19] benefits from multiple diversity.”
BackgroundMulti‐dimensional predictions of psychotic symptoms via machine learning
Taylor, J. A., Larsen, K. M., & Garrido, M. I. (2020) · Human Brain Mapping · doi:10.1002/hbm.25181 · Cited by 17
“We employed a multi-modal fusion approach which has previously been applied to neuroimaging data for combining different types of structural and functional images (Calhoun & Adali, 2009; Calhoun & Sui, 2016; Lahat, Adali, & Jutten, 2015).”
MethodMultidimensional harmonic retrieval based on Vandermonde tensor train
Zniyed, Y., Boyer, R., de Almeida, A. L. F., & Favier, G. (2019) · Signal Processing · doi:10.1016/j.sigpro.2019.05.007 · Cited by 17
“Moreover, it is likely that the joint exploitation of multi-diversity/modality sensing technologies for data fusion [12, 13, 14] further increases the data tensor order.”
BackgroundAndrade, D. F., de Almeida, E., de Carvalho, H. W. P., Pereira‐Filho, E. R., & Amarasiriwardena, D. D. (2020) · Talanta · doi:10.1016/j.talanta.2020.122025 · Cited by 18
Liu, Y., Ma, L., Dou, Y., Zhu, Z., Ma, L., & Liu, Z. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104088 · Cited by 17
Gezimati, M., & Singh, G. K. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3547084 · Cited by 16
“The application of deep learning for processing and fusion of multisource medical information will enable the development of connected and smart systems for next-generation health-care [77], [78], [79], [80], [81], [82], [83], [84], [85], [86], [87], [88], [89], [90], [91], [92], [93].”
Automatic fused multimodal deep learning for plant identification
Lapkovskis, A., Нефедова, Н. А., & Beikmohammadi, A. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1616020 · Cited by 15
“In this regard, the study first identifies the inadequacy of unimodal DL for plant classification, prompting an exploration of multimodal learning, particularly through multimodal fusion.”
Multimodal Wearable Biosensing Meets Multidomain AI: A Pathway to Decentralized Healthcare
Liu, C., Fan, H., Kim, M., Zhou, T., Yang, P., Zhao, L., Wang, Y., Che, Z., Liu, C., Li, B., et al. (2026) · Advanced Science · doi:10.1002/advs.202522900 · Cited by 31
CAMul: Calibrated and Accurate Multi-view Time-Series Forecasting
Kamarthi, H., Kong, L., Rodríguez, A., Zhang, C., & Prakash, B. A. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512037 · Cited by 34
“In order to integrate these representations, most methods employ simple summation or concatenation methods [31, 37, 50, 54, 56] either at the inital layers of model (early fusion) or last layers (late fusion) [13, 32, 39, 53].”
MethodIndependent vector analysis: Model, applications, challenges
Luo, Z. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109376 · Cited by 16
Predictive Control of a Human–in–the–Loop Network System Considering Operator Comfort Requirements
Sadowska, A. D., Maestre, J. M., Kassing, R., van Overloop, P. J., & Schutter, B. D. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3253962 · Cited by 16
“In particular, the observer must fuse the measurements obtained by the operators in unevenly spaced sampling steps (see [43]).”
BackgroundLiu, Y., Ding, X., Chi, M., Wu, J., & Ma, L. (2023) · Information Technology & Tourism · doi:10.1007/s40558-023-00280-x · Cited by 20
“Automatic feature extraction is based on deep learning but lacks features to capture multimodal data from different perspectives (Lahat et al. 2015).”
Deep learning-based geological map generation using geological routes
Li, C., Li, F., Liu, C., Tang, Z., Fu, S., Lin, M., Lv, X., Liu, S., & Liu, Y. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114214 · Cited by 20
Deep learning of multimodal networks with topological regularization for drug repositioning
Ohnuki, Y., Akiyama, M., & Sakakibara, Y. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-024-00897-y · Cited by 19
“…methods like Iwata et al.’s approach [27] using gene expression profile data from LINCS [28] and Wang et al.’s DeepDRK [29], which integrates multi-omics data for predicting cancer cell drug responses, demonstrate the value of extracting and integrating modality-specific information [30].”
MethodAsish, S. M., Kulshreshth, A. K., Borst, C. W., & Sutradhar, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456207 · Cited by 25
“There are three techniques used for multimodal data fusion: (1) Early fusion or fusion at the data level, (2) Decision-level fusion or late fusion, and (3) intermediate fusion [24].”
MethodUnveiling and advancing grassland degradation research using a BERTopic modelling approach
Li, T., Cui, L., Wu, Y., Pandey, R., Liu, H., Dong, J., Wang, W., Xu, Z. H., Song, X., Hao, Y., et al. (2024) · Journal of Integrative Agriculture · doi:10.1016/j.jia.2024.11.008 · Cited by 19
Local Canonical Correlation Analysis for Nonlinear Common Variables Discovery
Yair, O., & Talmon, R. (2016) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2628348 · Cited by 15
“Yet, the ability to gain a deep understanding of the true state of the system from the overwhelming amount of collected data from multiple (usually different) sources of information is challenging; it calls for the development of new technologies and novel ways to observe the system of interest and to fuse the availab…”
BackgroundFusion of Deep Neural Networks for Activity Recognition: A Regular Vine Copula Based Approach
Zhang, S., Geng, B., Varshney, P. K., & Rangaswamy, M. (2019) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion43075.2019.9011282 · Cited by 19
“Thus, the data compatibility issues among multiple modalities can be avoided [4], [5].”
BackgroundCombining MWL and MSG SEVIRI Satellite Signals for Rainfall Detection and Estimation
Kumah, K. K., Hoedjes, J. C. B., David, N., Maathuis, B. H. P., Gao, H. O., & Su, Z. (2020) · Atmosphere · doi:10.3390/atmos11090884 · Cited by 16
“In this context, data fusion [101,102] and statistical machine learning offer techniques to explore relationships (linear and non-linear) between multiple large input variables.”
BackgroundMulti-Scale Temporal Fusion Network for Real-Time Multimodal Emotion Recognition in IoT Environments
Yoon, S., & Kim, B. (2025) · Sensors · doi:10.3390/s25165066 · Cited by 17
Sensing in Smart Cities: A Multimodal Machine Learning Perspective
Sadiq, T., & Omlin, C. W. P. (2025) · Smart Cities · doi:10.3390/smartcities9010003 · Cited by 24
“Data alignment ensures coherent integration across modalities, while fusion strategies such as early, late, and hybrid approaches offer trade-offs in accuracy, robustness, and cost [10,11].”
InfluentialMafata, M., Brand, J., Kidd, M., Medvedovici, A. V., & Buica, A. (2022) · Beverages · doi:10.3390/beverages8040066 · Cited by 15
Liu, P., Zhang, Y., Xiong, Z., Wang, Y., & Qing, L. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1001885 · Cited by 19
“accurate than traditional single-modal emotional recognition, because information if integrated from different modalities (Lahat et al., 2015; Yoon et al., 2015; Wang et al., 2019).”
BackgroundMethodUnrestricted Anchor Graph Based GCN for Incomplete Multi-View Clustering
Zhao, L., Wang, Z., Yuan, Y., & Ding, F. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096284 · Cited by 15
“With the development of data acquisition equipment, real data often appears in multiple ways or comes from multiple sources [1][3].”
BackgroundAdvancing classroom fatigue recognition: A multimodal fusion approach using self-attention mechanism
Cao, L., Wang, W., Dong, Y., & Fan, C. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105756 · Cited by 17
Gao, P., Liu, T., Liu, J., Lu, B., & Zheng, W. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447194 · Cited by 25
Social media-based multi-modal ensemble framework for forecasting soybean futures price
An, W., Wang, L., & Zeng, Y. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109439 · Cited by 15
Fu, W.-S., Zhang, J., Zhang, D., Li, T., Lan, M., & Liu, N. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331241313126 · Cited by 22
“Compared to data obtained from a single modality, multimodal data can leverage complementarity (Lahat et al., 2015), thereby offering greater potential for exploring human cognitive ability (Mangaroska et al., 2021).”
Domain adaptive representation learning for facial action unit recognition
Sankaran, N., Mohan, D. D., Lakshminarayana, N. N., Setlur, S., & Govindaraju, V. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.107127 · Cited by 17
Decker, K. T., & Borghetti, B. J. (2022) · Remote Sensing · doi:10.3390/rs14092113 · Cited by 15
“The motivation behind multimodal remote sensing is that a single acquisition modality rarely provides a complete understanding of the phenomenon under study [2].”
BackgroundΜεδίτσκος, Γ., Kontopoulos, E., Vrochidis, S., & Kompatsiaris, I. Y. (2019) · Expert Systems · doi:10.1111/exsy.12378 · Cited by 18
Multimodal Named Data Discovery With Interest Broadcast Suppression for Vehicular CPS
Bouk, S. H., Ahmed, S. H., Eun, Y., & Park, K. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.2971479 · Cited by 15
“The term modality of the data is defined in [5] as; “ each acquisition of data about the same phenomenon generated by different types of sensors or detectors, at different conditions, in multiple subjects or experiments, among others ”.”
BackgroundLearning 2-in-1: Towards Integrated EEG-fMRI-NeurofeedbackPreprint
Perronnet, L., Lécuyer, A., Mano, M., Fleury, M., Lioi, G., Cury, C., Clerc, M., Lotte, F., & Barillot, C. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/397729 · Cited by 20
Mapping of multimodality data for manufacturing analyses in automated fiber placement
Brasington, A., Halbritter, J., Godbold, M., Kirkpatrick, M. B., Sacco, C., & Harik, R. (2023) · Composites Part B Engineering · doi:10.1016/j.compositesb.2023.110881 · Cited by 14
Natraj, S., Kojovic, N., Maillart, T., & Schaer, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308388 · Cited by 22
Narong, T. N., Zachko, Z. N., Torrisi, S. B., & Billinge, S. J. L. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01589-3 · Cited by 28
“This is common in scientific literature where many characterizations are done on the same sample, but a greater challenge is to use multiple experiments to refine parameters from a single model, a process known as complex modeling [6] or multimodal data fusion [7].”
Afifi, I., Elgendy, M. A., Abdelfatah, M., & El–Sappagh, S. (2025) · Brain Informatics · doi:10.1186/s40708-025-00286-7 · Cited by 17
Multimodal Model to Predict Tissue-to-Blood Partition Coefficients of Chemicals in Mammals and Fish
Zhang, S., Wang, Z., Chen, J., Luo, X., & Mai, B. (2024) · Environmental Science & Technology · doi:10.1021/acs.est.3c08016 · Cited by 16
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.