Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Tensor Decomposition for Signal Processing and Machine Learning
Sidiropoulos, N. D., De Lathauwer, L., Fu, X., Huang, K., Papalexakis, E. E., & Faloutsos, C. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2690524 · Cited by 1,617
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
“For the purposes of our review, we adopt the definition provided by Lahat et al. [2], where we consider phenomena or systems that are observed using multiple sensors and each sensor output can be termed a modality associated with a single data set.”
MethodA survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821
“The reasonable fusion of these multimodal data can help us better understand the event of interest, especially when one modality is incomplete (Khaleghi, Khamis, Karray, & Razavi, 2013; Lahat, Adali, & Jutten, 2015).”
BackgroundGeneralizing from a Few Examples: A Survey on Few-Shot Learning
Wang, Y., Yao, Q., Kwok, J. T.-Y., & Ni, L. M.-S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05046 · Cited by 772
“in E to make the learning of the supervised targetT feasible. Existing FSL methods use prior knowledge from a single modality, which is the output of a sensor for observing some phenomena or systems [61], such as images, texts, and videos. However, multi-modality prior knowledge can be used to provide a prior knowledg…”
MethodDeep learning in multimodal remote sensing data fusion: A comprehensive review
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102926 · Cited by 630
A Survey of Multi-View Representation Learning
Li, Y., Yang, M., & Zhang, Z. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2872063 · Cited by 524
“Recently, several related surveys [4], [24], [25], [26], [27] of”
Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision
Queralta, J. P., Taipalmaa, J., Can Pullinen, B., Sarker, V. K., Nguyen Gia, T., Tenhunen, H., Gabbouj, M., Raitoharju, J., & Westerlund, T. (2020) · IEEE Access · doi:10.1109/access.2020.3030190 · Cited by 526
“…parallel data integration with high-level decision making, (b) sequential processing of modalities when different modalities have difference conficende or quality levels, (c) true fusion with high-level features or with multivariate features, and (d) true fusion with minimal reduction [208], [209].”
BackgroundMultimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level) 30 .”
BackgroundLiu, W., Qiu, J.-L., Zheng, W.-L., & Lu, B.-L. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3071170 · Cited by 452
“image segmentation, and video classification [36].”
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456
Fan, C., Zhang, C., Yahja, A., & Mostafavi, A. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.102049 · Cited by 434
Data clustering: application and trends
Oyewole, G. J., & Thopil, G. A. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10325-y · Cited by 312
“Lahat et al. (2015) and Li and Wang (2021) discussed the complementary and diverse attributes of multi-modal data (e.g. the same data from text, image audio, and video) and also provided similar challenges of complexity resulting from the fusion of such data.”
BackgroundA comprehensive survey on multimodal medical signals fusion for smart healthcare systems
Muhammad, G., Alshehri, F., Karray, F., Saddik, A. E., Alsulaiman, M., & Falk, T. H. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.007 · Cited by 287
Deep learning based multimodal biomedical data fusion: an overview and comparative review
Duan, J., Xiong, J., Li, Y., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102536 · Cited by 284
Emotion Based Music Recommendation System Using Wearable Physiological Sensors
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2018) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2018.2844736 · Cited by 280
“[22] defines data fusion as the analysis of several datasets such that different datasets can interact and inform each other.”
BackgroundOuhami, M., Hafiane, A., Es-Saady, Y., El Hajji, M., & Canals, R. (2021) · Remote Sensing · doi:10.3390/rs13132486 · Cited by 264
“Multimodal data fusion is a challenging task because it deals with a combination of heterogenous data from different modalities (images, signals, times series, etc.) [113].”
BackgroundDeep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261
Task-Oriented Multi-User Semantic Communications for VQA
Xie, H., Qin, Z., & Li, G. Y. (2022) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2021.3136045 · Cited by 243
“Compared with the situations with single-modal data, multimodal data can provide more information for intelligent tasks, introduce new degrees of freedom, and improve performance of intelligent tasks [10], [11].”
BackgroundWang, S., Celebi, M. E., Zhang, Y.-D., Yu, X., Lu, S., Yao, X., Zhou, Q., Miguel, M.-G., Tian, Y., Gorriz, J. M., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.001 · Cited by 241
From signals to knowledge: A conceptual model for multimodal learning analytics
Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12288 · Cited by 237
Lv, Z., Huang, H., Li, X., Zhao, M., Benediktsson, J. A., Sun, W., & Falco, N. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3219376 · Cited by 232
“Then, the single image is more informative and accurate than any single temporal image [67], [68], [69], [70].”
BackgroundOgier du Terrail, J., Leopold, A., Joly, C., Béguier, C., Andreux, M., Maussion, C., Schmauch, B., Tramel, E. W., Bendjebbar, E., Zaslavskiy, M., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02155-w · Cited by 217
“We note that this strategy might be suboptimal as there exists a multitude of possible fusion methods for multimodal data 90,91 .”
BackgroundModern views of machine learning for precision psychiatry
Chen, Z. S., Kulkarni, P. (., Galatzer-Levy, I. R., Bigio, B., Nasca, C., & Zhang, Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100602 · Cited by 210
“Multimodal data fusion techniques present a framework to infer information how di ff erent data modalities interact and can be integrated for improved disease prediction [254, 255, 43].”
BackgroundRice-Fusion: A Multimodality Data Fusion Framework for Rice Disease Diagnosis
Patil, R. R., & Kumar, S. (2022) · IEEE Access · doi:10.1109/access.2022.3140815 · Cited by 208
Artificial Intelligence and Biosensors in Healthcare and Its Clinical Relevance: A Review
(2023) · doi:10.1109/access.2023.3285596 · Cited by 205
“Biological data is usually diverse, and a complete understanding of a complex biological system may require an ensemble of related data sets to extract hidden data dependencies [70].”
BackgroundA Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
“Multimodality extracts and combines vital information from the respective modality source and solves a given problem with richer representation and performance than the individual modalities [9].”
BackgroundMultimodal data as a means to understand the learning experience
Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.02.003 · Cited by 202
“The seamless integration and combination of different apparatuses, as well as the harmonization and sense-making of multimodal data-streams to support the learning experience, is an extremely challenging process (Lahat et al., 2015).”
BackgroundYOLOrs: Object Detection in Multimodal Remote Sensing Imagery
Sharma, M., Dhanaraj, M., Karnam, S., Chachlakis, D. G., Ptucha, R., Markopoulos, P. P., & Saber, E. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3041316 · Cited by 193
“Fusion of data collected across multiple modalities (multi-modal data) can allow for enhanced inference [36]–[39].”
BackgroundReview of Data Fusion Methods for Real-Time and Multi-Sensor Traffic Flow Analysis
Kashinath, S. A., Mostafa, S. A., Mustapha, A., Mahdin, H., Lim, D., Mahmoud, M. A., Mohammed, M. A., Al-Rimy, B. A. S., Fudzee, M. F. M., & Yang, T. J. (2021) · IEEE Access · doi:10.1109/access.2021.3069770 · Cited by 192
“Identifying critical data to fit the research purpose is a real challenge [60], [150].”
BackgroundEmerging Trends in Machine Learning: A Polymer Perspective
Martin, T. B., & Audus, D. J. (2023) · ACS Polymers Au · doi:10.1021/acspolymersau.2c00053 · Cited by 187
Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review
Debie, E., Fernandez Rojas, R., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M., & Abbass, H. A. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2939399 · Cited by 184
Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems
Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-019-00505-7 · Cited by 182
Touati, R., Mignotte, M., & Dahmane, M. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2933747 · Cited by 179
“Finally, let us add that the different imaging modalities may be complementary and this sensor fusion technique could potentially be exploited (not only in Geoscience imaging [7]) for further improving the change detection and analysis of land surfaces with complex properties subject to extreme conditions (e.”
BackgroundHu, J.-W., Zheng, B.-Y., Wang, C., Zhao, C.-H., Hou, X.-L., Pan, Q., & Xu, Z. (2020) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1900518 · Cited by 175
Imagining the future of bioimage analysis
Meijering, E., Carpenter, A. E., Peng, H., Hamprecht, F. A., & Olivo-Marin, J.-C. (2016) · Nature Biotechnology · doi:10.1038/nbt.3722 · Cited by 175
“Although the development of methods for single-source data analysis remains important, methods that properly account for the links between data sets from multiple sources have the potential to achieve gains that go far beyond those possible when each data set is analyzed separately 47 .”
MethodCeolini, E., Frenkel, C., Shrestha, S. B., Taverni, G., Khacef, L., Payvand, M., & Donati, E. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00637 · Cited by 172
“Multi-sensor data fusion is a direct consequence of the wellaccepted paradigm that certain natural processes and phenomena are expressed under completely different physical guises (Lahat et al., 2015).”
BackgroundInfluentialZhu, J., Zhang, X., Wang, R., Wang, M., Chen, P., Cheng, L., Wu, Z., Wang, Y., Liu, Q., & Liu, M. (2022) · Advanced Materials · doi:10.1002/adma.202200481 · Cited by 173
Fu, Y., Li, C., Yu, F. R., Luan, T. H., & Zhang, Y. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3083927 · Cited by 169
Sun, Y., Lei, L., Li, X., Tan, X., & Kuang, G. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3053571 · Cited by 166
CentralNet: A Multilayer Approach for Multimodal Fusion
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11024-6_44 · Cited by 166
Zhao, Y., Zhang, W., & Liu, X. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111362 · Cited by 163
Graph neural networks for financial fraud detection: a review
Cheng, D., Zou, Y., Xiang, S., & Jiang, C. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40474-y · Cited by 227
“However, this diversity also o ff ers significant opportunities; by combining these di ff erent types of data, a more comprehensive and nuanced view of financial activities can be obtained, leading to more e ff ective fraud detection strategies [43,91].”
Edge Video Analytics for Public Safety: A Review
Zhang, Q., Sun, H., Wu, X., & Zhong, H. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2925910 · Cited by 156
“[149] D.”
BackgroundVision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets
Charan, G., Osman, T., Hredzak, A., Thawdar, N., & Alkhateeb, A. (2022) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc51071.2022.9771835 · Cited by 153
Chemometrics in analytical chemistry—part I: history, experimental design and data analysis tools
Brereton, R. G., Jansen, J. J., Lopes, J. A., Marini, F., Pomerantsev, A. L., Rodionova, O. Y., Roger, J., Walczak, B., & Tauler, R. (2017) · Analytical and Bioanalytical Chemistry · doi:10.1007/s00216-017-0517-1 · Cited by 158
A systematic review of data fusion techniques for optimized structural health monitoring
Hassani, S., Dackermann, U., Mousavi, M., & Li, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102136 · Cited by 182
Ahmed, A., Xi, R., Hou, M., Shah, S. A., & Hameed, S. (2023) · IEEE Access · doi:10.1109/access.2023.3323574 · Cited by 198
“Since the inception of the fusion concept several methods [33], [56], [96], [137], [161], [193], [194] have been proposed to deal with the fusion of multiple data types for such we prepared a comprehensive taxonomy view for fusion handling techniques as presented in Figure 12.”
BackgroundMethodMachine Learning Techniques in Adaptive and Personalized Systems for Health and Wellness
Oyebode, O., Fowles, J. R., Steeves, D., & Orji, R. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2089085 · Cited by 190
“Multimodality refers to an approach in which data is obtained or acquired from more than one modality or source (Lahat et al., 2015).”
BackgroundMulti-modality approaches for medical support systems: A systematic review of the last decade
Salvi, M., Loh, H. W., Seoni, S., Barua, P. D., García, S., Molinari, F., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102134 · Cited by 144
“However, developing approaches to fuse heterogeneous, high-dimensional multimodal data poses unique technical challenges due to data incompatibility [12].”
BackgroundCheng, J., Deng, C., Su, Y., An, Z., & Wang, Q. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.03.012 · Cited by 175
Multi-task learning for dangerous object detection in autonomous driving
Chen, Y., Zhao, D., Lv, L., & Zhang, Q. (2017) · Information Sciences · doi:10.1016/j.ins.2017.08.035 · Cited by 141
Liu, T., Huang, J., Liao, T., Pu, R., Liu, S., & Peng, Y. (2022) · IRBM · doi:10.1016/j.irbm.2020.12.002 · Cited by 137
Anchors Bring Ease: An Embarrassingly Simple Approach to Partial Multi-View Clustering
Guo, J. F., & Ye, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301118 · Cited by 138
“Multi-view data has already been widely studied in the past few years (Lahat, Adali, and Jutten 2015; Bai et al. 2016; Bai et al. 2017).”
MethodNakach, F.-Z., Idri, A., & Göçeri, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10984-z · Cited by 168
A review on data fusion in multimodal learning analytics and educational data mining
Chango, W. G., Lara, J. A., Cerezo, R., & Romero, C. (2022) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1458 · Cited by 166
Qi, W., & Aliverti, A. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2963048 · Cited by 139
“denoising and deep learning (DL) [1].”
BackgroundSensor Fusion and Smart Sensor in Sports and Biomedical Applications
Mendes, J. J. A., Vieira, M., Pires, M., & Stevan, S. L. (2016) · Sensors · doi:10.3390/s16101569 · Cited by 138
“Using space-time signals obtained by encephalography (EEG) and magnetoencephalography (MEG), adding functional magnetic resonance imaging (fMRI) enables the analysis of how different parts of the brain contribute to activities related to perception and cognition [106].”
BackgroundWang, M., Tan, K., Jia, X., Wang, X., & Chen, Y. (2020) · Remote Sensing · doi:10.3390/rs12020205 · Cited by 135
Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots
Silva, D., Roche, J., & KONDOZ, A. M. (2018) · Sensors · doi:10.3390/s18082730 · Cited by 128
“Lahat et al. [20], identified several challenges that are imposed by multimodal data.”
BackgroundMethodInfluentialData fusion in cyber-physical-social systems: State-of-the-art and perspectives
Wang, P., Yang, L. T., Li, J., Chen, J., & Hu, S. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.11.002 · Cited by 127
Rolinger, L., Rüdt, M., & Hubbuch, J. (2020) · Analytical and Bioanalytical Chemistry · doi:10.1007/s00216-020-02407-z · Cited by 126
“When multiple or multimodal sensors are involved in a measurement, different data fusion strategies can be utilized for model building [116].”
BackgroundGas Detection and Identification Using Multimodal Artificial Intelligence Based Sensor Fusion
Narkhede, P., Walambe, R. A., Mandaokar, S., Chandel, P., Kotecha, K. V., & Ghinea, G. (2021) · Applied System Innovation · doi:10.3390/asi4010003 · Cited by 131
“Early fusion combines the raw data or the features extracted from the raw data [29].”
BackgroundLong, S., Tan, J., Mao, B., Tang, F., Li, Y., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3526606 · Cited by 165
Hussain, E., Bhaskar, A., & Chung, E. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103044 · Cited by 118
İlçi, V., & Toth, C. K. (2020) · Sensors · doi:10.3390/s20030899 · Cited by 137
A Survey of Collaborative Machine Learning Using 5G Vehicular Communications
Balkus, S. V., Wang, H., Cornet, B., Mahabal, C., Ngo, H. T., & Fang, H. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3149714 · Cited by 129
Systematically Understanding the Cyber Attack Business: A Survey
Huang, K., Siegel, M., & Madnick, S. (2019) · ACM Computing Surveys · doi:10.1145/3199674 · Cited by 112
“where MDF refers to the multimodal data fusion [82] that can be used to manage and analyze the collected data.”
BackgroundSun, Y., Zhang, X., Xin, Q., & Huang, J. (2018) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2018.06.005 · Cited by 117
Kolar, P., Benavidez, P. J., & Jamshidi, M. M. (2020) · Sensors · doi:10.3390/s20082180 · Cited by 125
“The effective temporal, spatial and geometrical alignment of this suite of heterogeneous sensors and the diversity utilization is called sensor data fusion [38,39].”
BackgroundInfluentialAI Empowered Wireless Communications: From Bits to Semantics
Qin, Z., Liang, L., Wang, Z., Jin, S., Tao, X., Tong, W., & Li, G. Y. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3437730 · Cited by 125
“In that work, LSTM and CNN-based semantic-channel coding schemes were used for text and image transmission, respectively, and then, the receiver predicted the answers by fusing different correlated semantics together [155].”
MethodA survey of multimodal deep generative models
Suzuki, M., & Matsuo, Y. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2035253 · Cited by 109
“In a multimodal learning framework, the term modality originally referred to each scheme and situation in which data are perceived [5, 45], i.”
A Fractal Projection and Markovian Segmentation-Based Approach for Multimodal Change Detection
Mignotte, M. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2986239 · Cited by 107
“…may be complementary and by this fact, this imaging modality fusion technique could potentially be exploited (not only in geoscience imaging [5]) for further improving the change detection and analysis of land surfaces with complex properties subject to extreme conditions (e.g.,…”
BackgroundZubair, M., Hussain, M., Albashrawi, M. A., Bendechache, M., & Owais, M. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.109014 · Cited by 169
Few-shot Learning: A SurveyPreprint
Wang, Y., & Yao, Q. (2019) · arXiv (Cornell University) · Cited by 112
Multimodal Data Fusion Using Source Separation: Application to Medical Imaging
Adalı, T., Levin‐Schwartz, Y., & Calhoun, V. D. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2461601 · Cited by 105
Wang, P.-Y., Chen, C.-T., Su, J.-W., Wang, T.-Y., & Huang, S.-H. (2021) · IEEE Access · doi:10.1109/access.2021.3071306 · Cited by 105
“In addition, multimodal data fusion across various domains has attracted considerable scholarly attention in recent years [30].”
BackgroundHGMF: Heterogeneous Graph-based Fusion for Multimodal Data with Incompleteness
Chen, J., & Zhang, A. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403182 · Cited by 144
“Data across multiple sources can provide complementary information that reveals the fundamental characteristics of real-world subjects and phenomenons [11].”
BackgroundMethodInfluentialMultimodal hyperspectral remote sensing: an overview and perspective
Gu, Y., Liu, T., Gao, G., Ren, G., Ma, Y., Chanussot, J., & Jia, X. (2021) · Science China Information Sciences · doi:10.1007/s11432-020-3084-1 · Cited by 110
Barua, A., Ahmed, M. U., & Begum, S. (2023) · IEEE Access · doi:10.1109/access.2023.3243854 · Cited by 123
“Articles [6], [23], [24], and [25] present an overview of the methods of fusion with challenges and prospects.”
BackgroundMachine Learning as a Tool for Hypothesis Generation
Ludwig, J., & Mullainathan, S. (2024) · The Quarterly Journal of Economics · doi:10.1093/qje/qjad055 · Cited by 185
Multimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101
“However, information from these images can be complemented by higher resolution medical images that do not have a time component, such as structural MRI and diffusion tensor imaging (Lahat et al. , 2015).”
BackgroundInfluentialA review of image fusion: Methods, applications and performance metrics
Singh, S., Singh, H., Bueno, G., Déniz, Ó., Singh, S., Monga, H., Hrisheekesha, P. N., & Pedraza, A. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2023.104020 · Cited by 107
Common and distinct components in data fusion
Smilde, A. K., Måge, I., Næs, T., Hankemeier, T., Lips, M. A., Kiers, H. A. L., Acar, E., & Bro, R. (2017) · Journal of Chemometrics · doi:10.1002/cem.2900 · Cited by 101
“For 2 blocks, it derives directly a decomposition according to X1 = TCPT1C + T1DP T 1D + E1 = X1C + X1D + E1, X2 = TCPT2C + T2DP T 2D + E2 = X2C + X2D + E2, which fits directly in our framework and Equation 9.”
InfluentialHuang, L., Ruan, S., Decazes, P., & Denœux, T. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102648 · Cited by 137
Ni, X., Ouyang, W., Jeong, H., Kim, J., Tzavelis, A., Mirzazadeh, A., Wu, C., Lee, J. Y., Keller, M., Mummidisetty, C. K., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2026610118 · Cited by 101
“The results offer many possibilities in data fusion for precision healthcare, including but not constrained to COVID-19 (19, 57, 58).”
A systematic review of intermediate fusion in multimodal deep learning for biomedical applications
Guarrasi, V., Aksu, F., Caruso, C. M., Di Feola, F., Rofena, A., Ruffini, F., & Soda, P. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105509 · Cited by 95
Multimodal Deep Learning Framework for Mental Disorder Recognition
Zhang, Z., Lin, W., Liu, M., & Mahmoud, M. (2020) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg47880.2020.00033 · Cited by 118
“[6] - that cannot be obtained from any of other modalities.”
BackgroundMultimodality in meta-learning: A comprehensive survey
Ma, Y., Zhao, S., Wang, W., Li, Y., & King, I. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108976 · Cited by 99
Chango, W. G., Cerezo, R., & Romero, C. (2020) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2020.106908 · Cited by 117
“Multimodal data fusion is the combination/integration of data from different/several sources/modalities/contexts in order to obtaining a better understanding of the teaching-learning process [11].”
BackgroundA systematic review of neurophysiological sensing for the assessment of acute pain
Rojas, R. F., Brown, N. A. T., Waddington, G. S., & Goecke, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00810-1 · Cited by 122
Odusami, M., Maskeliūnas, R., Damaševičius, R., & Misra, S. (2023) · Cognitive Neurodynamics · doi:10.1007/s11571-023-09993-5 · Cited by 113
“Although decision-level-fusion aimed to support accepted interpretations and comprehen-sions, the limitation resides in the requirement for prior knowledge-making algorithms to be very complex (Jin et al. 2020; Lahat et al. 2015).”
BackgroundData fusion approaches in spectroscopic characterization and classification of PDO wine vinegars
Ríos‐Reina, R., Callejón, R. M., Savorani, F., Amigo, J. M., & Cocchi, M. (2019) · Talanta · doi:10.1016/j.talanta.2019.01.100 · Cited by 85
A review of cancer data fusion methods based on deep learning
Zhao, Y., Li, X., Zhou, C., Peng, H., Zheng, Z., Chen, J., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102361 · Cited by 95
Salau, B. A., Rawal, A., & Rawat, D. B. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3170449 · Cited by 90
“[120], [121] E. Security and Primacy in AI enabled Wireless IoT/CPS AI security is also a vital challenge facing IoT/CPS.”
Multimodal semantic communication accelerated bidirectional caching for 6G MEC
Wang, C., Yu, X., Xu, L., Wang, Z., & Wang, W. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.10.036 · Cited by 90
BIM and IoT data fusion: The data process model perspective
Huang, X., Liu, Y., Huang, L., Onstein, E., & Merschbrock, C. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104792 · Cited by 100
Learning the geometry of common latent variables using alternating-diffusion
Lederman, R. R., & Talmon, R. (2015) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2015.09.002 · Cited by 79
Multimodal Neuroimaging: Basic Concepts and Classification of Neuropsychiatric Diseases
Tülay, E. E., Metin, B., Tarhan, N., & Arıkan, M. K. (2018) · Clinical EEG and Neuroscience · doi:10.1177/1550059418782093 · Cited by 92
Multi-Modality Sensing and Data Fusion for Multi-Vehicle Detection
Roy, D., Li, Y., Jian, T., Tian, P., Chowdhury, K. R., & Ioannidis, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3145663 · Cited by 97
“Additionally, fusion among diverse modalities has its own challenges, such as ensuring diversity among different modalities or the availability of multi-domain sensor data in a time-synchronized manner (see, e.g., Lahat et al. [24], [25]).”
BackgroundDeep Learning in Remote Sensing Image Fusion: Methods, protocols, data, and future perspectives
Vivone, G., Deng, L.-J., Deng, S., Hong, D., Jiang, M., Li, C., Li, W., Shen, H., Wu, X., Xiao, J.-L., et al. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3495516 · Cited by 124
“A fundamental characteristic of multimodality is its complementarity, meaning that each modality contributes unique value to the overall understanding that cannot be obtained from any other modality [2].”
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.