Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 101–200 of 1,001
Deep Learning in Multimodal Fusion for Sustainable Plant Care: A Comprehensive Review
Yang, Z., Li, Y., Wang, R.-F., Hu, P., & Su, W. (2025) · Sustainability · doi:10.3390/su17125255 · Cited by 115
Sun, H., Chen, Y., & Lin, L. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233070 · Cited by 93
“Inmultimodal learning, it is proven that the information learned from different modalities should be integrated to describe the semantics more holistically [3].”
BackgroundZhang, Z., Zhang, L., Wu, J., & Guo, W. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3404506 · Cited by 110
Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities
Abdali, S., Shaham, S., & Krishnamachari, B. (2024) · ACM Computing Surveys · doi:10.1145/3697349 · Cited by 145
“In what follows, we summarize and categorize the existing works into the aforementioned categories.”
Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors
Bernal, E. A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., & Bala, R. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2726187 · Cited by 74
“For more extensive treatments on multimodal data fusion, we refer the reader to [15] and [58].”
BackgroundA Novel Multimodal Approach for Hybrid Brain–Computer Interface
Sun, Z., Huang, Z., Duan, F., & Liu, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2994226 · Cited by 80
“A tensor is a higher-order array that represents signals from different types of sensors [49]–[51].”
BackgroundDeep Learning Approach for Detecting Work-Related Stress Using Multimodal Signals
Seo, W., Kim, N., Park, C., & Park, S. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3170915 · Cited by 81
Hunyadi, B., Dupont, P. J., Paesschen, W. V., & Huffel, S. V. (2016) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1197 · Cited by 74
Vehicle Tracking Using Surveillance With Multimodal Data Fusion
Zhang, Y., Song, B., Du, X., & Guizani, M. (2018) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2787101 · Cited by 73
“data fusion [6] to combine the knowledge from different aspects, so that a more comprehensive and accurate result can”
BackgroundAI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71
Optimal estimation of low-rank factors via feature level data fusion of multiplex signal systems
Li, H., Wang, Z., Pei, J., Cao, J., & Shi, Y. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3015914 · Cited by 74
Heterogeneous Network Access and Fusion in Smart Factory: A Survey
Xia, D., Jiang, C. S., Wan, J., Jin, J., Leung, V. C. M., & Martínez-García, M. (2022) · ACM Computing Surveys · doi:10.1145/3530815 · Cited by 78
“Therefore, most manufacturing data cannot be used directly, and it is necessary to carry out data cleaning and fusion within the edge of smart factory networks [130].”
BackgroundEmploying Multimodal Machine Learning for Stress Detection
Walambe, R. A., Nayak, P., Bhardwaj, A., & Kotecha, K. V. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/9356452 · Cited by 86
RetractedLuo, X., Jia, N., Ouyang, E., & Fang, Z. (2024) · Strategic Management Journal · doi:10.1002/smj.3597 · Cited by 82
Bühne, H. S. T., & Pettorelli, N. (2017) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12942 · Cited by 74
Multimodal Emotion Recognition Using Deep Canonical Correlation AnalysisPreprint
Liu, W., Qiu, J., Zheng, W., & Lu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05349 · Cited by 120
“We select the output dimension from the set [5, 10, 15, 20, 25, 30] and the EEG weight α1 in [0, 0.”
BackgroundMethodA Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem
Pereira, L. M., Salazar, A., & Vergara, L. (2023) · IEEE Access · doi:10.1109/access.2023.3296098 · Cited by 69
“Although some approaches are general [1], most of the reviews have different perspectives: modality fusion [2], [3], [4], sensor fusion [5], [6] or classifier combination [7], [8].”
BackgroundXu, Y., Zhang, J., & Wang, Y. (2022) · Food Chemistry · doi:10.1016/j.foodchem.2022.133939 · Cited by 72
A Fused CP Factorization Method for Incomplete Tensors
Wu, Y., Tan, H., Li, Y., Zhang, J., & Chen, X. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2851612 · Cited by 63
“further contribute to interpretability, robustness, uniqueness, and other desired properties [16].”
BackgroundInfluentialA New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow MMLA
Worsley, M. B., Martínez‐Maldonado, R., & D’Angelo, C. M. (2021) · Journal of Learning Analytics · doi:10.18608/jla.2021.7361 · Cited by 79
“This type of analysis takes a fairly high level approach, in contrast to lower-level data fusion approaches where multimodal data are combined before passing the data through a classifier (see Lahat, Adali, & Jutten (2015) for a more detailed discussion of multimodal fusion approaches).”
Abnormal event detection for video surveillance using an enhanced two-stream fusion method
Yang, Y., Fu, Z., & Naqvi, S. M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126561 · Cited by 73
SNSJam: Road traffic analysis and prediction by fusing data from multiple social networks
Alkouz, B., & Aghbari, Z. A. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102139 · Cited by 63
Mulder, D., Aarts, E., Vásquez, A. A., & Bloemendaal, M. (2023) · Molecular Psychiatry · doi:10.1038/s41380-023-02146-4 · Cited by 77
“…assess if and how an intervention affects the gut microbiota or brain connectivity, which microbes are causally involved, if intervention-induced changes in the microbiota result in altered brain connectivity, and – importantly – whether this is also re fl ected in altered cognition and behavior.”
BackgroundYang, J., Xu, Y., Chen, P., Li, J., Liu, D., & Chu, X. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.139771 · Cited by 76
Husformer : A Multimodal Transformer for Multimodal Human State Recognition
Wang, R., Jo, W., Zhao, D., Wang, W., Gupte, A., Yang, B., Chen, G., & Min, B. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3357618 · Cited by 87
“…MMResLSTM , with the highest recognition result achieved using a single modality on each dataset in terms of multiclass average accuracy (Acc). by integrating metrics from multiple sources, which is not possible with a single source [54], and enhancing the signal quality by minimizing noise [6].”
MethodSurvey on multimodal approaches to emotion recognition
Aruna Gladys, A., & Vetriselvi, V. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126693 · Cited by 82
Neurosymbolic AI for Reasoning Over Knowledge Graphs: A Survey
DeLong, L. N., Mir, R. F., & Fleuriot, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3420218 · Cited by 134
“For example, a news article might contain both text and images to describe a story [46], [84].”
A Perspective on Foundation Models in Chemistry
Choi, J., Nam, G., Choi, J., & Jung, Y. (2025) · JACS Au · doi:10.1021/jacsau.4c01160 · Cited by 106
Multimodal deep learning based on multiple correspondence analysis for disaster management
Pouyanfar, S., Tao, Y., Tian, H., Chen, S., & Shyu, M. (2018) · World Wide Web · doi:10.1007/s11280-018-0636-4 · Cited by 66
“Among them, multimodal data has attracted a lot of attention since single data types may not provide complete insights and knowledge of the subject of interest [27].”
BackgroundFoundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 186
RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey
Gao, M., Jiang, J., Zou, G., John, V., & Liu, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2907071 · Cited by 60
Ortiz‐Villanueva, E., Benavente, F., Piña, B., Sanz‐Nebot, V., Tauler, R., & Jaumot, J. (2017) · Analytica Chimica Acta · doi:10.1016/j.aca.2017.04.049 · Cited by 63
Double Coupled Canonical Polyadic Decomposition for Joint Blind Source Separation
Gong, X., Lin, Q., Cong, F., & Lathauwer, L. D. (2018) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2018.2830317 · Cited by 58
“We refer to [40] and references therein for an overview.”
BackgroundA Comprehensive Survey of Artificial Intelligence Techniques for Talent Analytics
Qin, C., Zhang, L., Cheng, Y., Zha, R., Shen, D., Zhang, Q., Chen, X., Sun, Y., Zhu, C., Zhu, H., et al. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3572744 · Cited by 172
“Recently, multimodal learning has been used to achieve multimodal data representation, translation, alignment, fusion, and co-learning in various domains, such as commercial, social, and biomedical [280], [281].”
Bokade, R., Navato, A. P., Ouyang, R., Jin, X., Chou, C.-A., Ostadabbas, S., & Mueller, A. V. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113885 · Cited by 71
Ding, X., Bucholc, M., Wang, H., Glass, D. H., Wang, H., Clarke, D. H., Bjourson, A. J., Dowey, L. R. C., O’Kane, M., Prasad, G., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-27997-8 · Cited by 56
Li, X., Yuan, Y., Liu, G., Han, Z. Y., & Stouffs, R. M. F. A. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2023.113876 · Cited by 54
Menezes, G. L., Mazon, G., Ferreira, R. E. P., Cabrera, V. E., & Dórea, J. R. R. (2024) · Animal Frontiers · doi:10.1093/af/vfae048 · Cited by 89
Self-evolving intelligent algorithms for facilitating data interoperability in IoT environments
Nawaratne, R., Alahakoon, D., Silva, D. D., Chhetri, P., & Chilamkurti, N. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.02.049 · Cited by 62
DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis
Verma, S., Wang, C., Zhu, L., & Liu, W. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/503 · Cited by 55
Multi-modality encoded fusion with 3D inception U-net and decoder model for brain tumor segmentation
Punn, N. S., & Agarwal, S. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09271-0 · Cited by 55
“The fusion of multi-modalities offers the multi-information of the brain tissues for comprehensive analysis to improve upon the tumor segmentation results [18].”
BackgroundAttallah, O. (2023) · Chemosensors · doi:10.3390/chemosensors11070364 · Cited by 61
“The three types of model-agnostic fusion are early, late, and intermediate fusion [60].”
BackgroundMuse, E. D., & Topol, E. J. (2024) · Cell Metabolism · doi:10.1016/j.cmet.2024.02.002 · Cited by 50
Multimodal Sensor System for Pressure Ulcer Wound Assessment and Care
Chang, M., Yu, T., Luo, J., Duan, K., Tu, P., Zhao, Y., Nagraj, N., Rajiv, V., Priebe, M. M., Wood, E. A., et al. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2782213 · Cited by 57
“MULTIMODAL sensor fusion has gained much attention due to the benefits it provides for various detection and recognition tasks [1], [2].”
BackgroundWeather Visibility Prediction Based on Multimodal Fusion
Zhang, C., Wu, M., Chen, J., Chen, K., Zhang, C., Xie, C., Huang, B., & He, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2920865 · Cited by 54
Clark, E., Brunton, S. L., & Kutz, J. N. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3013094 · Cited by 56
“[34] advocate for the benefits of multimodality for state estimation, and the structure monitoring community makes use of multiple types of sensors for fault detection and response reconstruction.”
BackgroundMultimodal Representation Learning: Advances, Trends and Challenges
Zhang, S., Zhai, J., Xie, B., Zhan, Y., & Wang, X. (2019) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc48188.2019.8949228 · Cited by 52
“Due to the limitation of pages, the interested reader can refer the references [1] and [2].”
BackgroundGraph-Based Data Fusion Applied to: Change Detection and Biomass Estimation in Rice Crops
Jimenez-Sierra, D. A., Benítez-Restrepo, H. D., Cardona, H. D. V., & Chanussot, J. (2020) · Remote Sensing · doi:10.3390/rs12172683 · Cited by 51
“This analysis provides fundamental data for environmental protection, sustainable development, and the maintenance of ecological balance [13].”
BackgroundUnidimensional and Multidimensional Methods for Recurrence Quantification Analysis with crqa
Coco, M. I., Mønster, D., Leonardi, G. G., Dale, R. A. C., & Wallot, S. (2021) · The R Journal · doi:10.32614/rj-2021-062 · Cited by 65
Sun, G., Chang, V. I. C., Yang, G., & Liao, D. (2017) · Information Sciences · doi:10.1016/j.ins.2017.08.021 · Cited by 50
Review of Various Image Fusion Algorithms and Image Fusion Performance Metric
Singh, S., Mittal, N., & Singh, H. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-020-09518-x · Cited by 48
Deep-learning-enabled multimodal data fusion for lung disease classification
Kumar, S., Ivanova, O., Melyokhin, A., & Tiwari, P. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101367 · Cited by 52
Yuanyuan, Z., Zhang, B., Cheng, S., Haolu, L., Jicheng, H., Tian, K., & Zhong, T. (2024) · International journal of agricultural and biological engineering · doi:10.25165/j.ijabe.20241702.8596 · Cited by 47
Zhu, F., Zhang, J., Dang, R., Hu, B., & Wang, Q. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106996 · Cited by 61
Multiclass Alpha Integration of Scores from Multiple Classifiers
Safont, G., Salazar, A., & Vergara, L. (2019) · Neural Computation · doi:10.1162/neco_a_01169 · Cited by 49
An open-access simultaneous electrocardiogram and phonocardiogram database
Kazemnejad, A., Karimi, S., Gordany, P., Clifford, G. D., & Sameni, R. (2024) · Physiological Measurement · doi:10.1088/1361-6579/ad43af · Cited by 51
Multimodal Sensor Fusion with Differentiable Filters
Lee, M. A., Yi, B., Martin-Martin, R., Savarese, S., & Bohg, J. (2020) · 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341579 · Cited by 46
“and multimodal state estimation for decades [2, 7, 21].”
Unsupervised Multimodal Anomaly Detection With Missing Sources for Liquid Rocket Engine
Feng, Y., Liu, Z., Chen, J., Lv, H., Wang, J., & Zhang, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3162949 · Cited by 53
“As concerns the concept of multimodality, some scholars define it as data from multiple sensors [24], [52], whereas some emphasize the different types of data [9], such as visual and audio modality.”
BackgroundLi, Q., Wang, Y., Zhang, Y., Zuo, Z., Chen, J., & Wang, W. W. (2024) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2024.3400861 · Cited by 57
“Traditional methods often involve weighting based on the relevance between source and target domains [32], [33], assigning greater weight to aspects similar to the target domain to mitigate domain shift issues [34], [35].”
Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data
Menegotto, A. B., Becker, C. D. L., & Cazella, S. C. (2021) · Health Information Science and Systems · doi:10.1007/s13755-021-00151-x · Cited by 51
“Multimodal Deep Learning (MDL) is an approach that allows the use of more than one data modality in DL architectures [24], enabling the creation of models for CAD of HCC, which combines, for instance, imaging and patient’s context attributes as suggested by the AASLD guidelines.”
MethodZhang, Y., Zhao, T., Gao, S., & Raubal, M. (2023) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2023.2234959 · Cited by 44
“These multiple sources endow context data with multiformity in modalities (Lahat et al. 2015).”
BackgroundInfluentialA unified industrial large knowledge model framework in Industry 4.0 and smart manufacturing
Lee, J., & Su, H. (2024) · International Journal of AI for Materials and Design · doi:10.36922/ijamd.3681 · Cited by 81
“In the second step of the ILKM framework, the focus shifts to transforming domain-specific data (human-interpretable data from LKL) into structured domain in-2 [8].”
BackgroundStreaming Data Fusion for the Internet of Things
Kenda, K., Kažič, B., Novak, E., & Mladenić, D. (2019) · Sensors · doi:10.3390/s19081955 · Cited by 52
“Frameworks that have the potential to be applied in various use cases need to be domain knowledge agnostic (purely data-driven), at least with the modeling algorithm [33].”
BackgroundLarge language model applications in disaster management: An interdisciplinary review
Xu, F., Ma, J., Li, N., & Cheng, J. C. P. (2025) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2025.105642 · Cited by 63
Multimodal Remote Sensing Image Registration With Accuracy Estimation at Local and Global Scales
Uss, M., Vozel, B., Лукин, В. В., & Chehdi, K. (2016) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2016.2587321 · Cited by 45
“Such a transformation allows fusion of multiview, multitemporal, multichannel, and multimodal images [3].”
BackgroundAthreya, A. P., Iyer, R. K., Neavin, D. R., Wang, L., Weinshilboum, R. M., Kaddurah‐Daouk, R., Rush, A. J., Frye, M. A., & Bobo, W. V. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840660 · Cited by 43
“It is important to note that several other researchers have proposed the combination of other modalities of biological data [28]–[30], but it remains to be explored whether combination becomes less effective when patient-reported data are used, since there is considerable heterogeneity in subject-reported measures.”
Estimation of sugar content in sugar beet root based on UAV multi-sensor data
Wang, Q., Che, Y., Shao, K., Jinyu, Z., Wang, R., Sui, Y., Guo, Y., Li, B., Meng, L., & Ma, Y. (2022) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2022.107433 · Cited by 42
Cocchi, M. (2019) · Data handling in science and technology · doi:10.1016/b978-0-444-63984-4.00001-6 · Cited by 46
Collective Affinity Learning for Partial Cross-Modal Hashing
Guo, J. F., & Zhu, W. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2941858 · Cited by 47
“Each group of features can be conceived as a particular modality [6], [7].”
BackgroundSELMA: SEmantic Large-Scale Multimodal Acquisitions in Variable Weather, Daytime and Viewpoints
Testolina, P., Barbato, F., Michieli, U., Giordani, M., Zanuttigh, P., & Zorzi, M. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3257086 · Cited by 41
“accuracy and performance of learning tasks [26].”
Multi-Level Confidence Learning for Trustworthy Multimodal Classification
Zheng, X., Tang, C., Wan, Z., Hu, C., & Zhang, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26346 · Cited by 70
Ahmed, A., Ramesh, J., Ganguly, S., Aburukba, R., Sagahyroon, A. A., & Aloul, F. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3303436 · Cited by 45
“Many of the stated challenges affect both unimodal and multimodal approaches; however, their impact on the presence of multimodality is particularly amplified, and their potential solutions’ effect diminished [4].”
BackgroundInfluentialMultimodal Deep Learning Crime Prediction Using Tweets
Sakirin, T., & Tanrıöver, Ö. Ö. (2023) · IEEE Access · doi:10.1109/access.2023.3308967 · Cited by 58
“as an effective approach in recent years [9].”
BackgroundMulti-Modal Non-Line-of-Sight Passive Imaging
Beckus, A., Tamasan, A. C., & Atia, G. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2896517 · Cited by 40
“Our approach is physics-driven in the sense that we use established physics-based models from the theory of light propagation and scattering to describe the transformation between the source image and the measurements [25].”
Objective Measures of Cognitive Load Using Deep Multi-Modal Learning: A Use-Case in Aviation
Wilson, J. C., Nair, S., Scielzo, S., & Larson, E. C. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3448111 · Cited by 52
“They take advantage of several key concepts from within machine learning: (1) multi-task learning [75, 85], (2) multimodal learning [59, 79], (3) variational autoencoders [55], and (4) x-vectors.”
BackgroundTable Tennis Tutor: Forehand Strokes Classification Based on Multimodal Data and Neural Networks
Sanusi, K. A. M., Mitri, D. D., Limbu, B. H., & Klemke, R. (2021) · Sensors · doi:10.3390/s21093121 · Cited by 50
“Moreover, these sensors are typically synchronised with human activities so that one can analyse historical evidence of learning activities and the description of the learning process [23].”
BackgroundLooking beyond the cancer cell for effective drug combinations
Dry, J. R., Yang, M., & Sáez-Rodríguez, J. (2016) · Genome Medicine · doi:10.1186/s13073-016-0379-8 · Cited by 41
“derived by comparing the analysis of the combined data sets with separate analyses of each data set alone [70].”
MethodToward multimodal signal detection of adverse drug reactions
Harpaz, R., DuMouchel, W., Schuemie, M., Bodenreider, O., Friedman, C., Horvitz, E., Ripple, A., Sorbello, A., White, R. W., Winnenburg, R., et al. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.10.013 · Cited by 39
Conati, C., Lallé, S., Rahman, M. A., & Toker, D. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3301400 · Cited by 51
“Research on interactive systems has examining the value of combining different data sources in a variety of user modelling tasks (see References [66, 88] for an overview); however, few of these works specifically investigated the combination of interaction and eye-tracking data.”
BackgroundCohesive Multi-Modality Feature Learning and Fusion for COVID-19 Patient Severity Prediction
Zhou, J., Zhang, X., Zhu, Z., Lan, X., Fu, L., Wang, H., & Wen, H. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3063952 · Cited by 38
“The appropriate fusion of available information from multi-modality data is still an important way to increase the prediction accuracy of the COVID-19 patient severity, even if some data missing issues (e.g. modality missing) may be encountered [9], [10].”
BackgroundTensor-based fusion of EEG and FMRI to understand neurological changes in schizophrenia
Acar, E., Levin‐Schwartz, Y., Calhoun, V. D., & Adalı, T. (2017) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2017.8050303 · Cited by 39
Employing multimodal co-learning to evaluate the robustness of sensor fusion for industry 5.0 tasks
Rahate, A., Mandaokar, S., Chandel, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2022) · Soft Computing · doi:10.1007/s00500-022-06802-9 · Cited by 40
“from the early stages of data processing is concatenated (Lahat et al. 2015).”
BackgroundTorkamani‐Azar, M., Lee, A., & Bednarik, R. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3182869 · Cited by 41
“Multimodal data collection necessities the implementation of data fusion techniques to overcome different sampling rates, physiological properties, spatio-temporal resolutions, sample sizes, and misaligned or non-synchronized streams [138].”
BackgroundMultimodal fusion techniques: Review, data representation, information fusion, and application areas
Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 59
Schizophrenia Shows Disrupted Links between Brain Volume and Dynamic Functional Connectivity
Abrol, A., Rashid, B., Rachakonda, S., Damaraju, E., & Calhoun, V. D. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00624 · Cited by 39
“…leveraging the above mentioned multivariate approaches have revealed significant information on clinical aspects of schizophrenia as discussed in several recent reviews on multimodal fusion (Biessmann et al., 2011; Schultz et al., 2012; Sui et al., 2012a; Lahat et al., 2015; Calhoun and Sui, 2016).”
BackgroundDetecting Mistakes in CPR Training with Multimodal Data and Neural Networks
Mitri, D. D., Schneider, J., Specht, M. M., & Drachsler, H. (2019) · Sensors · doi:10.3390/s19143099 · Cited by 41
“In support to the diffusion of the multimodal approach, comes as well the progress of contiguous scientific research areas such as body sensor networks [5], social signal processing [6], multimodal machine learning [7] and multimodal data fusion [8].”
BackgroundLatent common manifold learning with alternating diffusion: Analysis and applications
Talmon, R., & Wu, H. (2018) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2017.12.006 · Cited by 45
Exploring Shared Perception and Control in Cooperative Vehicle-Intersection Systems: A Review
Bejarbaneh, E. Y., Du, H., & Naghdy, F. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3432634 · Cited by 62
“More recently, multimodal sensor fusion technology which leverages multiple types of sensors such as LiDAR and radars, has become an emerging research trend in the field of autonomous driving cooperative perception to enhance perception capabilities and mitigate costs [97], [98], [99].”
BackgroundA machine learning approach for multimodal data fusion for survival prediction in cancer patients
Νικολάου, Ν., Salazar, D., RaviPrakash, H., Gonçalves, M., Mulla, R., Burlutskiy, N., Markuzon, N., & Jacob, E. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-00917-6 · Cited by 65
Anomalous behaviour detection based on heterogeneous data and data fusion
Ali, A. M., & Angelov, P. P. (2018) · Soft Computing · doi:10.1007/s00500-017-2989-5 · Cited by 44
“However, the most challenging part of data fusion is working with heterogeneous data (Lahat et al. 2015).”
BackgroundZhang, Y., Zhang, H., Xiao, L., Bai, Y., Calhoun, V. D., & Wang, Y.-P. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3161828 · Cited by 41
“It provides richer information than a single modality by leveraging complementary information from multiple modalities, which is now widely used in various areas such as sensor networks, video processing, medical imaging and intelligent system design [1]–[5].”
BackgroundHeterogeneous data integration methods for patient similarity networks
Gliozzo, J., Mesiti, M., Notaro, M., Petrini, A., Patak, A., Gallardo, A. P., Paccanaro, A., Valentini, G., & Casiraghi, E. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac207 · Cited by 46
Liang, R., Huang, C., Zhang, C., Li, B., Saydam, S., & Canbulat, I. (2023) · IEEE Access · doi:10.1109/access.2023.3267813 · Cited by 51
“Rockburst risk assessment research has also integrated various experiment data for prediction, such as UCS, geological parameters, tunnel shape, etc. [215], [216] McGaughey [16], [80] proposed data fusion analytics with geological data and static rock mass data for hazard prediction.”
BackgroundMultisensor Information Fusion: Future of Environmental Perception in Intelligent Vehicles
Zhang, Y., Tu, C., Gao, K., & Wang, L. (2024) · Journal of Intelligent and Connected Vehicles · doi:10.26599/jicv.2023.9210049 · Cited by 57
Xu, S., Wang, H., Liang, X. M., & Lu, H. (2024) · Foods · doi:10.3390/foods13233917 · Cited by 36
Hadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 59
“The data distribution from a different sensor is called modality [110].”
Joint Independent Subspace Analysis Using Second-Order Statistics
Lahat, D., & Christian Jutten, B. (2016) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2526960 · Cited by 36
Self-citationLevin‐Schwartz, Y., Calhoun, V. D., & Adalı, T. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2678483 · Cited by 35
“emphasizing importance of allowing datasets to fully interact with each other in an analysis [2], [66].”
BackgroundMultilevel Sensor Fusion With Deep Learning
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2018) · IEEE Sensors Letters · doi:10.1109/lsens.2018.2878908 · Cited by 38
Falco, N., Wainwright, H. M., Dafflon, B., Léger, E., Peterson, J., Steltzer, H., Wilmer, C., Rowland, J. C., Williams, K. H., & Hubbard, S. S. (2019) · Journal of Geophysical Research Biogeosciences · doi:10.1029/2018jg004394 · Cited by 34
“A feature level fusion (Lahat et al., 2015) via image concatenation was then applied to the various stacks of images (i.e., the original spectral data set and the two SDAPs), whose combination gave an image vector composed of 16 images.”
Method
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.