Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessThe DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Abstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Items 901–1,000 of 1,001
Towards LLMCI - Multimodal AI for LLM-Vision UI OperationPreprint
Barham, H., & Fasha, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4653823/v1 · Cited by 4
A Methodology to Manage Structured and Semi-structured Data in Knowledge Oriented Graph
Bellandi, V., Ceravolo, P., D’Andrea, G., Maghool, S., & Siccardi, S. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-08223-8_18 · Cited by 1
Collaborative Recognition over Distributed Underwater Acoustic Network
Sun, X., Wang, H., Shen, X., & Hua, F. (2022) · MTS/IEEE OCEANS Conference (OCEANS) · doi:10.1109/oceans47191.2022.9977264 · Cited by 1
“value of different information, improve network fusion performance, and increase the credibility of the fusion result is an urgent problem to be solved [15].”
BackgroundADL based Framework For Multimodal Data Fusion in Traffic Jam prediction
Godhbani, S., Elkosantini, S., Suh, W., & Lee, S. M. (2022) · International Conference on Software, Knowledge, Information Management and Applications (SKIMA) · doi:10.1109/skima57145.2022.10029488 · Cited by 6
“Therefore, addressing multi-source heterogeneous data fusion in the era of big data has become a research hotspot in recent years [1-5].”
BackgroundMultimodal Language Data and Platform Construction for LCTLs Teaching
Wang, Y., Jin, K., & Liu, W. (2023) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3587716.3587754 · Cited by 1
“Therefore, it can be solved by expanding the size of the corpus [14] [15] [16].”
BackgroundDeep Learning Based Multi-Modal Data Fusion for Car Crash Detection
Bhowmik, T. K., & Rateria, R. (2023) · International Conference on Computers and Devices for Communication · doi:10.1109/codec60112.2023.10466222 · Cited by 1
“Multimodality, according to Lahat et al. [2], combines data from multiple sensors for more effective problem-solving, creating a more robust data representation than individual sensors.”
BackgroundAging-friendly learning platform based on multimodal interaction Interface Design Study
Liu, K., & Liu, Y. (2024) · International Conference on Intelligent Education and Computer Technology (IECT) · doi:10.1145/3687311.3687394 · Cited by 1
“Modality itself is an objectively existing and representable symbol system, for example, human voice, expression, eyes, gestures, body movements, etc. can be called a modality after transcoding [3].”
AI-Ready Multimodal Data Pipeline to Enrich Cancer Care
Rashid, R., Hashtarkhani, S., Kumsa, F. A., Chinthala, L. K., White, B. M., Zink, J. A., Brett, C. L., Davis, R. L., Schwartz, D. L., & Shaban‐Nejad, A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825353 · Cited by 2
“Integrating these diverse data types into a unified AI-ready format poses significant challenges driven by heterogeneity in formats, inconsistent data quality, and the need for scalable reproducible solutions [6].”
Simone, A. D., Gravina, M., & Sansone, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-80507-3_10 · Cited by 1
Wei, T., Chen, H., Wang, J., & Liu, W. (2024) · IEEE International Conference on Signal, Information and Data Processing (ICSIDP) · doi:10.1109/icsidp62679.2024.10868650 · Cited by 1
“Due to the gap in multimodal data, exploring the relationship between modality-share and modality-specific information is essential to multimodal fusion tasks [14].”
Sajjad, N., & Mumtaz, R. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-6255-6.ch002 · Cited by 1
Multiscale astrobiology with the Vera C. Rubin Observatory Legacy Survey of Space and Time
Kovačević, A. B., Mason, N. J., & Ćiprijanović, A. (2025) · Frontiers in Astronomy and Space Sciences · doi:10.3389/fspas.2025.1594485 · Cited by 1
Two-Stage Dynamic Fusion Framework for Multimodal Classification Tasks
Ge, S., & Chen, Y. H. (2025) · INFORMS journal on computing · doi:10.1287/ijoc.2023.0448 · Cited by 2
PARAFAC2-Based Coupled Matrix and Tensor Factorizations With Constraints
Schenker, C., Wang, X., Horner, D. A., Rasmussen, M. A., & Acar, E. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3579651 · Cited by 4
“Joint analysis of such datasets from multiple sources, also referred to as data fusion, has the potential to exploit these complementary measurements and enhance knowledge discovery [1], [5].”
InfluentialDevelop intelligent waste bin prototype based on fusion feature recognition of sounds and RGB images
Chen, W., Chiang, C.-F., Lin, Y., & Hsieh, H.-J. (2025) · Waste Management · doi:10.1016/j.wasman.2025.114959 · Cited by 2
Machine Learning Models for Crop Management
Awate, P. L., & Nagne, A. D. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-93087-4_6 · Cited by 1
Aydoğan, Y., Haleem, B., Schulz, D., & Akanyeti, O. (2025) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/dsp65409.2025.11074868 · Cited by 1
“Software-based post hoc data synchronization offers an alternative to multimodal data alignment problems, provided that appropriate and empirically validated methods are available [5].”
A Target-Oriented Image Fusion Method for Quality Inspection of Heat-Sealed Packages
Garbacz, P. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-01773-4_3 · Cited by 1
Hyperspectral Imaging in Forensic Science: An Overview, Application Areas and Recent Advancements
Garapati, K., Maram, S. S., & Manikandan, V. M. (2025) · Intelligent systems reference library · doi:10.1007/978-3-031-83127-0_6 · Cited by 1
Cen, X. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2551708 · Cited by 1
“Deep Belief Network (DBN) [22], a generative model to learn multi-level feature representations, has slightly lower but similar kind of performance on all metrics, also representing its strength in such unsupervised feature learning of comprehension in language.”
InfluentialRetractedMulti-Modal Side-Channel Analysis Based on Isometric Compression and Combined Clustering
Kou, X., Yang, W. Y., Li, L., & Zhang, G. (2025) · Symmetry · doi:10.3390/sym17091483 · Cited by 1
Hao, X., Xu, H., Wang, X., Wang, T., Hämäläinen, T., & Cong, F. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112302 · Cited by 1
Toward an Early Warning System: Feature Selection Mechanism for Food Fraud from Fishing Activities
Klados, G. A., Tsitiridis, A., Chachampis, N., & Perakis, Κ. (2025) · International Conference in Electronic Engineering & Information Technology (EEITE) · doi:10.1109/eeite65381.2025.11166253 · Cited by 1
Mortlock, T., Chen, L., Smereka, J. M., Khargonekar, P. P., & Faruque, M. A. A. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3612873 · Cited by 3
Yao, S., Feng, Z., Liang, Q., Li, H., Jiang, Y., Xie, W., Qin, Y., Wang, W., Rong, P., & Luo, A. (2025) · EngMedicine · doi:10.1016/j.engmed.2025.100104 · Cited by 1
Cai, X., & Wilson, M. D. (2025) · Remote Sensing · doi:10.3390/rs17213591 · Cited by 4
Extracellular vesicle omics assisted disease management
Liu, P., Zhuang, W., Sun, J., & Xie, H.-Y. (2025) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2025.118621 · Cited by 1
Path-aware routing system for multimodal vigilance estimation: A structured fusion perspective
Sun, Y., Li, S., Bie, Y., Wang, L., Jin, T., Guo, M., & Yang, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129757 · Cited by 4
Multimodal Data Fusion for Enhanced Smart Contract Reputability Analysis
Malik, C., Ellul, J., & Bajada, J. (2025) · IEEE International Conference on Blockchain and Cryptocurrency (ICBC) · doi:10.1109/icbc64466.2025.11114521 · Cited by 1
“Multimodal data fusion combines multiple data sources to provide a more comprehensive and accurate understanding of a phenomenon [25].”
Yang, K., Yang, K., Zeng, S., Zhang, Y., & Zhang, D. (2025) · International Electronic Conference on Sensors and Applications (ECSA) · doi:10.3390/ecsa-12-26605 · Cited by 1
“At present, most related research focuses on simple nearshore structures (such as cross-Eng sea bridges), while multimodal fusion monitoring for complex structures like deep-sea platforms and submarine pipelines is still in the exploratory stage [19–21].”
Exploring cognitive workload recognition using CogRepLKNet with EEG-fMRI
Shao, Y. W., Zhou, Y., Wen, X., Gong, P., Dai, Q., & Zhang, D. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108575 · Cited by 3
Yang, F., & Gong, F. (2026) · Scientific Reports · doi:10.1038/s41598-026-36023-1 · Cited by 2
Singh, A. K., & Shukla, A. K. (2025) · IEEE International Conference on Intelligent Signal Processing and Effective Communication Technologies (INSPECT) · doi:10.1109/inspect67393.2025.11350529 · Cited by 1
“The generation of high-level representations for facial expressions and micro-movements was then performed and completed by ResNet50 [30].”
Stable and Generalizable Neural Network for Spatiotemporal Electromagnetic Modeling
Zheng, X., Zhang, Y., Peng, T. J., Wang, S. S., Hou, J., Chen, Q. C., Zhang, J., Mao, Y., You, J. W., & Cui, T. J. (2026) · IEEE Transactions on Antennas and Propagation · doi:10.1109/tap.2026.3658540 · Cited by 1
“First, information fusion denotes the integration of input data with multiple sources or modalities, aiming at providing complete knowledge of the phenomenon of interest [35].”
GPR-GSLAM: Gaussian Process Regression-Enhanced Real-Time RGB-D SLAM Using Gaussian Splatting
Qi, X., Zhang, Y., Fu, H., Zhang, L., Guo, X., & Xinyuan, J. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3664645 · Cited by 1
“1) Depth Completion: We use Gaussian Process Regression (GPR) [31] to complete missing depths in each RGB–”
Wilson, A., & Anitha, J. (2026) · Journal of Bionic Engineering · doi:10.1007/s42235-026-00851-8 · Cited by 1
PWLF: A Robust Weighted Late Multimodal Fusion Approach
Nirala, V., & Ratneshwer (2025) · International Conference on Intelligent Computing and Systems at the Edge (ICEdge) · doi:10.1109/icedge67252.2025.11412698 · Cited by 1
“Ashighlighted by Lahat et al. [1], relying solely on a single modality often limits our ability to fully understand or model real-world processes.”
Tirabassi, N., Kumar, S. B. R., Jha, S. K., & Ramanathan, A. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00043 · Cited by 1
“However, the integration of multimodal data presents significant challenges [4].”
Multimodal Human Action Recognition for Ubiquitous Systems: Cross-Attention of Skeleton and Audio
Boudmagh, M., Kerboua, A., & Redjimi, M. (2026) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v20i05.58381 · Cited by 1
Predictive GIS Modelling for Renewable Energy Site Suitability Using LLM Reasoning Pipelines
Dawar, D., Khandpur, S., Roychowdhury, S., Sharma, A., & Jha, D. (2025) · International Conference on Artificial Intelligence and Speech Technology (AIST) · doi:10.1109/aist68591.2025.11441593 · Cited by 1
“In absence of available high-resolution geospatial datasets, our system integrates matrix Earth observation (EO) imagery, tabular environmental attributes, and GIS layers into our pipeline, applying deep learning with feature fusion for site suitability estimation [8].”
Wang, J., Li, H., Wu, S., Yu, Y., Nie, G., & Fan, Z. (2026) · Remote Sensing · doi:10.3390/rs18081115 · Cited by 1
Eacuello, S. S., Prasad, R., & Sodhi, M. S. (2026) · Sensors · doi:10.3390/s26082405 · Cited by 1
Yang, J., ZHANG, Q.-D., Liu, Y., Li, J.-Y., Chen, P., Xu, Y.-P., Liu, D., & Chu, X. (2026) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2026.118864 · Cited by 1
A survey on opinion dynamics: From rule-based models to data-driven and hybrid approaches
Wang, J., Zhang, J., Dong, Z., & Jin, L. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133705 · Cited by 1
AN OVERVIEW OF MULTIMODAL LEARNING: CONCEPTS, CHALLENGES, APPLICATIONS AND DATASETS
Ateş, A. H., & Yurtsever, U. (2026) · Konya Journal of Engineering Sciences · doi:10.36306/konjes.1716813 · Cited by 1
Zhang, L., Yang, Y., & Wu, Y. (2026) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2026.117160 · Cited by 2
He, J., Zhang, Y., Chen, J., Jiang, Q., Liu, Y., Bi, G., & Zhang, Y. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.6c05993
Bian, X., Zhou, J., Fu, L., Jin, A., & Zhang, W. (2026) · Processes · doi:10.3390/pr14162537 · Cited by 2
Heikal, F. N., Ismail, T., Alharbi, F., El-Azab, J. M., & Hamdy, O. (2026) · IEEE Access · doi:10.1109/access.2026.3724948 · Cited by 1
“Multimodal sensing has gained increasing attention because different sensing techniques provide complementary information about tissue condition [55], [56].”
Nikolenko, S. (2026) · IEEE Access · doi:10.1109/access.2026.3741122
“…with numerous studies of multimodal fusion for fault diagnostics [4], [5], which can The associate editor coordinating the review of this manuscript and approving it for publication was Mark Kok Yew Ng . be thought of as part of a broader multimodal program in machine learning [6], [7], [8], [9].”
Multispectral imaging for improved liquid classification in security sensor systems
Burns, A., & Bajwa, W. U. (2018) · Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery · doi:10.1117/12.2304696 · Cited by 1
“, understanding language through audio-visual information), there are challenges involved in using multimodal data in machine learning due to the difficulties of fusing various modes of different dimensions and data types.(3) One example of multimodal data, which have been successfully used in different types of class…”
BackgroundContext-Based Multimodal Output for Human-Robot Collaboration
Kaiser, M., & Bürckert, C. (2018) · Conference on Human System Interactions · doi:10.1109/hsi.2018.8431264 · Cited by 1
Joint Independent Subspace Analysis by Coupled Block Decomposition: Non-Identifiable Cases
Lahat, D., & Christian Jutten, B. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461358 · Cited by 1
“Therefore, this minimal model allows us to isolate the contribution of the multiset diversity [18].”
BackgroundSelf-citationHazarika, A., & Bhuyan, M. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-815553-0.00003-3 · Cited by 1
Attention-Based Audio-Visual Fusion for Video Summarization
Fang, Y., Zhang, J., & Lu, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-36711-4_28 · Cited by 1
Dongo, I., Jáuregui, D. A. G., & Couture, N. (2019) · Conference on l'Interaction Homme-Machine (IHM) · doi:10.1145/3366550.3372259 · Cited by 1
“Afin de résoudre ce problème, des méthodes plus adaptées de fusion des données multi-modales [18] ou de fusion de classifieurs [19] peuvent être intégrés pour améliorer la précision de la reconnaissance des gestes à partir des deux capteurs.”
BackgroundProbabilistic Dynamic Non-negative Group Factor Model for Multi-source Text Mining
Lu, C., Peltonen, J., Nummenmaa, J., & Järvelin, K. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411956 · Cited by 1
“Analyzing data from multiple sources has attracted increasing attention in the machine learning community [11].”
BackgroundMultidimensional Performance Degradation Data Fusion and Residual Life Prediction for Machine Tools
Jiang, R. (2020) · Global Reliability and Prognostics and Health Management Conference (PHM) · doi:10.1109/phm-shanghai49105.2020.9280976 · Cited by 1
“Relative to the LS-SVM and the composite scale approach, other advantages of the proposed method include (a) its modeling process is relatively transparent, (b) it has better identifiability when the number of indicators is large [29], and (c) its resulting model has better interpretability [29].”
MethodA distributed simulator platform for rapid industrial user experience prototyping
Filho, R. S. S., Carroll, A. K., & Brooks, J. D. (2017) · Winter Simulation Conference · doi:10.1109/wsc.2017.8247889 · Cited by 1
Bayrak, Ş., & Yücel, E. (2022) · Studies in computational intelligence · doi:10.1007/978-981-19-2416-3_1 · Cited by 1
Multimodal recommendation of messenger channels
Koshchenko, E., Klimov, E., & Kovalenko, V. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528441 · Cited by 1
“Combining such data can provide a good representation of social and technical connections within organizations [2].”
BackgroundMultimodal Affect Recognition Using Temporal Convolutional Neural Networks
Ayoub, I., Heiries, V., & Osman, H. A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892145 · Cited by 1
“In decision-level fusion [12], multiple models are trained on unimodal features.”
BackgroundMethodPratiwi, S. H., Shaniya, P., Jati, G., & Jatmiko, W. (2023) · Jurnal Ilmu Komputer dan Informasi · doi:10.21609/jiki.v16i1.1073 · Cited by 1
Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates
Pester, A., Sulema, Y. S., Dychka, I. A., & Sulema, O. (2023) · Algorithms · doi:10.3390/a16040186 · Cited by 2
Data Driven Joint Sensor Fusion and Regression Based on Geometric Mean Squared Error
Lopez, C. A., & Riba, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095018 · Cited by 1
Marinoni, A., Mercier, M., Shi, Q., Selvakumaran, S., & Girolami, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095086 · Cited by 1
“Technological advancements over the last decade have enabled the use of sensors, which can collect huge volumes of records, to describe virtually any phenomena and event of everyday life [1, 2].”
BackgroundWork-in-Progress: Multimodal Odour and Image Recognition on Synthetic Dataset
Monori, F., & Tisan, A. (2023) · IEEE International Symposium on Industrial Electronics (ISIE) · doi:10.1109/isie51358.2023.10228134 · Cited by 1
“The idea behind multimodal learning originates from the fact that humans perceive the world through many different senses [6].”
Data Fusion for Prediction of Variations in Students Grades
Teixeira, R., Marcondes, F. S., Lima, H., Durães, D. A., & Nováis, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43078-7_24 · Cited by 1
The Application of Multimodal Learning to Enhance Language Proficiency in Oral English Teaching
Zeng, Y. (2023) · Adult and Higher Education · doi:10.23977/aduhe.2023.051806 · Cited by 5
“text-based, reading an overview document or slides about what the product they are going to be selling does[2].”
BackgroundOptimizing Vehicle Classification using YOLOv8: Strategies to Counter Overfitting
Jha, A., Saxena, I., Kadian, K., Indu, Sharma, V., & Singh, S. (2023) · 2023 3rd International Conference on Advancement in Electronics & Communication Engineering (AECE) · doi:10.1109/aece59614.2023.10428452 · Cited by 2
“This research sets the stage for safer, more efficient, and more dependable automated systems in the field of vehicle recognition, with implications reaching across diverse industries [22].”
BackgroundSuccessful Integration of IoT and Blockchain Technologies Using Several Machine Learning Algorithms
Calarany, C., Indumathy, M., Senthilraja, P., Suganya, D., Vardhan, V., & Rahul, Y. S. (2024) · IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT) · doi:10.1109/ic2pct60090.2024.10486595 · Cited by 1
Page, J., Oh, H., Chacko, T. K., Samuel, I. B. H., Lu, C., Forsten, R. D., Reinhard, M. J., Costanzo, M. E., & Broderick, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61063-9_8 · Cited by 1
Audio-Visual Cross-Modal Generation with Multimodal Variational Generative Model
Xu, Z., Wang, T., Liu, D., Hu, D., Zeng, H., & Cao, J. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10557902 · Cited by 2
“Video Understanding (VU) through audio and visual modality fusion has been widely studied [1]–[4].”
BackgroundDistributed Estimation by Two Agents with Different Feature Spaces
Raghavan, A., & Johansson, K. H. (2024) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc60939.2024.10644868 · Cited by 1
“In the context of the experiment, the set of all possible models or patterns is referred to as the knowledge space (KS).”
BackgroundTensor Kernel Learning for Classification of Alzheimer’s Conditions using Multimodal Data
Thanh, V. D., Thanh, L. T., Tuấn, P. M., Trung, N. L., Abed‐Meraim, K., Adel, M., Nguyen, V.-D., Trung, N. T., Long, D. D., & Chén, O. Y. (2024) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr63514.2024.10661014 · Cited by 4
“In tensor fusion, a common approach involves concatenating multiple data tensors into a single one and employing TD to obtain shared features [30], [31].”
MethodVTFEFN: An End-to-End Visual-Tactile Feature Extraction and Fusion Network
Tian, S., Zhang, Y., Zou, Z., Shu, L., & Hao, J. (2024) · International Symposium on Computer and Information Processing Technology (ISCIPT) · doi:10.1109/iscipt61983.2024.10672780 · Cited by 1
“[11] employ deep learning models to effectively integrate information from various modalities, thereby improving the understanding of complex environments.”
BackgroundLopez, C. A., & Riba, J. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3488554 · Cited by 1
Cement classification and characterization using Non-Invasive techniques
Romero, E., Ferreira, D. D. S., Pereira, F. M. V., Olivieri, A. C., Pereira‐Filho, E. R., & Arancibia, J. A. (2024) · Talanta · doi:10.1016/j.talanta.2024.127212 · Cited by 1
STFEnc: A Novel Deep Encoder for Brain-Computer Interface Based on Interpretable Brain Features
Du, Z., & Lai, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889278 · Cited by 1
“Deep learning’s ability to handle multimodal, high-dimensional data has made it a popular choice for EEG analysis [17].”
Chemical-sensitive Electron Tomography for Nanomaterials
Liu, L., Feng, S., & Han, L. (2025) · Chemical Research in Chinese Universities · doi:10.1007/s40242-025-4239-1 · Cited by 1
Echocardiographic view classification using ECG-assisted multiphase image sampling
Ghods, Z., Farrag, N. A., Heschl, A., Labib, D., Saedi, M., Nizamani, T., Hossain, A., Nguyen, N.-T., Feng, Y., Fine, N. M., et al. (2025) · SPIE Medical Imaging: Ultrasonic Imaging and Tomography · doi:10.1117/12.3046885 · Cited by 2
“…extracting key frames representing unique cardiac phases may support the learning of temporally dependent features with minimal input data, allowing models to train efficiently with consistent positional features and alignment, 17–20 this also reducing computational resources at time of inference.”
Data-driven approaches in concrete science: applications, challenges and future prospects
Barbhuiya, S., Das, B. B., & Adak, D. (2025) · Proceedings of the Institution of Civil Engineers - Construction Materials · doi:10.1680/jcoma.24.00018 · Cited by 2
Spatiotemporal Analysis of Forest Fires in Cyprus Using Earth Observation and Climate Data
Προδρόμου, Μ., Girtsou, S., Leventis, G., Charalampous, G., Apostolakis, A., Tzouvaras, M., Mettas, C., Giannopoulos, G., Kontoes, C., & Hadjimitsis, D. G. (2025) · International Conference on Meteorology, Climatology and Atmospheric Physics (COMECAP) · doi:10.3390/eesp2025035054 · Cited by 1
A Concept Representation Language (CRL)
Brown, P. C. (2018) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2018.00010 · Cited by 1
“This requires relating models from multiple domains to formulate a complete situational representation [6, 7].”
MethodGÖRMÜŞ, E. T., & Akar, Ö. (2019) · Geomatik · doi:10.29128/geomatik.538838 · Cited by 1
Multi-Level Sensor Fusion with Deep Learning
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2018) · HAL · doi:10.48550/arxiv.1811.02447 · Cited by 1
Data and Model Fusion in Chemometrics
Brown, S. D. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-409547-2.14586-x · Cited by 2
Intelligent Maritime Information Acquisition and Representation for Decision Support
Kyriakides, I., Hayes, D. R., & Tsiantis, P. (2020) · Progress in IS · doi:10.1007/978-3-030-50892-0_22 · Cited by 1
An Approach to Spatiotemporal Trajectory Clustering Based on Community Detection
Wang, X., Niu, X., Zhu, J., & Liu, Z. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/5582341 · Cited by 1
“, audio [1], video [2], text [3], and image [4]), and can be analyzed jointly or separately [5, 6].”
BackgroundDeep Feature Fusion for Face Analytics
Sankaran, N., Mohan, D. D., Tulyakov, S., Setlur, S., & Govindaraju, V. (2021) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-030-74697-1_9 · Cited by 1
Ellison, C. L., & Fields, W. R. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9565056 · Cited by 1
Input Prediction Using Consensus Driven SOMs
Gonnier, N., Boniface, Y., & Frezza-Buet, H. (2021) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi53840.2021.9654851 · Cited by 1
On the Pareto-Optimal Solutions in the Multimodal Clustering Problem
Bogatyrev, M., Orlov, D., & Shestaka, T. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-15168-2_15 · Cited by 1
A Survey on Visual Analysis of Massive Open Online Courses (MOOC) Data
Zhou, Z., Wang, S., Liu, Y., Wang, Y., & Wang, Y. (2022) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2022.19536 · Cited by 1
Slizovskaia, O., Gómez, E., & Haro, G. (2017) · Repositori digital de la UPF (Universitat Pompeu Fabra) · doi:10.5281/zenodo.1067687 · Cited by 1
Computer-Assisted Analysis of Biomedical ImagesPreprint
Rundo, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04381 · Cited by 3
Fu, D., Weber, C., Guochun, Y., Kerzel, M., Nan, W., Barros, P. V. A., Wu, H., Liu, X., & Wermter, S. (2019) · doi:10.31234/osf.io/s3u9j · Cited by 1
“, 2010; Shinn-Cunningham, 2008; Lee and Choo, 2013), most of them only focus either on a single modality or on general crossmodal processing (Lahat et al., 2015; Ramachandram and Taylor, 2017).”
BackgroundTask-Oriented Multi-User Semantic Communications for Multimodal DataPreprint
Xie, H., Qin, Z., & Li, G. Y. (2021) · arXiv (Cornell University) · Cited by 1
Singularity Expansion Method in Radar Multimodal Signal Processing and Antenna Characterization
Bhattacharyya, N., & Siddiqui, J. Y. (2021) · doi:10.1002/9781119571452.ch10 · Cited by 1
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.