The DOI matches its Crossref record; 214 of the 218 references it cites were found in a registry or an index.
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Open accessAbstract
In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term “modality” for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or “challenges,” are common to multiple domains. This paper deals with two key issues: “why we need data fusion” and “how we perform it.” The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, “diversity” is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level)30.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [30]
In the citing paper
“These methods vary considerably in their models and applications but generally involve extensions of traditional multivariate analysis techniques such as independent component analysis, canonical correlation analysis, and singular value decomposition (Lahat et al. 2015).”
From Similarity-based multimodal regression (2024) · cites this paper as Lahat et al. 2015
In the citing paper
“While late fusion requires less data and allows for easier integration of multiple sensing strategies and algorithms, it may not fully leverage the complementary strengths of different modalities115.”
From AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis (2025) · cites this paper as [115]
In the citing paper
“but to the best of our knowledge, there is no recent work that directly addresses the latest advances in deep multimodal learning particularly for the computer vision community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
In the citing paper
“Furthermore, some cutting-edge works [34, 36] have been proposed in recent years that address the mechanism of integrating and fusing multimodal representations inside deep learning architectures by showing the reader the possibilities this opens up for the artificial intelligence community.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [36]
Citations
Cites
Horn, R. A., & Johnson, C. R. (1985) · Cited by 22,203
Complex brain networks: graph theoretical analysis of structural and functional systems
Bullmore, E., & Sporns, O. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2575 · Cited by 11,820 · cited 2×
“Bullmore and Sporns [165] study brain networks by first constructing separate models of structural and functional networks based on several brain imaging modalities and fuse them using a graph-theoretical framework.”
BackgroundTensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301 · cited 2×
“More properties of tensor decompositions and their uses in various engineering applications can be found for example in [64], [65], [106] and references therein.”
BackgroundHorn, R. A., & Johnson, C. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810817 · Cited by 10,280
An Introduction to Multivariate Statistical Analysis
Muirhead, R. J., & Anderson, T. W. (1986) · Journal of Business & Economic Statistics · doi:10.2307/1391399 · Cited by 9,082
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205 · cited 2×
“Under these assumptions, separation can be achieved if the statistically independent sources are non-stationary, non-white, or nonGaussian [71], [80]–[82].”
Partial least-squares regression: a tutorial
Geladi, P., & Kowalski, B. R. (1986) · Analytica Chimica Acta · doi:10.1016/0003-2670(86)80028-9 · Cited by 6,992 · cited 1×
“We mention PLS [62], [203] and its multilinear extensions N -way PLS [62], [150] and higher-order PLS (HOPLS) [141].”
MethodPlanck 2013 results. XVI. Cosmological parameters
Planck Collaboration, Ade, P. A. R., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J., Baccigalupi, C., Banday, A. J., et al. (2014) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201321591 · Cited by 6,521 · cited 1×
“surveys, galaxy clustering [54], [55], and stacked gravitational lensing, also serve as important cosmological probes [51], [52].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 4×
“Most human verbal communication involves seeing the speaker [18].”
BackgroundInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Research works printed but not linked: 11
- lederman, 2015, Common manifold learning using alternating-diffusion
- stompor, 0, Data analysis of massive data sets a Planck example, LOFAR Workshop
- seyyedi, 2010, Comparing satellite derived rainfall with ground based radar for North-Western Europe
- harshman, 1972, PARAFAC2: Mathematical and technical notes, UCLA Working Papers Phonet, 22, 30
- cardoso, 0, The three easy routes to independent component analysis; contrasts and geometry, Proc ICA, 1
- sorber, 2014, Tensorlab v2 0
- 0, Joint data analysis for enhanced knowledge discovery in metabolomics
- sørensen, 2013, Multidimensional harmonic retrieval via coupled canonical polyadic decomposition
- comon, 0, Supervised classification, a probabilistic approach, Proc Eur Symp Artif Neural Netw, 111
- harshman, 0, Models for analysis of asymmetrical relationships among $N$ objects or stimuli, Proc Joint Meeting of the Psychometric Society and the Society for Mathematical Psychology, 1
- adal?, 2013, IEEE Trans Multimedia, 15
Cited by
Tensor Decomposition for Signal Processing and Machine Learning
Sidiropoulos, N. D., De Lathauwer, L., Fu, X., Huang, K., Papalexakis, E. E., & Faloutsos, C. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2690524 · Cited by 1,617
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
“For the purposes of our review, we adopt the definition provided by Lahat et al. [2], where we consider phenomena or systems that are observed using multiple sensors and each sensor output can be termed a modality associated with a single data set.”
MethodA survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821
“The reasonable fusion of these multimodal data can help us better understand the event of interest, especially when one modality is incomplete (Khaleghi, Khamis, Karray, & Razavi, 2013; Lahat, Adali, & Jutten, 2015).”
BackgroundGeneralizing from a Few Examples: A Survey on Few-Shot Learning
Wang, Y., Yao, Q., Kwok, J. T.-Y., & Ni, L. M.-S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05046 · Cited by 772
“in E to make the learning of the supervised targetT feasible. Existing FSL methods use prior knowledge from a single modality, which is the output of a sensor for observing some phenomena or systems [61], such as images, texts, and videos. However, multi-modality prior knowledge can be used to provide a prior knowledg…”
MethodDeep learning in multimodal remote sensing data fusion: A comprehensive review
Li, J., Hong, D., Gao, L., Yao, J., Zheng, K., Zhang, B., & Chanussot, J. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102926 · Cited by 630
Metadata
- Authors
- Lahat, D., Adali, T., & Jutten, C.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- ORCID
- Lahat, Dana
- Venue
- Proceedings of the IEEE
- Published
- 2015-09
- Volume
- 103
- Issue
- 9
- Pages
- 1449-1477
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W1539811621
- Cited by
- 1,296
- References
- 218
- Citations by year
- 2026: 155 · 2025: 239 · 2024: 177 · 2023: 135 · 2022: 123 · 2021: 102 · 2020: 105 · 2019: 93 · 2018: 69 · 2017: 63 · 2016: 31 · 2015: 4
- FWCI
- 18.08
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Locations
- Proceedings of the IEEE (publishedVersion)
- HAL (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Funders
- Project CHESS (2012-ERC-AdG-320684))
- National Science Foundation (NSF) (IIS 1017718, CCF 1117056)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, modality, diversity, matrix factorization, tensor decomposition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects. https://app.cochat.ai/papers/10.1109/jproc.2015.2460697
- The paper
- Lahat, D., Adali, T., & Jutten, C. (2015). Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects. Proceedings of the IEEE. https://doi.org/10.1109/jproc.2015.2460697
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.