Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 200 of the 203 references it cites were found in a registry or an index; 22 of the 44 sources its claims cite were checked for support.

Multimodal machine learning in precision health: A scoping review

Open access
Kline, Adrienne, Wang, Hanyin, Li, Yikuan, Dennis, Saya, Hutch, Meghan, Xu, Zhenxing, Wang, Fei, Cheng, Feixiong, Luo, Yuan · 2022 · npj Digital Medicine

Abstract

Machine learning is frequently being leveraged to tackle problems in the health sector including utilization for clinical decision-support. Its use has historically been focused on single modal data. Attempts to improve prediction and mimic the multimodal nature of clinical expert decision-making has been met in the biomedical field of machine learning by fusing disparate data. This review was conducted to summarize the current studies in this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA extension for Scoping Reviews to characterize multi-modal data fusion in health. Search strings were established and used in databases: PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 128 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. Early fusion was the most common data merging strategy. Notably, there was an improvement in predictive performance when using data fusion. Lacking from the papers were clear clinical deployment strategies, FDA-approval, and analysis of how using multimodal approaches from diverse sub-populations may improve biases and healthcare disparities. These findings provide a summary on multimodal data fusion as applied to health diagnosis/prognosis problems. Few papers compared the outputs of a multimodal approach with a unimodal prediction. However, those that did achieved an average increase of 6.4% in predictive performance (as measured by AUC) over the single best unimodal model.

Overview

CoChat Summary

## TL;DR

This scoping review summarizes 128 articles on multimodal machine learning in health from 2011–2021 and finds that early fusion was the most common data-merging strategy. Across the included studies, multimodal approaches generally improved predictive performance over unimodal methods, but clinical translation, FDA approval, and rigorous reporting were largely lacking.

## Key findings

  • The final cohort included 128 articles, and the most common health areas were neurology and oncology.
  • 126/128 papers explicitly reported a validation procedure; the most common were N-fold cross validation (55), train-test split (51), leave-one-out cross validation (10), and external dataset validation (10).
  • Early fusion was the most common data-merging strategy; late fusion was much less common, with n = 20 papers.
  • Among papers that compared multimodal vs unimodal prediction, the multimodal approach achieved an average increase of 6.4% in predictive accuracy/AUC.
  • Less than half of the papers were published in journals intended for a clinical audience (37.6%), and none had FDA-approved clinical tools.

## Methods

  • The authors conducted a PRISMA-ScR scoping review of multimodal/multiview machine learning in health, searching PubMed, Google Scholar, and IEEEXplore for studies published 2011–2021 in English.
  • Inclusion criteria required original research on multimodal or multiview machine learning for diagnostic or prognostication applications in human-derived data; 128 articles met criteria after 2-person verification.
  • They extracted study characteristics, health topics, modality combinations, fusion type, machine-learning methods, validation, outcomes, and limitations, and summarized the literature descriptively.

## Limitations

  • The paper states that it was not a systematic review, so some eligible titles may have been missed.
  • The authors note that they did not perform a critical appraisal of methodological quality of the included studies.
  • Evident from the text: many included studies had common methodological issues such as single-site cohorts, small sample sizes, retrospective data, imbalanced samples, missing-data handling, confounding, and limited interpretability, which likely affect generalizability and clinical use.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-022-00712-8

Cites

200 of 200 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Multimodal precision health; the flow of information. Information moves in a cyclical pattern from health centers to information commons, where it can be transformed and algorithmic modeling performed. These algorithms provide insight into many different health outcomes such as clinical trials, phenotyping, drug discovery, etc. These insights should return to health centers and practitioners to provide the most efficient, evidence-based medicine possible.
Fig. 1 Multimodal precision health; the flow of information. Information moves in a cyclical pattern from health centers to information commons, where it can be transformed and algorithmic modeling performed. These algorithms provide insight into many different health outcomes such as clinical trials, phenotyping, drug discovery, etc. These insights should return to health centers and practitioners to provide the most efficient, evidence-based medicine possible.CC BY · PubMed Central
Fig. 2 Early, intermediate, and late fusion; flow ofinformation from information commons to model structure to outcomes. Information fusion can occur in a myriad of ways. In machine learning, early, intermediate, and late fusion is typified by if all the information flows into a single model (early), a step-wise fashion where outputs from one model become inputs for another (intermediate), and lastly, where all unique data types undergo separate modelling after which ensembling and/or voting occurs (late).
Fig. 2 Early, intermediate, and late fusion; flow ofinformation from information commons to model structure to outcomes. Information fusion can occur in a myriad of ways. In machine learning, early, intermediate, and late fusion is typified by if all the information flows into a single model (early), a step-wise fashion where outputs from one model become inputs for another (intermediate), and lastly, where all unique data types undergo separate modelling after which ensembling and/or voting occurs (late).CC BY · PubMed Central
Fig. 3 Topic and Modality Modeling. Neurology, and in particular, Alzheimer’s disease investigations accounted for the most papers published on this topic (n = 22). With the onset of the COVID-19 pandemic, several primary research articles were dedicated to this topic, which can be arrived at through the respiratory or infectious disease hierarchies. All papers noted in this review used either two or three disparate data sources when fusing their data, and specifically that of imaging and EHR (n = 52), was the most prevalent.
Fig. 3 Topic and Modality Modeling. Neurology, and in particular, Alzheimer’s disease investigations accounted for the most papers published on this topic (n = 22). With the onset of the COVID-19 pandemic, several primary research articles were dedicated to this topic, which can be arrived at through the respiratory or infectious disease hierarchies. All papers noted in this review used either two or three disparate data sources when fusing their data, and specifically that of imaging and EHR (n = 52), was the most prevalent.CC BY · PubMed Central
Fig. 4 Meta-data from the review process. a Heat map of fusion type broken down into the coding platforms papers used by summing over paper counts (those that mentioned platform used), the most popular being the Python platform and early fusion. Of note, 37 of the papers did not explicitly mention a platform. b Total number of original research papers published in this sphere in the last 10 years. c Continental breakdown of author contributions (note some papers have authors from multiple continents). d Breakdown of publication type (clinical/non-clinical journal). Less than half (37.6%) of the papers were published in a journal intended for a clinical audience. e Sex breakdown of populations studied. Both men and women were represented in the papers, however, the degree of representation varied within an individual studies.
Fig. 4 Meta-data from the review process. a Heat map of fusion type broken down into the coding platforms papers used by summing over paper counts (those that mentioned platform used), the most popular being the Python platform and early fusion. Of note, 37 of the papers did not explicitly mention a platform. b Total number of original research papers published in this sphere in the last 10 years. c Continental breakdown of author contributions (note some papers have authors from multiple continents). d Breakdown of publication type (clinical/non-clinical journal). Less than half (37.6%) of the papers were published in a journal intended for a clinical audience. e Sex breakdown of populations studied. Both men and women were represented in the papers, however, the degree of representation varied within an individual studies.CC BY · PubMed Central
Fig. 5 Limitations to multimodal fusion in health and proposed future directions of the fields. Limitations to multimodal fusion implementation are stratified by their location in the workflow. These include issues associated with the underlying data, the modeling that arises from that data, and finally how these are ported back to health systems to provide translational decision support.
Fig. 5 Limitations to multimodal fusion in health and proposed future directions of the fields. Limitations to multimodal fusion implementation are stratified by their location in the workflow. These include issues associated with the underlying data, the modeling that arises from that data, and finally how these are ported back to health systems to provide translational decision support.CC BY · PubMed Central
Fig. 6 Overview of our PRIMSA-SCR process. a Health-related keyword, Multimodal-related keyword, machine learning-related keywords, |: or. For example, “health + heterogeneous data + machine learning” would be one of the search strings. b Overview of study inclusion process. c Research questions posed.
Fig. 6 Overview of our PRIMSA-SCR process. a Health-related keyword, Multimodal-related keyword, machine learning-related keywords, |: or. For example, “health + heterogeneous data + machine learning” would be one of the search strings. b Overview of study inclusion process. c Research questions posed.CC BY · PubMed Central

Verification

Claims the paper makes

16 Valid · 3 Not valid · 14 Not checked · 40 Own results and interpretations

32 more claims · 1 with their cited sources checked

Create a free account to see all 73 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 22
  • It may state two separate claims as one. 18

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

6 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 203 references

Research works printed but not linked: 8

  1. Alan, N. S., Christopher, L. B. & Franklin, E. W. Revisions to the JDL data fusion model. Sens. Fusion.: Architectures, Algorithms, Appl. III 3719, 430 (1999). publisher = SPIE.
  2. Xiaosong, W., Yifan, P., Le, L., Zhiyong, L. & Ronald, M. S. TieNet: Text-Image embedding network for common thorax disease classification and reporting in chest X-rays. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 9049–9058 (IEEE Computer Society, 2018).
  3. Yanbo, X., Siddharth, B., Shriprasad, R. D., Kevin, O. M. & Jimeng, S. RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2565–2573 (Association for Computing Machinery, 2018).
  4. Part of 13th International Joint Conference on Biomedical Engineering Systems and Technologies, BIOSTEC 2020 421-428 (2020).
  5. Wenhuan, Z., Anupam, G. & Daniel, H. H. On the application of advanced machine learning methods to analyze enhanced, multimodal data from persons infected with covid-19. Computation 9, 1–15 (2021).
  6. Cao, W. et al. Brits: Bidirectional Recurrent Rmputation for Time Series. NeurIPS, 31 (2018).
  7. Luo, Y. & Mao, C. ScanMap: Supervised confounding aware non-negative matrix factorization for polygenic risk modeling. In Machine Learning for Healthcare Conference; 2020: PMLR 27–45 (2020).
  8. Kalavathy, R. & Suresh, R. M. Pharmacovigilance from electronic medical records to report adverse events. J. Chem. Pharm. Sci. 2015, 188–191 (2015).

Cited by

See all 528

Metadata

Authors
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y.
Affiliations
  • Northwestern University (US)
  • Cornell University (US)
  • Cleveland Clinic Lerner College of Medicine (US)
Corresponding author
Luo, Yuan
Venue
npj Digital Medicine
Published
2022-11-07
Volume
5
Issue
1
Pages
171-171
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4308885870
Cited by
528
References
203
Citations by year
2026: 162 · 2025: 173 · 2024: 114 · 2023: 64 · 2022: 3
FWCI
105.22
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
Keywords
multimodal fusion, precision health, clinical decision support, feature-level fusion, neurology, prediction accuracy, clinical deployment, health care disparities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal machine learning in precision health: A scoping review. https://app.cochat.ai/papers/10.1038/s41746-022-00712-8
The paper
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022). Multimodal machine learning in precision health: A scoping review. npj Digital Medicine. https://doi.org/10.1038/s41746-022-00712-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.