The DOI matches its Crossref record; 200 of the 203 references it cites were found in a registry or an index; 22 of the 44 sources its claims cite were checked for support.
Multimodal machine learning in precision health: A scoping review
Open accessAbstract
Machine learning is frequently being leveraged to tackle problems in the health sector including utilization for clinical decision-support. Its use has historically been focused on single modal data. Attempts to improve prediction and mimic the multimodal nature of clinical expert decision-making has been met in the biomedical field of machine learning by fusing disparate data. This review was conducted to summarize the current studies in this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA extension for Scoping Reviews to characterize multi-modal data fusion in health. Search strings were established and used in databases: PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 128 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. Early fusion was the most common data merging strategy. Notably, there was an improvement in predictive performance when using data fusion. Lacking from the papers were clear clinical deployment strategies, FDA-approval, and analysis of how using multimodal approaches from diverse sub-populations may improve biases and healthcare disparities. These findings provide a summary on multimodal data fusion as applied to health diagnosis/prognosis problems. Few papers compared the outputs of a multimodal approach with a unimodal prediction. However, those that did achieved an average increase of 6.4% in predictive performance (as measured by AUC) over the single best unimodal model.
Overview
CoChat Summary
## TL;DR
This scoping review summarizes 128 articles on multimodal machine learning in health from 2011–2021 and finds that early fusion was the most common data-merging strategy. Across the included studies, multimodal approaches generally improved predictive performance over unimodal methods, but clinical translation, FDA approval, and rigorous reporting were largely lacking.
## Key findings
- The final cohort included 128 articles, and the most common health areas were neurology and oncology.
- 126/128 papers explicitly reported a validation procedure; the most common were N-fold cross validation (55), train-test split (51), leave-one-out cross validation (10), and external dataset validation (10).
- Early fusion was the most common data-merging strategy; late fusion was much less common, with n = 20 papers.
- Among papers that compared multimodal vs unimodal prediction, the multimodal approach achieved an average increase of 6.4% in predictive accuracy/AUC.
- Less than half of the papers were published in journals intended for a clinical audience (37.6%), and none had FDA-approved clinical tools.
## Methods
- The authors conducted a PRISMA-ScR scoping review of multimodal/multiview machine learning in health, searching PubMed, Google Scholar, and IEEEXplore for studies published 2011–2021 in English.
- Inclusion criteria required original research on multimodal or multiview machine learning for diagnostic or prognostication applications in human-derived data; 128 articles met criteria after 2-person verification.
- They extracted study characteristics, health topics, modality combinations, fusion type, machine-learning methods, validation, outcomes, and limitations, and summarized the literature descriptively.
## Limitations
- The paper states that it was not a systematic review, so some eligible titles may have been missed.
- The authors note that they did not perform a critical appraisal of methodological quality of the included studies.
- Evident from the text: many included studies had common methodological issues such as single-site cohorts, small sample sizes, retrospective data, imbalanced samples, missing-data handling, confounding, and limited interpretability, which likely affect generalizability and clinical use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
16 Valid · 3 Not valid · 14 Not checked · 40 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Data from different sources and file formats are rarely uniform, and this is especially the case with clinical data21.”
§ Multimodal machine learning in precision health: A scoping review > Introduction
What the sources say
- Supportedcites [21] Kohane IS What every reader should know about studies using electronic health record data but may be afraid to ask J. Med. Internet Res. 2021 23 e22219 10.2196/22219 33600347 PMC7927948 source
The Data Collection and Handling section says clinical data elements can use different units and codes across hospitals and even within one health system, requiring harmonization. The Data Type section also describes differences between codified records and clinical narrative text. Together, these support the cited point that clinical data from different sources are often not uniform. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Medication/drug topics present an underrepresented area, with only two papers being published in this field50,66.”
§ Multimodal machine learning in precision health: A scoping review > Discussion
What the sources say
- Supportedcites [50] Sivan K Predicting risk for Alcohol Use Disorder using longitudinal data with multimodal biomarkers and family history: A machine learning study Mol. Psychiatry 2021 26 1133 1141 10.1038/s41380-019-0534-x 31595034 PMC7138692 source
The Methods (Participants; page not supplied) describe this as a study predicting later alcohol use disorder, so the source serves as an example of a drug-related paper. The statement that there were only two such papers is the citing authors’ count, not a finding attributed to this individual study. (LLM, conf=0.86)
- Supportedcites [66] Yi, Z. et al. Predicting adverse drug reactions of combined medication from heterogeneous pharmacologic databases. BMC Bioinform.19, 517 (2018). PMC6311930 30598065 source
The Abstract Background and Background and motivation identify this paper as research on adverse drug reactions from combined medication. The citation identifies one of the two medication/drug papers counted by the citing authors; it does not attribute the literature count to this source. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Seldom were machine learning investigations on prospective data, an issue endemic in the field84.”
§ Multimodal machine learning in precision health: A scoping review > Discussion
What the sources say
- Partly supportedcites [84] Niha, B. et al. Radiogenomic analysis of hypoxia pathway is predictive of overall survival in Glioblastoma. Sci. Rep.8, 7 (2018). PMC5758516 29311558 sourceExperimental — accuracy not yet measured
The Methods (Study Population) describe this machine-learning study as a retrospective analysis, and the Discussion identifies retrospective data collection as a limitation. That supports one example of non-prospective research, but the paper does not establish that prospective machine-learning investigations are seldom conducted across the field. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Multiple studies have now pointed out that if the data an ML model is trained on is biased this often yields bias in the predictions186,187.”
§ Multimodal machine learning in precision health: A scoping review > Discussion
What the sources say
- Partly supportedcites [186] Obermeyer Z Powers B Vogeli C Mullainathan S Dissecting racial bias in an algorithm used to manage the health of populations Science 2019 366 447 53. 10.1126/science.aax2342 31649194 sourceExperimental — accuracy not yet measured
The paper explains that using health care costs as a training label produced racially biased predictions of health needs because costs reflect unequal access to care (p. 2; Mechanism of bias, p. 4). This supports the claim that bias in training data can lead to biased predictions, but the study does not establish that this *often* happens across ML models. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [187] Wang H Using machine learning to integrate sociobehavioral factors in predicting cardiovascular-related mortality risk Stud. Health Technol. Inf. 2019 264 433 437 31437960 source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“While incorporating disparate data does lend itself to seemingly better predictions139, as knowledge around certain diseases accumulates, data fusion in healthcare is an evolving target that warrants proactively adapting to the dynamic landscape190.”
§ Multimodal machine learning in precision health: A scoping review > Discussion
What the sources say
- Supportedcites [139] Nikolaos K Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: A multimodal, multisite machine learning analysis JAMA Psychiatry 2018 75 1156 1172 10.1001/jamapsychiatry.2018.2165 30267047 PMC6248111 source
The Discussion reports improved prognostic performance from blending structural MRI and functional data, particularly for patients with ambiguous clinical decision scores. It describes these results as evidence of improvements from multimodal risk assessment, supporting the citing sentence’s qualified claim. The supplied passages do not provide page numbers. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [190] Luo Y Wunderink RG Lloyd-Jones D Proactive vs reactive machine learning in health care: Lessons from the COVID-19 pandemic JAMA 2022 327 623 624 10.1001/jama.2021.24935 35085382 source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
32 more claims · 1 with their cited sources checked
Create a free account to see all 73 claims and every source verdict.
- Probably not a substantial claim. 22
- It may state two separate claims as one. 18
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI can analyse a patient’s genetic makeup, lifestyle and medical history to predict how they might respond to different treatments [48].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning models that integrate task-relevant multimodal EHR have been shown to yield superior performance compared to those relying on a single modality2–6.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using various fusion strategies designed to integrate information from multiple data modalities effectively, multimodal models allow for the compensation of information that may be missing or incomplete in any single modality, thus providing more comprehensive and clinically relevant insights into a patient’s condition that can match the cognitive behavior of healthcare providers [7,8,16,17].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In multimodal AI, variations in clinical practice, resource availability, and patient characteristics can lead to systematic underrepresentation of certain patient groups or clinical scenarios in the data, resulting in biased model training and reduced fairness [8,17].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“which aligns with the 6.4 percentage points improvement reported previously by Kline et al. (2022).”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Kline et al. (2022)
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
(2017) · HAL · Cited by 10,512
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092 · cited 1×
“This is likely fostered by curated databases that lend themselves to multi-modality predictions such as Alzheimer ’ s Disease Neuroimaging Initiative 165 and The Cancer Genome Atlas Program 166 .”
BackgroundDissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
By the 2019 American Geriatrics Society Beers Criteria® Update Expert Panel (2019) · Journal of the American Geriatrics Society · doi:10.1111/jgs.15767 · Cited by 3,163
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Deep learning for healthcare: review, opportunities and challenges
Miotto, R., Wang, F., Wang, S., Jiang, X., & Dudley, J. T. (2018) · Briefings in Bioinformatics · doi:10.1093/bib/bbx044 · Cited by 3,076 · cited 1×
“Clinical decision support has long been an aim for those implementing algorithms and machine learning in the health sphere 1 – 3 .”
Self-citationEnsemble based systems in decision making
Polikar, R. (2006) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2006.1688199 · Cited by 2,938
Alzheimer's Disease Neuroimaging Initiative (ADNI)
Petersen, R. C., Aisen, P. S., Beckett, L. A., Donohue, M. C., Gamst, A. C., Harvey, D. J., Jack, C. R., Jagust, W. J., Shaw, L. M., Toga, A. W., et al. (2010) · Neurology · doi:10.1212/wnl.0b013e3181cb3e25 · Cited by 2,572
Research works printed but not linked: 8
- Alan, N. S., Christopher, L. B. & Franklin, E. W. Revisions to the JDL data fusion model. Sens. Fusion.: Architectures, Algorithms, Appl. III 3719, 430 (1999). publisher = SPIE.
- Xiaosong, W., Yifan, P., Le, L., Zhiyong, L. & Ronald, M. S. TieNet: Text-Image embedding network for common thorax disease classification and reporting in chest X-rays. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 9049–9058 (IEEE Computer Society, 2018).
- Yanbo, X., Siddharth, B., Shriprasad, R. D., Kevin, O. M. & Jimeng, S. RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2565–2573 (Association for Computing Machinery, 2018).
- Part of 13th International Joint Conference on Biomedical Engineering Systems and Technologies, BIOSTEC 2020 421-428 (2020).
- Wenhuan, Z., Anupam, G. & Daniel, H. H. On the application of advanced machine learning methods to analyze enhanced, multimodal data from persons infected with covid-19. Computation 9, 1–15 (2021).
- Cao, W. et al. Brits: Bidirectional Recurrent Rmputation for Time Series. NeurIPS, 31 (2018).
- Luo, Y. & Mao, C. ScanMap: Supervised confounding aware non-negative matrix factorization for polygenic risk modeling. In Machine Learning for Healthcare Conference; 2020: PMLR 27–45 (2020).
- Kalavathy, R. & Suresh, R. M. Pharmacovigilance from electronic medical records to report adverse events. J. Chem. Pharm. Sci. 2015, 188–191 (2015).
Cited by
The emergence of AI-based wearable sensors for digital health technology: a review
Shajari, S., Kuruvinashetti, K., Komeili, A., & Sundararaj, U. (2023) · Sensors · doi:10.3390/s23239498 · Cited by 535
Reddy, S. (2024) · Implementation Science · doi:10.1186/s13012-024-01357-9 · Cited by 554
“Generative AI can analyse a patient’s genetic makeup, lifestyle and medical history to predict how they might respond to different treatments [48].”
BackgroundTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
“FMs can be designed to accept a wide range of data modalities (e.g., structured codes, lab values, clinical text, images, speech patterns, etc.) as inputs and incorporate them into a single uni fi ed representation 47 .”
BackgroundMultimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“Finally, multimodality characterizes systems designed to generate outputs in >1 modality, such as systems capable of producing both textual and image-based content [40].”
Metadata
- Authors
- Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y.
- Affiliations
- Northwestern University (US)
- Cornell University (US)
- Cleveland Clinic Lerner College of Medicine (US)
- Corresponding author
- Luo, Yuan
- Venue
- npj Digital Medicine
- Published
- 2022-11-07
- Volume
- 5
- Issue
- 1
- Pages
- 171-171
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC9640667
- OpenAlex
- W4308885870
- Cited by
- 528
- References
- 203
- Citations by year
- 2026: 162 · 2025: 173 · 2024: 114 · 2023: 64 · 2022: 3
- FWCI
- 105.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Keywords
- multimodal fusion, precision health, clinical decision support, feature-level fusion, neurology, prediction accuracy, clinical deployment, health care disparities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal machine learning in precision health: A scoping review. https://app.cochat.ai/papers/10.1038/s41746-022-00712-8
- The paper
- Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022). Multimodal machine learning in precision health: A scoping review. npj Digital Medicine. https://doi.org/10.1038/s41746-022-00712-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.