The DOI matches its Crossref record; 175 of the 182 references it cites were found in a registry or an index; 96 of the 129 sources its claims cite were checked for support.
Multimodal biomedical AI
Open accessAbstract
The increasing availability of biomedical data from large biobanks, electronic health records, medical imaging, wearable and ambient biosensors, and the lower cost of genome and microbiome sequencing have set the stage for the development of multimodal artificial intelligence solutions that capture the complexity of human health and disease. In this Review, we outline the key applications enabled, along with the technical and analytical challenges. We explore opportunities in personalized medicine, digital clinical trials, remote monitoring and care, pandemic surveillance, digital twin technology and virtual health assistants. Further, we survey the data, modeling and privacy challenges that must be overcome to realize the full potential of multimodal artificial intelligence in health. Multimodal artificial intelligence models could unlock many exciting applications in health and medicine; this Review outlines the most promising uses and the technical pitfalls to avoid.
Overview
CoChat Summary
TL;DR
This Review surveys how multimodal biomedical AI—integrating omics, EHR, imaging, wearables, ambient sensors, and other data—could enable precision health, digital clinical trials, remote monitoring, pandemic surveillance, digital twins, and virtual health assistants. It also catalogs technical, data, and privacy challenges that must be overcome before these applications can be validated and deployed.
Key findings
- Multimodal data can improve prediction over single modalities: a DETECT Health study follow-up found that jointly using self-reported symptoms and wearable sensor metrics reached an AUC of 0.80 (95% CI 0.73–0.86) for classifying COVID-19-positive versus COVID-19-negative status.
- Large multimodal resources are expanding: UK Biobank enrolled over 500,000 participants with plans to follow for at least 30 years; All of Us enrolled about 400,000 diverse participants toward a 1 million target; MIMIC contains de-identified data from thousands of critical care patients and about 380,000 records in the table.
- Fusion of modalities can outperform single-modality models: imaging plus EHR data improved pulmonary embolism detection, and a DeepMind EHR-based model projected 620,000 dimensions into 800 dimensions over a 6-h time frame to predict acute kidney injury.
- The field faces major unresolved challenges: high dimensionality and “dataset blind spots,” missing data, selection and racial bias, privacy/re-identification risks, and the need for randomized trials to validate remote monitoring and virtual health assistants.
Methods
- Design: narrative Review article; the authors state they outline key applications, technical and analytical challenges, and promising strategies, and explicitly do not discuss basic AI/machine learning concepts.
- Data or sample: no primary dataset or systematic sample; the paper synthesizes published literature, examples, and existing multimodal resources such as UK Biobank, China Kadoorie Biobank, Biobank Japan, Million Veteran Program, TOPMed, All of Us, Project Baseline, American Gut Project, MIMIC, MIPACT, and the North American Prodrome Longitudinal Study.
- Main analysis: conceptual and narrative synthesis of opportunities (personalized omics, digital clinical trials, remote monitoring, pandemic surveillance, digital twins, virtual health assistants) and challenges (modeling, data collection/harmonization, bias, privacy, data ownership).
Limitations
- The paper states that many multimodal AI use cases require further validation and replication, that remote patient monitoring for hospital-at-home is “not yet validated” and needs randomized trials versus hospital admission, and that most virtual health assistant applications have been tested only in small observational studies and need randomized clinical trials.
- The paper states that current methods are unlikely to be sufficient to overcome all major challenges and that further innovation is required; it also notes that false positives and clinician interpretation of risks were outside the scope of the Review.
- Evident from the text: as a narrative Review, it does not report a systematic search strategy, formal inclusion/exclusion criteria, or quantitative meta-analysis, so its claims are illustrative rather than systematically synthesized.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
80 Valid · 0 Not valid · 17 Not checked · 42 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A follow-up study from this program showed that jointly considering participant self-reported symptoms and sensor metrics improved performance relative to either modality alone, reaching an area under the receiver operating curve value of 0.80 (95% confidence interval 0.73-0.86) for classifying COVID-19-positive versus COVID-19-negative status 61 .”
What the sources say
- Partly supportedcites [61] Quer, G. et al. Wearable sensor data and self-reported symptoms for COVID-19 detection. Nat. Med. 27, 73–77 (2020). sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent progress with biosensors, continuous monitoring and analytics raises the possibility of simulating the hospital setting in a person's home.”
What the sources say
- Partly supportedcites [48] Noah, B. et al. Impact of remote patient monitoring on clinical outcomes: an updated meta-analysis of randomized controlled trials. NPJ Digit. Med. 1, 20172 (2018). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says wearable biosensors can continuously track health data and transmit it for patient or provider assessment, supporting the monitoring aspect of the claim. It does not say that these advances could simulate a hospital setting in a person's home, and the retrieved passages do not establish that possibility. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Data from wearable technology (including heart rate, sleep, physical activity, electrocardiography, oxygen saturation and glucose monitoring) and smartphone-enabled self-reported questionnaires can be useful for monitoring clinical trial patients, identifying adverse events or ascertaining trial outcomes 34 .”
What the sources say
- Partly supportedcites [34] Inan, O. T. et al. Digitizing clinical trials. NPJ Digit. Med. 3, 101 (2020). sourceExperimental — accuracy not yet measured
The Digital health data collection and Digital monitoring and biomarkers sections describe smartphone-collected patient-reported outcomes, wearable measurements, and their potential use as trial endpoints. Safety monitoring in digital clinical trials says continuous digital data might improve detection of infrequent events and reporting of adverse events. The paper supports the overall claim, but the cited passages do not identify oxygen-saturation monitoring, one of the examples in the citing sentence. (LLM, conf=0.84)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, geographic, sociocultural and economic disparities in access to enrollment, have led to a remarkable underrepresentation of several groups in these studies.”
What the sources say
- Partly supportedcites [32] Loree, J. M. et al. Disparity of race reporting and representation in clinical trials leading to cancer drug approvals from 2008 to 2018. JAMA Oncol. 5, e191870 (2019). sourceExperimental — accuracy not yet measured
The Discussion reports substantial underrepresentation of Black and Hispanic patients and describes distance, transportation, income, and distrust as barriers to participation. It says socioeconomic status may explain part of the disparity and that the reasons are complex; it does not establish that these factors caused the underrepresentation as the claim states. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“One study has also demonstrated the utilization of resting heart rate and sleep minutes tracked using wearable devices to improve surveillance of influenza-like illness in the USA 60 .”
What the sources say
- Supportedcites [60] Radin, J. M., Wineinger, N. E., Topol, E. J. & Steinhubl, S. R. Harnessing wearable device data to improve state-level real-time surveillance of influenza-like illness in the USA: a population-based study. Lancet Digit. Health 2, e85–e93 (2020). source
The Methods (Data collection) state that the wearable-device dataset included daily resting heart rate and sleep minutes. The Discussion reports improved prediction of state-level influenza-like illness rates when sleep and heart-rate measures were incorporated. (LLM, conf=0.94)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
40 more claims · 17 with their cited sources checked
Create a free account to see all 139 claims and every source verdict.
- It may state two separate claims as one. 51
- Probably not a substantial claim. 61
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Joint fusion learns a shared embedding space across modalities, which can represent cross-modal relationships beyond those considered in early or late fusion.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By using various fusion strategies designed to integrate information from multiple data modalities effectively, multimodal models allow for the compensation of information that may be missing or incomplete in any single modality, thus providing more comprehensive and clinically relevant insights into a patient’s condition that can match the cognitive behavior of healthcare providers [7,8,16,17].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This large volume of heterogeneous data offers substantial potential for advancing precision medicine to inform clinical decision-making in the ICU, yet it exceeds the ability of clinicians to process and interpret all available information in real time [7,8].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In multimodal AI, variations in clinical practice, resource availability, and patient characteristics can lead to systematic underrepresentation of certain patient groups or clinical scenarios in the data, resulting in biased model training and reduced fairness [8,17].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Multimodal data harmonization is particularly challenging, as differences in data formats, medical devices, variable definitions, and measurement units across centers make it difficult to effectively combine and analyze diverse datasets for robust model development and validation [8,107].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [8]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
A global reference for human genetic variation
The 1000 Genomes Project Consortium, authors, C., Auton, A., Abecasis, G. R., Steering committee, Altshuler, D. M., Durbin, R. M., Abecasis, G. R., Bentley, D. R., Chakravarti, A., et al. (2015) · Nature · doi:10.1038/nature15393 · Cited by 18,247
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Self-citationThe UK Biobank resource with deep phenotyping and genomic data
Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L. T., Sharp, K., Motyer, A., Vukcevic, D., Delaneau, O., O’Connell, J., et al. (2018) · Nature · doi:10.1038/s41586-018-0579-z · Cited by 8,876 · cited 1×
“In the UK, the UK Biobank initiated enrollment in 2006, reaching a final participant count of over 500,000, and plans to follow participants for at least 30 years after enrollment 81 .”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Medical AI algorithms using demographic features such as race as inputs can learn to perpetuate historical human biases, thereby resulting in harm when deployed 156 .”
MethodAn Integrated Genomic Analysis of Human Glioblastoma Multiforme
Parsons, D. W., Jones, S., Zhang, X., Lin, J. C.-H., Leary, R. J., Angenendt, P., Mankoo, P., Carter, H., Siu, I.-M., Gallia, G. L., et al. (2008) · Science · doi:10.1126/science.1164382 · Cited by 5,857 · cited 1×
“Genetic and molecular markers of malignant tumors have been integrated into clinical practice 5,6 , with the US Food and Drug Administration (FDA) providing approval for several companion diagnostic devices and nucleic acid-based tests 7,8 .”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312
Research works printed but not linked: 3
- Zhang, X., Zeman, M., Tsiligkaridis, T. & Zitnik, M. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representation (ICLR, 2022).
- Luo, Z. et al. Computer vision-based descriptive analytics of seniors’ daily activities for long-term health monitoring. In Proc. Machine Learning Research Vol. 85, 1–18 (PMLR, 2018).
- Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M. Retrieval augmented language model pre-training. In Proc. 37th International Conference on Machine Learning (eds. Iii, H. D. & Singh, A.) vol. 119, 3929–3938 (PMLR, 13–18 July 2020).
Cited by
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Self-citationScientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
The next generation of evidence-based medicine
Subbiah, V. (2023) · Nature Medicine · doi:10.1038/s41591-022-02160-z · Cited by 544
Artificial intelligence in drug development
Zhang, K., Yang, X., Wang, Y., Yu, Y., Huang, N., Li, G., Li, X., Wu, J. C., & Yang, S. (2025) · Nature Medicine · doi:10.1038/s41591-024-03434-4 · Cited by 498
Harrer, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104512 · Cited by 480
“…will run into the ‘ big data ’ problem, i.e., will have to face the fact that while theoretically electronic health records are a multimodal bottomless ocean of big data poised to be analysed using deep learning models, 74 when it comes to getting access to such data, they very much are not.”
Background
Metadata
- Authors
- Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J.
- Affiliations
- Yale University (US)
- Harvard University (US)
- Scripps Research Institute (US)
- Corresponding author
- Rajpurkar, Pranav
- Topol, Eric J.
- Venue
- Nature Medicine
- Published
- 2022-09
- Volume
- 28
- Issue
- 9
- Pages
- 1773-1784
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4295951577
- Semantic Scholar
- dda118e8154765f73cb8f5e2b1b8daa75faf726f
- Cited by
- 1,365
- References
- 182
- Citations by year
- 2026: 474 · 2025: 387 · 2024: 306 · 2023: 186 · 2022: 12
- FWCI
- 15.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Medicine (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Funders
- National Center for Advancing Translational Science (NCATS)/National Institutes of Health
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal AI, biomedical data integration, biobanks, electronic health records, diagnostic imaging, wearable biosensors, microbiome sequencing, personalized medicine, remote patient monitoring, pandemic surveillance, digital twin, virtual health assistants, data privacy
- MeSH terms
- Artificial Intelligence, Humans, Privacy, Electronic Health Records, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal biomedical AI. https://app.cochat.ai/papers/10.1038/s41591-022-01981-2
- The paper
- Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022). Multimodal biomedical AI. Nature Medicine. https://doi.org/10.1038/s41591-022-01981-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.