Health system-scale language models are all-purpose prediction engines
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Physicians make critical time-constrained decisions every day. Clinical predictive models can help physicians and administrators make decisions by forecasting clinical and operational events. Existing structured data-based clinical predictive models have limited use in everyday practice owing to complexity in data processing, as well as model development and deployment 1–3 . Here we show that unstructured clinical notes from the electronic health record can enable the training of clinical language models, which can be used as all-purpose clinical predictive engines with low-resistance development and deployment. Our approach leverages recent advances in natural language processing 4,5 to train a large language model for medical language (NYUTron) and subsequently fine-tune it across a wide range of clinical and operational predictive tasks. We evaluated our approach within our health system for five such tasks: 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction. We show that NYUTron has an area under the curve (AUC) of 78.7–94.9%, with an improvement of 5.36–14.7% in the AUC compared with traditional models. We additionally demonstrate the benefits of pretraining with clinical text, the potential for increasing generalizability to different sites through fine-tuning and the full deployment of our system in a prospective, single-arm trial. These results show the potential for using clinical language models in medicine to read alongside physicians and provide guidance at the point of care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large Language Models (LLMs) have attained human-level accuracy on medical question-answer (QA) benchmarks.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, fine-tuned LLMs have been shown to solve real-world problems such as predicting clinical outcomes just based on unstructured text data in EHRs13.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [13]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“These AI systems facilitate real-time image analysis, providing the surgeon with enhanced visualizations of the surgical site and critical anatomical structures, thus improving decision-making and reducing intraoperative risks.102”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [102]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Thus, the streamlined incorporation of these regulatory requirements into business processes via the QMS assures enduring safety, effectiveness, ethicality, regulatory compliance, and alignment with organizational and user needs as AI-enabled methodologies, such as LLMs, evolve10.”
From Implementing quality management systems to close the AI translation gap and facilitate safe, ethical, and effective health AI solutions (2023) · cites this paper as [10]
Experimental — accuracy not yet measuredIn the citing paper
“NYUTron has an AUC of 78.7-94.9%.”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [77]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Charlson, M. E., Pompei, P., Ales, K. L., & MacKenzie, C. R. (1987) · Journal of Chronic Diseases · doi:10.1016/0021-9681(87)90171-8 · Cited by 49,742
APACHE II: A severity of disease classification system
KNAUS, W. A., DRAPER, E. A., WAGNER, D. P., & ZIMMERMAN, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 13,664
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Our approach leverages recent advances in natural language processing that suggest that sufficiently scaled, self-supervised LLMs can out-perform strongly supervised approaches on non-medical predictive tasks 4,5,13 .”
MethodMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Transection of the oesophagus for bleeding oesophageal varices
Pugh, R. N. H., Murray-Lyon, I. M., Dawson, J. L., Pietroni, M. C., & Williams, R. (1973) · Journal of British Surgery · doi:10.1002/bjs.1800600817 · Cited by 7,941
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773
APACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034
THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED
WELCH, B. L. (1947) · Biometrika · doi:10.1093/biomet/34.1-2.28 · Cited by 4,135
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Research works printed but not linked: 11
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. in Proc. 2019 NAACL: Human Language Technologies (eds Burstein, J., Doran, C. & Solorio, T.) 4171–4186 (Association for Computational Linguistics, 2019).
- Brown, T. B. et al. Language models are few-shot learners. in Proc. NeurIPS (eds Wallach, H. et al.) 1877–1901 (Neural Information Processing Systems, 2020).
- Child, C. G. & Turcotte, J. G. Surgery and portal hypertension. Major Prob. Clin. Surg. 1, 1–85 (1964).
- van Walraven, C., Wong, J. & Forster, A. J. LACE+ index: extension of a validated index to predict early death or urgent readmission after hospital discharge using administrative data. Open Med. 6, 80–90 (2012).
- Perez, E., Kiela, D. & Cho, K. True few-shot learning with language models. in Proc. NeurIPS (eds Ranzato, M. et al.) 11054–11070 (Neural Information Processing Systems, 2021).
- Wolf, T. et al. Transformers: state-of-the-art natural language processing. in Proc. 2020 EMNLP (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) 38–45 (Association for Computational Linguistics, 2020).
- Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. ICLR https://openreview.net/forum?id=Bkg6RiCqY7 (2019).
- Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. ICLR https://arxiv.org/abs/1412.6980 (2017). · arXiv
- Pedregosa, F. et al. Scikit-Learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
- NCBI Literature Resources. Download PubMed data. https://pubmed.ncbi.nlm.nih.gov/download/ (2022).
- National Library of Medicine. PubMed Central: PMC article datasets. https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/ (2022).
Cited by
A whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615
Self-citationZack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00225-x · Cited by 510
The application of large language models in medicine: A scoping review
Meng, X., Yan, X., Zhang, K., Liu, D., Cui, X., Yang, Y., Zhang, M., Cao, C., Wang, J., Wang, X., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109713 · Cited by 414
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
“A recent study 32 examined an LLM developed at New York University, i.e., NYUTron, and our previously developed Gator-Tron 15 for prediction of readmission, in-hospital mortality, comorbidity, length of stay, and insurance denial, demonstrating the potential utility of LLMs in healthcare.”
Self-citation
Metadata
- Authors
- Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., Miceli, M., Kim, N. C., Orillac, C., Schnurman, Z., Livia, C., Weiss, H., Kurland, D., Neifert, S., Dastagirzada, Y., Kondziolka, D., Cheung, A. T. M., Yang, G., Cao, M., Flores, M., Costa, A. B., Aphinyanaphongs, Y., Cho, K., & Oermann, E. K.
- Affiliations
- NYU Langone Health (US)
- Nvidia (United States) (US)
- Canadian Institute for Advanced Research (CA)
- Corresponding author
- Oermann, Eric Karl
- ORCID
- Jiang, Lavender Yao
- Nasir-Moin, Mustafa
- Wang, Duo
- Abidin, Anas
- Eaton, Kevin
- Riina, Howard Antony
- Laufer, Ilya
- Kim, Nora C.
- Schnurman, Zane
- Livia, Christopher
- Weiss, Hannah
- Kurland, David
- Neifert, Sean
- Dastagirzada, Yosef
- Kondziolka, Douglas
- Cheung, Alexander T. M.
- Cao, Ming
- Flores, Mona
- Costa, Anthony B.
- Aphinyanaphongs, Yindalon
- Cho, Kyunghyun
- Oermann, Eric Karl
- Venue
- Nature
- Published
- 2023-06-07
- Volume
- 619
- Issue
- 7969
- Pages
- 357-362
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC10338337
- OpenAlex
- W4379769651
- Cited by
- 460
- References
- 35
- Citations by year
- 2026: 101 · 2025: 163 · 2024: 158 · 2023: 37
- FWCI
- 43.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- W. M. Keck Foundation
- National Cancer Institute (P30CA016087)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical language models, large language models, electronic health records, unstructured clinical notes, 30-day readmission prediction, in-hospital mortality prediction, comorbidity index, length of stay prediction, fine-tuning, clinical decision support
- MeSH terms
- Clinical Decision-Making, Clinical Trials as Topic, Humans, Length of Stay, Natural Language Processing, Patient Readmission, Physicians, Comorbidity, Hospital Mortality, Point-of-Care Systems, Insurance Coverage, Area Under Curve, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Health system-scale language models are all-purpose prediction engines. https://app.cochat.ai/papers/10.1038/s41586-023-06160-y
- The paper
- Jiang, L. Y., Liu, X. C., Nejatian, N. P., Nasir-Moin, M., Wang, D., Abidin, A., Eaton, K., Riina, H. A., Laufer, I., Punjabi, P., Miceli, M., Kim, N. C., Orillac, C., Schnurman, Z., Livia, C., Weiss, H., Kurland, D., Neifert, S., Dastagirzada, Y., . . . Oermann, E. K. (2023). Health system-scale language models are all-purpose prediction engines. Nature. https://doi.org/10.1038/s41586-023-06160-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.