Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department
Open accessAbstract
Importance: The introduction of large language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4; OpenAI), has generated significant interest in health care, yet studies evaluating their performance in a clinical setting are lacking. Determination of clinical acuity, a measure of a patient’s illness severity and level of required medical attention, is one of the foundational elements of medical reasoning in emergency medicine. Objective: To determine whether an LLM can accurately assess clinical acuity in the emergency department (ED). Design, Setting, and Participants: This cross-sectional study identified all adult ED visits from January 1, 2012, to January 17, 2023, at the University of California, San Francisco, with a documented Emergency Severity Index (ESI) acuity level (immediate, emergent, urgent, less urgent, or nonurgent) and with a corresponding ED physician note. A sample of 10 000 pairs of ED visits with nonequivalent ESI scores, balanced for each of the 10 possible pairs of 5 ESI scores, was selected at random. Exposure: The potential of the LLM to classify acuity levels of patients in the ED based on the ESI across 10 000 patient pairs. Using deidentified clinical text, the LLM was queried to identify the patient with a higher-acuity presentation within each pair based on the patients’ clinical history. An earlier LLM was queried to allow comparison with this model. Main Outcomes and Measures: Accuracy score was calculated to evaluate the performance of both LLMs across the 10 000-pair sample. A 500-pair subsample was manually classified by a physician reviewer to compare performance between the LLMs and human classification. Results: From a total of 251 401 adult ED visits, a balanced sample of 10 000 patient pairs was created wherein each pair comprised patients with disparate ESI acuity scores. Across this sample, the LLM correctly inferred the patient with higher acuity for 8940 of 10 000 pairs (accuracy, 0.89 [95% CI, 0.89-0.90]). Performance of the comparator LLM (accuracy, 0.84 [95% CI, 0.83-0.84]) was below that of its successor. Among the 500-pair subsample that was also manually classified, LLM performance (accuracy, 0.88 [95% CI, 0.86-0.91]) was comparable with that of the physician reviewer (accuracy, 0.86 [95% CI, 0.83-0.89]). Conclusions and Relevance: In this cross-sectional study of 10 000 pairs of ED visits, the LLM accurately identified the patient with higher acuity when given pairs of presenting histories extracted from patients’ first ED documentation. These findings suggest that the integration of an LLM into ED workflows could enhance triage processes while maintaining triage quality and warrants further investigation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most studies applying large language models on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data.
“Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [18]
Large language models (LLMs) can be integrated to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks.
“Moreover, large language models (LLMs) can be integrated into this principle to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks [23, 48–52].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449 · see citing works
Ambient artificial intelligence scribes to alleviate the burden of clinical documentation.
Tierney, A. A., Gayre, G., Hoberman, B., Mattern, B., Ballesca, M., Kipnis, P., Liu, V., & Lee, K. (2024) · NEJM Catalyst · doi:10.1056/cat.23.0404 · Cited by 400 · see citing works
Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-43436-9 · Cited by 319 · see citing works
Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer
Fink, M. A., Bischoff, A., Fink, C. A., Moll, M., Kroschke, J., Dulz, L., Heußel, C. P., Kauczor, H., & Weber, T. F. (2023) · Radiology · doi:10.1148/radiol.231362 · Cited by 278 · see citing works
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · see citing works
Assessing Diagnostic Reasoning: A Consensus Statement Summarizing Theory, Practice, and Future Needs
Ilgen, J. S., Humbert, A. J., Kuhn, G. J., Hansen, M. L., Norman, G. R., Eva, K. W., Charlin, B., & Sherbino, J. (2012) · Academic Emergency Medicine · doi:10.1111/acem.12034 · Cited by 79 · see citing works
Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models
Alsentzer, E., Rasmussen, M. J., Fontoura, R., Cull, A. L., Beaulieu-Jones, B., Gray, K. J., Bates, D. W., & Kovacheva, V. P. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00957-x · Cited by 69 · see citing works
Do We Still Need Clinical Language Models?
Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., & Alsentzer, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08091 · Cited by 62 · see citing works
Radhakrishnan, L., Schenk, G., Muenzen, K., et al. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad045 · Cited by 54 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 4
- [7]Emergency Nurses Association. Emergency Severity Index Handbook. 5th ed. Emergency Nurses Association; 2023. Accessed January 30, 2024. https://californiaena.org/wp-content/uploads/2023/05/ESI-Handbook-5th-Edition-3-2023.pdf
- [12]Sushil M, Kennedy VE, Miao BY, Mandair D, Zack T, Butte AJ. Extracting detailed oncologic history and treatment plan from medical oncology notes with large language models. arXiv. Preprint posted online August 7, 2023.
- [13]Ouyang L, Wu J, Jiang X,. Training language models to follow instructions with human feedback. arXiv. Preprint posted online March 4, 2022.
- [14]Miao BY, Williams CY, Chinedu-Eneh E,. Identifying reasons for contraceptive switching from real-world data using large language models. arXiv. Preprint posted online February 5, 2024.
Cited by
Sorted by citations
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203 · see citing works
Williams, C. Y. K., Miao, B. Y., Kornblith, A. E., & Butte, A. J. (2024) · Nature Communications · doi:10.1038/s41467-024-52415-1 · Cited by 119 · see citing works
Self-citationChen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28 · see citing works
Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R., Hong, P., Bates, D. W., & Zhou, L. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106091 · Cited by 26 · see citing works
Urquhart, E., Ryan, J., Hartigan, S., Nita, C., Hanley, C., Moran, P., Bates, J., Jooste, R., Judge, C., Laffey, J. G., et al. (2024) · Intensive Care Medicine Experimental · doi:10.1186/s40635-024-00656-1 · Cited by 18 · see citing works
Bejan, C. A., Reed, A. M., Mikula, M., Zhang, S., Xu, Y., Fabbri, D., Embí, P. J., & Hsi, R. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-86632-5 · Cited by 8 · see citing works
Gautam, S., Chandra, M., De, A., Chakravorti, T., Malik, G., & De Choudhury, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06849 · Cited by 7 · see citing works
Dash, D., Macalalad, J., Lum, D. L., Baker, M. W., Dalsey, W. C., Sangal, R. B., Dayton, J. R., Waseem, T. C., Sikka, N. K., & Chu, A. L. (2026) · JACEP Open · doi:10.1016/j.acepjo.2026.100503
- Authors
- Williams, C. Y. K., Zack, T., Miao, B. Y., Sushil, M., Wang, M., Kornblith, A. E., & Butte, A. J.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Williams, Christopher Y. K.
- Butte, Atul J.
- ORCID
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2024-05-07
- Volume
- 7
- Issue
- 5
- Pages
- e248895
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC11077390
- OpenAlex
- W4396691546
- Cited by
- 148
- References
- 14
- Citations by year
- 2026: 54 · 2025: 69 · 2024: 25
- FWCI
- 13.84
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (K23HD110716)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
- Keywords
- large language models, Emergency Severity Index, emergency department, GPT-4, emergency medicine, clinical text classification, model comparison, adult patients, health equity, health care utilization, health care access, quality of care
- MeSH terms
- Adult, Cross-Sectional Studies, Emergency Service, Hospital, Female, Humans, Male, Middle Aged, San Francisco, Severity of Illness Index, Patient Acuity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
