The Triage and Diagnostic Accuracy of Frontier Large Language Models: Updated Comparison to Physician Performance
Open accessPublished version of a preprint
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Earlier vignette studies found high diagnostic accuracy for language models on Semigran-style cases, and one found their stated confidence reasonably calibrated by Brier score.
“Earlier vignette studies found high diagnostic accuracy for language models on Semigran-style cases, and one found their stated confidence reasonably calibrated by Brier score [5,6]”
From Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models (2026) · cites this paper as [6]
The Methods (Overview; page unavailable) describe testing LLMs on 48 synthetic clinical vignettes, and the Results (page unavailable) report high diagnostic accuracy. The copy does not identify these as Semigran-style cases, and a search for Brier score and confidence calibration found no such assessment. Thus, it supports the accuracy portion but not the case provenance or calibration portion. (LLM, conf=0.84)
Experimental — accuracy not yet measuredGeneral-purpose language models on standardized clinical vignettes diagnose and triage at levels approaching those of physicians.
“On standardized clinical vignettes, general-purpose language models already diagnose and triage at levels approaching those of physicians [5,6]”
From Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models (2026) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [2]Saab K Tu T Weng WH Tanno R Stutz D Wulczyn E Zhang F Capabilities of Gemini models in medicine arXiv Preprint posted online on April 29, 2024 10.48550/arXiv.2404.18416 · DOI · Paper page
- [3]Sorich MJ Menz BD Hopkins AM Quality and safety of artificial intelligence generated health information BMJ 2024 03 20 384 q596 10.1136/bmj.q596 38508683 38508683 · DOI
- [4]Levine DM Tuwani R Kompa B Varma A Finlayson SG Mehrotra A Beam A The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study Lancet Digit Health 2024 08 6 8 e555 e561 10.1016/S2589-7500(24)00097-9 39059888 S2589-7500(24)00097-9 39059888 · DOI · Paper page
- [5]Williams CYK Zack T Miao BY Sushil M Wang M Kornblith AE Butte AJ Use of a large language model to assess clinical acuity of adults in the emergency department JAMA Netw Open 2024 05 01 7 5 e248895 10.1001/jamanetworkopen.2024.8895 38713466 2818387 38713466 PMC11077390 · DOI · Paper page
- [6]Masanneck L Schmidt L Seifert A Kölsche Tristan Huntemann N Jansen R Mehsin M Bernhard M Meuth SG Böhm Lennert Pawlitzki M Triage performance across large language models, ChatGPT, and untrained doctors in emergency medicine: comparative study J Med Internet Res 2024 06 14 26 e53297 10.2196/53297 38875696 v26i1e53297 38875696 PMC11214027 · DOI
Cited by
Sorted by citations
Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?
Yuan, G. C., Zhang, X., Kim, S. E., & Rajpurkar, P. (2026) · Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026) · doi:10.18653/v1/2026.healing-1.1 · Cited by 1 · see citing works
“However, existing mixed-vendor approaches in medicine mostly rely on independent model runs, combining models from different providers via voting or simple aggregation (Barabucci et al., 2024; Sorich et al., 2024).”
Deobhakta, A., Chang, J., Chen, R. W. S., Jaipalli, S., Pinhas, A., Tania Tai, T. Y., & Rosen, R. B. (2026) · doi:10.64898/2026.10.09.26365153
- Authors
- Sorich, M. J., Mangoni, A. A., Bacchi, S., Menz, B. D., & Hopkins, A. M.
- Affiliations
- Flinders University (AU)
- Harvard University (US)
- Corresponding author
- Sorich, Michael Joseph
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-12-06
- Volume
- 26
- Pages
- e67409
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11662182
- OpenAlex
- W4404334341
- Cited by
- 26
- References
- 1
- Citations by year
- 2026: 19 · 2025: 7
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Medical Research Council
- National Health and Medical Research Council (2008119, 2030913)
- Cancer Council South Australia (APP2008119)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- triage, diagnostic accuracy, physician performance, clinical decision support, emergency department triage
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
