The DOI matches its Crossref record; the references it cites are not checked yet.
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Open accessAbstract
Importance: Artificial intelligence (AI) applications in health care have been effective in many areas of medicine, but they are often trained for a single task using labeled data, making deployment and generalizability challenging. Whether a general-purpose AI language model can perform diagnosis and triage is unknown. Objective: Compare the general-purpose Generative Pre-trained Transformer 3 (GPT-3) AI model’s diagnostic and triage performance to attending physicians and lay adults who use the Internet. Design: We compared the accuracy of GPT-3’s diagnostic and triage ability for 48 validated case vignettes of both common (e.g., viral illness) and severe (e.g., heart attack) conditions to lay people and practicing physicians. Finally, we examined how well calibrated GPT-3’s confidence was for diagnosis and triage. Setting and Participants: The GPT-3 model, a nationally representative sample of lay people, and practicing physicians. Exposure: Validated case vignettes (<60 words; <6 th grade reading level). Main Outcomes and Measures: Correct diagnosis, correct triage. Results: Among all cases, GPT-3 replied with the correct diagnosis in its top 3 for 88% (95% CI, 75% to 94%) of cases, compared to 54% (95% CI, 53% to 55%) for lay individuals (p<0.001) and 96% (95% CI, 94% to 97%) for physicians (p=0.0354). GPT-3 triaged (71% correct; 95% CI, 57% to 82%) similarly to lay individuals (74%; 95% CI, 73% to 75%; p=0.73); both were significantly worse than physicians (91%; 95% CI, 89% to 93%; p<0.001). As measured by the Brier score, GPT-3 confidence in its top prediction was reasonably well-calibrated for diagnosis (Brier score = 0.18) and triage (Brier score = 0.22). Conclusions and Relevance: A general-purpose AI language model without any content-specific training could perform diagnosis at levels close to, but below physicians and better than lay individuals. The model was performed less well on triage, where its performance was closer to that of lay individuals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When examined in the field of diagnosis and triage in medicine, it was found that Generative Pre-trained Transformer 3 (GPT-3), an artificial intelligence application, was better than non-physicians, and close to, but less successful than physicians [8].”
From Is there any room for ChatGPT AI bot in speech-language pathology? (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredIn the citing paper
“The findings indicated that GPT-3’s performance nearly approached that of professional medical practitioners.”
From A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis (2024) · cites this paper as [146]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 32 · Cited by 106
Metadata
- Authors
- Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A.
- Affiliations
- Brigham and Women's Hospital (US)
- Harvard University (US)
- Corresponding author
- Beam, Andrew
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-02-01
- PMC
- PMC9915829
- OpenAlex
- W4318765555
- Cited by
- 106
- References
- 32
- Citations by year
- 2026: 6 · 2025: 23 · 2024: 44 · 2023: 33
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by-nc-nd)
- Funders
- National Heart, Lung, and Blood Institute (K01HL141771)
- National Human Genome Research Institute (T32HG002295)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- GPT-3, large language models, diagnosis, triage, diagnostic accuracy, confidence calibration, Brier score, physician comparison, lay public, case vignettes, general-purpose AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model. https://app.cochat.ai/papers/10.1101/2023.01.30.23285067
- The paper
- Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023). The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model. medRxiv. https://doi.org/10.1101/2023.01.30.23285067
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.