CoChat
Sign Up
PreprintCited by 135Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareClinical Reasoning and Diagnostic Skills

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow

Open access
Rao, Arya, Pang, Michael, Kim, John, Kamineni, Meghana, Lie, Winston, Prasad, Anoop K., Landman, Adam, Dreyer, Keith J, Succi, Marc D.·2023·medRxiv
DOI 10.1101/2023.02.21.23285886PMC PMC9980239
Open this paper in CoChat — free accountOpen PDF

Abstract

IMPORTANCE: Large language model (LLM) artificial intelligence (AI) chatbots direct the power of large training datasets towards successive, related tasks, as opposed to single-ask tasks, for which AI already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as virtual physicians, has not yet been evaluated. OBJECTIVE: To evaluate ChatGPT's capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. DESIGN: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. SETTING: ChatGPT, a publicly available LLM. PARTICIPANTS: Clinical vignettes featured hypothetical patients with a variety of age and gender identities, and a range of Emergency Severity Indices (ESIs) based on initial clinical presentation. EXPOSURES: MSD Clinical Manual vignettes. MAIN OUTCOMES AND MEASURES: We measured the proportion of correct responses to the questions posed within the clinical vignettes tested. RESULTS: ChatGPT achieved 71.7% (95% CI, 69.3% to 74.1%) accuracy overall across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI, 67.8% to 86.1%), and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI, 54.2% to 66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%, p<0.001) and clinical management (β=-7.4%, p=0.02) type questions. CONCLUSIONS AND RELEVANCE: ChatGPT achieves impressive accuracy in clinical decision making, with particular strengths emerging as it has more clinical information at its disposal.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.02.21.23285886

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Research works printed but not linked: 1

  1. Case studies. Merck Manuals Professional Edition. Accessed February 1, 2023. https://www.merckmanuals.com/professional/pages-with-widgets/case-studies?mode=list

Cited by

Sorted by citations

Items 101–135 of 135

Related

See all

Metadata

Authors
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K. J., & Succi, M. D.
Affiliations
  • Harvard University (US)
  • Brigham and Women's Hospital (US)
  • Massachusetts General Hospital (US)
Corresponding author
Succi, Marc D.
ORCID
Venue
medRxiv
Publisher
openRxiv
Published
2023-02-26
DOI
10.1101/2023.02.21.23285886
PMC
PMC9980239
OpenAlex
W4322208207
Cited by
134
References
24
Citations by year
2026: 8 · 2025: 20 · 2024: 55 · 2023: 50
Type
posted content
Language
en
Open access
Locations
  • medRxiv (acceptedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
Funders
  • National Institutes of Health
  • National Institute of General Medical Sciences (T32GM144273)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
ChatGPT, large language models, clinical decision support, differential diagnosis, clinical vignettes, emergency medicine, clinical reasoning, diagnostic accuracy, clinical management, Emergency Severity Index

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review