Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

Open access
Benoit, James R. A. · 2023 · medRxiv

Abstract

Objective: To determine the capabilities of ChatGPT for rapidly generating, rewriting, and evaluating (via diagnostic and triage accuracy) sets of clinical vignettes. Design: We explored the capabilities of ChatGPT for generating and rewriting vignettes. First, we gave it natural language prompts to generate 10 new sets of 10 vignettes, each set for a different common childhood illness. Next, we had it generate 10 sets of 10 vignettes given a set of symptoms from which to draw. We then had it rewrite 15 existing pediatric vignettes at different levels of health literacy. Fourth, we asked it to generate 10 vignettes written as a parent, and rewrite these vignettes as a physician, then at a grade 8 reading level, before rewriting them from the original parent’s perspective. Finally, we evaluated ChatGPT for diagnosis and triage for 45 clinical vignettes previously used for evaluating symptom checkers. Setting and participants: ChatGPT, a publicly available, free chatbot. Main outcome measures: Our main outcomes for de novo vignette generation were whether ChatGPT followed vignette creation instructions consistently, correctly, and listed reasonable symptoms for the disease being described. For generating vignettes from pre-existing symptom sets, we examined whether the symptom sets were used without introducing extra symptoms. Our main outcome for rewriting existing standardized vignettes to match patient demographics, and rewriting vignettes between styles, was whether symptoms were dropped or added outside the original vignette. Finally, our main outcomes examining diagnostic and triage accuracy on 45 standardized patient vignettes were whether the correct diagnosis was listed first, and if the correct triage recommendation was made. Results: ChatGPT was able to quickly produce varied contexts and symptom profiles when writing vignettes based on an illness name, but overused some core disease symptoms. It was able to use given symptom lists as the basis for vignettes consistently, adding one additional (though appropriate) symptom from outside the list for one disease. Pediatric vignettes rewritten at different levels of health literacy showed more complex symptoms being dropped when writing at low health literacy in 87.5% of cases. While writing at high health literacy, it added a diagnosis to 80% of vignettes (91.7% correctly diagnosed). Symptoms were retained in 90% of cases when rewriting vignettes between viewpoints. When presented with 45 vignettes, ChatGPT identified illnesses with 75.6% (95% CI, 62.6% to 88.5%) first-pass diagnostic accuracy and 57.8% (95% CI, 42.9% to 72.7%) triage accuracy. Its use does require monitoring and has caveats, which we discuss. Conclusions: ChatGPT was capable, with caveats and appropriate review, of generating, rewriting, and evaluating clinical vignettes.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.02.04.23285478

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 40 references

Research works printed but not linked: 8

  1. An overview of chatbot technology. Artificial Intelligence Applications and Innovations: 16th IFIP WG 125 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16; 2020: Springer.
  2. Chen M , Tworek J , Jun H , Yuan Q , Pinto HPdO , Kaplan J , Edwards H , Burda Y , Joseph N , Brockman G. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:210703374.
  3. Daras G , Dimakis AG . 2022. Discovering the hidden vocabulary of dalle-2. arXiv preprint arXiv:220600169.
  4. Poole B , Jain A , Barron JT , Mildenhall B. 2022. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:220914988.
  5. Villegas R , Babaeizadeh M , Kindermans P-J , Moraldo H , Zhang H , Saffar MT , Castro S , Kunze J , Erhan D. 2022. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:221002399.
  6. Alayrac J-B , Donahue J , Luc P , Miech A , Barr I , Hasson Y , Lenc K , Mensch A , Millican K , Reynolds M. 2022. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:220414198.
  7. Gozalo-Brizuela R , Garrido-Merchan EC . 2023. Chatgpt is not all you need. A state of the art review of large generative ai models. arXiv preprint arXiv:230104655.
  8. 2010, Storytelling as a communication tool for health consumers: Development of an intervention for parents of children with croup, Stories to communicate health information. BMC Pediatr, 10, 64

Cited by

See all 74

Metadata

Authors
Benoit, J. R. A.
Affiliations
University of Alberta (CA)
Corresponding author
Benoit, James R. A.
Venue
medRxiv
Publisher
openRxiv
Published
2023-02-08
OpenAlex
W4319454867
Cited by
74
References
40
Citations by year
2026: 6 · 2025: 13 · 2024: 27 · 2023: 28
Type
posted content
Language
en
Open access
Licence
cc-by
Locations
medRxiv (acceptedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
ChatGPT, clinical vignettes, symptom checkers, diagnostic accuracy, triage accuracy, health literacy, patient perspectives, physician attitudes, natural language processing, large language models, childhood diseases, patient-provider communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ChatGPT for Clinical Vignette Generation, Revision, and Evaluation. https://app.cochat.ai/papers/10.1101/2023.02.04.23285478
The paper
Benoit, J. R. A. (2023). ChatGPT for Clinical Vignette Generation, Revision, and Evaluation. medRxiv. https://doi.org/10.1101/2023.02.04.23285478

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.