The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Open accessAbstract
Objective: To determine the capabilities of ChatGPT for rapidly generating, rewriting, and evaluating (via diagnostic and triage accuracy) sets of clinical vignettes. Design: We explored the capabilities of ChatGPT for generating and rewriting vignettes. First, we gave it natural language prompts to generate 10 new sets of 10 vignettes, each set for a different common childhood illness. Next, we had it generate 10 sets of 10 vignettes given a set of symptoms from which to draw. We then had it rewrite 15 existing pediatric vignettes at different levels of health literacy. Fourth, we asked it to generate 10 vignettes written as a parent, and rewrite these vignettes as a physician, then at a grade 8 reading level, before rewriting them from the original parent’s perspective. Finally, we evaluated ChatGPT for diagnosis and triage for 45 clinical vignettes previously used for evaluating symptom checkers. Setting and participants: ChatGPT, a publicly available, free chatbot. Main outcome measures: Our main outcomes for de novo vignette generation were whether ChatGPT followed vignette creation instructions consistently, correctly, and listed reasonable symptoms for the disease being described. For generating vignettes from pre-existing symptom sets, we examined whether the symptom sets were used without introducing extra symptoms. Our main outcome for rewriting existing standardized vignettes to match patient demographics, and rewriting vignettes between styles, was whether symptoms were dropped or added outside the original vignette. Finally, our main outcomes examining diagnostic and triage accuracy on 45 standardized patient vignettes were whether the correct diagnosis was listed first, and if the correct triage recommendation was made. Results: ChatGPT was able to quickly produce varied contexts and symptom profiles when writing vignettes based on an illness name, but overused some core disease symptoms. It was able to use given symptom lists as the basis for vignettes consistently, adding one additional (though appropriate) symptom from outside the list for one disease. Pediatric vignettes rewritten at different levels of health literacy showed more complex symptoms being dropped when writing at low health literacy in 87.5% of cases. While writing at high health literacy, it added a diagnosis to 80% of vignettes (91.7% correctly diagnosed). Symptoms were retained in 90% of cases when rewriting vignettes between viewpoints. When presented with 45 vignettes, ChatGPT identified illnesses with 75.6% (95% CI, 62.6% to 88.5%) first-pass diagnostic accuracy and 57.8% (95% CI, 42.9% to 72.7%) triage accuracy. Its use does require monitoring and has caveats, which we discuss. Conclusions: ChatGPT was capable, with caveats and appropriate review, of generating, rewriting, and evaluating clinical vignettes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thus, ChatGPT can be useful in health care education including enhanced communication skills given proper academic mentoring [42,57,67].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [67]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, a recent preprint by Benoit showed the promising potential of ChatGPT in rapidly crafting consistent realistic clinical vignettes of variable complexities that can be a valuable educational source with lower costs [67].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [67]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Within the context of professional skills enhancement, a recent study demonstrated the potential of ChatGPT in producing vignettes based on specific diseases, which can serve as an important educational source [4].”
From Health profession students' perceptions of ChatGPT in healthcare and education: insights from a mixed-methods study (2025) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“However, the copyright issues should be taken into account regarding the ChatGPT-generated clinical vignettes, aside from the issue of inaccurate references [67].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 5×
“We began exploring ChatGPT because of its potential to generate large, standardized, high-quality databases of clinical vignettes relevant to our work with common childhood illnesses.(21) In this study, we”
BackgroundMethodInfluentialAudiolm: A language modeling approach to audio generation
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3288409 · Cited by 432
A critical review of state‐of‐the‐art chatbot designs and applications
Luo, B., Lau, R. Y. K., Li, C., & Si, Y. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1434 · Cited by 224
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 186
Research works printed but not linked: 8
- An overview of chatbot technology. Artificial Intelligence Applications and Innovations: 16th IFIP WG 125 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16; 2020: Springer.
- Chen M , Tworek J , Jun H , Yuan Q , Pinto HPdO , Kaplan J , Edwards H , Burda Y , Joseph N , Brockman G. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:210703374.
- Daras G , Dimakis AG . 2022. Discovering the hidden vocabulary of dalle-2. arXiv preprint arXiv:220600169.
- Poole B , Jain A , Barron JT , Mildenhall B. 2022. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:220914988.
- Villegas R , Babaeizadeh M , Kindermans P-J , Moraldo H , Zhang H , Saffar MT , Castro S , Kunze J , Erhan D. 2022. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:221002399.
- Alayrac J-B , Donahue J , Luc P , Miech A , Barr I , Hasson Y , Lenc K , Mensch A , Millican K , Reynolds M. 2022. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:220414198.
- Gozalo-Brizuela R , Garrido-Merchan EC . 2023. Chatgpt is not all you need. A state of the art review of large generative ai models. arXiv preprint arXiv:230104655.
- 2010, Storytelling as a communication tool for health consumers: Development of an intervention for parents of children with croup, Stories to communicate health information. BMC Pediatr, 10, 64
Cited by
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Sallam, M., Salim, N., Barakat, M., & Al-Tammemi, A. (2023) · Narra J · doi:10.52225/narra.v3i1.103 · Cited by 331
Sallam, M. (2023) · doi:10.1101/2023.02.19.23286155 · Cited by 3,123
“Thus, ChatGPT can be useful in healthcare education including enhanced communication skills given proper academic mentoring [22,45,67].”
BackgroundAn exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · doi:10.1101/2023.03.31.23287979 · Cited by 238
Metadata
- Authors
- Benoit, J. R. A.
- Affiliations
- University of Alberta (CA)
- Corresponding author
- Benoit, James R. A.
- ORCID
- Benoit, James R. A.
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-02-08
- OpenAlex
- W4319454867
- Cited by
- 74
- References
- 40
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 27 · 2023: 28
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- medRxiv (acceptedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT, clinical vignettes, symptom checkers, diagnostic accuracy, triage accuracy, health literacy, patient perspectives, physician attitudes, natural language processing, large language models, childhood diseases, patient-provider communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChatGPT for Clinical Vignette Generation, Revision, and Evaluation. https://app.cochat.ai/papers/10.1101/2023.02.04.23285478
- The paper
- Benoit, J. R. A. (2023). ChatGPT for Clinical Vignette Generation, Revision, and Evaluation. medRxiv. https://doi.org/10.1101/2023.02.04.23285478
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.