ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Objective: To determine the capabilities of ChatGPT for rapidly generating, rewriting, and evaluating (via diagnostic and triage accuracy) sets of clinical vignettes. Design: We explored the capabilities of ChatGPT for generating and rewriting vignettes. First, we gave it natural language prompts to generate 10 new sets of 10 vignettes, each set for a different common childhood illness. Next, we had it generate 10 sets of 10 vignettes given a set of symptoms from which to draw. We then had it rewrite 15 existing pediatric vignettes at different levels of health literacy. Fourth, we asked it to generate 10 vignettes written as a parent, and rewrite these vignettes as a physician, then at a grade 8 reading level, before rewriting them from the original parent’s perspective. Finally, we evaluated ChatGPT for diagnosis and triage for 45 clinical vignettes previously used for evaluating symptom checkers. Setting and participants: ChatGPT, a publicly available, free chatbot. Main outcome measures: Our main outcomes for de novo vignette generation were whether ChatGPT followed vignette creation instructions consistently, correctly, and listed reasonable symptoms for the disease being described. For generating vignettes from pre-existing symptom sets, we examined whether the symptom sets were used without introducing extra symptoms. Our main outcome for rewriting existing standardized vignettes to match patient demographics, and rewriting vignettes between styles, was whether symptoms were dropped or added outside the original vignette. Finally, our main outcomes examining diagnostic and triage accuracy on 45 standardized patient vignettes were whether the correct diagnosis was listed first, and if the correct triage recommendation was made. Results: ChatGPT was able to quickly produce varied contexts and symptom profiles when writing vignettes based on an illness name, but overused some core disease symptoms. It was able to use given symptom lists as the basis for vignettes consistently, adding one additional (though appropriate) symptom from outside the list for one disease. Pediatric vignettes rewritten at different levels of health literacy showed more complex symptoms being dropped when writing at low health literacy in 87.5% of cases. While writing at high health literacy, it added a diagnosis to 80% of vignettes (91.7% correctly diagnosed). Symptoms were retained in 90% of cases when rewriting vignettes between viewpoints. When presented with 45 vignettes, ChatGPT identified illnesses with 75.6% (95% CI, 62.6% to 88.5%) first-pass diagnostic accuracy and 57.8% (95% CI, 42.9% to 72.7%) triage accuracy. Its use does require monitoring and has caveats, which we discuss. Conclusions: ChatGPT was capable, with caveats and appropriate review, of generating, rewriting, and evaluating clinical vignettes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thus, ChatGPT can be useful in health care education including enhanced communication skills given proper academic mentoring [42,57,67].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [67]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, a recent preprint by Benoit showed the promising potential of ChatGPT in rapidly crafting consistent realistic clinical vignettes of variable complexities that can be a valuable educational source with lower costs [67].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [67]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Within the context of professional skills enhancement, a recent study demonstrated the potential of ChatGPT in producing vignettes based on specific diseases, which can serve as an important educational source [4].”
From Health profession students' perceptions of ChatGPT in healthcare and education: insights from a mixed-methods study (2025) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“However, the copyright issues should be taken into account regarding the ChatGPT-generated clinical vignettes, aside from the issue of inaccurate references [67].”
From ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns (2023) · cites this paper as [67]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“These early results are corroborated by ChatGPT offering near-passing performance on all sections of the USMLE (United States Medical Licensing Exam).(44) However, providing appropriate triage advice of these models can and should still be improved.”
ResultGilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 5×
“We began exploring ChatGPT because of its potential to generate large, standardized, high-quality databases of clinical vignettes relevant to our work with common childhood illnesses.(21) In this study, we”
BackgroundMethodInfluentialAudiolm: A language modeling approach to audio generation
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3288409 · Cited by 432
A critical review of state‐of‐the‐art chatbot designs and applications
Luo, B., Lau, R. Y. K., Li, C., & Si, Y. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1434 · Cited by 224
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review
Wallace, W., Chan, C., Chidambaram, S., Hanna, L., Iqbal, F. M., Acharya, A., Normahani, P., Ashrafian, H., Markar, S. R., Sounderajah, V., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00667-w · Cited by 186
Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022) · doi:10.1101/2022.12.23.22283901 · Cited by 172
Gourlay, A., Mshana, G., Birdthistle, I., Bulugu, G., Zaba, B., & Urassa, M. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-21 · Cited by 160
Comparison of Physician and Computer Diagnostic Accuracy
Semigran, H. L., Levine, D. M., Nundy, S., & Mehrotra, A. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.6001 · Cited by 145 · cited 1×
“43 Interestingly, these results show there is a benefit of providing context to ChatGPT’s diagnostic accuracy, raising questions around the appropriateness of using symptom-only diagnosis in future symptom checkers.”
BackgroundJungmann, S. M., Klan, T., Kuhn, S., & Jungmann, F. (2019) · JMIR Formative Research · doi:10.2196/13863 · Cited by 127
Accuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125
Haider, A. H., Schneider, E. B., Sriram, N., Scott, V. K., Swoboda, S. M., Zogg, C. K., Dhiman, N., Haut, E. R., Efron, D. T., Pronovost, P. J., et al. (2015) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2015.01.065 · Cited by 112
ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B., Ricke, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.14882 · Cited by 97
Parents’ experiences and information needs related to childhood fever: A systematic review
Thompson, A. P., Nesari, M., Hartling, L., & Scott, S. D. (2019) · Patient Education and Counseling · doi:10.1016/j.pec.2019.10.004 · Cited by 84
Meherali, S. M., Campbell, A., Hartling, L., & Scott, S. D. (2018) · Journal of Patient Experience · doi:10.1177/2374373518771362 · Cited by 82
Tremblay, D., Turcotte, A., Touati, N., Poder, T. G., Kilpatrick, K., Bilodeau, K., Roy, M., Richard, P. O., Lessard, S., & Giordano, É. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057095 · Cited by 76
Stacey, D., Brière, N., Robitaille, H., Fraser, K., Desroches, S., & Légaré, F. (2014) · Journal of Interprofessional Care · doi:10.3109/13561820.2014.911157 · Cited by 72
Cheng, C., Elsworth, G. R., & Osborne, R. H. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.604401 · Cited by 72
Evidence and practice: a review of vignettes in qualitative research
Murphy, J., Hughes, J. A., Read, S., & Ashby, S. (2021) · Nurse Researcher · doi:10.7748/nr.2021.e1787 · Cited by 71
Scott, S. D., Brett-MacLean, P., Archibald, M. M., & Hartling, L. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-19 · Cited by 66
Developing and Establishing Content Validity of Vignettes for Health Care Education and Research
Marie, B. S., Jimmerson, A., Perkhounkova, Y., & Herr, K. (2020) · Western Journal of Nursing Research · doi:10.1177/0193945920969693 · Cited by 53
Reid, K., Hartling, L., Ali, S., Le, A., Norris, A., & Scott, S. D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8877 · Cited by 49
Extensible Chatbot Architecture Using Metamodels of Natural Language Understanding
Matić, R., Kabiljo, M., Zivkovic, M. Z., & Čabarkapa, M. (2021) · Electronics · doi:10.3390/electronics10182300 · Cited by 46
A Randomized Controlled Trial of Storytelling as a Communication Tool
Hartling, L., Scott, S. D., Johnson, D. W., Bishop, T., & Klassen, T. P. (2013) · PLoS ONE · doi:10.1371/journal.pone.0077800 · Cited by 44
Scott, S. D., Albrecht, L., Given, L. M., Hartling, L., Johnson, D. W., Jabbour, M., & Klassen, T. P. (2017) · Canadian Journal of Emergency Medicine · doi:10.1017/cem.2016.406 · Cited by 41
Stories – a novel approach to transfer complex health information to parents: A qualitative study
Scott, S. D., Hartling, L., O’Leary, K., Archibald, M. M., & Klassen, T. P. (2012) · Arts & Health · doi:10.1080/17533015.2012.656203 · Cited by 39
Li, M.-H., Yang, Y.-S., Wang, K.-C., Chiang, Y., Shen, P., Lai, W., Guo, T., & Chen, P. (2017) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.7b12367 · Cited by 34
Assessing Complex Emergency Management with Clinical Case-Vignettes: A Validation Study
Rousseau, A., Rozenberg, P., & Ravaud, P. (2015) · PLoS ONE · doi:10.1371/journal.pone.0138663 · Cited by 31
Clinical vignettes improve performance in anatomy practical assessment
Ikah, D. S. K., Finn, G. M., Swamy, M., White, P. M., & McLachlan, J. C. (2014) · Anatomical Sciences Education · doi:10.1002/ase.1471 · Cited by 29
In vitro antimicrobial activity of a novel compound, Mul-1867, against clinically important bacteria
Tetz, G. V., & Tetz, V. (2015) · Antimicrobial Resistance and Infection Control · doi:10.1186/s13756-015-0088-x · Cited by 27
Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study
Ben-Shabat, N., Sharvit, G., Meimis, B., Ben Joya, D., Sloma, A., Kiderman, D., Shabat, A., Tsur, A. M., Watad, A., & Amital, H. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104897 · Cited by 27
Archibald, M. M., & Scott, S. D. (2019) · Nursing Open · doi:10.1002/nop2.369 · Cited by 24
Automated Item Generation: The Future of Medical Education Assessment
Royal, K. D., Hedgpeth, M., Jeon, T., & Colford, C. M. (2018) · EMJ Innovation · doi:10.33590/emjinnov/10313113 · Cited by 21
Parents’ information needs and preferences related to bronchiolitis: a qualitative study
Campbell, A., Hartling, L., Louie‐Poon, S., & Scott, S. D. (2019) · CMAJ Open · doi:10.9778/cmajo.20190092 · Cited by 20
Vignette element analysis for automated generation of vignettes in pharmacy education
McLaughlin, J. E., Ma, C., & Hubal, R. C. (2020) · Education in the Health Professions · doi:10.4103/ehp.ehp_3_20 · Cited by 6
Avey: An Accurate AI Algorithm for Self-DiagnosisPreprint
Hammoud, M., Douglas, S., Darmach, M., Alawneh, S., Sanyal, S., & Kanbour, Y. (2022) · medRxiv · doi:10.1101/2022.03.08.22272076 · Cited by 6
Research works printed but not linked: 8
- An overview of chatbot technology. Artificial Intelligence Applications and Innovations: 16th IFIP WG 125 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16; 2020: Springer.
- Chen M , Tworek J , Jun H , Yuan Q , Pinto HPdO , Kaplan J , Edwards H , Burda Y , Joseph N , Brockman G. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:210703374.
- Daras G , Dimakis AG . 2022. Discovering the hidden vocabulary of dalle-2. arXiv preprint arXiv:220600169.
- Poole B , Jain A , Barron JT , Mildenhall B. 2022. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:220914988.
- Villegas R , Babaeizadeh M , Kindermans P-J , Moraldo H , Zhang H , Saffar MT , Castro S , Kunze J , Erhan D. 2022. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:221002399.
- Alayrac J-B , Donahue J , Luc P , Miech A , Barr I , Hasson Y , Lenc K , Mensch A , Millican K , Reynolds M. 2022. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:220414198.
- Gozalo-Brizuela R , Garrido-Merchan EC . 2023. Chatgpt is not all you need. A state of the art review of large generative ai models. arXiv preprint arXiv:230104655.
- 2010, Storytelling as a communication tool for health consumers: Development of an intervention for parents of children with croup, Stories to communicate health information. BMC Pediatr, 10, 64
Cited by
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Sallam, M. (2023) · doi:10.1101/2023.02.19.23286155 · Cited by 3,123
“Thus, ChatGPT can be useful in healthcare education including enhanced communication skills given proper academic mentoring [22,45,67].”
BackgroundSallam, M., Salim, N., Barakat, M., & Al-Tammemi, A. (2023) · Narra J · doi:10.52225/narra.v3i1.103 · Cited by 331
An exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · doi:10.1101/2023.03.31.23287979 · Cited by 238
Metadata
- Authors
- Benoit, J. R. A.
- Affiliations
- University of Alberta (CA)
- Corresponding author
- Benoit, James R. A.
- ORCID
- Benoit, James R. A.
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-02-08
- OpenAlex
- W4319454867
- Cited by
- 74
- References
- 36
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 27 · 2023: 28
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- medRxiv (acceptedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT, clinical vignettes, symptom checkers, diagnostic accuracy, triage accuracy, health literacy, patient perspectives, physician attitudes, natural language processing, large language models, childhood diseases, patient-provider communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.