Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Background: Case studies have shown ChatGPT can run clinical simulations at the medical student level. However, no data have assessed ChatGPT’s reliability in meeting desired simulation criteria such as medical accuracy, simulation formatting, and robust feedback mechanisms. Objective: This study aims to quantify ChatGPT’s ability to consistently follow formatting instructions and create simulations for preclinical medical student learners according to principles of medical simulation and multimedia educational technology. Methods: Using ChatGPT-4 and a prevalidated starting prompt, the authors ran 360 separate simulations of an acute asthma exacerbation. A total of 180 simulations were given correct answers and 180 simulations were given incorrect answers. ChatGPT was evaluated for its ability to adhere to basic simulation parameters (stepwise progression, free response, interactivity), advanced simulation parameters (autonomous conclusion, delayed feedback, comprehensive feedback), and medical accuracy (vignette, treatment updates, feedback). Significance was determined with χ ² analyses using 95% CIs for odds ratios. Results: In total, 100% (n=360) of simulations met basic simulation parameters and were medically accurate. For advanced parameters, 55% (200/360) of all simulations delayed feedback, while the Correct arm (157/180, 87%) delayed feedback was significantly more than the Incorrect arm (43/180, 24%; P <.001). A total of 79% (285/360) of simulations concluded autonomously, and there was no difference between the Correct and Incorrect arms in autonomous conclusion (146/180, 81% and 139/180, 77%; P =.36). Overall, 78% (282/360) of simulations gave comprehensive feedback, and there was no difference between the Correct and Incorrect arms in comprehensive feedback (137/180, 76% and 145/180, 81%; P =.31). ChatGPT-4 was not significantly more likely to conclude simulations autonomously ( P =.34) and provide comprehensive feedback ( P =.27) when feedback was delayed compared to when feedback was not delayed. Conclusions: These simulations have the potential to be a reliable educational tool for simple simulations and can be evaluated by a novel 9-part metric. Per this metric, ChatGPT simulations performed perfectly on medical accuracy and basic simulation parameters. It performed well on comprehensive feedback and autonomous conclusion. Delayed feedback depended on the accuracy of user inputs. A simulation meeting one advanced parameter was not more likely to meet all advanced parameters. Further work must be done to ensure consistent performance across a broader range of simulation scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent advancements in generative artificial intelligence (gAI) have demonstrated great potential in medical education, particularly for knowledge acquisition [25,26].”
From Development of a Clinical Clerkship Mentor Using Generative AI and Evaluation of Its Effectiveness in a Medical Student Trial Compared to Student Mentors: 2-Part Comparative Study (2025) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Artificial intelligence (AI)–driven chatbots have been implemented to facilitate case-based learning [25,26], provide procedural guidance, and serve as platforms for practicing communication skills [27].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 1×
“Delayed feedback best mirrors real-life clinical scenarios where immediate feedback is not always available to redirect incorrect thought processes [36,37].”
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Such studies indicated that LLMs can pass medical exams and possess solid foundations in medical reasoning [3,6-8].”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 2×
“However, even given the use of proper prompt engineering techniques, perfect performance is surprising given ChatGPT’s known inconsistencies [4,34,35].”
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1×
“An expanding body of literature has focused on LLMs in medical education, including the perspectives of both students and seasoned clinicians [9-13].”
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · cited 1×
“Early investigations assessed whether large language models (LLMs) could pass medical trainee licensing examinations [1-5].”
Cognitive Theory of Multimedia Learning
Mayer, R. E. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.005 · Cited by 1,976
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“However, even given the use of proper prompt engineering techniques, perfect performance is surprising given ChatGPT’s known inconsistencies [4,34,35].”
ChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628 · cited 1×
“Such studies indicated that LLMs can pass medical exams and possess solid foundations in medical reasoning [3,6-8].”
Boscardin, C. K., Gin, B., Golde, P. B., & Hauer, K. E. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005439 · Cited by 620
Mayer, R. E., & Fiorella, L. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.015 · Cited by 364
Csikszentmihalyi, M., Abuhamdeh, S., & Nakamura, J. (2014) · Flow and the Foundations of Positive Psychology · doi:10.1007/978-94-017-9088-8_15 · Cited by 321
Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments
Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B. S., Charney, A. W., Nadkarni, G., & Klang, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-43436-9 · Cited by 319 · cited 1×
“Early investigations assessed whether large language models (LLMs) could pass medical trainee licensing examinations [1-5].”
The Self-Explanation Principle in Multimedia Learning
Wylie, R., & Chi, M. T. H. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.021 · Cited by 296 · cited 1×
“This is supported by Mayer’s self-explanation principle, which shows strong learning when previous knowledge is integrated into current learning by explaining one’s reasoning [23].”
A systematic review of large language models and their implications in medical education
Lucas, H. C., Upperman, J. S., & Robinson, J. R. (2024) · Medical Education · doi:10.1111/medu.15402 · Cited by 287 · cited 1×
“Despite this promising result, these technologies’ unknown accuracy and precision in displaying desirable parameters limits their broader applicability [19-21].”
Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations
Ali, R., Tang, O. Y., Connolly, I. D., Zadnik Sullivan, P. L., Shin, J. H., Fridley, J. S., Asaad, W. F., Cielo, D., Oyelese, A. A., Doberstein, C. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002632 · Cited by 273 · cited 2×
“However, even given the use of proper prompt engineering techniques, perfect performance is surprising given ChatGPT’s known inconsistencies [4,34,35].”
Prompt Engineering in Medical Education
Heston, T., & Khun, C. (2023) · International Medical Education · doi:10.3390/ime2030019 · Cited by 232 · cited 1×
“Frameworks such as RISE (role, input, steps, expectations; ie, inputting the LLM’s role, anticipated input, required steps, and desired expectations) exist for prompt engineering but have been infrequently applied to medical student LLM use [17].”
The Signaling (or Cueing) Principle in Multimedia Learning
van Gog, T. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.014 · Cited by 231 · cited 1×
“Signaling, or indicating what information is vital, highlights specific learning points rather than generalizing about a user’s performance [31].”
The Expertise Reversal Principle in Multimedia Learning
Kalyuga, S. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139547369.028 · Cited by 151 · cited 1×
“It is also in-line with the expertise reversal effect that suggests novice learners will do better with more intense educational guidance, while advanced learners do better with less [38].”
Alkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132
ChatGPT interactive medical simulations for early clinical education: Case study
Scherr, R., Halaseh, F. F., Spina, A., Andalib, S., & Rivera, R. (2023) · JMIR Medical Education · doi:10.2196/49877 · Cited by 122
Self-citationThe Pros and Cons of Using ChatGPT in Medical Education: A Scoping Review
Mohammad, B., Supti, T., Alzubaidi, M., Shah, H., Alam, T., Shah, Z., & Househ, M. (2023) · Studies in Health Technology and Informatics · doi:10.3233/shti230580 · Cited by 114 · cited 1×
“Such studies indicated that LLMs can pass medical exams and possess solid foundations in medical reasoning [3,6-8].”
Embracing ChatGPT for Medical Education: Exploring Its Impact on Doctors and Medical Students
Wu, Y., Zheng, Y., Feng, B., Yang, Y., Kang, K., & Zhao, A. (2024) · JMIR Medical Education · doi:10.2196/52483 · Cited by 92
The Modality Principle in Multimedia Learning
Low, R., & Sweller, J. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.012 · Cited by 78
Magalhães Araujo, S., & Cruz-Correia, R. (2024) · JMIR Medical Education · doi:10.2196/51151 · Cited by 72
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63 · cited 1×
“Studies have also suggested that ChatGPT can be an effective tool for students when entering the clinical wards [15].”
The Feedback Principle in Multimedia Learning
Johnson, C. I., & Priest, H. A. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.023 · Cited by 62
The Generative Drawing Principle in Multimedia Learning
Leutner, D., & Schmeck, A. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.022 · Cited by 59
ChatGPT in Medical Education and Research: A Boon or a Bane?
Jeyaraman, M., K, S. P., Jeyaraman, N., Nallakumarasamy, A., Yadav, S., & Bondili, S. K. (2023) · Cureus · doi:10.7759/cureus.44316 · Cited by 58
Multimedia Learning with Simulations and Microworlds
Plass, J. L., & Schwartz, R. N. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139547369.036 · Cited by 56
Society for Simulation in Healthcare Guidelines for Simulation Training
Stefanidis, D., Cook, D. A., Motamedi, M. K., Muret-Wagstaff, S. L., Calhoun, A. W., Lauridsen, K. G., Paige, J. T., Lockey, A. S., Donoghue, A. J., Hall, A. K., et al. (2024) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000776 · Cited by 56 · cited 2×
“The Society for Simulation in Healthcare recommends simulation sessions be high or low fidelity with low physical realism, making AI simulation sessions acceptable when high-fidelity sessions are unavailable [30].”
Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study
Nakao, T., Miki, S., Nakamura, Y., Kikuchi, T., Nomura, Y., Hanaoka, S., Yoshikawa, T., & Abe, O. (2024) · JMIR Medical Education · doi:10.2196/54393 · Cited by 56 · cited 1×
“However, acute asthma exacerbation is not an exceed-ingly complex problem and has well-established management guidelines that likely exist in ChatGPT’s training data, so its medical accuracy cannot be generalized to more complex clinical simulations like intensive care management [2,4].”
Multimedia Learning in e-Courses
Clark, R. C. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139547369.040 · Cited by 47
Finding the Place of ChatGPT in Medical Education
van de Ridder, J. M. M., Shoja, M. M., & Rajput, V. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005254 · Cited by 41
ChatGPT’s Role in Improving Education Among Patients Seeking Emergency Medical Treatment
Halaseh, F. F., Yang, J. S., Danza, C., Halaseh, R., & Spiegelman, L. (2024) · Western Journal of Emergency Medicine · doi:10.5811/westjem.18650 · Cited by 24
Self-citationChamberland, M., Setrakian, J., St‐Onge, C., Bergeron, L., Mamede, S., & Schmidt, H. G. (2019) · BMC Medical Education · doi:10.1186/s12909-019-1638-3 · Cited by 22 · cited 1×
“Delayed feedback best mirrors real-life clinical scenarios where immediate feedback is not always available to redirect incorrect thought processes [36,37].”
OpenAI’s ChatGPT Clinical Simulation
Musallam, E., Ali, A. A., & Alkhafaji, M. (2024) · Nurse Educator · doi:10.1097/nne.0000000000001657 · Cited by 16
Fifteen Common but Questionable Principles of Multimedia Learning
Feldon, D. F., Jeong, S., & Clark, R. E. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108894333.005 · Cited by 15 · cited 1×
“This is also congruent with the authentic learning environments principle which suggests that learning can happen equivalently in any environment as long as the design adheres to effective learning principles [32].”
Acute Asthma Exacerbations: Management Strategies.
Dabbs, W. S., Bradley, M. H., & Chamberlin, S. M. (2024) · PubMed · Cited by 10
Kim, H. W., Hong, J. W., Nam, E. J., Kim, K. Y., Kim, J. H., & Kang, J. I. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277059 · Cited by 8 · cited 1×
“Moving from theoretical, classroom-based instruction to hands-on patient care introduces new challenges [16].”
Research works printed but not linked: 1
- Dabbs, Am Fam Physician, 109, 43
Cited by
A review of ChatGPT in medical education: exploring advantages and limitations
Cheng, Y., & Zhu, L. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002505 · Cited by 29
Ebihara, H., Kasai, H., Shimizu, I., Shikino, K., Tajima, H., Kimura, Y., & Ito, S. (2025) · JMIR Medical Education · doi:10.2196/76702 · Cited by 10
“Artificial intelligence (AI)–driven chatbots have been implemented to facilitate case-based learning [25,26], provide procedural guidance, and serve as platforms for practicing communication skills [27].”
van Velzen, M., van der Willigen, R. F., de Beer, V. J., de Graaf-Waar, H. I., Janssen, E., van Leeuwen, S., van der Willigen, M. F., van der Willigen, M. J., Renardus, G., Maaroufi, R. E., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1644084 · Cited by 9
Kıyak, Y. S., İş-Kara, T., & Emekli, E. (2026) · Medical Science Educator · doi:10.1007/s40670-025-02621-3 · Cited by 13
Zouakia, Z., Logak, E., Szymczak, A., Jaïs, J., Burgun, A., & Tsopra, R. (2026) · JMIR Medical Education · doi:10.2196/82116 · Cited by 11
“Generation of clinical simulations (ie, scenarios given to students with interaction, such as simulated patients): LLMs demonstrated their ability to generate accurate cases in asthma [30], as well as realistic cases in the nursing field [31].”
Metadata
- Authors
- Scherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R.
- Affiliations
- University of California, Irvine (US)
- Stanford Medicine (US)
- Tufts University (US)
- Corresponding author
- Scherr, Riley
- ORCID
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-02-27
- Volume
- 9
- Pages
- e66478-e66478
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC11884304
- OpenAlex
- W4407042205
- Cited by
- 10
- References
- 39
- Citations by year
- 2026: 6 · 2025: 4
- FWCI
- 2.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- ChatGPT-4, large language models, clinical simulation, medical education, delayed feedback, medical accuracy, acute asthma exacerbation, preclinical medical students, AI-generated educational content, feedback mechanisms, multimedia educational technology, simulation-based training, simulation-based education
- MeSH terms
- Simulation Training, Formative Feedback, Generative Artificial Intelligence, Clinical Competence, Education, Medical, Undergraduate, Educational Measurement, Humans, Students, Medical, Reproducibility of Results, Patient Simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.