The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 5 of the 25 sources its claims cite were checked for support.
Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial
Open accessAbstract
Background: Clinical decision-making (CDM) refers to physicians’ ability to gather, evaluate, and interpret relevant diagnostic information. An integral component of CDM is the medical history conversation, traditionally practiced on real or simulated patients. In this study, we explored the potential of using Large Language Models (LLM) to simulate patient-doctor interactions and provide structured feedback. Methods: We developed AI prompts to simulate patients with different symptoms, engaging in realistic medical history conversations. In our double-blind randomized design, the control group participated in simulated medical history conversations with AI patients (control group), while the intervention group, in addition to simulated conversations, also received AI-generated feedback on their performances (feedback group). We examined the influence of feedback based on their CDM performance, which was evaluated by two raters (ICC = 0.924) using the Clinical Reasoning Indicator – History Taking Inventory (CRI-HTI). The data was analyzed using an ANOVA for repeated measures. Results: Our final sample included 21 medical students (age mean = 22.10 years, semester mean = 4, 14 females). At baseline, the feedback group (mean = 3.28 ± 0.09 [standard deviation]) and the control group (3.21 ± 0.08) achieved similar CRI-HTI scores, indicating successful randomization. After only four training sessions, the feedback group (3.60 ± 0.13) outperformed the control group (3.02 ± 0.12), F (1,18) = 4.44, p = .049 with a strong effect size, partial η 2 = 0.198. Specifically, the feedback group showed improvements in the subdomains of CDM of creating context ( p = .046) and securing information ( p = .018), while their ability to focus questions did not improve significantly ( p = .265). Conclusion: The results suggest that AI-simulated medical history conversations can support CDM training, especially when combined with structured feedback. Such training format may serve as a cost-effective supplement to existing training methods, better preparing students for real medical history conversations.
Overview
CoChat Summary
TL;DR
This randomized controlled trial found that medical students who received ChatGPT-generated feedback after simulated patient conversations improved more in clinical decision making than students who only completed the AI simulation. The feedback group finished with higher CRI-HTI scores after four scenarios (3.604 vs 3.017), with a statistically significant and strong effect.
Key findings
- 29 students initially agreed to participate; 24 began after 5 dropped out before randomization, and 21 were included in the final analysis after 3 technical exclusions.
- After four scenarios, the feedback group outperformed the control group on clinical decision making: mean CRI-HTI 3.604 ± 0.130 vs 3.017 ± 0.118; F(1,20) = 4.436, p = .049; partial η² = 0.198.
- The feedback group improved significantly in the CRI-HTI subdomains “creating context” (p = .046) and “securing information” (p = .018), but not “focusing questions” (p = .265).
- Participants rated the simulations as highly accurate (8.67/10 ± 2.01), feasible (8.38/10 ± 1.88), and realistic (7.81/10 ± 1.40).
Methods
- Two-group randomized controlled study conducted at the University of Münster during winter term 2023/2024, using OpenAI ChatGPT 3.5 in the Playground interface.
- Sample: enrolled medical students from the University of Münster who attended in person and gave consent; final analyzed sample was 21 participants, aged 19–35, with 11 in control and 10 in feedback.
- Participants completed four 6-minute simulated medical-history scenarios (herniated lumbar disc, stroke, meningitis, concussion); the feedback group received ChatGPT-generated CRI-HTI-based feedback plus three improvement suggestions after each case.
- Outcomes were rated by two blinded human raters using the German CRI-HTI; main analysis was repeated-measures ANOVA with group as between-subject factor, CRI-HTI over 3 levels as within-subject factor, and gender, age, and semester as covariates.
Limitations
- The paper states that ChatGPT 3.5’s training data only extends to January 2022, the model continues to change over time, and replication with newer versions has not been tested.
- The conversations were conducted in German, and the authors note responses may have been more precise in English because ChatGPT was originally trained on English texts.
- The sample was relatively small, limiting statistical power; this is explicitly stated by the authors.
- The authors observed that some participants adapted to model latency by combining multiple questions into one prompt, which they note does not reflect a realistic medical-history conversation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
2 Valid · 2 Not valid · 9 Not checked · 49 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Among the few studies that exist on LLM-generated patient simulations [ 21 – 24 ], none have implemented feedback or debriefing at the time of reporting.”
What the sources say
- Could not verifycites [21] Holderried F, Stegemann-Philipps C, Herschbach L, et al. A generative pretrained transformer (GPT)-powered chatbot as a simulated patient to practice history taking: prospective, mixed methods study. JMIR Med Educ. 2024;10(1). https://doi.org/10.2196/53961. sourceExperimental — accuracy not yet measured
Our copy confirms only the framing that this is one of the first LLM/GPT-powered simulated-patient studies: the Limitations section states 'As this was the first study using GPT as a simulated patient...' and the Comparison With Prior Work section notes that most prior chatbot studies 'were performed before the broad accessibility of LLMs, such as GPT,' restricted in authenticity, adaptability and transferability. No passage in our copy — including Comparison With Prior Work, Principal Findings, Limitations or Conclusion — addresses whether those LLM-generated patient simulation studies implemented feedback or debriefing; searches for 'feedback', 'debriefing', 'reflection' and automated learner feedback after the conversation returned only CUQ usability items, interface comparisons and the absence of objective skill measurement in this study itself. The specific claim that none of the cited studies reported feedback or debriefing therefore cannot be checked against this source. (LLM, conf=0.72)
- Not checked yetcites [22] Benfatah M, Marfak A, Saad E, Hilali A, Nejjari C, Youlyouz-Marfak I. Assessing the efficacy of ChatGPT as a virtual patient in nursing simulation training: a study on nursing students’ experience. Teach Learn Nurs Published Online. 2024. https://doi.org/10.1016/j.teln.2024.02.005. source
- Disputedcites [23] Scherr R, Halaseh FF, Spina A, Andalib S, Rivera R. ChatGPT interactive medical simulations for early clinical education: case study. JMIR Med Educ. 2023;9(1). https://doi.org/10.2196/49877. sourceExperimental — accuracy not yet measured
Our copy of Scherr et al. (2023) explicitly builds and reports a feedback mechanism: the Methods state they 'specified that we wanted feedback about the answers only at the end of the simulation', and Principal Findings report that 'Feedback on the efficacy of treatments at each stage is given at the end of the simulation and can be either brief (Textbox 1) or detailed (Textbox 2 and 3)', with Textbox 4 generated specifically 'to exhibit simulation feedback mechanisms' including per-step feedback and alternatives to incorrect answers on request (Limitations, Preclinical-Clinical Appropriateness). The paper does not use the word 'debriefing', but the claim's assertion that none of the cited LLM-patient-simulation studies implemented feedback is contrary to what this source reports. Thus citing it among [21-24] as a study lacking feedback/debriefing is contradicted by its own text. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
- Not checked yetcites [24] Vaughn J, Ford SH, Scott M, Jones C, Lewinski A. Enhancing healthcare education: leveraging ChatGPT for innovative simulation scenarios. Clin Simul Nurs. 2024;87. https://doi.org/10.1016/j.ecns.2023.101487. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the feedback group showed improvements in the CDM-subdomains creating context ( p = .046) and securing information ( p = .018), while their ability to focus questions did not improve significantly ( p = .265).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the feedback group showed improvements in the CDM-subdomains creating context ( p = .046) and securing information ( p = .018), while their ability to focus questions did not improve significantly ( p = .265).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After four scenarios, participants in the feedback group ( M = 3.604 ± 0.130; 95% CI : 3.331, 3.878) outperformed the control group ( M = 3.017 ± 0.118; 95% CI : 2.770, 3.264), in CDM, F(1,20) = 4.436, p = .049, with a strong effect size, partial η 2 = 0.198.”
- Also stated in: § Discussion · boosted
“Using an experimental design, we found that the group receiving LLM-generated feedback demonstrated a significant improvement across four clinical scenarios compared to the control group.”
boostedstates a causen = 4 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, the feedback group showed improvements in the CDM-subdomains creating context ( p = .046) and securing information ( p = .018), while their ability to focus questions did not improve significantly ( p = .265).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
7 more claims
Create a free account to see all 62 claims and to verify the 20 references still open.
- Probably not a substantial claim. 42
- It may state two separate claims as one. 18
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, most existing research remains conceptual or exploratory, with limited empirical evidence on the practical implementation of GPT-based systems in specific skill domains.”
From Feasibility study of using GPT for history-taking training in medical education: a randomized clinical trial (2025) · cites this paper as [7]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Brügge et al. [7] reported improved decision-making through AI-facilitated simulations.”
From Feasibility study of using GPT for history-taking training in medical education: a randomized clinical trial (2025) · cites this paper as [7]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(1,20) = 4.436, p = .049”
Reported p = .049 · recomputed p = .048
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(1, 20) = 4.436reported p = .049 · recomputed p = .048
“F(1,20) = 4.436, p = .049”
- Consistentt(18) = 0.219reported p = .829 · recomputed p = .829
“t(18) = 0.219, p = .829”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“F(1,20) = 4.436”
“partial η 2 = 0.198”
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
How does communication heal? Pathways linking clinician–patient communication to health outcomes
Street, R. L., Makoul, G., Arora, N. K., & Epstein, R. M. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2008.11.015 · Cited by 2,465
An overview of the uses of standardized patients for teaching and evaluating clinical skills. AAMC
Barrows, H. S. (1993) · Academic Medicine · doi:10.1097/00001888-199306000-00002 · Cited by 928
Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724
Clinical Reasoning Strategies in Physical Therapy
Edwards, I., Jones, M., Carr, J., Braunack-Mayer, A., & Jensen, G. M. (2004) · Physical Therapy · doi:10.1093/ptj/84.4.312 · Cited by 472
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
Cited by
Generative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119
“Differences emerged after just four sessions—highlighting the potential of foundation models to improve the provision of tailored clinical education 96 .”
Chow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
A systematic review of ethical considerations of large language models in healthcare and medicine
Fareed, M., Fatima, M., Uddin, J., Ahmed, A., & Sattar, M. A. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1653631 · Cited by 63
Generative AI for decision-making: A multidisciplinary perspective
Albashrawi, M. A. (2025) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2025.100751 · Cited by 83
Rädel-Ablass, K., Schliz, K., Schlick, C., Meindl, B., Pahr-Hosbach, S., Schwendemann, H., Rupp, S., Roddewig, M., & Miersch, C. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06756-0 · Cited by 60
“Further research supports the integration of AI and virtual training in medical education, particularly for developing clinical reasoning skills, highlighting the effectiveness of AI tools in improving students’ ability to interact with patients and make informed clinical decisions [8–10].”
Metadata
- Authors
- Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D.
- Affiliations
- University Hospital Münster (DE)
- Harvard University (US)
- University of Münster (DE)
- Corresponding author
- Darici, Dogus
- Venue
- BMC Medical Education
- Published
- 2024-11-28
- Volume
- 24
- Issue
- 1
- Pages
- 1391-1391
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC11605890
- OpenAlex
- W4404795474
- Cited by
- 137
- References
- 34
- Citations by year
- 2026: 96 · 2025: 41
- FWCI
- 12.83
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Universität Münster
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Keywords
- large language models, clinical decision-making, medical history, patient simulation, AI-generated feedback, medical students, clinical reasoning, AI prompts, structured feedback
- MeSH terms
- Clinical Decision-Making, Formative Feedback, Adult, Artificial Intelligence, Clinical Competence, Double-Blind Method, Education, Medical, Undergraduate, Feedback, Female, Humans, Male, Medical History Taking, Physician-Patient Relations, Students, Medical, Patient Simulation, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial. https://app.cochat.ai/papers/10.1186/s12909-024-06399-7
- The paper
- Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D. (2024). Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial. BMC Medical Education. https://doi.org/10.1186/s12909-024-06399-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.