Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 137Artificial Intelligence in Healthcare and EducationClinical Reasoning and Diagnostic SkillsSimulation-Based Education in Healthcare

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; 5 of the 25 sources its claims cite were checked for support.

Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial

Open access

Abstract

Background: Clinical decision-making (CDM) refers to physicians’ ability to gather, evaluate, and interpret relevant diagnostic information. An integral component of CDM is the medical history conversation, traditionally practiced on real or simulated patients. In this study, we explored the potential of using Large Language Models (LLM) to simulate patient-doctor interactions and provide structured feedback. Methods: We developed AI prompts to simulate patients with different symptoms, engaging in realistic medical history conversations. In our double-blind randomized design, the control group participated in simulated medical history conversations with AI patients (control group), while the intervention group, in addition to simulated conversations, also received AI-generated feedback on their performances (feedback group). We examined the influence of feedback based on their CDM performance, which was evaluated by two raters (ICC = 0.924) using the Clinical Reasoning Indicator – History Taking Inventory (CRI-HTI). The data was analyzed using an ANOVA for repeated measures. Results: Our final sample included 21 medical students (age mean = 22.10 years, semester mean = 4, 14 females). At baseline, the feedback group (mean = 3.28 ± 0.09 [standard deviation]) and the control group (3.21 ± 0.08) achieved similar CRI-HTI scores, indicating successful randomization. After only four training sessions, the feedback group (3.60 ± 0.13) outperformed the control group (3.02 ± 0.12), F (1,18) = 4.44, p = .049 with a strong effect size, partial η 2 = 0.198. Specifically, the feedback group showed improvements in the subdomains of CDM of creating context ( p = .046) and securing information ( p = .018), while their ability to focus questions did not improve significantly ( p = .265). Conclusion: The results suggest that AI-simulated medical history conversations can support CDM training, especially when combined with structured feedback. Such training format may serve as a cost-effective supplement to existing training methods, better preparing students for real medical history conversations.

Overview

CoChat Summary

TL;DR

This randomized controlled trial found that medical students who received ChatGPT-generated feedback after simulated patient conversations improved more in clinical decision making than students who only completed the AI simulation. The feedback group finished with higher CRI-HTI scores after four scenarios (3.604 vs 3.017), with a statistically significant and strong effect.

Key findings

  • 29 students initially agreed to participate; 24 began after 5 dropped out before randomization, and 21 were included in the final analysis after 3 technical exclusions.
  • After four scenarios, the feedback group outperformed the control group on clinical decision making: mean CRI-HTI 3.604 ± 0.130 vs 3.017 ± 0.118; F(1,20) = 4.436, p = .049; partial η² = 0.198.
  • The feedback group improved significantly in the CRI-HTI subdomains “creating context” (p = .046) and “securing information” (p = .018), but not “focusing questions” (p = .265).
  • Participants rated the simulations as highly accurate (8.67/10 ± 2.01), feasible (8.38/10 ± 1.88), and realistic (7.81/10 ± 1.40).

Methods

  • Two-group randomized controlled study conducted at the University of Münster during winter term 2023/2024, using OpenAI ChatGPT 3.5 in the Playground interface.
  • Sample: enrolled medical students from the University of Münster who attended in person and gave consent; final analyzed sample was 21 participants, aged 19–35, with 11 in control and 10 in feedback.
  • Participants completed four 6-minute simulated medical-history scenarios (herniated lumbar disc, stroke, meningitis, concussion); the feedback group received ChatGPT-generated CRI-HTI-based feedback plus three improvement suggestions after each case.
  • Outcomes were rated by two blinded human raters using the German CRI-HTI; main analysis was repeated-measures ANOVA with group as between-subject factor, CRI-HTI over 3 levels as within-subject factor, and gender, age, and semester as covariates.

Limitations

  • The paper states that ChatGPT 3.5’s training data only extends to January 2022, the model continues to change over time, and replication with newer versions has not been tested.
  • The conversations were conducted in German, and the authors note responses may have been more precise in English because ChatGPT was originally trained on English texts.
  • The sample was relatively small, limiting statistical power; this is explicitly stated by the authors.
  • The authors observed that some participants adapted to model latency by combining multiple questions into one prompt, which they note does not reflect a realistic medical-history conversation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12909-024-06399-7

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

2 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Fig. 1 Study design. The diagram illustrates the timeline of the study. The control group completed four patient cases, with a 2-minute break between each case. In contrast, the feedback group received feedback from ChatGPT during these breaks instead of pausing. After fulfilling the tasks, the participants received a survey with no time limit
Fig. 1 Study design. The diagram illustrates the timeline of the study. The control group completed four patient cases, with a 2-minute break between each case. In contrast, the feedback group received feedback from ChatGPT during these breaks instead of pausing. After fulfilling the tasks, the participants received a survey with no time limitCC BY · PubMed Central
Fig. 2 Example of a conversation between the participant and ChatGPT. The prompt of the fourth scenario instructed ChatGPT to simulate a patient with a concussion. Italicized text represents the responses given by ChatGPT
Fig. 2 Example of a conversation between the participant and ChatGPT. The prompt of the fourth scenario instructed ChatGPT to simulate a patient with a concussion. Italicized text represents the responses given by ChatGPTCC BY · PubMed Central
Fig. 3 Example of the feedback using the CRI-HTI-Score and the individual feedback given by ChatGPT. The participant received the feedback after completing scenario 2. This is the same participant as in Fig. 3. Italicized text represents the responses given by ChatGPT
Fig. 3 Example of the feedback using the CRI-HTI-Score and the individual feedback given by ChatGPT. The participant received the feedback after completing scenario 2. This is the same participant as in Fig. 3. Italicized text represents the responses given by ChatGPTCC BY · PubMed Central
Fig. 4 Effects of AI-generated feedback on clinical decision-making and history taking. Median CRI-HT Score (ICC = 0.924) with the individual values is shown for the control group (red) and feedback group (blue) over four subsequent AI-simulated history taking scenarios
Fig. 4 Effects of AI-generated feedback on clinical decision-making and history taking. Median CRI-HT Score (ICC = 0.924) with the individual values is shown for the control group (red) and feedback group (blue) over four subsequent AI-simulated history taking scenariosCC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 2 Not valid · 9 Not checked · 49 Own results and interpretations

7 more claims

Create a free account to see all 62 claims and to verify the 20 references still open.

See all claims — free account
  • Probably not a substantial claim. 42
  • It may state two separate claims as one. 18

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(1, 20) = 4.436reported p = .049 · recomputed p = .048

    “F(1,20) = 4.436, p = .049”

  • Consistentt(18) = 0.219reported p = .829 · recomputed p = .829

    “t(18) = 0.219, p = .829”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 34 references

Cited by

See all 137

Metadata

Authors
Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D.
Affiliations
  • University Hospital Münster (DE)
  • Harvard University (US)
  • University of Münster (DE)
Corresponding author
Darici, Dogus
Venue
BMC Medical Education
Published
2024-11-28
Volume
24
Issue
1
Pages
1391-1391
ISSN
1472-6920 (ISSN-L)
OpenAlex
W4404795474
Cited by
137
References
34
Citations by year
2026: 96 · 2025: 41
FWCI
12.83
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Universität Münster
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Social Sciences › Education › Simulation-Based Education in Healthcare
Keywords
large language models, clinical decision-making, medical history, patient simulation, AI-generated feedback, medical students, clinical reasoning, AI prompts, structured feedback
MeSH terms
Clinical Decision-Making, Formative Feedback, Adult, Artificial Intelligence, Clinical Competence, Double-Blind Method, Education, Medical, Undergraduate, Feedback, Female, Humans, Male, Medical History Taking, Physician-Patient Relations, Students, Medical, Patient Simulation, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial. https://app.cochat.ai/papers/10.1186/s12909-024-06399-7
The paper
Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D. (2024). Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial. BMC Medical Education. https://doi.org/10.1186/s12909-024-06399-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.