Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 2 of the 3 sources its claims cite were checked for support.

AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training

Open access
Saggar, Arpita, Dimitrova, Vania, Sarikaya, Duygu, Hogg, David, Darling, Jonathan C · 2026 · Archives of Disease in Childhood

Abstract

BACKGROUND: Simulated medical scenarios are useful for evaluating and developing clinical competencies but scheduling them is expensive and time-consuming. Large language models show promise in role-playing tasks. We investigated the fidelity with which ChatGPT can mimic patients, clinicians and examiners in educational settings. OBJECTIVE: To determine the realism with which ChatGPT can portray patient, doctor and examiner roles, and the utility of these agents in clinical education. METHOD: We selected four paediatric scenarios from mock objective structured clinical examinations (OSCEs) and set up separate patient, doctor and examiner ChatGPT agents for each. The patient and doctor agents conversed with each other in written format. The examiner agent marked the doctor agent based on this conversation. Patients and clinicians familiar with the OSCE assessed the dialogues. RESULTS: The patient agent was judged to be true to character most of the time and good at expressing emotion. The doctor agent was reported to be an effective communicator but occasionally used jargon. Both agents tended to produce repetitive responses which undermined realism. The examiner agent had good correlation with human clinicians. There was moderate support for using the simulated interactions for educational purposes. CONCLUSION: Although the realism of the agents can be improved, ChatGPT can generate plausible proxies of participants in medical scenarios and could be useful for complementing standardised patient-based training.

Overview

CoChat Summary

TL;DR

ChatGPT-based patient, doctor, and examiner agents produced plausible but imperfect simulations of paediatric OSCE consultations, with patient and doctor agents showing moderate realism and examiner scoring correlating somewhat with human clinicians. The authors conclude these agents could complement standardised-patient training, but low fidelity in difficult scenarios requires human oversight.

Key findings

  • PatientGPT was judged faithful to role in 15/24 combined assessments and authentic in 16/24; emotion expression was rated highly by both groups (11/12 patients, 10/12 clinicians). Patient assessors were more favourable than clinicians for faithfulness (9/12 vs 6/12) and authenticity (10/12 vs 6/12).
  • DoctorGPT was rated empathetic in 20/24 combined assessments and received passing grades from all assessors in the MMR vaccine and heart murmur stations; meningitis was borderline passing, while self-harm had no clinician marking overlap but a human majority favoured passing grades. It failed to mention drug use or sexual relationships in the HEADSSS history.
  • ExaminerGPT had higher average agreement with clinicians than clinicians did with each other (mean κ=0.39 vs 0.33), but tended to mark leniently and overlapped with majority overall grade in only one case (meningitis diagnosis).
  • Just over half of combined assessors (14/24) agreed the conversations could be used for educational purposes, with patients more favourable than clinicians (8/12 vs 6/12).

Methods

  • Four paediatric OSCE scenarios from Year 4 MBChB mock stations at the University of Leeds were simulated using GPT-3.5 Turbo in a zero-shot setting via the OpenAI API. Separate PatientGPT, DoctorGPT, and ExaminerGPT agents were created for each scenario; PatientGPT and DoctorGPT conversed in written format, then ExaminerGPT graded DoctorGPT.
  • Three paediatric consultants and three expert patient volunteers assessed the dialogues using 5-point and 4-point scales, free-text feedback, and pedagogical utility ratings. Clinicians also graded DoctorGPT using the same criteria as ExaminerGPT.
  • Agreement between each clinician pair and between each clinician and ExaminerGPT was measured with weighted Cohen’s kappa.

Limitations

  • Stated: findings may have limited generalisability due to the small number of scenarios and small assessor group; the study was limited to written format and did not include non-verbal communication such as tone or body language; OSCE time limits were not considered and results would likely differ if incorporated.
  • Evident from the text: the AI–AI interaction setup may have amplified echoing, repetition, and verbosity, and the study used only one off-the-shelf model (GPT-3.5 Turbo) without human–AI interactions.
  • Evident from the text: no demographic data were collected, and the assessor sample was small, which may limit interpretation of subgroup comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1136/archdischild-2025-329846

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Expert evaluations for the PatientGPT agent across all scenarios. The 'Only Clinicians’ and 'Only Patients’ bars represent a sample size of 12, while the bar for 'Combined’ depicts percentages based on 24 samples.
Figure 1 Expert evaluations for the PatientGPT agent across all scenarios. The 'Only Clinicians’ and 'Only Patients’ bars represent a sample size of 12, while the bar for 'Combined’ depicts percentages based on 24 samples.CC BY · PubMed Central
Figure 2 Evaluation for DoctorGPT’s empathy, communication skills and organisational skills. The 'Only Clinicians’ and 'Only Patients’ bars represent a sample size of 12, while the bar for 'Combined’ shows percentages corresponding to 24 samples.
Figure 2 Evaluation for DoctorGPT’s empathy, communication skills and organisational skills. The 'Only Clinicians’ and 'Only Patients’ bars represent a sample size of 12, while the bar for 'Combined’ shows percentages corresponding to 24 samples.CC BY · PubMed Central
Figure 3 Station-specific marking for DoctorGPT done by three human clinicians and ExaminerGPT. The agent had good overlap with clinician judgement most of the time, but struggled in cases where a stricter assessment was requisite.
Figure 3 Station-specific marking for DoctorGPT done by three human clinicians and ExaminerGPT. The agent had good overlap with clinician judgement most of the time, but struggled in cases where a stricter assessment was requisite.CC BY · PubMed Central
Figure 4 Assessor responses on the usefulness of the simulated dialogues as a learning aid. The 'Only Clinicians’ and 'Only Patients’ bars represent a sample size of 12, while the bar for 'Combined’ shows percentages corresponding to 24 samples.
Figure 4 Assessor responses on the usefulness of the simulated dialogues as a learning aid. The 'Only Clinicians’ and 'Only Patients’ bars represent a sample size of 12, while the bar for 'Combined’ shows percentages corresponding to 24 samples.CC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 0 Not valid · 1 Not checked · 41 Own results and interpretations

16 more claims

Create a free account to see all 44 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 7

Create a free account to see these claims.

Citations

Cites

Show all 21 references

Research works printed but not linked: 1

  1. Goldenring, 2004, Getting into adolescent heads: An essential update, Contemp Pediatr, 21, 64

Cited by

See all 5

Metadata

Authors
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C.
Affiliations
  • University of Leeds (GB)
  • Computer Science, University of Leeds, Leeds, UK
Corresponding author
Saggar, Arpita
Venue
Archives of Disease in Childhood
Publisher
BMJ
Published
2026-03-06
Volume
111
Issue
7
Pages
638-644
ISSN
  • 0003-9888 (ISSN-L)
  • 1468-2044
OpenAlex
W7134131229
Cited by
4
References
21
Citations by year
2026: 4
FWCI
5.20
Type
journal article
Language
en
Open access
Locations
  • Archives of Disease in Childhood (publishedVersion, cc-by)
  • medRxiv (submittedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
Funders
Engineering and Physical Sciences Research Council (EP/S024336/1)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Simulation-Based Education in Healthcare
  • Social Sciences › Education › Innovations in Medical Education
Keywords
ChatGPT, large language models, medical education, standardized patients, clinical communication, simulation-based training, AI literacy
MeSH terms
Large Language Models, Generative Artificial Intelligence, Child, Clinical Competence, Female, Humans, Pediatrics, Physician-Patient Relations, Patient Simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training. https://app.cochat.ai/papers/10.1136/archdischild-2025-329846
The paper
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026). AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training. Archives of Disease in Childhood. https://doi.org/10.1136/archdischild-2025-329846

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.