The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 2 of the 3 sources its claims cite were checked for support.
AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Open accessAbstract
BACKGROUND: Simulated medical scenarios are useful for evaluating and developing clinical competencies but scheduling them is expensive and time-consuming. Large language models show promise in role-playing tasks. We investigated the fidelity with which ChatGPT can mimic patients, clinicians and examiners in educational settings. OBJECTIVE: To determine the realism with which ChatGPT can portray patient, doctor and examiner roles, and the utility of these agents in clinical education. METHOD: We selected four paediatric scenarios from mock objective structured clinical examinations (OSCEs) and set up separate patient, doctor and examiner ChatGPT agents for each. The patient and doctor agents conversed with each other in written format. The examiner agent marked the doctor agent based on this conversation. Patients and clinicians familiar with the OSCE assessed the dialogues. RESULTS: The patient agent was judged to be true to character most of the time and good at expressing emotion. The doctor agent was reported to be an effective communicator but occasionally used jargon. Both agents tended to produce repetitive responses which undermined realism. The examiner agent had good correlation with human clinicians. There was moderate support for using the simulated interactions for educational purposes. CONCLUSION: Although the realism of the agents can be improved, ChatGPT can generate plausible proxies of participants in medical scenarios and could be useful for complementing standardised patient-based training.
Overview
CoChat Summary
TL;DR
ChatGPT-based patient, doctor, and examiner agents produced plausible but imperfect simulations of paediatric OSCE consultations, with patient and doctor agents showing moderate realism and examiner scoring correlating somewhat with human clinicians. The authors conclude these agents could complement standardised-patient training, but low fidelity in difficult scenarios requires human oversight.
Key findings
- PatientGPT was judged faithful to role in 15/24 combined assessments and authentic in 16/24; emotion expression was rated highly by both groups (11/12 patients, 10/12 clinicians). Patient assessors were more favourable than clinicians for faithfulness (9/12 vs 6/12) and authenticity (10/12 vs 6/12).
- DoctorGPT was rated empathetic in 20/24 combined assessments and received passing grades from all assessors in the MMR vaccine and heart murmur stations; meningitis was borderline passing, while self-harm had no clinician marking overlap but a human majority favoured passing grades. It failed to mention drug use or sexual relationships in the HEADSSS history.
- ExaminerGPT had higher average agreement with clinicians than clinicians did with each other (mean κ=0.39 vs 0.33), but tended to mark leniently and overlapped with majority overall grade in only one case (meningitis diagnosis).
- Just over half of combined assessors (14/24) agreed the conversations could be used for educational purposes, with patients more favourable than clinicians (8/12 vs 6/12).
Methods
- Four paediatric OSCE scenarios from Year 4 MBChB mock stations at the University of Leeds were simulated using GPT-3.5 Turbo in a zero-shot setting via the OpenAI API. Separate PatientGPT, DoctorGPT, and ExaminerGPT agents were created for each scenario; PatientGPT and DoctorGPT conversed in written format, then ExaminerGPT graded DoctorGPT.
- Three paediatric consultants and three expert patient volunteers assessed the dialogues using 5-point and 4-point scales, free-text feedback, and pedagogical utility ratings. Clinicians also graded DoctorGPT using the same criteria as ExaminerGPT.
- Agreement between each clinician pair and between each clinician and ExaminerGPT was measured with weighted Cohen’s kappa.
Limitations
- Stated: findings may have limited generalisability due to the small number of scenarios and small assessor group; the study was limited to written format and did not include non-verbal communication such as tone or body language; OSCE time limits were not considered and results would likely differ if incorporated.
- Evident from the text: the AI–AI interaction setup may have amplified echoing, repetition, and verbosity, and the study used only one off-the-shelf model (GPT-3.5 Turbo) without human–AI interactions.
- Evident from the text: no demographic data were collected, and the assessor sample was small, which may limit interpretation of subgroup comparisons.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
2 Valid · 0 Not valid · 1 Not checked · 41 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“This behaviour may be attributed to the issue of sycophancy in LLMs,27 where models trained on human feedback tailor responses by repeating or reinforcing user preferences (the user in this scenario being DoctorGPT).”
§ AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training > Results > PatientGPT
What the sources say
- Supportedcites [27] Perez E , Ringer S , Lukosiute K , et al . Discovering language model behaviors with model-written evaluations. In: Rogers A , Boyd-Graber J , Okazaki N , eds. Findings of the Association for Computational Linguistics; Association for Computational Linguistics, 13387–434. Toronto, Canada. doi:10.18653/v1/2023.findings-acl.847 source
Section 4 (p. 6) describes sycophancy as RLHF models tailoring responses to appear preferable to human evaluators. Section 4.2 (p. 7) reports that models repeat users’ views and that RLHF may incentivize this behavior. The proposed explanation for PatientGPT is hedged and consistent with that general finding. (LLM, conf=0.91)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agent also failed to recommend resources for further information, where needed.”
§ AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training > Results > DoctorGPT
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DoctorGPT received passing grades from all assessors in the MMR vaccine and heart murmur stations.”
§ AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training > Results > DoctorGPT
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The patient agent was judged to be true to character most of the time and good at expressing emotion.”
- Also stated in: § PatientGPT · plain
“PatientGPT was judged to be moderately faithful to its assigned character (15/24 in agreement overall), with the patient assessment being more favourable than the clinician assessment (9/12 vs 6/12 in agreement, respectively).”
§ AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training > Abstract > Results
plainn = 24- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § PatientGPT · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DoctorGPT was reported to be an empathetic communicator overall (20/24 in agreement), with the patient ratings being slightly more varied across the 5-point scale than those from the clinicians (10/12 in agreement for both groups).”
§ AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training > Results > DoctorGPT
plainn = 24- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
16 more claims
Create a free account to see all 44 claims and every source verdict.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 7
Citations
Cites
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1×
“⇒ Disparities observed between (human) patient and clinician assessment for the GPT agents suggest that diverse viewpoints are needed to fully capture the experiential learning associated with clinical communication.”
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Assessment of clinical competence using an objective structured clinical examination (OSCE)
HARDEN, R. M., & GLEESON, F. A. (1979) · Medical Education · doi:10.1111/j.1365-2923.1979.tb00918.x · Cited by 1,509
Evaluating large language models for accuracy incentivizes hallucinations
Kalai, A. T., Nachum, O., Vempala, S., & Zhang, E. (2026) · Nature · doi:10.1038/s41586-026-10549-w · Cited by 786
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.818634 · Cited by 490
Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366 · cited 2×
“A notable exception which incorporates this assessment is the Articulate Medical Intelligence Explorer (AMIE) system by Google, 18 which fine-tunes an LLM for clinical reasoning, but is not available for use outside of Google Research.”
A systematic review of large language model (LLM) evaluations in clinical medicine
Shool, S., Adimi, S., Saboori Amleshi, R., Bitaraf, E., Golpira, R., & Tara, M. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02954-4 · Cited by 346 · cited 1×
“9 14 While useful for establishing clinical knowledge, these evaluations rely solely on quantitative metrics and fail to explore the capabilities of LLMs in open-ended discourse.”
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 344
Research works printed but not linked: 1
- Goldenring, 2004, Getting into adolescent heads: An essential update, Contemp Pediatr, 21, 64
Cited by
Beyond innovation: equity considerations in AI-driven paediatric education
Munive, A. C., Bermúdez, I. A. B., & Triana, G. A. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2026-330725 · Cited by 1
LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment
Li, L., Li, D., Chen, C., Ma, R., Yu, R., Lin, M., Yin, R., Fan, L., Shyr, C., Ma, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.25273 · Cited by 1
“Minor changes in prompt wording can alter grading strictness, score distributions, and rationale quality [118].”
Tre personaggi in cerca di autore (e un solo attore)
Ventura, A. (2026) · Medico e Bambino · doi:10.53126/meb45351
Havalad, V., Brennan, G., Cangellaris, O. V., Huma, L., Carter, K., Greening, H., & Rissman, L. (2026) · doi:10.2196/preprints.103300
Large language models in paediatric medical education: equity must come first
Green, Z., & Kadhim, A. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2026-330994
Metadata
- Authors
- Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C.
- Affiliations
- University of Leeds (GB)
- Computer Science, University of Leeds, Leeds, UK
- Corresponding author
- Saggar, Arpita
- Venue
- Archives of Disease in Childhood
- Publisher
- BMJ
- Published
- 2026-03-06
- Volume
- 111
- Issue
- 7
- Pages
- 638-644
- ISSN
- 0003-9888 (ISSN-L)
- 1468-2044
- PMC
- PMC13311969
- OpenAlex
- W7134131229
- Cited by
- 4
- References
- 21
- Citations by year
- 2026: 4
- FWCI
- 5.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Archives of Disease in Childhood (publishedVersion, cc-by)
- medRxiv (submittedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/S024336/1)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- ChatGPT, large language models, medical education, standardized patients, clinical communication, simulation-based training, AI literacy
- MeSH terms
- Large Language Models, Generative Artificial Intelligence, Child, Clinical Competence, Female, Humans, Pediatrics, Physician-Patient Relations, Patient Simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training. https://app.cochat.ai/papers/10.1136/archdischild-2025-329846
- The paper
- Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026). AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training. Archives of Disease in Childhood. https://doi.org/10.1136/archdischild-2025-329846
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.