Virtual Patients Using Large Language Models: Scalable, Contextualized Simulation of Clinician-Patient Dialogue With Feedback
Open accessPublished version of a preprint
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
Background: Virtual patients (VPs) are computer screen–based simulations of patient-clinician encounters. VP use is limited by cost and low scalability. Objective: We aimed to show that VPs powered by large language models (LLMs) can generate authentic dialogues, accurately represent patient preferences, and provide personalized feedback on clinical performance. We also explored using LLMs to rate the quality of dialogues and feedback. Methods: We conducted an intrinsic evaluation study rating 60 VP-clinician conversations. We used carefully engineered prompts to direct OpenAI’s generative pretrained transformer (GPT) to emulate a patient and provide feedback. Using 2 outpatient medicine topics (chronic cough diagnosis and diabetes management), each with permutations representing different patient preferences, we created 60 conversations (dialogues plus feedback): 48 with a human clinician and 12 “self-chat” dialogues with GPT role-playing both the VP and clinician. Primary outcomes were dialogue authenticity and feedback quality, rated using novel instruments for which we conducted a validation study collecting evidence of content, internal structure (reproducibility), relations with other variables, and response process. Each conversation was rated by 3 physicians and by GPT. Secondary outcomes included user experience, bias, patient preferences represented in the dialogues, and conversation features that influenced authenticity. Results: The average cost per conversation was US $0.51 for GPT-4.0-Turbo and US $0.02 for GPT-3.5-Turbo. Mean (SD) conversation ratings, maximum 6, were overall dialogue authenticity 4.7 (0.7), overall user experience 4.9 (0.7), and average feedback quality 4.7 (0.6). For dialogues created using GPT-4.0-Turbo, physician ratings of patient preferences aligned with intended preferences in 20 to 47 of 48 dialogues (42%-98%). Subgroup comparisons revealed higher ratings for dialogues using GPT-4.0-Turbo versus GPT-3.5-Turbo and for human-generated versus self-chat dialogues. Feedback ratings were similar for human-generated versus GPT-generated ratings, whereas authenticity ratings were lower. We did not perceive bias in any conversation. Dialogue features that detracted from authenticity included that GPT was verbose or used atypical vocabulary (93/180, 51.7% of conversations), was overly agreeable (n=56, 31%), repeated the question as part of the response (n=47, 26%), was easily convinced by clinician suggestions (n=35, 19%), or was not disaffected by poor clinician performance (n=32, 18%). For feedback, detractors included excessively positive feedback (n=42, 23%), failure to mention important weaknesses or strengths (n=41, 23%), or factual inaccuracies (n=39, 22%). Regarding validation of dialogue and feedback scores, items were meticulously developed (content evidence), and we confirmed expected relations with other variables (higher ratings for advanced LLMs and human-generated dialogues). Reproducibility was suboptimal, due largely to variation in LLM performance rather than rater idiosyncrasies. Conclusions: LLM-powered VPs can simulate patient-clinician dialogues, demonstrably represent patient preferences, and provide personalized performance feedback. This approach is scalable, globally accessible, and inexpensive. LLM-generated ratings of feedback quality are similar to human ratings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-generated simulations demonstrate clinically relevant psychiatric exchanges [23-25] and generate automated feedback aligned with expert judgment [26], offering a cost-efficient and scalable alternative to real patients for clinical training.”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [23]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior studies have demonstrated the technical feasibility of text-based VSPs and AI-assisted automated scoring systems.”
In the citing paper
“These findings, echoed by Cook et al [23], who observed that LLM-powered virtual patients generated personalized, high-quality performance feedback rated comparably to human evaluators, substantiate the potential of AI-assisted formative assessment in clinical education.”
From Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study (2026) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“To appraise rating reproducibility, we estimated variance components and calculated a single-rating intraclass correlation coefficient (ICC), which was interpreted using criteria from Landis and Koch [66] (ie, 0-0.2=slight; 0.21-0.4=fair; 0.41-0.6=moderate; and 0.61-0.8=substantial).”
InfluentialThe FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423 · cited 1×
“LLMs represent a disruptive technology [31], offering an unprecedented opportunity to transform VP production and use, enabling scalable, accessible (ie, inexpensive and low expertise), interoperable, and reusable [32] simulations of patient-clinician encounters.”
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 1×
“VPs—computer screen–based simulations of patient-clinician encounters [7]—have demonstrated efficacy in teaching, assessing, and studying clinical reasoning [8] and could also support validation of decision-support tools before clinical implementation [9,10].”
The Experience of Presence: Factor Analytic Insights
Schubert, T., Friedmann, F., & Regenbrecht, H. (2001) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474601300343603 · Cited by 2,303 · cited 1×
“We added a third empirically derived framework for evaluating “presence” in virtual reality (ie, realness, involvement, and spatial “physical” presence) [54,55].”
Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application
Cook, D. A., & Beckman, T. J. (2006) · The American Journal of Medicine · doi:10.1016/j.amjmed.2005.10.036 · Cited by 1,745 · cited 1×
“For the 3 novel instruments, we conducted a validation study collecting validity evidence from 4 of 5 potential sources [39,40]: content (ie, grounding of the instruments in theory and prior empirical work); internal structure (ie, rating reproducibility); relations with other variables (ie,…”
Self-citationFeedback in Clinical Medical Education
Ende, J. (1983) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1983.03340060055026 · Cited by 1,329 · cited 1×
“Providing performance feedback to clinicians is also essential in clinical skill development [23], yet it is commonly of low quality or simply absent [24-27].”
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 1×
“Bias—“skew that produces a type of harm toward different social groups” [62]—is a well-known risk in AI generally and natural language generation specifically [62-65].”
A three-talk model for shared decision making: multistage consultation process
Elwyn, G., Durand, M. A., Song, J., Aarts, J., Barr, P. J., Berger, Z., Cochran, N., Frosch, D., Galasiński, D., Gulbrandsen, P., et al. (2017) · BMJ · doi:10.1136/bmj.j4891 · Cited by 1,018
What every teacher needs to know about clinical reasoning
Eva, K. W. (2005) · Medical Education · doi:10.1111/j.1365-2929.2004.01972.x · Cited by 930 · cited 1×
“As we seek to prevent errors in clinical practice [1,2] and promote high-value care [3,4], we need to better understand clinical reasoning and how to support its development and application [2,5].”
Research works printed but not linked: 3
- Tong, Y, 2014, The standards for educational and psychological testing, 11
- Roller, S, arXiv, 2006.12442 · arXiv
- Adiwardana, D, arXiv
Cited by
Zeng, J., Qi, W., Shen, S., Liu, X., Li, S., Wang, B., Dong, C., Zhu, X., Shi, Y., Lou, X., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/79091 · Cited by 76
“Among studies using ChatGPT, only 2 mentioned setting the “temperature” hyperparameter [49,51], with one of them exploring whether the temperature affects LLM-based virtual patients’performance.”
InfluentialA large language model digital patient system enhances ophthalmology history taking skills
Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01841-6 · Cited by 39
Gin, B. C., LaForge, K., Burk-Rafel, J., & Boscardin, C. K. (2025) · Academic Medicine · doi:10.1097/acm.0000000000006117 · Cited by 36
Li, D., & Lutfi, S. L. (2025) · JMIR Medical Informatics · doi:10.2196/79039 · Cited by 45
The feasibility of using generative artificial intelligence for history taking in virtual patients
Yi, Y., & Kim, K.-J. (2025) · BMC Research Notes · doi:10.1186/s13104-025-07157-8 · Cited by 28
“In particular, further studies are warranted to inform the effective use of prompts and behavioral components of the chatbot that optimizes its performance across various clinical contexts [8, 9, 10].”
Metadata
- Authors
- Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A.
- Affiliations
- Mayo Clinic (US)
- University of New Mexico (US)
- Corresponding author
- Cook, David A
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-04-04
- Volume
- 27
- Pages
- e68486
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC12008702
- OpenAlex
- W4406327404
- Cited by
- 96
- References
- 65
- Citations by year
- 2026: 72 · 2025: 24
- FWCI
- 23.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Edinburgh Napier University
- Gentofte Hospital
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- virtual patients, large language models, GPT, personalized feedback, patient preferences, GPT-3.5-turbo, diabetes management, simulation scalability, user experience, prompt engineering
- MeSH terms
- Large Language Models, Communication, Diabetes Mellitus, Feedback, Humans, Language, Physician-Patient Relations, User-Computer Interface, Patient Simulation, Patient Preference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
