A Language Model–Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Background: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback. Objective: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students’ performance in history taking with a simulated patient. Methods: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients’ responses and provide immediate feedback on the comprehensiveness of the students’ history taking. Students’ interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback. Results: Most of the study’s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4’s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed “almost perfect” agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model’s assessments were overly specific or diverged from human judgement. Conclusions: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have highlighted the potential of LLMs to support clinical reasoning, generate patient narratives, and even demonstrate performance approaching passing thresholds on medical licensing examinations [6, 7].”
From Utilizing generative artificial intelligence to create simulated patient for history taking in gynecology (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Conventional methods—whether involving real standardized patients or static computer-based cases – are resource-intensive and offer limited scenario variability49.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [49]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By providing continuous feedback, personalized review recommendations, and adaptive practice tasks, AI digital tutors can help students identify knowledge gaps and reinforce key concepts during the learning process [28, 29].”
From Effectiveness of a generative AI-powered digital tutor integrated with a knowledge graph in anatomy education for nursing students: a randomized controlled trial (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, the AI tutoring system enables personalized learning pathways by analyzing students’ learning behavior and performance data [29, 35].”
From Effectiveness of a generative AI-powered digital tutor integrated with a knowledge graph in anatomy education for nursing students: a randomized controlled trial (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior studies have demonstrated the technical feasibility of text-based VSPs and AI-assisted automated scoring systems.”
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“For total feedback, we found an interrater reliability, measured by Cohen κ , of 0.832 (95% CI 0.816-0.848), indicating an “almost perfect” agreement [32].”
InfluentialSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“…that GPT-4, given its capabilities in medical knowledge [24-26] and reasoning [13], can accurately assess students’ performance in history taking despite potential limitations such as logical errors [27] and AI’s propensity to generate nonsensical content, known as “hallucinations” [28].”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Last, we did not measure any educational outcomes (ie, skill acquisition), and thus, cannot state whether the AI-generated feedback in fact improved students’ performance.”
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 1,435 · cited 1×
“Whereas traditional virtual patients often yield low levels of feedback [10], the linguistic capabilities of LLMs can provide students with higher-quality feedback [19].”
Hampton, J. R., Harrison, M. J., Mitchell, J. R., Prichard, J. S., & Seymour, C. (1975) · BMJ · doi:10.1136/bmj.2.5969.486 · Cited by 1,018
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840 · cited 1×
“Consequently, the recent emergence of LLMs such as GPT has been heralded as having the potential to revolutionize how students learn [39].”
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 633 · cited 1×
“LLMs have also demonstrated the capability of providing feedback in other circumstances, including argumentation [20], writing [21], and scientific papers [22].”
Research works printed but not linked: 4
- Peterson, MC, 1992, West J Med, 156, 163
- 10.1056/aioa2400196 · doi:10.1056/aioa2400196 · Paper page
- 2023, R: A Language and Environment for Statistical Computing
- Bowman, SR, 2023, arXiv
Cited by
Items 101–152 of 152
Glinkowski, W., Jacennik, B., Jankowska, K., Cedro, T., Wilk, S., & Doniec, R. (2026) · Applied Sciences · doi:10.3390/app16104858 · Cited by 1
Samaraweera, S., Zaid, Z., Maheshi, B., Martínez‐Maldonado, R., & Dharmaratne, A. T. (2026) · doi:10.2196/preprints.101710
BaHammam, F. A. (2026) · JMIR Formative Research · doi:10.2196/91021 · Cited by 1
Sun, Y., & Wu, L.-Y. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1841351 · Cited by 3
“In parallel, conversational agents and AI-enabled simulation tools can be employed to recreate health consultation scenarios, allowing learners to engage in interactive decision-making processes and apply acquired knowledge in practice-oriented contexts (37–39).”
Aljedaani, S., Sultan, I. E., & El-Tarhouny, S. (2026) · International Medical Education · doi:10.3390/ime5020051 · Cited by 1
“Simulating patient encounters provides medical students with a safe and controlled environment to practice history-taking skills, free from the constraints of clinical settings [9].”
Kundurti, N., & Schweizer, J. (2026) · doi:10.2196/preprints.101964
Let’s Talk Systems - What Do We Really Mean by a Systems Approach in AI-Enhanced Medical Education?
Surapaneni, K. M. (2026) · Medical Science Educator · doi:10.1007/s40670-026-02783-8
“A student might use AI deeply to clarify concepts or refine clinical reasoning, benefiting in ways the system did not deliberately design [19, 20 ].”
Dhillon, I. K., Lee, G. K. Y., & Hu, S. (2026) · JMIR Medical Education · doi:10.2196/83376 · Cited by 1
“A strength of this study was the use of real-life clinical encounters, as opposed to previous studies’ use of simulated clinical encounters [6,7,9,17].”
Authors’ Reply: Methodological Concerns in AI Medical Education Frameworks
Izquierdo‐Condoy, J. S., Arias-Intriago, M., Corrales, L. M., & Ortiz‐Prado, E. (2026) · JMIR Medical Education · doi:10.2196/104447
AI and Human Peer Feedback in Introductory Programming: A Mixed Methods Study
Çallı, E., & Er, E. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29794-5_50
Liu, Y., Zhu, Y., Shi, C., Lu, X., Wu, L., Yue, M., Hong, X., Xia, O., & Zhang, W. (2026) · JMIR Medical Education · doi:10.2196/101425 · Cited by 1
“In AI-supported medical and medical-education settings, emerging work further links trust and acceptance not only to perceived usefulness but also to technology-related factors affecting confidence in system outputs, interaction processes, and feedback quality [10,15].”
AI for Clinical Competency Assessment: Scoping Review of Methods and Applications
Zhang, W., Daniels, B., Mita, C., Nguyen, H., & Duong, D. (2026) · JMIR Medical Education · doi:10.2196/92826 · Cited by 2
“…[39], SOPHIE [42,43], AMTES [44], Turing Interactive Case Learning [45], 2-Sigma [46,47], VR-AI-ECS [48], and MedSimAI [49], while others described the intervention more generally as a “virtual patient,” “simulated patient chatbot,” or “AI-supported online platform” [31,33,35-38, 40, 41, 50].”
InfluentialMethodological Concerns in AI Medical Education Frameworks
Fatima, S. U., Fatima, E., & Kumari, R. (2026) · JMIR Medical Education · doi:10.2196/101696
“Critically, both studies revealed unresolved AI performance gaps: 8 of 45 feedback categories showed low concordance with human raters in Holderried et al [4], and AI scoring was systematically stricter than expert grading, by 3.51 points, in Luordo et al [5]—concerns that FACETS fails to address.”
InfluentialGenerative AI in health sciences education and practice. Part 4: career development
Reimers, S. J. (2026) · British Journal of Cardiac Nursing · doi:10.12968/bjca.2026.0040
Hu, Y., Wang, Y., Watanabe, B., Gao, M., Qin, Z., Shi, K., Zhang, Z., Shambanna, S., Xue, Z., Zou, T., et al. (2026) · doi:10.2196/preprints.106505
Dai, D. W., & Consoli, S. (2026) · Language and Intercultural Communication · doi:10.1080/14708477.2026.2692971
“A common approach seems to develop AI agents/avatars for communication practices (Holderried et al., 2024).”
Artificial intelligence generated virtual patients in health professions education: a scoping review
Fang, E., Boon, Y., Ho, J. Y., Tey, N., & Low, M. (2026) · BMC Medical Education · doi:10.1186/s12909-026-10028-w · Cited by 2
Alhaskir, M., Haven, H., Langner, M., Arasteh, S. T., Heidemeyer, H., Sacic, N., Majeed, R. W., Peters, A., Müller, N., Fries, M., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10282455/v1
The Future of Anesthesiology Education
Chen, F., Martinelli, S. M., & Isaak, R. J. (2026) · Anesthesiology Clinics · doi:10.1016/j.anclin.2026.06.011
Zhang, J., Zhu, Y., Lin, D., Yang, X., Yu, S., Liu, X., Wu, N., & Zhao, M. (2026) · Research Square · doi:10.21203/rs.3.rs-10452795/v1
Dai, H., Liu, M., Li, T., Jiancheng, W. A. N. G., & Fang, S. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1916714
Making AI Agents Practical in Health Professions Education: An Implementation Framework (Preprint)Preprint
Rejeleene, R., & Mehta, N. (2026) · doi:10.2196/preprints.108871
Alefragkis, D., Aloizou, D., & Chougia, V. (2026) · Cureus · doi:10.7759/cureus.114402
“Other studies have explored LLM-based simulated patients and virtual patient systems for communication practice and automated feedback [10,11].”
InfluentialGoyal, K., Noori, S., Malhan, N., & Moyo, S. (2026) · doi:10.2196/preprints.108381
Huang, Y., Xiao, X., Sheng, X., & Wang, C. (2026) · Cureus · doi:10.7759/cureus.114656
“As LLMs have advanced rapidly, a growing body of research has introduced them into the medical interview domain [12,13].”
Kamaleddin, M. A., Mirjalili, M., Barzegar, R., Le, N. T., Cote, Z., Winkler, O., Burback, L., Zhang, Y., Zai͏̈ane, O. R., Monson, C. M., et al. (2026) · JMIR Medical Education · doi:10.2196/92964 · Cited by 1
“This structure is useful for formative simulation design because it makes the basis for feedback explicit and reviewable [20,42].”
Gandhi, B., Sheth, U., McCarthy, J., & Sibbald, M. (2026) · International Medical Education · doi:10.3390/ime5030085
“Comparable work has also shown that VPs may support history-taking and diagnostic tasks when the simulation is deliberately designed around those objectives [8,9,16].”
Alvarado, J. E. G., Lagos, J. C. E., & Oyarvide, W. R. V. (2026) · Revista Tribunal · doi:10.59659/revistatribunal.v6i16.521
CHARLIE, Can You Touch Your Nose?
Grasnick, A., Becker, D., Bleser, G., Taetz, B., Drejer, J., Khan, A. M., Koch, S., & Dodiya, J. (2026) · International Conference on Virtual Reality · doi:10.1109/icvr70658.2026.11672302
“GPT-4 patient simulators can conduct plausible history-taking dialogues and generate structured feedback [9], but they operate without a body, without nonverbal behavior, and without the ability to express physical limitations.”
Artificial intelligence in emergency medicine education: A narrative review
Pelletier, J., Arango-Ibañez, J. P., Taylor, R. A., Smith, M. E., Gottlieb, M. A., Jones, D. D., Colletti, J. E., Rego, A., & Long, B. (2026) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2026.09.006
Zhao, Z., Wu, B., Tian, S., Hu, H., & Hao, P. (2026) · JMIR Medical Education · doi:10.2196/95039
“Early work in health professions education has explored generative AI for functions such as content support, explanation, feedback, and learner assistance [13-15].”
Wang, X., Sang, A., Zhang, J., Tang, H., Liu, Y., Zhao, Z., Liu, J., Xu, M., & Li, X. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1927674 · Cited by 1
Xu, J., Yu, X., Fang, Z., Wang, X., Zhang, L., & Cai, Y. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1947067
“Leveraging learning analytics to capture clinical reasoning and decision-making strategies through virtual patient platform logs and sequential data provides detailed feedback mechanisms and supports deliberate practice for mastery achievement (103).”
Wang, J., Jia, Z., Wu, K., Liu, P., Du, J., & Li, L. (2026) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-026-04006-w
Tang, X., Filonczuk, A., Zhang, X., & Cheng, Y. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_19
Kelleher, M. E., Zhou, C. Y., Weber, D., Overla, S., Bowen, J., Generoso, J. R., Zheng, W., Santen, S. A., & Turner, L. (2026) · Diagnosis · doi:10.1515/dx-2026-0069
“Single LLM chatbots have demonstrated the ability to generate plausible histories and responses when given structured prompts [25, 26].”
Generative artificial intelligence and the transformation of medical education: a scoping review
Zhao, J., Yang, K., Gao, H., & Jiang, G. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1907589 · Cited by 1
Zhao, W., Li, X., Gao, Z., Chen, Z., & Sang, L. (2026) · Research Square · doi:10.21203/rs.3.rs-10590073/v1
Situational Context Embedding and Knowledge-Based Grounding for LLMs in Surgical Training Assistance
Malayeri, A. B., Seibold, M., Oettl, F. C., Cavalcanti, N., Gerth, T., Lekar, R., Hoch, A., Zingg, P., & Fürnstahl, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38233-7_47
Ismail, N. A. S., Azman, A. H. A., Zafira, A., Faiz, M. F. A., Shaharir, S. S., Mustafar, R., Shamsuddin, C. M., Ma’arof, D. Z., & Sahran, S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1895321
Choi, Y.-M., Hur, S., & Kim, S.-H. (2026) · Journal of Digital Contents Society · doi:10.9728/dcs.2026.27.9.2573
Shawon, M. T. R., Irbaz, M. S., Elyazori, H. R. A., Resapu, K. R., Lin, Y., Cardenas, V. F., Eklou, K. P., Alemi, F., & Lybarger, K. (2026) · JMIR AI · doi:10.2196/100772 · Cited by 1
“Recent systems have integrated LLMs to enhance expressiveness and flexibility [37], constructed structured knowledge bases from clinical notes to support medical intake tasks [38], and embedded risk-aware feedback mechanisms [34].”
Brace, M. M., Furman, G., Diewald, L. K., Shenkman, R., Felgoise, S. H., & Oliver, T. (2026) · Advances in Simulation · doi:10.1186/s41077-026-00464-5 · Cited by 1
“In other words, without more detailed guidance, the AI lacked the precision needed to distinguish between superficial responses and the specific communication behaviors the rubric intended to capture [9].”
InfluentialBayona, A., Manrique, R., & Grévisse, C. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02472-w
Zhang, J., Zhu, Y., Lin, D., Yang, X., Yu, S., Liu, X., Wu, N., & Zhao, M. (2026) · Research Square · doi:10.21203/rs.3.rs-11023382/v1
Hao, S., & Huang, J. (2026) · International Conference on Big Data and Intelligent Medicine (BDIMed) · doi:10.1145/3816325.3843665
Bostraeten, P. V., Mertens, L., Jaeken, J., Bossen, J. K. J., Toelen, J., Bekkering, G. E., Delvaux, N., & Aertgeerts, B. (2026) · JMIR Formative Research · doi:10.2196/102449
Quan el xatbot és el pacient: com el futur personal sanitari aprèn amb IA
Soler, O. (2026) · The Conversation · doi:10.64628/adg.a5yfyhfqj
Jaffri, S. U., Koo, A. C., Ting, C., & Hussain, S. (2026) · Healthcare · doi:10.3390/healthcare14213643
Guo, X. (2026) · International Conference on Intelligent Education and Computer Technology (IECT) · doi:10.1145/3845354.3845498
“Future work will expand pediatric case scenarios, introduce multimodal behavior data, and combine large language model-powered virtual patients and automated feedback mechanisms to improve system performance in open-ended consultation, clinical reasoning, and personalized guidance [10].”
Metadata
- Authors
- Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M.
- Affiliations
- University of Tübingen (DE)
- Universitätsklinikum Tübingen (DE)
- Corresponding author
- Holderried, Friederike
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2024-08-16
- Volume
- 10
- Pages
- e59213
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC11364946
- OpenAlex
- W4400079353
- Cited by
- 152
- References
- 43
- Citations by year
- 2026: 95 · 2025: 53 · 2024: 4
- FWCI
- 14.14
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- medical history, standardized patients, chatbots, large language models, GPT-4, automated feedback, medical students, clinical interviewing skills, inter-rater reliability, artificial intelligence in medical education
- MeSH terms
- Artificial Intelligence, Clinical Competence, Education, Medical, Undergraduate, Feedback, Female, Humans, Male, Medical History Taking, Prospective Studies, Students, Medical, Reproducibility of Results, Patient Simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.