A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Background: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions Objective: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. Methods: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). Results: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). Conclusions: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“GPT is constantly updated with the latest medical research, guidelines, and case studies, ensuring that the educational content remains current and accurate.”
From Feasibility study of using GPT for history-taking training in medical education: a randomized clinical trial (2025) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The use of chatbots has been increasingly reported [40, 112–115]. Gorashi et al. analyzed their strengths, weaknesses, opportunities, and threats in this context [107]. Chatbots have assisted students in history-taking practice [116] and preparing for objective structured clinical examinations (OSCE) [117]. In one study comparing the use of ChatGPT with a traditional textbook for learning medical”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [116]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“and Holderried et al. found that a ChatGPT chatbot acting as a simulated patient produced highly plausible history responses51.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [51]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Among the few studies that exist on LLM-generated patient simulations [ 21 – 24 ], none have implemented feedback or debriefing at the time of reporting.”
From Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial (2024) · cites this paper as [21]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“However, LLMs, such as GPT, are language models using a next-word prediction paradigm [29] and are thus prone to “hallucinations” (ie. producing nonsensical content) [30] .”
MethodEffective physician-patient communication and health outcomes: a review.
(1995) · Europe PMC (PubMed Central) · Cited by 4,085
Fagin, R., Halpern, J. Y., Moses, Y., & Vardi, M. (1995) · The MIT Press eBooks · doi:10.7551/mitpress/5803.001.0001 · Cited by 2,610
How does communication heal? Pathways linking clinician–patient communication to health outcomes
Street, R. L., Makoul, G., Arora, N. K., & Epstein, R. M. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2008.11.015 · Cited by 2,465
Psychosomatic medicine in Germany: more timely than ever
Zipfel, S., Herzog, W., Kruse, J., & Henningsen, P. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000447701 · Cited by 2,323
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Psychology of Reasoning: Structure and Content
Wason, P. C., & Johnson-Laird, P. N. (1972) · Medical Entomology and Zoology · Cited by 1,784
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Wason, P. C. (1968) · Quarterly Journal of Experimental Psychology · doi:10.1080/14640746808400161 · Cited by 1,476
Virtual reality and the transformation of medical education
Pottle, J. (2019) · Future Healthcare Journal · doi:10.7861/fhj.2019-0036 · Cited by 985
Nadarzynski, T., Miles, O., Cowie, A., & Ridge, D. (2019) · DIGITAL HEALTH · doi:10.1177/2055207619871808 · Cited by 810 · cited 1×
“Chatbots have thus become valuable tools in health care; their nonjudgmental and easily accessible nature makes them particularly well suited for responding to patient inquiries and concerns [19,20].”
BackgroundBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · cited 1×
“5 is known to perform reasonably well in both abductive and commonsense reasoning tasks [46,47]; our data confirmed these observations.”
ResultBhayana, R., Krishna, S., & Bleakney, R. R. (2023) · Radiology · doi:10.1148/radiol.230582 · Cited by 460
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
The doctor-patient relationship: Challenges, opportunities, and strategies
Goold, S. D., & Lipkin, M. (1999) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1999.00267.x · Cited by 422
Endpoints in medical communication research, proposing a framework of functions and outcomes
de Haes, H., & Bensing, J. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2008.12.006 · Cited by 402
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · cited 1×
“5 is known to perform reasonably well in both abductive and commonsense reasoning tasks [46,47]; our data confirmed these observations.”
ResultUK consensus statement on the content of communication curricula in undergraduate medical education
Von Fragstein, M., Silverman, J., Cushing, A., Quilligan, S., Salisbury, H., Wiskin, C., & on behalf of the UK Council for Clinical Communication Skills Teaching in Undergraduate Medical Education (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03137.x · Cited by 330
Approaches to studying formal and everyday reasoning.
Galotti, K. M. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.3.331 · Cited by 322
Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations
Ali, R., Tang, O. Y., Connolly, I. D., Zadnik Sullivan, P. L., Shin, J. H., Fridley, J. S., Asaad, W. F., Cielo, D., Oyelese, A. A., Doberstein, C. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002632 · Cited by 273
Teaching communication skills to medical students, a challenge in the curriculum?
Deveugele, M., Derese, A., Maesschalck, S. D., Willems, S., Driel, M. V., & Maeseneer, J. D. (2005) · Patient Education and Counseling · doi:10.1016/j.pec.2005.06.004 · Cited by 252
Liaw, S. Y., Tan, J. Z., Lim, S., Zhou, W., Yap, J., Ratan, R., Ooi, S. L., Wong, S. J., Seah, B., & Chua, W. L. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105718 · Cited by 223 · cited 2×
“Therefore, newer approaches integrate chatbots in virtual reality environments [54], paving the way for a more integrated learning experience.”
BackgroundBeyond a good story: from Hawthorne Effect to reactivity in health professions education research
Paradis, E., & Sutkin, G. (2017) · Medical Education · doi:10.1111/medu.13122 · Cited by 221
Teaching history taking to medical students: a systematic review
Keifenheim, K. E., Teufel, M., Ip, J., Speiser, N., Leehr, E. J., Zipfel, S., & Herrmann-Werner, A. (2015) · BMC Medical Education · doi:10.1186/s12909-015-0443-x · Cited by 218 · cited 1×
“To practice history taking, the learner is required to have an interactive encounter [14], and courses frequently rely on simulated or real patients [15].”
BackgroundSelf-citationEffective virtual patient simulators for medical communication training: A systematic review
Lee, J., Kim, H., Kim, K. H., Jung, D., Jowsey, T., & Webster, C. S. (2020) · Medical Education · doi:10.1111/medu.14152 · Cited by 210
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., & Mctear, M. (2019) · Proceedings of the 31st European Conference on Cognitive Ergonomics · doi:10.1145/3335082.3335094 · Cited by 206
Espejel, J. L., Ettifouri, E. H., Alassan, M. S. Y., Chouham, E. M., & Dahhane, W. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100032 · Cited by 191
Kaplonyi, J., Bowles, K., Nestel, D., Kiegaldie, D., Maloney, S., Haines, T., & Williams, C. (2017) · Medical Education · doi:10.1111/medu.13387 · Cited by 189
Moldt, J.-A., Festl-Wietek, T., Madany Mamlouk, A., Nieselt, K., Fuhl, W., & Herrmann-Werner, A. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2182659 · Cited by 186 · cited 2×
“In particular, chatbots are of interest tool-wise in the area of virtual patients [21,22].”
BackgroundSelf-citationChatbot-based heathcare service with a knowledge base for cloud computing
Chung, K., & Park, R. C. (2019) · Cluster Computing · doi:10.1007/s10586-018-2334-5 · Cited by 172
Stamer, T., Steinhäuser, J., & Flägel, K. (2023) · Journal of Medical Internet Research · doi:10.2196/43311 · Cited by 160
Chatbots as Conversational Healthcare Services
Jovanovic, M., Báez, M., & Casati, F. (2020) · IEEE Internet Computing · doi:10.1109/mic.2020.3037151 · Cited by 153
How clinical communication has become a core part of medical education in the UK
Brown, J. (2008) · Medical Education · doi:10.1111/j.1365-2923.2007.02955.x · Cited by 140
Personality Traits in Large Language ModelsPreprint
Serapio‐García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Abdulhai, M., Faust, A., & Matarić, M. J. (2023) · Research Square · doi:10.21203/rs.3.rs-3296728/v1 · Cited by 137
Bosse, H. M., Nickel, M., Huwendiek, S., Schultz, J. H., & Nikendei, C. (2015) · BMC Medical Education · doi:10.1186/s12909-015-0468-1 · Cited by 135
Hausberg, M. C., Hergert, A., Kröger, C., Bullinger, M., Rose, M., & Andreas, S. (2012) · BMC Medical Education · doi:10.1186/1472-6920-12-16 · Cited by 133
McHugh, C., & Way, J. (2017) · Mind · doi:10.1093/mind/fzw068 · Cited by 132
A systematic review of healthcare professionals' core competency instruments
Jabri, F. A., Kvist, T. A., Azimirad, M., & Turunen, H. E. (2021) · Nursing and Health Sciences · doi:10.1111/nhs.12804 · Cited by 114
Performance of ChatGPT and GPT-4 on Neurosurgery Written Board ExaminationsPreprint
Ali, R., Tang, O. Y., Connolly, I. D., et al. (2023) · medRxiv · doi:10.1101/2023.03.25.23287743 · Cited by 91
Maicher, K. R., Danforth, D. R., Price, A., Zimmerman, L., Wilcox, B., Liston, B. W., Cronau, H., Belknap, L., Ledford, C. H., Way, D. P., et al. (2016) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000195 · Cited by 90
Noble, L., Scott‐Smith, W., O’Neill, B., & Salisbury, H. (2018) · Patient Education and Counseling · doi:10.1016/j.pec.2018.04.013 · Cited by 83
Effects of Portraying Psychologically and Emotionally Complex Standardized Patient Roles
McNaughton, N., Tiberius, R. G., & Hodges, B. D. (1999) · Teaching and Learning in Medicine · doi:10.1207/s15328015tl110303 · Cited by 78
An Overview of the Use of Chatbots in Medical and Healthcare Education
Frangoudes, F., Hadjiaros, M., Schiza, E. C., Matsangidou, M., Tsivitanidou, O., & Neokleous, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77943-6_11 · Cited by 64
Principles and Practice of Case-based Clinical Reasoning Education
(2018) · Innovation and Change in Professional Education · doi:10.1007/978-3-319-64828-6 · Cited by 58
Developing communication curricula in healthcare education: An evidence-based guide
Bachmann, C., Pettit, J. E., & Rosenbaum, M. (2021) · Patient Education and Counseling · doi:10.1016/j.pec.2021.11.016 · Cited by 58
Core competencies of the European internist: A discussion paper
Pálsson, R., Kellett, J. G., Lindgren, S. C., Merino, J., Semple, C., & Séréni, D. (2007) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2006.10.002 · Cited by 49
Machine learning to extract communication and history‐taking skills in OSCE transcripts
Jani, K. H., Jones, K. A., Jones, G. W., Amiel, J., Barron, B., & Elhadad, N. (2020) · Medical Education · doi:10.1111/medu.14347 · Cited by 39 · cited 1×
“In addition to physician-patient communication skills, chatbots have been used for interprofessional communication [54] and for skill assessments [55].”
BackgroundBorowczyk, M., Stalmach–Przygoda, A., Doroszewska, A., Libura, M., Chojnacka-Kuraś, M., Małecki, Ł., Kowalski, Z., & Jankowska, K. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04533-5 · Cited by 39
A psychometric framework for evaluating and shaping personality traits in large language models
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01115-6 · Cited by 30
Towards Validating a Chatbot Usability Scale
Holmes, S. L., Bond, R., Moorhead, A., Zheng, J., COATES, V. E., & McTear, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35708-4_24 · Cited by 26
Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench
Huang, J.-T., Lam, M. H., Li, E. J., Ren, S., Wang, W., Jiao, W., Tu, Z., & Lyu, M. R. (2023) · arXiv · doi:10.48550/arxiv.2308.03656 · Cited by 14
Revisiting the Reliability of Psychological Scales on Large Language ModelsPreprint
Huang, J.-T., Jiao, W., Lam, M. H., Li, E. J., Wang, W., & Lyu, M. R.-T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.19926 · Cited by 10
Mitigating the Hawthorne Effect Using Computer Simulations
Holmes, S. Y. (2011) · SensePublishers eBooks · doi:10.1007/978-94-6091-329-7_11 · Cited by 8
Learning and Collaboration Technologies: Games and Virtual Environments for Learning
(2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77943-6 · Cited by 3
Singh, U., & Aarabhi, P. (2023) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai54212.2023.00097 · Cited by 3
Research works printed but not linked: 4
- Stewart, MA, 1995, CMAJ, 152, 1423
- 2023, R: A Language and Environment for Statistical Computing
- Serapio-GarcíaGSafdariMCrepyCSunLFitzSAbdulhaiMFaustAMatari?MPersonality traits in large language modelsResearch Square20232024-01-03https://www.researchsquare.com/article/rs-3296728/latest
- TavarnesiGLausAMazzaRAmbrosiniLCatenazziNVaniniSTuggenerDLearning with virtual patients in medical educationCEUR-WS2024-01-03https://ceur-ws.org/Vol-2193/paper4.pdf
Cited by
Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/59213 · Cited by 152
“We used the illness script, as described in [13] and already used in the prompt for history taking, to define the categories by which to provide feedback, called “feedback categories.””
InfluentialSelf-citationA scoping review of large language models for generative tasks in mental health care
Hua, Y., Na, H., Li, Z., Liu, F., Fang, X., Clifton, D., & Torous, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01611-4 · Cited by 151
Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06399-7 · Cited by 137
Artificial Intelligence in Medical Education: Transforming Learning and Practice
Sriram, A., Ramachandran, K., & Krishnamoorthy, S. (2025) · Cureus · doi:10.7759/cureus.80852 · Cited by 123
Metadata
- Authors
- Holderried, F., Stegemann–Philipps, C., Herschbach, L., Moldt, J.-A., Nevins, A., Griewatz, J., Holderried, M., Herrmann-Werner, A., Festl-Wietek, T., & Mahling, M.
- Affiliations
- University of Tübingen (DE)
- Stanford Medicine (US)
- Universitätsklinikum Tübingen (DE)
- Corresponding author
- Holderried, Friederike
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2024-01-16
- Volume
- 10
- Pages
- e53961
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC10828948
- OpenAlex
- W4390918905
- Cited by
- 156
- References
- 53
- Citations by year
- 2026: 65 · 2025: 74 · 2024: 17
- FWCI
- 14.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- generative pre-trained transformer, GPT-3.5, chatbots, standardized patients, medical history, medical communication training, large language models, medical students, prompt engineering, user experience, medical education technology, question-answer pairs, social desirability bias
- MeSH terms
- Adult, Communication, Humans, Language, Medical History Taking, Prospective Studies, Students, Medical, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.