A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Background: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions Objective: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. Methods: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). Results: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). Conclusions: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“GPT is constantly updated with the latest medical research, guidelines, and case studies, ensuring that the educational content remains current and accurate.”
From Feasibility study of using GPT for history-taking training in medical education: a randomized clinical trial (2025) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The use of chatbots has been increasingly reported [40, 112–115]. Gorashi et al. analyzed their strengths, weaknesses, opportunities, and threats in this context [107]. Chatbots have assisted students in history-taking practice [116] and preparing for objective structured clinical examinations (OSCE) [117]. In one study comparing the use of ChatGPT with a traditional textbook for learning medical”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [116]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“and Holderried et al. found that a ChatGPT chatbot acting as a simulated patient produced highly plausible history responses51.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [51]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Among the few studies that exist on LLM-generated patient simulations [ 21 – 24 ], none have implemented feedback or debriefing at the time of reporting.”
From Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial (2024) · cites this paper as [21]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“However, LLMs, such as GPT, are language models using a next-word prediction paradigm [29] and are thus prone to “hallucinations” (ie. producing nonsensical content) [30] .”
MethodEffective physician-patient communication and health outcomes: a review.
(1995) · Europe PMC (PubMed Central) · Cited by 4,085
Fagin, R., Halpern, J. Y., Moses, Y., & Vardi, M. (1995) · The MIT Press eBooks · doi:10.7551/mitpress/5803.001.0001 · Cited by 2,610
How does communication heal? Pathways linking clinician–patient communication to health outcomes
Street, R. L., Makoul, G., Arora, N. K., & Epstein, R. M. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2008.11.015 · Cited by 2,465
Psychosomatic medicine in Germany: more timely than ever
Zipfel, S., Herzog, W., Kruse, J., & Henningsen, P. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000447701 · Cited by 2,323
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Psychology of Reasoning: Structure and Content
Wason, P. C., & Johnson-Laird, P. N. (1972) · Medical Entomology and Zoology · Cited by 1,784
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Wason, P. C. (1968) · Quarterly Journal of Experimental Psychology · doi:10.1080/14640746808400161 · Cited by 1,476
Research works printed but not linked: 4
- Stewart, MA, 1995, CMAJ, 152, 1423
- 2023, R: A Language and Environment for Statistical Computing
- Serapio-GarcíaGSafdariMCrepyCSunLFitzSAbdulhaiMFaustAMatari?MPersonality traits in large language modelsResearch Square20232024-01-03https://www.researchsquare.com/article/rs-3296728/latest
- TavarnesiGLausAMazzaRAmbrosiniLCatenazziNVaniniSTuggenerDLearning with virtual patients in medical educationCEUR-WS2024-01-03https://ceur-ws.org/Vol-2193/paper4.pdf
Cited by
Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/59213 · Cited by 152
“We used the illness script, as described in [13] and already used in the prompt for history taking, to define the categories by which to provide feedback, called “feedback categories.””
InfluentialSelf-citationA scoping review of large language models for generative tasks in mental health care
Hua, Y., Na, H., Li, Z., Liu, F., Fang, X., Clifton, D., & Torous, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01611-4 · Cited by 151
Brügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06399-7 · Cited by 137
Artificial Intelligence in Medical Education: Transforming Learning and Practice
Sriram, A., Ramachandran, K., & Krishnamoorthy, S. (2025) · Cureus · doi:10.7759/cureus.80852 · Cited by 123
Metadata
- Authors
- Holderried, F., Stegemann–Philipps, C., Herschbach, L., Moldt, J.-A., Nevins, A., Griewatz, J., Holderried, M., Herrmann-Werner, A., Festl-Wietek, T., & Mahling, M.
- Affiliations
- University of Tübingen (DE)
- Stanford Medicine (US)
- Universitätsklinikum Tübingen (DE)
- Corresponding author
- Holderried, Friederike
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2024-01-16
- Volume
- 10
- Pages
- e53961
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC10828948
- OpenAlex
- W4390918905
- Cited by
- 156
- References
- 53
- Citations by year
- 2026: 65 · 2025: 74 · 2024: 17
- FWCI
- 14.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- generative pre-trained transformer, GPT-3.5, chatbots, standardized patients, medical history, medical communication training, large language models, medical students, prompt engineering, user experience, medical education technology, question-answer pairs, social desirability bias
- MeSH terms
- Adult, Communication, Humans, Language, Medical History Taking, Prospective Studies, Students, Medical, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.