A Natural Language Processing–Based Virtual Patient Simulator and Intelligent Tutoring System for the Clinical Diagnostic Process: Simulator Development and Case Study
Open accessPublished version of a preprint
Abstract
BACKGROUND: Shortage of human resources, increasing educational costs, and the need to keep social distances in response to the COVID-19 worldwide outbreak have prompted the necessity of clinical training methods designed for distance learning. Virtual patient simulators (VPSs) may partially meet these needs. Natural language processing (NLP) and intelligent tutoring systems (ITSs) may further enhance the educational impact of these simulators. OBJECTIVE: The goal of this study was to develop a VPS for clinical diagnostic reasoning that integrates interaction in natural language and an ITS. We also aimed to provide preliminary results of a short-term learning test administered on undergraduate students after use of the simulator. METHODS: We trained a Siamese long short-term memory network for anamnesis and NLP algorithms combined with Systematized Nomenclature of Medicine (SNOMED) ontology for diagnostic hypothesis generation. The ITS was structured on the concepts of knowledge, assessment, and learner models. To assess short-term learning changes, 15 undergraduate medical students underwent two identical tests, composed of multiple-choice questions, before and after performing a simulation by the virtual simulator. The test was made up of 22 questions; 11 of these were core questions that were specifically designed to evaluate clinical knowledge related to the simulated case. RESULTS: We developed a VPS called Hepius that allows students to gather clinical information from the patient's medical history, physical exam, and investigations and allows them to formulate a differential diagnosis by using natural language. Hepius is also an ITS that provides real-time step-by-step feedback to the student and suggests specific topics the student has to review to fill in potential knowledge gaps. Results from the short-term learning test showed an increase in both mean test score (P<.001) and mean score for core questions (P<.001) when comparing presimulation and postsimulation performance. CONCLUSIONS: By combining ITS and NLP technologies, Hepius may provide medical undergraduate students with a learning tool for training them in diagnostic reasoning. This may be particularly useful in a setting where students have restricted access to clinical wards, as is happening during the COVID-19 pandemic in many countries worldwide.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many existing LLM-based patient simulation systems remain primarily confined to textual interactions.
“While these systems demonstrate the power of LLMs for complex medical simulations and text-based dialogue training, many remain primarily confined to textual interactions.”
From LLM-Powered Virtual Patient Agents for Interactive Clinical Skills Training with Automated Feedback (2025) · cites this paper as [10]
The Discussion section, “Virtual Patient Simulators and Natural Language Processing” (page unavailable), says Hepius permits free-text interaction, supporting a textual-interaction example. It does not establish that many LLM-based patient simulators are primarily confined to text; instead, it describes most virtual patient simulators as using menus and predefined selections. (LLM, conf=0.85)
Experimental — accuracy not yet measuredAI was used in simulations for surgical simulation with a virtual operative assistant, simulated patient physician medical interviews, and interactions with avatars as standardized patients during clinical interviews.
“Various uses were described, including surgical simulation with a virtual operative assistant [132], simulated patient physician medical interviews [120–123, 130], and interactions with avatars as standardized patients during clinical interviews [124–126].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [130]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · see citing works
Kosko, B. (1986) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(86)80040-2 · Cited by 3,576 · see citing works
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324 · see citing works
The power of testing memory: Basic research and implications for educational practice
Roediger, H. L., & Karpicke, J. D. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00012.x · Cited by 2,037 · see citing works
Medical Student Education in the Time of COVID-19
Rose, S. (2020) · JAMA · doi:10.1001/jama.2020.5227 · Cited by 1,740 · see citing works
Building Watson: An Overview of the DeepQA Project
Ferrucci, D. A., Brown, E. W., Chu‐Carroll, J., Fan, J., Gondek, D. C., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J. M., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,523 · see citing works
Neural Networks and Deep Learning
Aggarwal, C. (2018) · Cambridge International Law Journal · doi:10.1007/978-3-319-94463-0 · Cited by 932 · see citing works
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930 · cited 2× · see citing works
“These programs have been proven to be effective as teaching tools within different educational fields [12,13].”
BackgroundAl-Balas, M., Al-Balas, H. I., Jaber, H. M., Obeidat, K. A., Al-Balas, H., Aborajooh, E. A., Al-Taher, R. N., & Al-Balas, B. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02257-4 · Cited by 706 · see citing works
Virtual patients: a critical literature review and proposed next steps
Cook, D. A., & Triola, M. M. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03286.x · Cited by 624 · see citing works
Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi, M. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2018.1558257 · Cited by 605 · see citing works
Kononowicz, A. A., Woodham, L. A., Edelbring, S., Stathakarou, N., Davies, D., Saxena, N., Tudor Car, L., Carlstedt-Duke, J., Car, J., & Zary, N. (2019) · Journal of Medical Internet Research · doi:10.2196/14676 · Cited by 600 · see citing works
Computerized Virtual Patients in Health Professions Education: A Systematic Review and Meta-Analysis
Cook, D. A., Erwin, P. J., & Triola, M. M. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181edfe13 · Cited by 570 · see citing works
AMEE Guide no. 34: teaching in the clinical environment
Ramani, S., & Leinster, S. (2008) · Medical Teacher · doi:10.1080/01421590802061613 · Cited by 531 · see citing works
Individualized Bayesian Knowledge Tracing Models
Yudelson, M. V., Koedinger, K. R., & Gordon, G. J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_18 · Cited by 502 · see citing works
Clinical cognition and diagnostic error: applications of a dual process model of reasoning
Croskerry, P. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9182-2 · Cited by 494 · see citing works
Design for classroom orchestration
Dillenbourg, P. (2013) · Computers & Education · doi:10.1016/j.compedu.2013.04.013 · Cited by 386 · see citing works
Sapci, A. H., & Sapci, H. A. (2020) · JMIR Medical Education · doi:10.2196/19285 · Cited by 377 · see citing works
Clinical Reasoning Assessment Methods: A Scoping Review and Practical Guidance
Daniel, M. M., Rencic, J. J., Durning, S. J., Holmboe, E. S., Santen, S. A., Lang, V. J., Ratcliffe, T. A., Gordon, D., Heist, B. S., Lubarsky, S., et al. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002618 · Cited by 358 · see citing works
How to improve the teaching of clinical reasoning: a narrative review and a proposal
Schmidt, H. G., & Mamede, S. (2015) · Medical Education · doi:10.1111/medu.12775 · Cited by 308 · see citing works
Concept maps in medical education: an analytical literature review
Daley, B. J., & Torre, D. M. (2010) · Medical Education · doi:10.1111/j.1365-2923.2010.03628.x · Cited by 307 · see citing works
The economics of health care quality and medical errors.
Andel, C., Davidow, S. L., Hollander, M., & Moreno, D. A. (2012) · PubMed · Cited by 254 · see citing works
The Transformational Effects of COVID-19 on Medical Education
Lucey, C. R., & Johnston, S. C. (2020) · JAMA · doi:10.1001/jama.2020.14136 · Cited by 240 · see citing works
Artificial Intelligence as an Effective Classroom Assistant
du Boulay, B. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.93 · Cited by 213 · cited 1× · see citing works
“However, in spite of its promising features, it is important to stress that Hepius cannot fully replace a skilled human tutor working one on one with a learner [65].”
BackgroundThe Role for Virtual Patients in the Future of Medical Education
Berman, N. B., Durning, S. J., Fischer, M. R., Huwendiek, S., & Triola, M. M. (2016) · Academic Medicine · doi:10.1097/acm.0000000000001146 · Cited by 211 · see citing works
Introduction to Natural Language Processing
Podrebarac, A. A. (2019) · Routledge eBooks · doi:10.4324/9781351027946-15 · Cited by 176 · see citing works
Garcia, S., & Miller, S. (2019) · American Mathematical Society eBooks · doi:10.1090/mbk/121/79 · Cited by 122 · see citing works
A Review of Natural Language Processing in Medical Education
Chary, M. A., Parikh, S., Manini, A. F., Boyer, E. W., & Radeous, M. (2018) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2018.11.39725 · Cited by 112 · see citing works
Medical education in the time of COVID-19
Wayne, D. B., Green, M. M., & Neilson, E. G. (2020) · Science Advances · doi:10.1126/sciadv.abc7110 · Cited by 112 · see citing works
Mixture Hidden Markov Models for Sequence Data: The seqHMM Package in R
Helske, S., & Helske, J. (2019) · Journal of Statistical Software · doi:10.18637/jss.v088.i03 · Cited by 111 · see citing works
Dict2vec : Learning Word Embeddings using Lexical Dictionaries
Tissier, J., Gravier, C., & Habrard, A. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1024 · Cited by 105 · see citing works
Twelve tips to support the development of clinical reasoning skills using virtual patient cases
Posel, N. H., McGee, J. B., & FLEISZER, D. M. (2014) · Medical Teacher · doi:10.3109/0142159x.2014.993951 · Cited by 101 · see citing works
Natural Language Processing and its Use in Education
Khaled, K. (2014) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2014.051210 · Cited by 94 · see citing works
WatsonPaths: Scenario‐Based Question Answering and Inference over Unstructured Information
Lally, A., Bagchi, S., Barborak, M., Buchanan, D. W., Chu‐Carroll, J., Ferrucci, D. A., Glass, M. R., Kalyanpur, A. A., Mueller, E. T., Murdock, J. W., et al. (2017) · AI Magazine · doi:10.1609/aimag.v38i2.2715 · Cited by 87 · see citing works
Escape from the Skinner Box: The case for contemporary intelligent learning environments
du Boulay, B. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12860 · Cited by 80 · see citing works
Weingart, S. N. (2000) · Western Journal of Medicine · doi:10.1136/ewjm.172.6.390 · Cited by 68 · see citing works
A possible future for next generation adaptive learning systems
Essa, A. (2016) · Smart Learning Environments · doi:10.1186/s40561-016-0038-y · Cited by 67 · see citing works
Adaptive systems in education: a review and conceptual unification
Wilson, C., & Scott, B. C. (2017) · International Journal of Information and Learning Technology · doi:10.1108/ijilt-09-2016-0040 · Cited by 56 · see citing works
El Saadawi, G. M., Tseytlin, E., Legowski, E., Jukic, D., Castine, M., Fine, J., Gormley, R., & Crowley, R. S. (2008) · Advances in Health Sciences Education · doi:10.1007/s10459-007-9081-3 · Cited by 48 · see citing works
Al-Balas, M., Al-Balas, H. I., Jaber, H. M., Obeidat, K. A., Al-Balas, H., Aborajooh, E. A., Al-Taher, R. N., & Al-Balas, B. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02428-3 · Cited by 46 · see citing works
Wu, G., & Lan, M. (2016) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s16-1135 · Cited by 20 · see citing works
Developing an intelligent tutoring system for robotic‐assisted surgery instruction
Julian, D., & Smith, R. (2019) · The International Journal of Medical Robotics and Computer Assisted Surgery · doi:10.1002/rcs.2037 · Cited by 19 · see citing works
Designing engaging intelligent tutoring systems in an age of cognitive computing
Sundararajan, S. C. M., & Nitta, S. (2015) · IBM Journal of Research and Development · doi:10.1147/jrd.2015.2464085 · Cited by 13 · see citing works
A novel approach to virtual patient simulation using natural language processing
Persad, A. R. L., Stroulia, E., & Forgie, S. E. D. (2016) · Medical Education · doi:10.1111/medu.13197 · Cited by 13 · see citing works
AI Medical School Tutor: Modelling and Implementation
Afzal, S., Dhamecha, T. I., Gagnon, P., Nayak, A., Shah, A., Carlstedt-Duke, J., Pathak, S., Mondal, S., Gugnani, A., Zary, N., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59137-3_13 · Cited by 10 · see citing works
(2019) · Oncology Times · doi:10.1097/01.cot.0000554500.76160.69 · Cited by 9 · see citing works
Schuelper, N., Ludwig, S., Anders, S., & Raupach, T. (2019) · JMIR Medical Education · doi:10.2196/13386 · Cited by 8 · see citing works
Helske, J., & Helske, S. (2021) · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- [14]Vanlehn K Lynch C Schulze K Shapiro JA Shelby R Taylor L Treacy D Weinstein A Wintersgill M The Andes physics tutoring system: Five years of evaluations Proceedings of the 12th International Conference on Artificial Intelligence in Education 2005 12th International Conference on Artificial Intelligence in Education July 18-22, 2005 Amsterdam, the Netherlands 678 685
- [32]Mueller J Thyagarajan A Siamese recurrent architectures for learning sentence similarity Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) 2016 Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16) February 12-17, 2016 Phoenix, AZ 2786 2792
- [33]Goldberg Y Levy O word2vec explained: Deriving Mikolov et al’s negative-sampling word-embedding method ArXiv. Preprint posted online on February 15, 2014.
- [35]Mikolov T Chen K Corrado G Dean J Efficient estimation of word representations in vector space Proceedings of the 1st International Conference on Learning Representations (ICLR 2013) 2013 1st International Conference on Learning Representations (ICLR 2013) May 2-4, 2013 Scottsdale, AZ
- [36]Mikolov T Sutskever I Chen K Corrado GS Dean J Distributed representations of words and phrases and their compositionality Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS'13) 2013 27th International Conference on Neural Information Processing Systems (NIPS'13) December 5-10, 2013 Lake Tahoe, NV 3111 3119
- [37]Le Q Mikolov T Distributed representations of sentences and documents Proceedings of the 31st International Conference on Machine Learning 2014 31st International Conference on Machine Learning June 21-26, 2014 Beijing, China 1188 1196
- [42]Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser Ł Polosukhin I Attention is all you need Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017) 2017 31st Conference on Neural Information Processing Systems (NIPS 2017) December 4-9, 2017 Long Beach, CA
- [60]Abdulnour RE Lieber J Faselis C Sternschein R Hayden RM Massaro A Rubin D Covid-19 Rx: Treatment simulations NEJM Group 2020 2021-03-19 https://covid19rx.nejm.org/landing/index.html
Cited by
Sorted by citations
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
Feigerlova, E., Hani, H., & Hothersall-Davies, E. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06719-5 · Cited by 166 · see citing works
Stamer, T., Steinhäuser, J., & Flägel, K. (2023) · Journal of Medical Internet Research · doi:10.2196/43311 · Cited by 160 · see citing works
AI in Medical Education: Global situation, effects and challenges
Zhang, W., Cai, M., Lee, H. J., Evans, R., Zhu, C., & Ming, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12009-8 · Cited by 148 · see citing works
Rangel-de Lázaro, G., & Duart, J. M. (2023) · Sustainability · doi:10.3390/su15043507 · Cited by 95 · see citing works
Wang, M., Sun, Z., Jia, M., Wang, Y., Wang, H., Zhu, X., Chen, L., & Ji, H. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01797-7 · Cited by 52 · see citing works
Artificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
“Various uses were described, including surgical simulation with a virtual operative assistant [132], simulated patient physician medical interviews [120–123, 130 ], and interactions with”
Maicher, K. R., Stiff, A., Scholl, M., White, M., Fosler-Lussier, E., Schuler, W., Serai, P., Sunder, V., Forrestal, H., Mendella, L., et al. (2023) · Medical Teacher · doi:10.1080/0142159x.2022.2130216 · Cited by 44 · see citing works
Lippitsch, A., Steglich, J., Ludwig, C., Kellner, J., Hempel, L., Stoevesandt, D., & Thews, O. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107964 · Cited by 8 · see citing works
LLM-Powered Virtual Patient Agents for Interactive Clinical Skills Training with Automated Feedback
Voigt, H., Sugamiya, Y., Lawonn, K., Zarrieß, S., & Takanishi, A. (2025) · arXiv · doi:10.48550/arxiv.2508.13943
- Authors
- Furlan, R., Gatti, M., Menè, R., Shiffer, D., Marchiori, C., Giaj Levra, A., Saturnino, V., Brunetta, E., & Dipaola, F.
- Affiliations
- Humanitas University (IT)
- IBM (Italy) (IT)
- IBM Research - Zurich (CH)
- Corresponding author
- Furlan, Raffaello
- ORCID
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2021-04-09
- Volume
- 9
- Issue
- 4
- Pages
- e24073
- ISSN
- 2291-9694 (ISSN-L)
- PMC
- PMC8041050
- OpenAlex
- W3139009262
- Cited by
- 67
- References
- 50
- Citations by year
- 2026: 17 · 2025: 21 · 2024: 14 · 2023: 7 · 2022: 5 · 2021: 3
- FWCI
- 2.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Informatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Humanitas University
- Humanitas Research Hospital
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Health Sciences › Medicine › Physiology › Simulation-Based Education in Healthcare
- Keywords
- virtual patient simulator, intelligent tutoring systems, natural language processing, differential diagnosis, medical education, distance education, COVID-19 pandemic, real-time feedback, knowledge gaps, undergraduate medical students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
