Designing a virtual patient dialogue system based on terminology-rich resources: Challenges and evaluation
Open accessAbstract
Virtual patient software allows health professionals to practise their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history-taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we designed a patient record model, a knowledge model for the task and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowledge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 terms, and dictionaries with over 959,000 word/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non-medical evaluators) interact with the system and use 35 cases from 18 specialities. We conducted a quantitative evaluation of all components by analysing interaction logs (11,834 turns). Natural language understanding achieved an F-measure of 95.8%. Dialogue management provided on average 74.3 (±9.5)% of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All evaluated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8% of their terms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current conversational agents are available via multiple digital platforms (telephones, mobile phones, tablets, computers) and in many virtual formats (chatbots, embodied conversational agents, three-dimensional avatars).
“Current agents are available via multiple digital platforms, including telephones, mobile phones, tablets, and computers, and in many virtual formats such as chatbots, embodied conversational agents, and three-dimensional avatars [2,7,8].”
From Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review (2020) · cites this paper as [7]
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
Huddleston, R. (1988) · Language · doi:10.2307/415437 · Cited by 7,587 · see citing works
A Comprehensive Grammar of the English Language
Beauvais, P. J., Quirk, R., Greenbaum, S., et al. (1987) · College Composition and Communication · doi:10.2307/357731 · Cited by 5,488 · see citing works
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · see citing works
Computerized Virtual Patients in Health Professions Education: A Systematic Review and Meta-Analysis
Cook, D. A., Erwin, P. J., & Triola, M. M. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181edfe13 · Cited by 570 · see citing works
SNOMED-CT: The advanced terminology and coding system for eHealth.
Donnelly, K. (2006) · PubMed · Cited by 541 · see citing works
PERFORMANCE MEASURES FOR INFORMATION EXTRACTION
Makhoul, J. I., Kubala, F., Schwartz, R. M., & Weischedel, R. (2007) · Cited by 503 · see citing works
The Information State Approach to Dialogue Management
Traum, D. R., & Larsson, S. (2003) · Text, speech and language technology · doi:10.1007/978-94-010-0019-2_15 · Cited by 344 · see citing works
Lexical methods for managing variation in biomedical terminologies.
McCray, A. T., Srinivasan, S., & Browne, A. C. (1994) · PubMed · Cited by 281 · see citing works
(2001) · PubMed · Cited by 275 · see citing works
Health dialog systems for patients and consumers
Bickmore, T., & Giorgino, T. (2006) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.12.004 · Cited by 242 · see citing works
The use of virtual patients to teach medical students history taking and communication skills
Stevens, A. O., Hernandez, J. A., Johnsen, K., Dickerson, R. F., Raij, A. B., Harrison, C., DiPietro, M., Allen, B., Ferdig, R. E., Foti, S., et al. (2006) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2006.03.002 · Cited by 221 · see citing works
PARADISE: A Framework for Evaluating Spoken Dialogue AgentsPreprint
Walker, M. A., Litman, D. J., Kamm, C., & Abella, A. (1997) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9704004 · Cited by 175 · see citing works
UMLS Concept Indexing for Production Databases: A Feasibility Study
Nadkarni, P., Chen, R., & Brandt, C. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080080 · Cited by 116 · see citing works
Jokinen, K., & McTear, M. (2010) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-02134-3 · Cited by 104 · see citing works
USING POMDPS FOR DIALOG MANAGEMENT
Young, S. J. (2006) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2006.326785 · Cited by 97 · see citing works
Maicher, K. R., Danforth, D. R., Price, A., Zimmerman, L., Wilcox, B., Liston, B. W., Cronau, H., Belknap, L., Ledford, C. H., Way, D. P., et al. (2016) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000195 · Cited by 90 · see citing works
Sorting Out the Virtual Patient
Talbot, T. B., Sagae, K., John, B., & Rizzo, A. (2012) · International Journal of Gaming and Computer-Mediated Simulations · doi:10.4018/jgcms.2012070101 · Cited by 84 · see citing works
Hubal, R. C., Kizakevich, P. N., Guinn, C. I., Merino, K. D., & West, S. L. (2000) · PubMed · Cited by 78 · see citing works
Virtual Patients and Problem-Based Learning in Advanced Therapeutics
Benedict, N. J. (2010) · American Journal of Pharmaceutical Education · doi:10.5688/aj7408143 · Cited by 78 · see citing works
Evaluation of Justina: A Virtual Patient with PTSD
Kenny, P. G., Parsons, T. D., Gratch, J. S., & Rizzo, A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85483-8_40 · Cited by 77 · see citing works
Un système de dictionnaires électroniques pour les mots simples du français
Courtois, B. (1990) · Langue française · doi:10.3406/lfr.1990.6323 · Cited by 72 · see citing works
On the means for clarification in dialogue
Purver, M., Ginzburg, J., & Healey, P. G. T. (2001) · SIGdial Workshop on Discourse and Dialogue (SIGDIAL) · doi:10.3115/1118078.1118094 · Cited by 72 · see citing works
Automated spoken dialogue system for hypertensive patient home management
Giorgino, T., Azzini, I., Rognoni, C., Quaglini, S., Stefanelli, M., Gretter, R., & Falavigna, D. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.04.026 · Cited by 69 · see citing works
Development of Virtual Patient Simulations for Medical Education
Danforth, D. R., Procter, M., Chen, R., Johnson, M. E., & Heller, R. (2009) · Journal of Virtual Worlds Research · doi:10.4101/jvwr.v2i2.707 · Cited by 65 · see citing works
A crowdsourcing method to develop virtual human conversational agents
Rossen, B., & Lok, B. C. L. (2012) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2011.11.004 · Cited by 65 · see citing works
Automatic generation of spoken dialogue from medical plans and ontologies
Beveridge, M. A., & Fox, J. (2006) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.12.008 · Cited by 57 · see citing works
Jokinen, K., & McTear, M. (2009) · Synthesis lectures on human language technologies · doi:10.2200/s00204ed1v01y200910hlt005 · Cited by 57 · see citing works
Methods for Evaluating Software Architecture: A Survey
Roy, B. (2008) · Cited by 55 · see citing works
Deep Learning in Natural Language Processing
Neterer, J. R., & Guzide, O. (2018) · Proceedings of the West Virginia Academy of Science · doi:10.55632/pwvas.v90i1.339 · Cited by 51 · see citing works
Rossen, B., Lind, S. D., & Lok, B. C. L. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04380-2_52 · Cited by 49 · see citing works
Empirical methods for evaluating dialog systems
Paek, T. (2001) · doi:10.3115/1118053.1118054 · Cited by 47 · see citing works
An Experimental Constructional Database : The MorTAL Project
Hathout, N., Dal, G., & Namer, F. (2002) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 47 · see citing works
Handling errors and determining confirmation strategies—An object-based approach
McTear, M., O’Neill, I., Hanna, P., & Liu, X. (2005) · Speech Communication · doi:10.1016/j.specom.2004.11.006 · Cited by 44 · see citing works
An ontology for clinical questions about the contents of patient notes
Patrick, J., & Li, M. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.11.008 · Cited by 44 · see citing works
Acquiring meaning for French medical terminology: contribution of morphosemantics
Fiammetta, N., & Zweigenbaum, P. (2004) · Studies in health technology and informatics · doi:10.3233/978-1-60750-949-3-535 · Cited by 43 · see citing works
Self-citationUMLF: a unified medical lexicon for French
Zweigenbaum, P., Baud, R. H., Burgun, A., et al. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.03.010 · Cited by 38 · see citing works
Self-citationTools For Research and Education In Speech Science
Cole, R. A. (1999) · Cited by 30 · see citing works
On the Means for Clarification in Dialogue
Purver, M., Ginzburg, J., & Healey, P. G. T. (2003) · Text, speech and language technology · doi:10.1007/978-94-010-0019-2_11 · Cited by 26 · see citing works
The LIMSI Participation in the QAst Track
Rosset, S., Galibert, O., Adda, G., et al. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85760-0_55 · Cited by 24 · see citing works
Self-citationEmbodied Conversational Virtual Patients
Kenny, P. G., & Parsons, T. D. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-617-6.ch011 · Cited by 21 · see citing works
Handling speech input in the ritel QA dialogue system
van Schooten, B. W., Rosset, S., Galibert, O., Max, A., den Akker, R. O., & Illouz, G. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-55 · Cited by 20 · see citing works
Self-citationIntegrating spoken dialog and question answering: the ritel project
Rosset, S., Galibert, O., Illouz, G., et al. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-526 · Cited by 19 · see citing works
Self-citationLessons learned in modeling virtual pediatric patients.
Hubal, R. C., Deterding, R. R., Frank, G. A., et al. (2003) · PubMed · Cited by 19 · see citing works
Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System
Jin, L., White, M., Jaffe, E., Zimmerman, L., & Danforth, D. R. (2017) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w17-5002 · Cited by 19 · see citing works
Managing Linguistic and Terminological Variation in a Medical Dialogue System
Llanos, L. C., Bouamor, D., Zweigenbaum, P., et al. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4w5j85o9imsc · Cited by 15 · see citing works
Self-citationA case based reasoning model for multilingual language generation in dialogues
Salazar, V. L., Eisman, E. M., Peña, J. L. C., & Zurita, J. M. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.01.085 · Cited by 14 · see citing works
Description of the PatientGenesys Dialogue System
Llanos, L. C., Bouamor, D., Bilinski, É., Ligozat, A., Zweigenbaum, P., & Rosset, S. (2015) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4660 · Cited by 14 · see citing works
Self-citationDesigning Useful Virtual Standardized Patient Encounters
Talbot, T. B., Sagae, K., John, B., & Rizzo, A. (2012) · Cited by 13 · see citing works
Natural Language Understanding Performance & Use Considerations in Virtual Medical Encounters.
Talbot, T. B., Kalisch, N., Christoffersen, K., et al. (2016) · PubMed · Cited by 12 · see citing works
A Deep Learning Methodology for Semantic Utterance Classification in Virtual Human Dialogue Systems
Datta, D., Brashers, V. L., Owen, J., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-47665-0_53 · Cited by 11 · see citing works
Oden, K., Nelson, M., & Lee, M. (2018) · International Journal for Innovation Education and Research · doi:10.31686/ijier.vol6.iss6.1059 · Cited by 11 · see citing works
Automatic classification of doctor-patient questions for a virtual patient record query task
Llanos, L. C., Rosset, S., & Zweigenbaum, P. (2017) · doi:10.18653/v1/w17-2343 · Cited by 8 · see citing works
Self-citationTransfer-Based Learning-to-Rank Assessment of Medical Term Technicality
Bouamor, D., Llanos, L. C., Ligozat, A., et al. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5962moy2xrns · Cited by 7 · see citing works
Language Understanding in Maryland Virtual Patient
Nirenburg, S., Beale, S., McShane, M. J., et al. (2008) · Cited by 6 · see citing works
A Corpus of Word-Aligned Asked and Anticipated Questions in a Virtual Patient Dialogue System
Gokcen, A., Jaffe, E., Erdmann, J., et al. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5cak39rrjyjy · Cited by 6 · see citing works
Hoxha, J., & Weng, C. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.04.003 · Cited by 5 · see citing works
Deep Learning for Spoken and Text Dialog Systems
Çelikyılmaz, A., Deng, L., & Hakkani-Tür, D. Z. (2017) · Cited by 3 · see citing works
How Can I Help You Today ? Guide de la Consultation Médicale et Paramédicale en Anglais
Pastore, F. (2016) · Cited by 1 · see citing works
Conversational Agents for Automated Inpatient and Outpatient Health Counseling.
Bickmore, T. (2015) · AMIA · Cited by 1 · see citing works
Approaches and methodologies for automatic Question-Answering in an open-domain, interactive setup.
GALIBERT, O., & ADDA-DECKER, M. (2009) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · see citing works
de Cock, C., Milne-Ives, M., van Velthoven, M. H., Alturkistani, A., Lam, C., & Meinert, E. (2020) · JMIR Research Protocols · doi:10.2196/16934 · Cited by 62 · see citing works
Borg, A., Georg, C., Jobs, B., Huss, V., Waldenlind, K., Ruiz, M., Edelbring, S., Skantze, G., & Parodis, I. (2025) · Journal of Medical Internet Research · doi:10.2196/63312 · Cited by 51 · see citing works
Das, A., Selek, S., Warner, A. R., Zuo, X., Hu, Y., Kuttichi Keloth, V., Li, J., Zheng, W. J., & Xu, H. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.27 · Cited by 47 · see citing works
Maicher, K. R., Stiff, A., Scholl, M., White, M., Fosler-Lussier, E., Schuler, W., Serai, P., Sunder, V., Forrestal, H., Mendella, L., et al. (2023) · Medical Teacher · doi:10.1080/0142159x.2022.2130216 · Cited by 44 · see citing works
Virtual Standardized Patients for Mental Health Education
Reger, G. M., Norr, A. M., Gramlich, M. A., & Buchman, J. M. (2021) · Current Psychiatry Reports · doi:10.1007/s11920-021-01273-5 · Cited by 38 · see citing works
Towards Visual Dialog for Radiology
Kovaleva, O., Shivade, C., Kashyap, S., Kanjaria, K., Wu, J., Ballah, D., Coy, A., Karargyris, A., Guo, Y., Beymer, D. B., et al. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.6 · Cited by 38 · see citing works
Chatbots in Healthcare Curricula: The Case of a Conversational Virtual Patient
Dolianiti, F., Tsoupouroglou, I., Antoniou, P., Konstantinidis, S., Anastasiades, S., & Bamidis, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60735-7_15 · Cited by 37 · see citing works
Lippitsch, A., Steglich, J., Ludwig, C., Kellner, J., Hempel, L., Stoevesandt, D., & Thews, O. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107964 · Cited by 8 · see citing works
LLM-Powered Virtual Patient Agents for Interactive Clinical Skills Training with Automated Feedback
Voigt, H., Sugamiya, Y., Lawonn, K., Zarrieß, S., & Takanishi, A. (2025) · arXiv · doi:10.48550/arxiv.2508.13943
- Authors
- Campillos-Llanos, L., Thomas, C., Bilinski, É., Zweigenbaum, P., & Rosset, S.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Venue
- Natural Language Engineering
- Publisher
- Cambridge University Press (CUP)
- Published
- 2019-07-15
- Volume
- 26
- Issue
- 2
- Pages
- 183-220
- ISSN
- 1351-3249 (ISSN-L)
- 1469-8110
- OpenAlex
- W2961793453
- Cited by
- 34
- References
- 64
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 5 · 2023: 2 · 2022: 8 · 2021: 5 · 2020: 6 · 2019: 1
- FWCI
- 2.31
- Type
- journal article
- Language
- en
- Open access
- green
- hal.science
- Licence
- Locations
- Natural Language Engineering (publishedVersion)
- HAL (submittedVersion, other-oa)
- DIGITAL.CSIC (Spanish National Research Council (CSIC)) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- virtual patients, natural language dialogue systems, medical history, medical terminology, French language, vocabulary coverage
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
