Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review
Open accessPublished version of a preprint
Abstract
Background: Conversational agents (also known as chatbots) have evolved in recent decades to become multimodal, multifunctional platforms with potential to automate a diverse range of health-related activities supporting the general public, patients, and physicians. Multiple studies have reported the development of these agents, and recent systematic reviews have described the scope of use of conversational agents in health care. However, there is scarce research on the effectiveness of these systems; thus, their viability and applicability are unclear. Objective: The objective of this systematic review is to assess the effectiveness of conversational agents in health care and to identify limitations, adverse events, and areas for future investigation of these agents. Methods: The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols will be used to structure this protocol. The focus of the systematic review is guided by a population, intervention, comparator, and outcome framework. A systematic search of the PubMed (Medline), EMBASE, CINAHL, and Web of Science databases will be conducted. Two authors will independently screen the titles and abstracts of the identified references and select studies according to the eligibility criteria. Any discrepancies will then be discussed and resolved. Two reviewers will independently extract and validate data from the included studies into a standardized form and conduct quality appraisal. Results: As of January 2020, we have begun a preliminary literature search and piloting of the study selection process. Conclusions: This systematic review aims to clarify the effectiveness, limitations, and future applications of conversational agents in health care. Our findings may be useful to inform the future development of conversational agents and promote the personalization of patient care. International Registered Report Identifier (IRRID): PRR1-10.2196/16934
AI snapshot
Written from the paper’s full text.
TL;DR
This is a protocol for a systematic review assessing the effectiveness of conversational agents in health care, including their limitations, adverse events, and future applications. As of January 2020, the authors had begun a preliminary literature search and piloting of the study selection process, but no review findings are reported.
Key findings
- No effectiveness results are reported; as of January 2020, the authors had begun a preliminary literature search and piloting of the study selection process.
- The planned review aims to assess whether intended health-related outcomes of conversational agents are fulfilled, whether effectiveness varies by population or function, what capabilities and resource impacts these agents have, and what limitations or gaps remain.
- Planned outcomes include clinical, behavioral, and psychosocial parameters; health literacy; shared decision making; practical improvement in health care provision; and user-based outcomes such as acceptability, usability, engagement, and satisfaction.
Methods
- Protocol for a systematic review using the Population, Intervention, Comparator, Outcomes (PICO) framework and PRISMA-P guidelines; planned searches include PubMed (Medline), Embase, CINAHL, ACM Digital, and Web of Science.
- Inclusion criteria cover interventional studies (e.g., randomized controlled trials, non-randomized studies) and observational studies (e.g., cross-sectional surveys, cohort studies, qualitative studies), published in English, with no year restriction; the conversational agent must interact with a human user and adapt responses using natural language processing.
- Two reviewers will independently screen, extract, and validate data; quality appraisal will use the Cochrane Risk of Bias tool, ROBINS-I, and the NIH-NHLBI quality assessment tool; synthesis will be descriptive/narrative because meta-analysis is anticipated to be unlikely.
Limitations
- The authors state that a meta-analysis is unlikely owing to the anticipated variety of study aims, methods, and reported outcomes.
- The authors state they do not anticipate any issues with implementation of the proposed protocol.
- Evident from the text: this is a protocol only, so no included studies, effectiveness estimates, adverse events, or review conclusions are yet available; the initial search may also lead to refinement or exclusion of certain study types.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 35 cited sources checked · 0 supported · 0 flagged
- Premisecitation
Investigated uses of conversational agents include triage, diagnostics, counseling, health promotion, and training of health care professionals.
“The investigated uses for these systems include triage, diagnostics, counseling, health promotion, and training of health care professionals [1,4,11-16].”
Quote in the paperNot a substantial claim0 of 8 sources checked- cites [1] source
Not checked in the free run.
- cites [4] source
Not checked in the free run.
- cites [11] Oliven, A, 2011, Stud Health Technol Inform
Not checked in the free run.
- cites [12] source
Not checked in the free run.
- cites [13] Ghosh, S, 2018, Stud Health Technol Inform
Not checked in the free run.
- cites [14] source
Not checked in the free run.
- cites [15] source
Not checked in the free run.
- cites [16] source
Not checked in the free run.
- Premisecitation
Few reviews have comprehensively examined the overall scope and progress in the field of conversational agents in health care.
“Despite a wealth of literature on conversational agents and their application to health care, the majority of reviews on the topic focus on a specific therapy area or function, whereas few reviews have comprehensively examined the overall scope and progress in the field [20-23].”
Quote in the paperSubstantial claim0 of 4 sources checked - Premisecitation
The majority of reviews on conversational agents in health care focus on a specific therapy area or function.
“Despite a wealth of literature on conversational agents and their application to health care, the majority of reviews on the topic focus on a specific therapy area or function, whereas few reviews have comprehensively examined the overall scope and progress in the field [20-23].”
Quote in the paperSubstantial claim0 of 4 sources checked - Premisecitation
The widespread availability of digital platforms through which conversational agents operate enables populations with limited health provision or health literacy to access these services.
“The widespread availability of the digital platforms through which these conversational agents operate enables populations with limited health provision or health literacy to access these services”
Quote in the paperNot a substantial claim0 of 2 sources checked - Premisecitation
Conversational agents are helping to provide patient-centered care by increasing patients' involvement in their health care and decision making.
“Finally, these agents are helping to provide patient-centered care by increasing the patients’ involvement in their health care and decision making [2,17,18].”
Quote in the paperSubstantial claim0 of 3 sources checked
- Premisecitation
Laranjo et al. conducted a systematic review of conversational agents in 2018 investigating characteristics, applications, and evaluation measures.
“Laranjo et al [24] conducted a systematic review of conversational agents in 2018, in which they investigated the characteristics, applications, and evaluation measures of conversational agents; however, this was limited to agents with unconstrained natural language input and systems that had been tested with human participants.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [24] source
Not checked in the free run.
- Interpretationreasoning
There is scarce research on the effectiveness of conversational agents in health care.
“However, there is scarce research on the effectiveness of these systems; thus, their viability and applicability are unclear.”
Quote in the paperNot a substantial claim - Premisecitation
Montenegro et al.'s survey did not address implications for users.
“Although they described a general taxonomy detailing the functions and architecture, the implications for the users were not addressed.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [25] source
Not checked in the free run.
- Classified as probably not a substantial claim. 7
- May combine two claims. 3
Classified as probably not a substantial claim.
PremisecitationWithin health care, conversational agents have been designed to automate specialized tasks to support health care professionals, patients, or at-risk populations.
“Within the health care field, conversational agents have been designed to automate specialized tasks to support health care professionals, patients, or at-risk populations [2,10-12].”
Quote in the paperSubstantial claim0 of 4 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationConversational agents mimic human interaction using natural language processing to analyze user inputs and respond appropriately using human language via auditory or textual methods.
“Conversational agents mimic human interaction using natural language processing to analyze user inputs and respond appropriately using human language via auditory or textual methods [5].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [5] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationCurrent conversational agents are available via multiple digital platforms (telephones, mobile phones, tablets, computers) and in many virtual formats (chatbots, embodied conversational agents, three-dimensional avatars).
“Current agents are available via multiple digital platforms, including telephones, mobile phones, tablets, and computers, and in many virtual formats such as chatbots, embodied conversational agents, and three-dimensional avatars [2,7,8].”
Quote in the paperNot a substantial claim0 of 3 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPersonalization features have been integrated into conversational agents to improve user satisfaction, user engagement, and dialogue quality.
“Personalization features have also been integrated into conversational agents to improve user satisfaction, user engagement, and dialogue quality [19].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [19] source
Not checked in the free run.
May combine two claims.
InterpretationreasoningThe viability and applicability of conversational agents in health care are unclear.
“However, there is scarce research on the effectiveness of these systems; thus, their viability and applicability are unclear.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMontenegro et al. surveyed the literature on conversational agents applied to health care with a focus on patterns, goals, and interactions.
“Similarly, in 2019, Montenegro et al [25] surveyed the literature related to conversational agents applied to health care with a focus on their patterns, goals, and interactions.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [25] source
Not checked in the free run.
May combine two claims.
PremisecitationLaranjo et al.'s review was limited to agents with unconstrained natural language input and systems tested with human participants.
“Laranjo et al [24] conducted a systematic review of conversational agents in 2018, in which they investigated the characteristics, applications, and evaluation measures of conversational agents; however, this was limited to agents with unconstrained natural language input and systems that had been tested with human participants.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [24] source
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone givenConversational agents have potential to alleviate current health care resource burdens by automating functions that previously required face-to-face interaction.
“Furthermore, these conversational agents have potential to alleviate current health care resource burdens by automating functions that previously required face-to-face interaction; thus, it is important to identify whether this is an observed outcome of the use of these technologies.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMontenegro et al. described a general taxonomy detailing the functions and architecture of conversational agents.
“Although they described a general taxonomy detailing the functions and architecture, the implications for the users were not addressed.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [25] source
Not checked in the free run.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The full methods for this review have been published in detail in a systematic review protocol.
“The full methods for this review have been published in detail in a systematic review protocol [24].”
From The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review (2020) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Data extraction is described as both independent and single-reviewer
MinorThe abstract says two reviewers will independently extract data, whereas the Data Extraction section assigns extraction to one reviewer and validation to a second.
“Two reviewers will independently extract and validate data from the included studies into a standardized form and conduct quality appraisal.”
“One reviewer will review the full text of all the papers included in the final selection and extract data that will be validated by a second reviewer.”
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817 · see citing works
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,577 · see citing works
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · see citing works
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · see citing works
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,765 · see citing works
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · see citing works
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388 · cited 1× · see citing works
“Laranjo et al [24] conducted a systematic review of conversational agents in 2018, in which they investigated the characteristics, applications, and evaluation measures of conversational agents; however, this was limited to agents with unconstrained natural language input and systems that had been tested with human pa…”
BackgroundChatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048 · see citing works
Embodied conversational agents in clinical psychology: a scoping review
Provoost, S., Lau, H. M., Ruwaard, J., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6553 · Cited by 433 · see citing works
The Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332 · cited 1× · see citing works
“Personalization features have also been integrated into conversational agents to improve user satisfaction, user engagement, and dialogue quality [19].”
BackgroundSurvey of conversational agents in health
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.054 · Cited by 322 · see citing works
A randomized controlled trial of an automated exercise coach for older adults
Bickmore, T. W., Silliman, R. A., Nelson, K., Cheng, D. M., Winter, M., Henault, L., & Paasche‐Orlow, M. K. (2013) · Journal of the American Geriatrics Society · doi:10.1111/jgs.12449 · Cited by 305 · cited 1× · see citing works
“The investigated uses for these systems include triage, diagnostics, counseling, health promotion, and training of health care professionals [1,4,11-16].”
BackgroundBickmore, T. W., Pfeifer, L. M., Byron, D., Forsythe, S., Henault, L. E., Jack, B. W., Silliman, R., & Paasche-Orlow, M. K. (2010) · Journal of Health Communication · doi:10.1080/10810730.2010.499991 · Cited by 249 · see citing works
Tanaka, H., Negoro, H., Iwasaka, H., & Nakamura, S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0182151 · Cited by 145 · see citing works
Philip, P., Micoulaud-Franchi, J.-A., Sagaspe, P., Sevin, E. D., Olive, J., Bioulac, S., & Sauteraud, A. (2017) · Scientific Reports · doi:10.1038/srep42656 · Cited by 128 · see citing works
Healthcare ex Machina: Are conversational agents ready for prime time in oncology?
Bibault, J.-E., Chaix, B., Nectoux, P., Pienkowski, A., Guillemasé, A., & Brouard, B. (2019) · Clinical and Translational Radiation Oncology · doi:10.1016/j.ctro.2019.04.002 · Cited by 84 · see citing works
Designing a spoken dialogue interface to an intelligent cognitive assistant for people with dementia
Wolters, M. K., Kelly, F., & Kilgour, J. (2016) · Health Informatics Journal · doi:10.1177/1460458215593329 · Cited by 79 · cited 2× · see citing works
“The investigated uses for these systems include triage, diagnostics, counseling, health promotion, and training of health care professionals [1,4,11-16].”
BackgroundAssistive Conversational Agent for Health Coaching: A Validation Study
Fadhil, A., Wang, Y., & Reiterer, H. (2019) · Methods of Information in Medicine · doi:10.1055/s-0039-1688757 · Cited by 77 · see citing works
Owens, O. L., Felder, T., Tavakoli, A. S., Revels, A. A., Friedman, D. B., Hughes-Halbert, C., & Hébert, J. R. (2019) · American Journal of Health Promotion · doi:10.1177/0890117118786866 · Cited by 65 · see citing works
Quro: Facilitating User Symptom Check Using a Personalised Chatbot-Oriented Dialogue System
Ghosh, S., Bhatia, S., & Bhatia, A. (2018) · Studies in health technology and informatics · doi:10.3233/978-1-61499-890-7-51 · Cited by 60 · see citing works
The potential of conversational agents to provide a rapid HIV counseling and testing services
van Heerden, A., Ntinga, X., & Vilakazi, K. (2017) · 2017 International Conference on the Frontiers and Advances in Data Science (FADS) · doi:10.1109/fads.2017.8253198 · Cited by 57 · see citing works
Dialogue Systems and Conversational Agents for Patients with Dementia: The Human–Robot Interaction
Russo, A., D'Onofrio, G., Gangemi, A., Giuliani, F., Mongiovi, M., Ricciardi, F., Greco, F., Cavallo, F., Dario, P., Sancarlo, D., et al. (2019) · Rejuvenation Research · doi:10.1089/rej.2018.2075 · Cited by 47 · see citing works
Oliven, A., Nave, R., Gilad, D., et al. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-806-9-233 · Cited by 40 · see citing works
Abdullah, A. S. M., Gaehde, S. A., & Bickmore, T. (2018) · Journal of Epidemiology and Global Health · doi:10.2991/j.jegh.2018.08.104 · Cited by 39 · see citing works
Campillos-Llanos, L., Thomas, C., Bilinski, É., Zweigenbaum, P., & Rosset, S. (2020) · Natural Language Engineering · doi:10.1017/s1351324919000329 · Cited by 34 · see citing works
Medical Shared Decision Making with a Virtual Agent
Zhang, Z., & Bickmore, T. (2018) · Proceedings of the 18th International Conference on Intelligent Virtual Agents · doi:10.1145/3267851.3267883 · Cited by 28 · see citing works
Evaluation of a virtual dialogue method for breast cancer patient education
Harless, W. G., Zier, M. A., Harless, M. G., Duncan, R. C., Braun, M. A., Willey, S., Isaacs, C., & Warren, R. D. (2009) · Patient Education and Counseling · doi:10.1016/j.pec.2009.02.006 · Cited by 24 · see citing works
Zhaopeng, X., Fei, Y., Mustafa, Q. Y. A., Ting, G., Jennifer, W., & Lixin, S. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190661 · Cited by 9 · see citing works
VALIDITY AND USABILITY OF AN AUTOMATED FALL RISK ASSESSMENT TOOL FOR OLDER ADULTS
Sun, R., Aldunate, R. G., Ratnam, R., Jain, S., Morrow, D. G., & Sosnoff, J. J. (2018) · Innovation in Aging · doi:10.1093/geroni/igy023.1338 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Oliven, A, 2011, Stud Health Technol Inform, 169, 233
- Ghosh, S, 2018, Stud Health Technol Inform, 252, 51
Cited by
Sorted by citations
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · see citing works
Self-citationKowatsch, T., Lohse, K.-M., Erb, V., Schittenhelm, L., Galliker, H., Lehner, R., & Huang, E. M. (2021) · Journal of Medical Internet Research · doi:10.2196/23612 · Cited by 39 · see citing works
Mancone, S., Diotaiuti, P., Valente, G., Corrado, S., Bellizzi, F., Vilarino, G. T., & Andrade, A. (2023) · Behavioral Sciences · doi:10.3390/bs13070550 · Cited by 14 · see citing works
- Authors
- de Cock, C., Milne-Ives, M., van Velthoven, M. H., Alturkistani, A., Lam, C., & Meinert, E.
- Affiliations
- Imperial College London (GB)
- University of Oxford (GB)
- Corresponding author
- Meinert, Edward
- ORCID
- Venue
- JMIR Research Protocols
- Publisher
- JMIR Publications Inc.
- Published
- 2020-03-09
- Volume
- 9
- Issue
- 3
- Pages
- e16934
- ISSN
- 1929-0748 (ISSN-L)
- PMC
- PMC7091022
- OpenAlex
- W2987917655
- Cited by
- 62
- References
- 29
- Citations by year
- 2026: 7 · 2025: 13 · 2024: 9 · 2023: 14 · 2022: 7 · 2021: 9 · 2020: 3
- FWCI
- 0.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Research Protocols (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Spiral (Imperial College London) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion)
- Funders
- National Institute for Health and Care Research (AI_AWARD01852)
- EIT Health
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- chatbots, virtual assistants, health care, effectiveness, adverse events, health technology adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
