Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis
Open accessThe DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index; 28 of the 46 sources its claims cite were checked for support.
Abstract
Clinicians and patients seeking electronic health applications face challenges in selecting effective solutions due to a high market failure rate. Conversational agent applications ("chatbots") show promise in increasing healthcare user engagement by creating bonds between the applications and users. It is unclear if chatbots improve patient adherence or if past trends to include chatbots in electronic health applications were due to technology hype dynamics and competitive pressure to innovate. We conducted a systematic literature review using Preferred Reporting Items for Systematic reviews and Meta-Analyses methodology on health chatbot randomized control trials. The goal of this review was to identify if user engagement indicators are published in eHealth chatbot studies. A meta-analysis examined patient clinical trial retention of chatbot apps. The results showed no chatbot arm patient retention effect. The small number of studies suggests a need for ongoing eHealth chatbot research, especially given the claims regarding their effectiveness made outside the scientific literatures.
Overview
CoChat Summary
## TL;DR
This systematic review and meta-analysis of 14 randomized chatbot trials in health applications found little published evidence on user-engagement indicators such as working alliance, acceptance, and adherence. Meta-analysis of loss to follow-up showed no statistically significant retention advantage for chatbot arms overall.
## Key findings
- The review identified 14 articles / 14 RCTs after screening 664 articles from database searches, with 67 full texts reviewed.
- Across all 14 studies, the meta-analysis found an overall effect size of 0.99 (95% CI 0.95–1.03; p < 0.01), indicating no statistically significant difference in loss to follow-up between chatbot and comparison arms.
- Heterogeneity was substantial: 64.3% of the variation was explained by between-study heterogeneity, and the homogeneity test was rejected (Q = 42.33, df = 13, p < 0.01).
- Only one study (Fitzpatrick) showed significantly higher chatbot retention (RR 1.31, 95% CI 1.03–1.67); the largest study (Pot et al.) showed significantly lower retention for the chatbot arm (RR 0.89, 95% CI 0.85–0.92).
- Few studies reported comparable UEI data: only two studies examined working alliance, and only two studies compared acceptance measures between chatbot and controls.
## Methods
- Systematic review and meta-analysis of health chatbot randomized controlled trials, following PRISMA methodology; the protocol was not registered.
- Searches were run in PubMed, ProQuest, and EBSCO CINAHL in March 2022, limited to studies published between January 1, 2016 and February 28, 2022, and restricted to adult (18+) patient-facing health chatbots with a control arm.
- The main quantitative analysis used random-effects meta-analysis in R (metafor package) with log relative risk for loss-to-follow-up, plus heterogeneity tests and Egger’s regression test for publication bias; moderators included age and participant incentives.
## Limitations
- The paper states that there were few effect sizes / only 14 studies, so results should be interpreted with caution.
- It also states there was significant heterogeneity among the 14 effect sizes, limiting certainty.
- The review did not include unpublished trials and did not examine studies outside multi-arm randomized trials; these are explicit limitations stated in the paper.
- Evident from the text: many UEI measures were not consistently collected, many outcomes used custom, non-comparable surveys, and the review’s meta-analysis is based on loss-to-follow-up as an indirect proxy for engagement rather than direct engagement measures.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 Summary of article selection process [39]](https://figures.cochat.ai/paper-figures/3d0519d0fcc14f8aa27aad207d256886/patient-engagement-with-conversational-agents-in-health-applications-20162022-a-figure-1.jpg)

Verification
Claims the paper makes
19 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior chatbot reviews did not compare user engagement indicators (UEI).”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > eHealth User Engagement Background Improving eHealth application uptake involves developing products in a holistic framework based on participatory development approaches, persuasive design techniques, and business modeling [24]. Randomized controlled trials (RCTs) are the gold standard method for determining causation of treatment to outcome. RCTs treat randomly selected groups of patients with different therapies (i.e. health chatbot apps) to compare treatment effectiveness based on their health outcomes. Patients in RCTs are randomly placed into groups called trial arms. We examined two-arm RCTs because they allowed comparisons of chatbot groups to other treatments or control groups using non-chatbot treatments. Researchers can also design RCTs to collect data on an app’s user engagement. RCTs can collect data on why and when study subjects cease participation in the trial. This is referred to as loss-to-follow up (“LTFU” or “loss” or “attrition”). Loss to follow-up (LTFU) occurs when a clinical trial study subject ceases participating in a trial protocol, and we hypothesized that chatbots could improve trial retention. LTFU differs from adherence, where LTFU is defined as when participants do not return to fill in trial follow-up questionnaires. We expected an increase in chatbot user engagement would appear as a positive effect on LTFU rates because participants rarely leave a trial due to random reasons [25]. Researchers can design trials to reduce LTFU through retention techniques like offering financial incentives to participants that complete all study requirements and remain in the trial for the entire study period. Prior chatbot reviews did not compare user engagement indicators (UEI). Eysenbach’s “law of attrition” observed that in eHealth trials, a substantial proportion of users drop out before completion or stop using the application [26]. Examples of UEIs include working alliance, acceptance, and adherence. Working alliance measures key aspects of user engagement such as (a) agreement on the tasks of therapy, (b) agreement on the goals of therapy, and (c) development of an affective bond [27]. Acceptance loss is a measure of user satisfaction and subject loss due to dissatisfaction, and is referred to as disenchantment/discontinuance rejection in clinical trials [26]. Adherence loss is where subjects stop using the chatbot during the trial and is called non-usage attrition in clinical trials [26].
What the sources say
- Could not verifycites [28] M. Milne-Ives, C. de Cock, E. Lim, M. H. Shehadeh, N. de Pennington, G. Mole, E. Normando, and E. Meinert, The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review, J Med Internet Res 22, e20346 (2020). sourceExperimental — accuracy not yet measured
The Results section compares studies on usability, satisfaction, and other evaluation outcomes, but does not identify or discuss user engagement indicators (UEI). Searches for UEI and for engagement measures such as usage frequency and retention found no passage establishing whether prior chatbot reviews compared those indicators. (LLM, conf=0.70)
- Partly supportedcites [29] M. M. Ng, J. Firth, M. Minen, and J. Torous, User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity, PS 70, 538 (2019). sourceExperimental — accuracy not yet measured
The Results section says UEI ratings could not be compared across the mental health app studies reviewed because their criteria and methods varied. It does not assess prior chatbot reviews or say whether those reviews compared UEI. (LLM, conf=0.82)
- Could not verifycites [30] L. Tudor Car, D. A. Dhinagaran, B. M. Kyaw, T. Kowatsch, S. Joty, Y.-L. Theng, and R. Atun, Conversational Agents in Health Care: Scoping Review and Conceptual Analysis, J Med Internet Res 22, e17158 (2020). sourceExperimental — accuracy not yet measured
The Results section, “Conversational Agent Evaluation” (page unavailable), discusses acceptability, effectiveness, interaction counts, and response rates, but does not say whether prior chatbot reviews compared user engagement indicators. Searches for “user engagement indicators” and comparisons of engagement measures in previous reviews found no passage addressing that claim. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Conversational agents, or ‘chatbots’, offer a potential solution to improve eHealth app uptake and treatment adherence [17–19].”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > Chatbot Patient Alliance Background
What the sources say
- Partly supportedcites [17] A. Abdulrahman and D. Richards, In Search of Embodied Conversational and Explainable Agents for Health Behaviour Change and Adherence, MTI 5, 56 (2021). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says conversational agents are potential interventions for persuading users to adhere to treatment. It discusses mobile-based health interventions but does not say conversational agents improve uptake of eHealth apps. (LLM, conf=0.86)
- Partly supportedcites [18] P. Henson, H. Wisniewski, C. Hollis, M. Keshavan, and J. Torous, Digital Mental Health Apps and the Therapeutic Alliance: Initial Review, BJPsych Open 5, e15 (2019). sourceExperimental — accuracy not yet measured
The Discussion (Interpretation of our findings) says smartphone apps used alongside therapy can increase engagement and adherence, and the Discussion (Limitations) says optimising the digital therapeutic relationship may improve adherence to app use. The latter section mentions chatbots as interactive interventions that attempt to humanise therapy, but does not say chatbots themselves improve app uptake or treatment adherence. Page numbers were not provided in our copy. (LLM, conf=0.85)
- Could not verifycites [19] V. Mai, C. Neef, and A. Richert, “Clicking vs. Writing”—The Impact of a Chatbot’s Interaction Method on the Working Alliance in AI-Based Coaching, Coaching Theor. Prax. (2022). sourceExperimental — accuracy not yet measured
The Introduction describes chatbots used to support mental or emotional health, and §5.1 discusses working alliance in chatbot coaching. Neither passage addresses whether conversational agents improve eHealth app uptake or treatment adherence; searches for those outcomes found no supporting passage in our copy. Page numbers were not available. (LLM, conf=0.75)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“therefore, there is a small evidence base to provide guidance assessing their effectiveness [21, 22].”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > Chatbot Patient Alliance Background
What the sources say
- Partly supportedcites [21] Q. Pham, D. Wiljer, and J. A. Cafazzo, Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods, JMIR Mhealth Uhealth 4, e107 (2016). sourceExperimental — accuracy not yet measured
The Discussion (Principal Findings; page unavailable) says only a fraction of publicly available mHealth apps have been evaluated, supporting a limited evidence base for apps generally. It does not establish that the evidence base for assessing chatbot effectiveness specifically is small. (LLM, conf=0.82)
- Partly supportedcites [22] O. Byambasuren, S. Sanders, E. Beller, and P. Glasziou, Prescribable mHealth Apps Identified from an Overview of Systematic Reviews | Npj Digital Medicine, Npj Digital Medicine 1, 12 (2018). sourceExperimental — accuracy not yet measured
The Principal finding section (page not provided) says that few effectiveness trials of stand-alone health apps had been conducted and that the evidence was of very low quality. This supports a limited evidence base for assessing health apps, but the paper does not establish that finding specifically for chatbots. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Systematic reviews have shown that there are few published chatbot randomized control trials (RCT) in the eHealth market [21]”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > Chatbot Patient Alliance Background
What the sources say
- Could not verifycites [21] Q. Pham, D. Wiljer, and J. A. Cafazzo, Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods, JMIR Mhealth Uhealth 4, e107 (2016). sourceExperimental — accuracy not yet measured
The Methods section describes a review of mHealth app trials registered on ClinicalTrials.gov, not a review of published chatbot trials. The Discussion (Principal Findings) says RCTs were the predominant design among the registered mHealth trials; the Limitations section says the search was not exhaustive and did not cover all published work. A search for chatbot trials and published eHealth RCTs found no passage addressing how many chatbot RCTs had been published. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“This is similar to other findings on chatbot arm clinical interventions that found a lack of evidence that chatbot primary effect is clinically important [37].”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Results > Results of Syntheses
What the sources say
- Supportedcites [37] A. A. Abd-Alrazaq, A. Rababeh, M. Alajlani, B. M. Bewick, and M. Househ, Effectiveness and Safety of Using Chatbots to Improve Mental Health: Systematic Review and Meta-Analysis, J Med Internet Res 22, e16021 (2020). source
The Discussion (Principal Findings) says the difference favoring chatbots on depression severity was statistically significant but not clinically important. The Discussion (Conclusion) likewise cites that finding as a reason definitive conclusions about chatbot effectiveness could not be drawn. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
58 more claims · 4 with their cited sources checked
Create a free account to see all 143 claims and to verify the 4 references still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 47
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“Normalizing adherence results Questionnaire (CSQ-8) [ 61 ] and System Usability Scale (SUS) [ 62 ].”
MethodKirk, J., MacDonald, A., Lavender, P., Dean, J., & Rubin, G. (2017) · Biomedicine Hub · doi:10.1159/000480347 · Cited by 3,887
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1×
“A newer generation artificial intelligence (AI)- based conversational large language model (LLM), such as ChatGPT, have potential to improve health care applications over traditional chatbots’ Natural Language Processing (NLP) [ 23].”
BackgroundAssessment of client/patient satisfaction: development of a general scale
Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90094-6 · Cited by 2,887 · cited 1×
“Normalizing adherence results Questionnaire (CSQ-8) [ 61 ] and System Usability Scale (SUS) [ 62 ].”
MethodFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 4×
“Raw adherence attrition proportions at different points in time during a trial can illustrate attrition curves and allow between trial arm and between study comparisons [26].”
BackgroundInfluential
Cited by
Pira, G. L., Ruini, C., Vescovelli, F., Baños, R., & Ventura, S. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02144-9 · Cited by 29
“For a review on conversational agent and patients’ engagement [59 ].”
Identifying healthcare needs with patient experience reviews using ChatGPT
Li, J., Yang, Y., Chen, R., Zheng, D., Pang, P. C., Lam, C. K., Wong, D., & Wang, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0313442 · Cited by 23
Ho, C. M., Shaowei, G., Mok, P. K.-L., Lam, C. H., Ho, W. Y., Mak, C. H., Qin, J., Wong, A. K. C., & Hui, V. C. C. (2025) · Journal of Medical Internet Research · doi:10.2196/74299 · Cited by 23
“For example, Cevasco et al [11] reported that these advancements exemplify the paradigm shift toward patient-centered care that emphasizes transparency, shared decision-making, and patient autonomy.”
Eiselt, A.-K., Kirkendall, S., Xiong, E., Langner, D., & Goldfarb, M. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1713334 · Cited by 21
Evaluation of Cancer Survivors’ Experience of Using AI-Based Conversational Tools: Qualitative Study
Khairat, S., Mehraby, H., Masoumi, S., Coffel, M., Rockey-Bartlett, C., Huang, A., Wood, W. A., & Basch, E. (2025) · JMIR Cancer · doi:10.2196/77390 · Cited by 16
“High user acceptance can enhance patient education, symptom management, and emotional well-being, ultimately improving health outcomes [22].”
Metadata
- Authors
- Cevasco, K. E., Morrison Brown, R. E., Woldeselassie, R., & Kaplan, S.
- Affiliations
- George Mason University (US)
- Corresponding author
- Cevasco, Kevin E.
- Venue
- Journal of Medical Systems
- Published
- 2024-04-10
- Volume
- 48
- Issue
- 1
- Pages
- 40-40
- ISSN
- 0148-5598 (ISSN-L)
- 1573-689X
- PMC
- PMC11004048
- OpenAlex
- W4394686921
- Cited by
- 35
- References
- 71
- Citations by year
- 2026: 19 · 2025: 16
- FWCI
- 3.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal of Medical Systems (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- chatbots, patient engagement, e-health applications, clinical trial retention, patient adherence, technology hype, health technology adoption
- MeSH terms
- Communication, Humans, Patient Participation, Telemedicine, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis. https://app.cochat.ai/papers/10.1007/s10916-024-02059-x
- The paper
- Cevasco, K. E., Morrison Brown, R. E., Woldeselassie, R., & Kaplan, S. (2024). Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis. Journal of Medical Systems. https://doi.org/10.1007/s10916-024-02059-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.