CoChat

Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis

Open access
Cevasco, Kevin E., Morrison Brown, Rachel E., Woldeselassie, Rediet, Kaplan, Seth · 2024 · Journal of Medical Systems

The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index; 28 of the 46 sources its claims cite were checked for support.

Abstract

Clinicians and patients seeking electronic health applications face challenges in selecting effective solutions due to a high market failure rate. Conversational agent applications ("chatbots") show promise in increasing healthcare user engagement by creating bonds between the applications and users. It is unclear if chatbots improve patient adherence or if past trends to include chatbots in electronic health applications were due to technology hype dynamics and competitive pressure to innovate. We conducted a systematic literature review using Preferred Reporting Items for Systematic reviews and Meta-Analyses methodology on health chatbot randomized control trials. The goal of this review was to identify if user engagement indicators are published in eHealth chatbot studies. A meta-analysis examined patient clinical trial retention of chatbot apps. The results showed no chatbot arm patient retention effect. The small number of studies suggests a need for ongoing eHealth chatbot research, especially given the claims regarding their effectiveness made outside the scientific literatures.

Overview

CoChat Summary

## TL;DR

This systematic review and meta-analysis of 14 randomized chatbot trials in health applications found little published evidence on user-engagement indicators such as working alliance, acceptance, and adherence. Meta-analysis of loss to follow-up showed no statistically significant retention advantage for chatbot arms overall.

## Key findings

  • The review identified 14 articles / 14 RCTs after screening 664 articles from database searches, with 67 full texts reviewed.
  • Across all 14 studies, the meta-analysis found an overall effect size of 0.99 (95% CI 0.95–1.03; p < 0.01), indicating no statistically significant difference in loss to follow-up between chatbot and comparison arms.
  • Heterogeneity was substantial: 64.3% of the variation was explained by between-study heterogeneity, and the homogeneity test was rejected (Q = 42.33, df = 13, p < 0.01).
  • Only one study (Fitzpatrick) showed significantly higher chatbot retention (RR 1.31, 95% CI 1.03–1.67); the largest study (Pot et al.) showed significantly lower retention for the chatbot arm (RR 0.89, 95% CI 0.85–0.92).
  • Few studies reported comparable UEI data: only two studies examined working alliance, and only two studies compared acceptance measures between chatbot and controls.

## Methods

  • Systematic review and meta-analysis of health chatbot randomized controlled trials, following PRISMA methodology; the protocol was not registered.
  • Searches were run in PubMed, ProQuest, and EBSCO CINAHL in March 2022, limited to studies published between January 1, 2016 and February 28, 2022, and restricted to adult (18+) patient-facing health chatbots with a control arm.
  • The main quantitative analysis used random-effects meta-analysis in R (metafor package) with log relative risk for loss-to-follow-up, plus heterogeneity tests and Egger’s regression test for publication bias; moderators included age and participant incentives.

## Limitations

  • The paper states that there were few effect sizes / only 14 studies, so results should be interpreted with caution.
  • It also states there was significant heterogeneity among the 14 effect sizes, limiting certainty.
  • The review did not include unpublished trials and did not examine studies outside multi-arm randomized trials; these are explicit limitations stated in the paper.
  • Evident from the text: many UEI measures were not consistently collected, many outcomes used custom, non-comparable surveys, and the review’s meta-analysis is based on loss-to-follow-up as an indirect proxy for engagement rather than direct engagement measures.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10916-024-02059-x

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Summary of article selection process [39]
Fig. 1 Summary of article selection process [39]CC BY · PubMed Central
Fig. 2 Forest plot comparing chatbot arm loss to follow-up with three types of control arms. Higher study subject trial retention has a higher risk ratio. The risk ratio is shown on a log scale indicated at the bottom of the figure
Fig. 2 Forest plot comparing chatbot arm loss to follow-up with three types of control arms. Higher study subject trial retention has a higher risk ratio. The risk ratio is shown on a log scale indicated at the bottom of the figureCC BY · PubMed Central

Verification

Claims the paper makes

19 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations

58 more claims · 4 with their cited sources checked

Create a free account to see all 143 claims and to verify the 4 references still open.

See all claims — free account
  • Probably not a substantial claim. 49
  • It may state two separate claims as one. 47

Create a free account to see these claims.

Citations

Cites

Cited by

See all 35

Metadata

Authors
Cevasco, K. E., Morrison Brown, R. E., Woldeselassie, R., & Kaplan, S.
Affiliations
George Mason University (US)
Corresponding author
Cevasco, Kevin E.
Venue
Journal of Medical Systems
Published
2024-04-10
Volume
48
Issue
1
Pages
40-40
ISSN
  • 0148-5598 (ISSN-L)
  • 1573-689X
OpenAlex
W4394686921
Cited by
35
References
71
Citations by year
2026: 19 · 2025: 16
FWCI
3.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Medical Systems (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
chatbots, patient engagement, e-health applications, clinical trial retention, patient adherence, technology hype, health technology adoption
MeSH terms
Communication, Humans, Patient Participation, Telemedicine, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.