Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis
Open accessThe DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index; 28 of the 46 sources its claims cite were checked for support.
Abstract
Clinicians and patients seeking electronic health applications face challenges in selecting effective solutions due to a high market failure rate. Conversational agent applications ("chatbots") show promise in increasing healthcare user engagement by creating bonds between the applications and users. It is unclear if chatbots improve patient adherence or if past trends to include chatbots in electronic health applications were due to technology hype dynamics and competitive pressure to innovate. We conducted a systematic literature review using Preferred Reporting Items for Systematic reviews and Meta-Analyses methodology on health chatbot randomized control trials. The goal of this review was to identify if user engagement indicators are published in eHealth chatbot studies. A meta-analysis examined patient clinical trial retention of chatbot apps. The results showed no chatbot arm patient retention effect. The small number of studies suggests a need for ongoing eHealth chatbot research, especially given the claims regarding their effectiveness made outside the scientific literatures.
Overview
CoChat Summary
## TL;DR
This systematic review and meta-analysis of 14 randomized chatbot trials in health applications found little published evidence on user-engagement indicators such as working alliance, acceptance, and adherence. Meta-analysis of loss to follow-up showed no statistically significant retention advantage for chatbot arms overall.
## Key findings
- The review identified 14 articles / 14 RCTs after screening 664 articles from database searches, with 67 full texts reviewed.
- Across all 14 studies, the meta-analysis found an overall effect size of 0.99 (95% CI 0.95–1.03; p < 0.01), indicating no statistically significant difference in loss to follow-up between chatbot and comparison arms.
- Heterogeneity was substantial: 64.3% of the variation was explained by between-study heterogeneity, and the homogeneity test was rejected (Q = 42.33, df = 13, p < 0.01).
- Only one study (Fitzpatrick) showed significantly higher chatbot retention (RR 1.31, 95% CI 1.03–1.67); the largest study (Pot et al.) showed significantly lower retention for the chatbot arm (RR 0.89, 95% CI 0.85–0.92).
- Few studies reported comparable UEI data: only two studies examined working alliance, and only two studies compared acceptance measures between chatbot and controls.
## Methods
- Systematic review and meta-analysis of health chatbot randomized controlled trials, following PRISMA methodology; the protocol was not registered.
- Searches were run in PubMed, ProQuest, and EBSCO CINAHL in March 2022, limited to studies published between January 1, 2016 and February 28, 2022, and restricted to adult (18+) patient-facing health chatbots with a control arm.
- The main quantitative analysis used random-effects meta-analysis in R (metafor package) with log relative risk for loss-to-follow-up, plus heterogeneity tests and Egger’s regression test for publication bias; moderators included age and participant incentives.
## Limitations
- The paper states that there were few effect sizes / only 14 studies, so results should be interpreted with caution.
- It also states there was significant heterogeneity among the 14 effect sizes, limiting certainty.
- The review did not include unpublished trials and did not examine studies outside multi-arm randomized trials; these are explicit limitations stated in the paper.
- Evident from the text: many UEI measures were not consistently collected, many outcomes used custom, non-comparable surveys, and the review’s meta-analysis is based on loss-to-follow-up as an indirect proxy for engagement rather than direct engagement measures.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 Summary of article selection process [39]](https://figures.cochat.ai/paper-figures/3d0519d0fcc14f8aa27aad207d256886/patient-engagement-with-conversational-agents-in-health-applications-20162022-a-figure-1.jpg)

Verification
Claims the paper makes
19 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior chatbot reviews did not compare user engagement indicators (UEI).”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > eHealth User Engagement Background Improving eHealth application uptake involves developing products in a holistic framework based on participatory development approaches, persuasive design techniques, and business modeling [24]. Randomized controlled trials (RCTs) are the gold standard method for determining causation of treatment to outcome. RCTs treat randomly selected groups of patients with different therapies (i.e. health chatbot apps) to compare treatment effectiveness based on their health outcomes. Patients in RCTs are randomly placed into groups called trial arms. We examined two-arm RCTs because they allowed comparisons of chatbot groups to other treatments or control groups using non-chatbot treatments. Researchers can also design RCTs to collect data on an app’s user engagement. RCTs can collect data on why and when study subjects cease participation in the trial. This is referred to as loss-to-follow up (“LTFU” or “loss” or “attrition”). Loss to follow-up (LTFU) occurs when a clinical trial study subject ceases participating in a trial protocol, and we hypothesized that chatbots could improve trial retention. LTFU differs from adherence, where LTFU is defined as when participants do not return to fill in trial follow-up questionnaires. We expected an increase in chatbot user engagement would appear as a positive effect on LTFU rates because participants rarely leave a trial due to random reasons [25]. Researchers can design trials to reduce LTFU through retention techniques like offering financial incentives to participants that complete all study requirements and remain in the trial for the entire study period. Prior chatbot reviews did not compare user engagement indicators (UEI). Eysenbach’s “law of attrition” observed that in eHealth trials, a substantial proportion of users drop out before completion or stop using the application [26]. Examples of UEIs include working alliance, acceptance, and adherence. Working alliance measures key aspects of user engagement such as (a) agreement on the tasks of therapy, (b) agreement on the goals of therapy, and (c) development of an affective bond [27]. Acceptance loss is a measure of user satisfaction and subject loss due to dissatisfaction, and is referred to as disenchantment/discontinuance rejection in clinical trials [26]. Adherence loss is where subjects stop using the chatbot during the trial and is called non-usage attrition in clinical trials [26].
What the sources say
- Could not verifycites [28] M. Milne-Ives, C. de Cock, E. Lim, M. H. Shehadeh, N. de Pennington, G. Mole, E. Normando, and E. Meinert, The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review, J Med Internet Res 22, e20346 (2020). sourceExperimental — accuracy not yet measured
The Results section compares studies on usability, satisfaction, and other evaluation outcomes, but does not identify or discuss user engagement indicators (UEI). Searches for UEI and for engagement measures such as usage frequency and retention found no passage establishing whether prior chatbot reviews compared those indicators. (LLM, conf=0.70)
- Partly supportedcites [29] M. M. Ng, J. Firth, M. Minen, and J. Torous, User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity, PS 70, 538 (2019). sourceExperimental — accuracy not yet measured
The Results section says UEI ratings could not be compared across the mental health app studies reviewed because their criteria and methods varied. It does not assess prior chatbot reviews or say whether those reviews compared UEI. (LLM, conf=0.82)
- Could not verifycites [30] L. Tudor Car, D. A. Dhinagaran, B. M. Kyaw, T. Kowatsch, S. Joty, Y.-L. Theng, and R. Atun, Conversational Agents in Health Care: Scoping Review and Conceptual Analysis, J Med Internet Res 22, e17158 (2020). sourceExperimental — accuracy not yet measured
The Results section, “Conversational Agent Evaluation” (page unavailable), discusses acceptability, effectiveness, interaction counts, and response rates, but does not say whether prior chatbot reviews compared user engagement indicators. Searches for “user engagement indicators” and comparisons of engagement measures in previous reviews found no passage addressing that claim. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Conversational agents, or ‘chatbots’, offer a potential solution to improve eHealth app uptake and treatment adherence [17–19].”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > Chatbot Patient Alliance Background
What the sources say
- Partly supportedcites [17] A. Abdulrahman and D. Richards, In Search of Embodied Conversational and Explainable Agents for Health Behaviour Change and Adherence, MTI 5, 56 (2021). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says conversational agents are potential interventions for persuading users to adhere to treatment. It discusses mobile-based health interventions but does not say conversational agents improve uptake of eHealth apps. (LLM, conf=0.86)
- Partly supportedcites [18] P. Henson, H. Wisniewski, C. Hollis, M. Keshavan, and J. Torous, Digital Mental Health Apps and the Therapeutic Alliance: Initial Review, BJPsych Open 5, e15 (2019). sourceExperimental — accuracy not yet measured
The Discussion (Interpretation of our findings) says smartphone apps used alongside therapy can increase engagement and adherence, and the Discussion (Limitations) says optimising the digital therapeutic relationship may improve adherence to app use. The latter section mentions chatbots as interactive interventions that attempt to humanise therapy, but does not say chatbots themselves improve app uptake or treatment adherence. Page numbers were not provided in our copy. (LLM, conf=0.85)
- Could not verifycites [19] V. Mai, C. Neef, and A. Richert, “Clicking vs. Writing”—The Impact of a Chatbot’s Interaction Method on the Working Alliance in AI-Based Coaching, Coaching Theor. Prax. (2022). sourceExperimental — accuracy not yet measured
The Introduction describes chatbots used to support mental or emotional health, and §5.1 discusses working alliance in chatbot coaching. Neither passage addresses whether conversational agents improve eHealth app uptake or treatment adherence; searches for those outcomes found no supporting passage in our copy. Page numbers were not available. (LLM, conf=0.75)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“therefore, there is a small evidence base to provide guidance assessing their effectiveness [21, 22].”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > Chatbot Patient Alliance Background
What the sources say
- Partly supportedcites [21] Q. Pham, D. Wiljer, and J. A. Cafazzo, Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods, JMIR Mhealth Uhealth 4, e107 (2016). sourceExperimental — accuracy not yet measured
The Discussion (Principal Findings; page unavailable) says only a fraction of publicly available mHealth apps have been evaluated, supporting a limited evidence base for apps generally. It does not establish that the evidence base for assessing chatbot effectiveness specifically is small. (LLM, conf=0.82)
- Partly supportedcites [22] O. Byambasuren, S. Sanders, E. Beller, and P. Glasziou, Prescribable mHealth Apps Identified from an Overview of Systematic Reviews | Npj Digital Medicine, Npj Digital Medicine 1, 12 (2018). sourceExperimental — accuracy not yet measured
The Principal finding section (page not provided) says that few effectiveness trials of stand-alone health apps had been conducted and that the evidence was of very low quality. This supports a limited evidence base for assessing health apps, but the paper does not establish that finding specifically for chatbots. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Systematic reviews have shown that there are few published chatbot randomized control trials (RCT) in the eHealth market [21]”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Introduction > Chatbot Patient Alliance Background
What the sources say
- Could not verifycites [21] Q. Pham, D. Wiljer, and J. A. Cafazzo, Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods, JMIR Mhealth Uhealth 4, e107 (2016). sourceExperimental — accuracy not yet measured
The Methods section describes a review of mHealth app trials registered on ClinicalTrials.gov, not a review of published chatbot trials. The Discussion (Principal Findings) says RCTs were the predominant design among the registered mHealth trials; the Limitations section says the search was not exhaustive and did not cover all published work. A search for chatbot trials and published eHealth RCTs found no passage addressing how many chatbot RCTs had been published. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“This is similar to other findings on chatbot arm clinical interventions that found a lack of evidence that chatbot primary effect is clinically important [37].”
§ Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis > Results > Results of Syntheses
What the sources say
- Supportedcites [37] A. A. Abd-Alrazaq, A. Rababeh, M. Alajlani, B. M. Bewick, and M. Househ, Effectiveness and Safety of Using Chatbots to Improve Mental Health: Systematic Review and Meta-Analysis, J Med Internet Res 22, e16021 (2020). source
The Discussion (Principal Findings) says the difference favoring chatbots on depression severity was statistically significant but not clinically important. The Discussion (Conclusion) likewise cites that finding as a reason definitive conclusions about chatbot effectiveness could not be drawn. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
58 more claims · 4 with their cited sources checked
Create a free account to see all 143 claims and to verify the 4 references still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 47
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“Normalizing adherence results Questionnaire (CSQ-8) [ 61 ] and System Usability Scale (SUS) [ 62 ].”
MethodKirk, J., MacDonald, A., Lavender, P., Dean, J., & Rubin, G. (2017) · Biomedicine Hub · doi:10.1159/000480347 · Cited by 3,887
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1×
“A newer generation artificial intelligence (AI)- based conversational large language model (LLM), such as ChatGPT, have potential to improve health care applications over traditional chatbots’ Natural Language Processing (NLP) [ 23].”
BackgroundAssessment of client/patient satisfaction: development of a general scale
Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90094-6 · Cited by 2,887 · cited 1×
“Normalizing adherence results Questionnaire (CSQ-8) [ 61 ] and System Usability Scale (SUS) [ 62 ].”
MethodFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Eysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 4×
“Raw adherence attrition proportions at different points in time during a trial can illustrate attrition curves and allow between trial arm and between study comparisons [26].”
BackgroundInfluentialMedication Adherence: WHO Cares?
Brown, M. T., & Bussell, J. K. (2011) · Mayo Clinic Proceedings · doi:10.4065/mcp.2010.0575 · Cited by 1,928 · cited 1×
“Adherence research in the pharmaceutical market has been underfunded, but companies are financially successful through insurance reimbursement despite 50% of patients not adhering to their prescribed medication regimens [68].”
BackgroundMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126
Chatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090 · cited 1×
“The prevalence of published chatbot studies increased dramatically starting in 2016 following chatbot cloud platform software releases by large technology companies [40 ].”
BackgroundPerspective: Linking Design Thinking with Innovation Outcomes through Cognitive Bias Reduction
Liedtka, J. (2015) · Journal of Product Innovation Management · doi:10.1111/jpim.12163 · Cited by 951
Loss to follow-up in cohort studies: how much is too much?
Kristman, V., Manno, M., & Côté, P. (2004) · European Journal of Epidemiology · doi:10.1023/b:ejep.0000036568.02655.f8 · Cited by 740 · cited 2×
“We expected an increase in chatbot user engagement would appear as a positive effect on LTFU rates because participants rarely leave a trial due to random reasons [25 ].”
Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018) · doi:10.2196/preprints.9782 · Cited by 715
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · cited 2×
“Additionally, two chatbot systematic reviews were identified with lists of trials referenced [ 28 , 38 ].”
BackgroundHow much loss to follow-up is acceptable in long-term randomised trials and prospective studies?
Fewtrell, M. S., Kennedy, K., Singhal, A., Martin, R. M., Ness, A., Hadders-Algra, M., Koletzko, B., & Lucas, A. (2008) · Archives of Disease in Childhood · doi:10.1136/adc.2007.127316 · Cited by 601 · cited 1×
“Chatbot study subject attrition was an important indica - tor because it may be a source of bias and systematic error leading to an incorrect estimation of association between exposure and outcome [36].”
BackgroundLinardon, J., & Fuller-Tyszkiewicz, M. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000459 · Cited by 573 · cited 1×
“Other factors have influenced LTFU in mobile app trials showing lower attrition rates associated with acceptance-based interventions, participant monetary compensation, younger age, and employing engagement reminders [ 34 , 35 ].”
BackgroundConversational Agents in Health Care: Scoping Review and Conceptual Analysis
Tudor Car, L., Dhinagaran, D. A., Kyaw, B. M., Kowatsch, T., Joty, S., Theng, Y.-L., & Atun, R. (2020) · Journal of Medical Internet Research · doi:10.2196/17158 · Cited by 553
Munder, T., Wilmers, F., Leonhart, R., Linster, H. W., & Barth, J. (2010) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.658 · Cited by 538 · cited 2×
“Hauser-Ulrich and Zwerenz studies captured working alliance measures using validated instruments (Working Alliance Inventory-Short Revised (WAI-SR) and Helping Alliance Questionnaire (HAQ)) [ 27 , 60].”
BackgroundMethodLi, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 505
Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020) · Journal of Medical Internet Research · doi:10.2196/16021 · Cited by 503 · cited 4×
“This is similar to other findings on chatbot arm clinical inter - ventions that found a lack of evidence that chatbot primary effect is clinically important [ 37 ].”
BackgroundResultInfluentialWhat is the clinical value of mHealth for patients?
Rowland, S. P., Fitzgerald, J. E., Holme, T., Powell, J., & McGregor, A. (2020) · npj Digital Medicine · doi:10.1038/s41746-019-0206-x · Cited by 454
The hype cycle model: A review and future directions
Dedehayir, O., & Steinert, M. (2016) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.04.005 · Cited by 425
The Revised Helping Alliance Questionnaire (HAq-II) : Psychometric Properties.
(1996) · PubMed Central · Cited by 394
User engagement in mental health apps: A review of measurement, reporting, and validity
Ng, M. M., Firth, J., Minen, M., & Torous, J. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201800519 · Cited by 293 · cited 1×
“Software monitoring tools and system logs enabled measuring trial subject UEIs for eHealth prod - ucts more thoroughly than RCTs for medication trials, but eHealth UEIs were not collected consistently [ 29].”
MethodPrescribable mHealth apps identified from an overview of systematic reviews
Byambasuren, O., Sanders, S., Beller, E., & Glasziou, P. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0021-9 · Cited by 291
Szinay, D., Jones, A., Chadborn, T., Brown, J., & Naughton, F. (2020) · Journal of Medical Internet Research · doi:10.2196/17572 · Cited by 289 · cited 1×
“A prior study found health and wellbeing smartphone apps were more frequently down-loaded by women than men [44] and all mental health articles had higher female than male participation.”
BackgroundGreer, S., Ramo, D., Chang, Y.-J., Fu, M., Moskowitz, J., & Haritatos, J. (2019) · JMIR mHealth and uHealth · doi:10.2196/15018 · Cited by 265
Darcy, A., Daniels, J., Salinger, D., Wicks, P., & Robinson, A. (2021) · JMIR Formative Research · doi:10.2196/27868 · Cited by 260 · cited 2×
“Per Eyesnbach’s Law of Attrition, dropout rates are high in eHealth trials, and UEI metrics as is supported by an observational study funded by the application’s owner (Woebot) [ 64].”
MethodResultMarwaha, J. S., Landman, A. B., Brat, G. A., Dunn, T., & Gordon, W. J. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00557-1 · Cited by 241 · cited 1×
“Health systems commonly require eHealth software to be approved through a governance board process before deployment into patient care settings [65].”
BackgroundCascella, M., Semeraro, F., Montomoli, J., Bellini, V., Piazza, O., & Bignami, E. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02045-3 · Cited by 219 · cited 2×
“(e.g. documentation management, summarizing literature) [ 67].”
MethodHauser-Ulrich, S., Künzli, H., Meier-Peterhans, D., & Kowatsch, T. (2020) · JMIR mHealth and uHealth · doi:10.2196/15806 · Cited by 210
Dahmen, J., Kayaalp, M. E., Ollivier, M., Pareek, A., Hirschmann, M. T., Karlsson, J., & Winkler, P. W. (2023) · Knee Surgery, Sports Traumatology, Arthroscopy · doi:10.1007/s00167-023-07355-6 · Cited by 193 · cited 1×
“The applications may lack context to provide the LLMs effective information about specific cases [71 ].”
BackgroundSibanda, E. L., Weller, I. V. D., Hakim, J. G., & Cowan, F. M. (2013) · AIDS · doi:10.1097/qad.0000000000000027 · Cited by 180 · cited 1×
“However, there was an opportunity to examine chatbot effects on RCT participant retention [25 , 31].”
Quantifying App Store Dynamics: Longitudinal Tracking of Mental Health Apps
Larsen, M. E., Nicholas, J., & Christensen, H. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.6020 · Cited by 177
Scaling evidence-based treatments through digital mental health.
Schueller, S. M., & Torous, J. (2020) · American Psychologist · doi:10.1037/amp0000654 · Cited by 175
Anan, T., Kajiki, S., Oka, H., Fujii, T., Kawamata, K., Mori, K., & Matsudaira, K. (2021) · JMIR mhealth and uhealth · doi:10.2196/27535 · Cited by 170
Lim, S. M., Shiau, C. W. C., Cheng, L. J., & Lau, Y. (2022) · Behavior Therapy · doi:10.1016/j.beth.2021.09.007 · Cited by 163 · cited 3×
“Additionally, two chatbot systematic reviews were identified with lists of trials referenced [ 28 , 38 ].”
BackgroundBeyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods
Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5720 · Cited by 162 · cited 1×
“Systematic reviews have shown that there are few published chatbot randomized control trials (RCT) in the eHealth market [ 21 ]; therefore, there is a small evi - dence base to provide guidance assessing their effectiveness [21 , 22 ].”
BackgroundA Study of Today's A.I. through Chatbots and Rediscovery of Machine Intelligence
Khanna, A., Pandey, B., Vashishta, K., Kalia, K., Pradeepkumar, B., & Das, T. (2015) · International Journal of u- and e-Service, Science and Technology · doi:10.14257/ijunesst.2015.8.7.28 · Cited by 162
Gong, E., Baptista, S., Russell, A., Scuffham, P., Riddell, M., Speight, J., Bird, D., Williams, E., Lotfaliany, M., & Oldenburg, B. (2020) · Journal of Medical Internet Research · doi:10.2196/20322 · Cited by 159
Digital mental health apps and the therapeutic alliance: initial review
Henson, P., Wisniewski, H., Hollis, C., Keshavan, M., & Torous, J. (2019) · BJPsych Open · doi:10.1192/bjo.2018.86 · Cited by 152
Baumel, A., & Kane, J. M. (2018) · Journal of Medical Internet Research · doi:10.2196/11491 · Cited by 131 · cited 2×
“Self-guided eHealth interventions that create clinician-patient therapeutic alliance and therapeutic persuasiveness are positively correlated with real-world usage of mobile apps and web-based programs [20].”
Burton, C., Szentagotai Tatar, A., McKinstry, B., Matheson, C., Matu, S., Moldovan, R., Macnab, M., Farrow, E., David, D., Pagliari, C., et al. (2016) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15609793 · Cited by 125
Good and bad market research: A critical review of Net Promoter Score
Fisher, N. I., & Kordupleski, R. E. (2019) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.2417 · Cited by 120 · cited 1×
“The Hauser-Ulrich and So studies measured acceptance through net promoter score (NPS) [ 63].”
BackgroundZwerenz, R., Becker, J., Knickenberg, R. J., Siepmann, M., Hagen, K., & Beutel, M. E. (2017) · Psychotherapy and Psychosomatics · doi:10.1159/000481177 · Cited by 108 · cited 1×
“The authors stated that the higher discharge HAQ scores suggest a desire of participants to interact with a chatbot in the same way they do with humans, and sup - ports the theory of media equation, which claims that people tend to treat computers or other media as if they were real people [58 ].”
BackgroundBerger, T., Urech, A., Krieger, T., Stolz, T., Schulz, A., Vincent, A., Moser, C. T., Moritz, S., & Meyer, B. (2017) · Psychological Medicine · doi:10.1017/s0033291716002270 · Cited by 98
Lutz, J., Offidani, E., Taraboanta, L., Lakhan, S. E., & Campellone, T. R. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.823977 · Cited by 89 · cited 1×
“The lack of blinding may be common where subjects are part of ongoing mental health therapy or when pharmacotherapy blinding designs may not translate to eHealth [45].”
BackgroundTop-Funded Digital Health Companies And Their Impact On High-Burden, High-Cost Conditions
Safavi, K. C., Mathews, S. C., Bates, D. W., Dorsey, E. R., & Cohen, A. B. (2019) · Health Affairs · doi:10.1377/hlthaff.2018.05081 · Cited by 79
The regulatory gap in digital health and alternative pathways to bridge it
Iqbal, J. D., & Biller‐Andorno, N. (2022) · Health Policy and Technology · doi:10.1016/j.hlpt.2022.100663 · Cited by 77
Jack, B. W., Bickmore, T., Yinusa-Nyahkoon, L. S., Reichert, M., Julce, C., Sidduri, N., Martin-Howard, J., Zhang, Z., Woodhams, E. J., Fernandez, J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30189-8 · Cited by 76
TensioBot: a chatbot assistant for self-managed in-house blood pressure checking
Echeazarra, L., Pereira, J., & Saracho, R. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01730-x · Cited by 66
Pot, M., Paulussen, T. G., Ruiter, R. A., Eekhout, I., de Melker, H. E., Spoelstra, M. E., & van Keulen, H. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7449 · Cited by 62
Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis
Day, S., Shah, V., Kaganoff, S., Powelson, S., & Mathews, S. C. (2022) · Journal of Medical Internet Research · doi:10.2196/37677 · Cited by 57 · cited 1×
“One challenge is that most eHealth applications are marketed without documented effectiveness, and few have regulatory approval [2–4].”
BackgroundSo, R., Furukawa, T. A., Matsushita, S., Baba, T., Matsuzaki, T., 古野, 悟., Okada, H., & Higuchi, S. (2020) · Journal of Gambling Studies · doi:10.1007/s10899-020-09935-4 · Cited by 55
Mantovani, A., Leopaldi, C., Nighswander, C. M., & Di Bidino, R. (2023) · Frontiers in Medical Technology · doi:10.3389/fmedt.2023.1101476 · Cited by 48 · cited 1×
“Payors require RCTs evidence to approve reimbursement which is necessary for the health system’s return-on-investment business case [66].”
BackgroundCan the outside‐view approach improve planning decisions in software development projects?
Shmueli, O., Pliskin, N., & Fink, L. (2015) · Information Systems Journal · doi:10.1111/isj.12091 · Cited by 44
Does hype create irreversibilities? Affective circulation and market investments in digital health
Geiger, S., & Gross, N. (2017) · Marketing Theory · doi:10.1177/1470593117692024 · Cited by 42 · cited 1×
“One challenge is that most eHealth applications are marketed without documented effectiveness, and few have regulatory approval [2–4].”
BackgroundSandoval, L. R., Buckey, J. C., Ainslie, R. C., Tombari, M. L., Stone, W. S., & Hegel, M. T. (2016) · Behavior Therapy · doi:10.1016/j.beth.2016.04.001 · Cited by 34
Mai, V., Neef, C., & Richert, A. (2022) · Coaching | Theorie & Praxis · doi:10.1365/s40896-021-00063-3 · Cited by 33
Crutzen, R., Viechtbauer, W., Kotz, D., & Spigt, M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.019 · Cited by 32 · cited 2×
“We expected chatbot intervention arms to show similar LTFU rates as non-chatbot control arms; because in peer-reviewed health research LTFU was normally similar across trial arms [32].”
ResultHingle, M. D., Patrick, H., Sacher, P. M., & Castro, C. M. (2018) · Health Education & Behavior · doi:10.1177/1090198118788600 · Cited by 32
The Fine Line Between Persuasion and Digital Addiction
Cemiloglu, D., Naiseh, M., Catania, M., Oinas-Kukkonen, H., & Ali, R. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-79460-6_23 · Cited by 28 · cited 1×
“Technology companies have been successful in video game uptake, to the extent that internet gaming disorder is stressed in the Statistical Manual of Mental Disorders (DSM-5) [ 69].”
Artificial intelligence in pediatric behavioral health
van Schalkwyk, G. (2023) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-023-00586-y · Cited by 25
From Evidence-based to Market-based mHealth: Itinerary of a Mobile (for) Development Project
Dahdah, M. A. (2019) · Science Technology & Human Values · doi:10.1177/0162243918824657 · Cited by 13
Abdulrahman, A., & Richards, D. (2021) · Multimodal Technologies and Interaction · doi:10.3390/mti5090056 · Cited by 11 · cited 2×
“Conversational agents, or ‘chatbots’, offer a potential solu - tion to improve eHealth app uptake and treatment adher - ence [17–19].”
BackgroundGrowth management of eHealth service start-ups
Saarela, M., Simunaniemi, A., Muhos, M., & Leviäkangas, P. (2017) · Journal of Advances in Management Research · doi:10.1108/jamr-06-2017-0073 · Cited by 8
Koch, A. R., Craemer, K. A., Garland, C. E., Fox, W. B., Jones, C. T., Qualls, A. C., Sterr, J. C., & Geller, S. E. (2024) · Journal of Women's Health · doi:10.1089/jwh.2023.0307 · Cited by 7 · cited 2×
“Other factors have influenced LTFU in mobile app trials showing lower attrition rates associated with acceptance-based interventions, participant monetary compensation, younger age, and employing engagement reminders [ 34 , 35 ].”
Background
Cited by
Pira, G. L., Ruini, C., Vescovelli, F., Baños, R., & Ventura, S. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02144-9 · Cited by 29
“For a review on conversational agent and patients’ engagement [59 ].”
Identifying healthcare needs with patient experience reviews using ChatGPT
Li, J., Yang, Y., Chen, R., Zheng, D., Pang, P. C., Lam, C. K., Wong, D., & Wang, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0313442 · Cited by 23
Ho, C. M., Shaowei, G., Mok, P. K.-L., Lam, C. H., Ho, W. Y., Mak, C. H., Qin, J., Wong, A. K. C., & Hui, V. C. C. (2025) · Journal of Medical Internet Research · doi:10.2196/74299 · Cited by 23
“For example, Cevasco et al [11] reported that these advancements exemplify the paradigm shift toward patient-centered care that emphasizes transparency, shared decision-making, and patient autonomy.”
Eiselt, A.-K., Kirkendall, S., Xiong, E., Langner, D., & Goldfarb, M. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1713334 · Cited by 21
Evaluation of Cancer Survivors’ Experience of Using AI-Based Conversational Tools: Qualitative Study
Khairat, S., Mehraby, H., Masoumi, S., Coffel, M., Rockey-Bartlett, C., Huang, A., Wood, W. A., & Basch, E. (2025) · JMIR Cancer · doi:10.2196/77390 · Cited by 16
“High user acceptance can enhance patient education, symptom management, and emotional well-being, ultimately improving health outcomes [22].”
Metadata
- Authors
- Cevasco, K. E., Morrison Brown, R. E., Woldeselassie, R., & Kaplan, S.
- Affiliations
- George Mason University (US)
- Corresponding author
- Cevasco, Kevin E.
- Venue
- Journal of Medical Systems
- Published
- 2024-04-10
- Volume
- 48
- Issue
- 1
- Pages
- 40-40
- ISSN
- 0148-5598 (ISSN-L)
- 1573-689X
- PMC
- PMC11004048
- OpenAlex
- W4394686921
- Cited by
- 35
- References
- 71
- Citations by year
- 2026: 19 · 2025: 16
- FWCI
- 3.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal of Medical Systems (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- chatbots, patient engagement, e-health applications, clinical trial retention, patient adherence, technology hype, health technology adoption
- MeSH terms
- Communication, Humans, Patient Participation, Telemedicine, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.