CoChat
Sign Up
Cited by 504Digital Mental Health InterventionsSocial Robot Interaction and HRIMental Health Treatment and Access

Effectiveness and safety of using chatbots to improve mental health: systematic review and meta-analysis

Open access
Abd-Alrazaq, Alaa Ali, Rababeh, Asma, Alajlani, Mohannad, Bewick, Bridgette M, Househ, Mowafa·2020·Journal of Medical Internet Research
DOI 10.2196/16021PMC PMC7385637

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The global shortage of mental health workers has prompted the utilization of technological advancements, such as chatbots, to meet the needs of people with mental health conditions. Chatbots are systems that are able to converse and interact with human users using spoken, written, and visual language. While numerous studies have assessed the effectiveness and safety of using chatbots in mental health, no reviews have pooled the results of those studies. Objective: This study aimed to assess the effectiveness and safety of using chatbots to improve mental health through summarizing and pooling the results of previous studies. Methods: A systematic review was carried out to achieve this objective. The search sources were 7 bibliographic databases (eg, MEDLINE, EMBASE, PsycINFO), the search engine “Google Scholar,” and backward and forward reference list checking of the included studies and relevant reviews. Two reviewers independently selected the studies, extracted data from the included studies, and assessed the risk of bias. Data extracted from studies were synthesized using narrative and statistical methods, as appropriate. Results: Of 1048 citations retrieved, we identified 12 studies examining the effect of using chatbots on 8 outcomes. Weak evidence demonstrated that chatbots were effective in improving depression, distress, stress, and acrophobia. In contrast, according to similar evidence, there was no statistically significant effect of using chatbots on subjective psychological wellbeing. Results were conflicting regarding the effect of chatbots on the severity of anxiety and positive and negative affect. Only two studies assessed the safety of chatbots and concluded that they are safe in mental health, as no adverse events or harms were reported. Conclusions: Chatbots have the potential to improve mental health. However, the evidence in this review was not sufficient to definitely conclude this due to lack of evidence that their effect is clinically important, a lack of studies assessing each outcome, high risk of bias in those studies, and conflicting results for some outcomes. Further studies are required to draw solid conclusions about the effectiveness and safety of chatbots. Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42019141219; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42019141219

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review and meta-analysis found that chatbots may improve some mental health outcomes, but the evidence is weak, often high risk of bias, and not sufficient for definitive conclusions. Only two included studies assessed safety, and both reported no adverse events or harms.

Key findings

  • Of 1048 citations, 12 studies examining 8 outcomes were included; only 4 RCTs were meta-analyzed. Chatbots showed a statistically significant effect on depression severity versus treatment as usual or information (SMD –0.55, 95% CI –0.87 to –0.23; P<.001), but this was not clinically important; heterogeneity was P=.99, I²=0%, and evidence quality was low.
  • For anxiety severity, 2 RCTs found no statistically significant difference between chatbot and information (MD –1.38, 95% CI –5.5 to 2.74; P=.55), with substantial heterogeneity (P=.02; I²=80%) and very low-quality evidence.
  • Narrative synthesis suggested chatbots significantly decreased distress and stress, and one RCT found significantly decreased acrophobia severity at 2- and 4-week follow-ups (P<.001). No statistically significant effect was found for subjective psychological wellbeing, while results were conflicting for positive and negative affect.
  • Safety was assessed in only 2 RCTs; both concluded chatbots were safe in mental health, with no adverse events, harm, distress, or worsening of depressive symptoms reported.

Methods

  • Systematic review reported per PRISMA and registered in PROSPERO (CRD42019141219). Searches covered 7 bibliographic databases plus Google Scholar (first 100 hits) and backward/forward reference checking, from June 8 to June 11, 2019.
  • Eligible studies were RCTs or quasiexperiments of chatbots as standalone software or web-based tools for mental health, excluding robotics, serious games, SMS, telephone systems, and human-operator-generated dialogue. Two reviewers independently selected studies, extracted data, and assessed risk of bias; agreement was very good (Cohen κ=0.85 and 0.89 for selection steps; κ=0.84 for data extraction; κ=0.75 for risk of bias).
  • Data were synthesized narratively and statistically using RevMan 5.3; continuous outcomes used mean difference or standardized mean difference with a random-effects model. Risk of bias was assessed with RoB 2 for RCTs and ROBINS-I for quasiexperiments, and evidence quality with GRADE.

Limitations

  • Stated by the paper: the review was restricted to chatbots in standalone software or web browsers, excluding robotics, serious games, SMS, telephone systems, and human-operator-controlled chatbots; it was restricted to English-language studies; most included studies had high risk of bias and meta-analyzed evidence ranged from very low to low quality.
  • Stated by the paper: meta-analysis used only post-intervention data because studies did not report enough pre-post data, and one-group pre-post trials were not meta-analyzed due to internal validity threats.
  • Evident from the text: only 4 of 12 included studies were meta-analyzed, several outcomes came from only 1–3 studies, sample sizes were often small (range 10–454; median 71.5), and safety evidence came from only 2 RCTs, limiting certainty and generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/16021

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 70 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    There was substantial heterogeneity in the anxiety meta-analysis (P=.02; I2=80%).

    “There was substantial heterogeneity (P=.02; I2=80%).”
    Quote in the paperSubstantial claim
  • Findingown data

    The meta-analysis of 4 RCTs showed a statistically significant difference favoring chatbots over treatment as usual or information on the severity of depression (SMD -0.55, 95% CI -0.87 to -0.23).

    “The meta-analysis showed a statistically significant difference (P<.001) favoring chatbots over treatment as usual or information on the severity of depression (SMD –0.55, 95% CI –0.87 to –0.23; Figure 4).”
    Quote in the paperSubstantial claim
  • Findingown data

    The quality of the evidence for anxiety was very low because it was downgraded by 3 levels due to a high risk of bias and heterogeneity.

    “The quality of the evidence was very low because it was downgraded by 3 levels due to a high risk of bias and heterogeneity (Multimedia Appendix 9).”
    Quote in the paperSubstantial claim
  • Findingown data

    The difference in depression severity was not clinically important, as the total effect (-0.55) was within the boundaries of a minimal clinically important difference (-4.7 to 4.7).

    “However, this difference was not clinically important, as the total effect (–0.55) was within the boundaries of a minimal clinically important difference (–4.7 to 4.7)”
    Quote in the paperSubstantial claim
  • Findingown data

    The heterogeneity of the evidence for depression was not a concern (P=.99; I2=0%).

    “The heterogeneity of the evidence was not a concern (P=.99; I2= 0%).”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 34
  • May combine two claims. 48

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Two additional studies were identified from forward reference list checking, and one study was identified by backward reference list checking.

    “Two additional studies were identified from forward reference list checking, and one study was identified by backward reference list checking.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Interrater agreement between reviewers for quality of evidence assessment was considerable (Cohen κ=0.86).

    “There was considerable interrater agreement between the reviewers (Cohen κ= 0.86) [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Altman, D, 1990, Practical statistics for medical research

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the remaining 83 studies, 9 studies were included after reading the full text.

    “Of the remaining 83 studies, 9 studies were included after reading the full text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Patient Health Questionnaire (PHQ-9) was the most used outcome measure in the included studies (4/12).

    “The Patient Health Questionnaire (PHQ-9) was the most used outcome measure in the included studies (4/12).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Interrater agreement between reviewers for risk of bias assessment was good (Cohen κ=0.75).

    “Interrater agreement between the reviewers was good (Cohen κ=0.75) [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Altman, D, 1990, Practical statistics for medical research

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The most common outcome assessed by the included studies was severity of depression (6/12).

    “The most common outcome assessed by the included studies was severity of depression (6/12).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The search retrieved 1048 citations.

    “The search retrieved 1048 citations (Figure 1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Interrater agreement between reviewers for data extraction was very good (Cohen κ=0.84).

    “Interrater agreement between the reviewers was very good (Cohen κ=0.84) [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Altman, D, 1990, Practical statistics for medical research

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Half of the included studies (6/12) were RCTs, and the other half were quasiexperimental.

    “half of the studies (6/12) were RCTs, while the other half were quasiexperimental.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Overall, 12 studies were included in the narrative synthesis, but only 4 of those studies were meta-analyzed.

    “Overall, 12 studies were included in the narrative synthesis, but only 4 of those studies were meta-analyzed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Interrater agreement between reviewers for study selection was 0.85 in the first step and 0.89 in the second step, indicating very good agreement.

    “Cohen κ [21] was calculated to assess interrater agreement between reviewers, which was 0.85 and 0.89 in the first and second step of the selection process, respectively, indicating a very good level of agreement [22].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [21] Higgins, J, 2008, J. Higgins, Green S. editors. Cochrane Handbook for Systematic Reviews of Interventions. ed source

      Not checked in the free run.

    • cites [22] Altman, D, 1990, Practical statistics for medical research

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    After removing 413 duplicates, 635 unique titles and abstracts remained.

    “After removing 413 duplicates, 635 unique titles and abstracts remained.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The effect size of chatbots on acrophobia in RCT [38] was 2.0, substantially higher than the total effect size of therapist-assisted exposure treatment on phobias (1.1) reported by a meta-analysis [46].

    “The effect size of chatbots on acrophobia in this RCT [38] was substantially higher than the total effect size of therapist-assisted exposure treatment on phobias reported by a meta-analysis (2.0 versus 1.1) [46].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [38] source

      Not checked in the free run.

    • cites [46] source

      Not checked in the free run.

    • Rests on an assumption: Effect sizes from different studies and interventions are comparable enough to compare directly.

  • May combine two claims.

    Findingcitation

    The chatbot in [29] was more advanced than the one in [28]; it depended on artificial intelligence and machine learning to generate responses to users.

    “Specifically, the chatbot in the first study [29] was more advanced than the one in the second study [28]; it depended on artificial intelligence and machine learning to generate responses to users, and this makes it more humanlike and lets users feel more socially connected [5].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [29] source

      Not checked in the free run.

    • cites [28] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Only two studies assessed the safety of chatbots and concluded that they are safe in mental health, as no adverse events or harms were reported.

    “Only two studies assessed the safety of chatbots and concluded that they are safe in mental health, as no adverse events or harms were reported.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The first study assessed the effect of the chatbot on positive and negative affect together, while the other study examined positive affect and negative affect separately.

    “With regards to the second difference, while the first study assessed the effect of the chatbot on positive and negative affect together [29], the other study examined the effect of the chatbot on positive affect and negative affect separately [28].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [29] source

      Not checked in the free run.

    • cites [28] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    The chatbot in the quasiexperiment [32] contained a virtual representation (embodiment), which enables chatbots to communicate with users verbally and nonverbally.

    “Second, in contrast to the 2 RCTs, the chatbot in the quasiexperiment [32] contained a virtual representation (ie, embodiment), which enables chatbots to communicate with users verbally and nonverbally (through body movements and facial expressions).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Low-quality evidence from 4 RCTs showed a statistically significant difference favoring chatbots over treatment as usual or information on the severity of depression, but this difference was not clinically important.

    “For the first outcome (depression), low-quality evidence from 4 RCTs showed a statistically significant difference favoring chatbots over treatment as usual or information on the severity of depression, but this difference was not clinically important.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The overall effect on depression in this review was -0.55 and was comparable to other reviews.

    “The overall effect on depression in this review (–0.55) was comparable to other reviews.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Effect sizes from different studies and interventions are comparable enough to compare directly.

  • May combine two claims.

    Findingcitation

    The second subjective psychological wellbeing study demonstrated a slight improvement after using chatbots, but this improvement was not statistically significant (P=.06).

    “The second study demonstrated a slight improvement in subjective psychological wellbeing after using chatbots, but this improvement was not statistically significant (P=.06) [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Demirci, H, 2018, User experience over time with conversational agents case study of woebot on supporting subjective well-being

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    In Ly et al [33], chatbots significantly reduced stress compared with waiting list in all participants (P=.03) and in adherent participants (P=.01).

    “Ly and colleagues [33] found a statistically significant difference favoring chatbots over the waiting list on stress when they analyzed data from all participants (P=.03) and from those who only adhered to the intervention (P=.01).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Some studies assessed outcomes before and after interventions, while other studies examined them only after interventions.

    “Further, while some studies assessed outcomes before and after interventions, other studies examined them only after interventions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both RCTs measuring safety concluded that chatbots are safe for use in mental health, as no adverse events or harm were reported when chatbots were used to treat users with depression and acrophobia.

    “Of the 2 RCTs measuring the safety of chatbots, both concluded that chatbots are safe for use in mental health, as no adverse events or harm were reported when chatbots were used to treat users with depression and acrophobia.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The evidence for each outcome came from only a few studies that also had small sample sizes.

    “Third, the evidence for each outcome came from only a few studies that also had small sample sizes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    This study is the first review of the literature that assessed the effectiveness and safety of chatbots in mental health.

    “This study is the first review of the literature that assessed the effectiveness and safety of chatbots in mental health.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: No previous reviews assessed the effectiveness of chatbots in mental health.

  • May combine two claims.

    Findingnone given

    Two-thirds of the chatbots in this review used predefined rules and decision trees to generate their responses, while the remaining chatbots used artificial intelligence.

    “Two-thirds of the chatbots in this review used predefined rules and decision trees to generate their responses, while the remaining chatbots used artificial intelligence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The risk of bias in the classification of the interventions was rated high in 3 quasiexperimental studies and serious in one study.

    “the risk of bias in the classification of the interventions was rated as high in those 3 studies. Further, the risk of bias in this domain was judged as serious in one study”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Most of the RCTs (5/6) used an appropriate random allocation sequence, concealed that allocation sequence, and had comparable groups, and were rated as having a low risk of bias in the randomization process.

    “Most of the RCTs (5/6) used an appropriate random allocation sequence, concealed that allocation sequence, and had comparable groups. These studies were rated as having a low risk of bias in the randomization process (Figure 2).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The protocol for this systematic review is registered at PROSPERO (CRD42019141219).

    “The protocol for this systematic review is registered at PROSPERO (number CRD42019141219).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The risk of bias due to missing outcome data was judged low in 3 quasiexperimental studies and moderate in the remaining 3 due to availability of less than 95% of participants' data.

    “The risk of bias due to missing outcome data was judged as low in 3 studies while it was rated as moderate in the remaining 3 studies due to availability of less than 95% of the participants’ data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Risk of bias in RCTs was assessed using the RoB 2 tool, and risk of bias in quasi-experiments was assessed using the ROBINS-I tool.

    “Risk of bias in RCTs was assessed using the Risk of Bias 2 (RoB 2) tool [23], and risk of bias in quasi-experiments was examined using the Risk Of Bias In Non-randomized Studies – of Interventions (ROBINS-I) tool [24].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [23] Higgins, JP, 2016, Cochrane Database of Systematic Reviews

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    According to the World Health Organization, mental health services reach 15% of those in need in developing countries and 45% in developed countries.

    “mental health services reach 15% and 45% of those in need in developing and developed countries, respectively [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The overall risk of bias was rated critical in 1 quasiexperimental study, moderate in 3, and serious in 2.

    “While the overall risk of bias was rated as critical in 1 study, it was judged as moderate and serious in 3 and 2 studies, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Adoption of new technology relies on first ascertaining the levels of safety and effectiveness.

    “Adoption of new technology, especially those heavily related to artificial intelligence and machine learning, relies on first ascertaining the levels of safety and effectiveness [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Schunemann, H, 2008, Higgins J, Green S. editors

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Two reviewers independently selected the studies, extracted data, and assessed risk of bias.

    “Two reviewers independently selected the studies, extracted data from the included studies, and assessed the risk of bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    There is a global shortage of mental health human resources, poor funding, and mental health illiteracy.

    “There is a shortage of mental health human resources, poor funding, and mental health illiteracy globally [5,6].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [5] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Low-income and middle-income countries have 0.1 psychiatrists per 1,000,000 people, compared to 90 psychiatrists per 1,000,000 people in high-income countries.

    “there are 0.1 psychiatrists per 1,000,000 people [7], compared to 90 psychiatrists per 1,000,000 people in high-income countries [8].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The overall quality of evidence was assessed using GRADE.

    “the overall quality of that evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Schunemann, H, 2008, Higgins J, Green S. editors

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The review is reported in line with the PRISMA statement.

    “This review is reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (Multimedia Appendix 1) [20].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The domain of missing outcome data was judged low risk in 4 RCTs and high risk in 2 RCTs due to high attrition rate, lack of analysis methods used to correct for bias, and differences between intervention groups in proportions of missing outcome data.

    “The domain of missing outcome data was judged as having a low risk of bias in 4 studies while it was rated as having a high risk of bias in the remaining 2 studies due to a high attrition rate, lack of analysis methods used to correct for bias, and presence of differences between intervention groups in the proportions of missing outcome data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The evidence in this review was not sufficient to definitely conclude that chatbots improve mental health due to lack of evidence that their effect is clinically important, a lack of studies assessing each outcome, high risk of bias in those studies, and conflicting results for some outcomes.

    “However, the evidence in this review was not sufficient to definitely conclude this due to lack of evidence that their effect is clinically important, a lack of studies assessing each outcome, high risk of bias in those studies, and conflicting results for some outcomes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Artificial intelligence and machine learning make chatbots more humanlike and let users feel more socially connected.

    “Specifically, the chatbot in the first study [29] was more advanced than the one in the second study [28]; it depended on artificial intelligence and machine learning to generate responses to users, and this makes it more humanlike and lets users feel more socially connected [5].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The conflicting results between this review and another review on smartphone mental health interventions may result from either differences in interventions or the number of meta-analyzed studies (2 versus 9).

    “These conflicting results may result from either differences in interventions (chatbots versus different mobile interventions) in both reviews or the number of meta-analyzed studies (2 versus 9).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Premisecitation

    Embodiment makes conversations with chatbots more empathetic and facilitates effective rapport with users.

    “It is purported that embodiment makes conversations with chatbots more empathetic and facilitates effective rapport with users [19,42,43].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [19] source

      Not checked in the free run.

    • cites [42] source

      Not checked in the free run.

    • cites [43] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    Evidence from the 2 studies on depression was synthesized narratively, so the review could not identify whether the decrease in depression was clinically important.

    “As evidence from the 2 studies was synthesized narratively, we could not identify whether this decrease in depression was clinically important.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The two studies on positive and negative affect were homogeneous in design, sample characteristics, comparator characteristics, and outcome measures but differed in chatbot type and data analysis, which may have led to contradictory findings.

    “Although the 2 studies were very homogenous in terms of study design, sample characteristics, comparator characteristics, and outcome measures, they were different in the type of chatbots and data analysis, and these differences may have led to contradictory findings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    A meta-analysis was not carried out for stress because one study [37] did not report data required for the analysis.

    “A meta-analysis was not carried out for this outcome as 1 study [37] did not report data required for the analysis.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] HuangJLiQXueYChengTXuSJiaJTeenchat: a chatterbot system for sensing and releasing adolescents' stresseditors2015International Conference on Health Information ScienceMay 28-30, 2015Melbourne, Australia133145

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Three previous reviews about chatbots applied these restrictions.

    “For this reason, 3 previous reviews about chatbots applied these restrictions [5,13,19].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [5] source

      Not checked in the free run.

    • cites [13] source

      Not checked in the free run.

    • cites [19] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The review cannot comment on the effectiveness of chatbots that involve human-generated content or those that use alternative modes of delivery.

    “Accordingly, this review cannot comment on the effectiveness of chatbots that involve human-generated content or those that use alternative modes of delivery.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    As this review focused on the effectiveness and safety of chatbots, the review excluded many studies that assessed the usability and acceptance of chatbots in mental health.

    “As this review focused on the effectiveness and safety of chatbots, we excluded many studies that assessed the usability and acceptance of chatbots in mental health.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    When possible, findings of the included studies were meta-analyzed; thereby, the review was able to increase the power of the studies and improve the estimates of the likely size of effect of chatbots on a variety of mental health outcomes.

    “When possible, findings of the included studies were meta-analyzed; thereby, we were able to increase the power of the studies and improve the estimates of the likely size of effect of chatbots on a variety of mental health outcomes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    This review was developed, executed, and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

    “This review was developed, executed, and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement [20].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    Such analysis was not carried out in this review as one-group trials are very vulnerable to several threats of internal validity.

    “However, such analysis was not carried out in this review as such trials are very vulnerable to several threats of internal validity, such as maturation threat, instrumentation threat, regression threat, and history threat [41,48].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [41] Bhattacherjee, A, 2012, Social science research: Principles, methods, and practices

      Not checked in the free run.

    • cites [48] Shadish, W, 2002, Experimental and quasi-experimental designs for generalized causal inference

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The search was not restricted to a certain type of chatbots, comparators, outcomes, year of publication, nor country of publication, and this makes the review more comprehensive.

    “The search was not restricted to a certain type of chatbots, comparators, outcomes, year of publication, nor country of publication, and this makes the review more comprehensive.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The review excluded studies that contained chatbots controlled by human operators.

    “Further, we excluded studies that contained chatbots controlled by human operators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    In contrast to rule-based chatbots, artificial intelligence chatbots can generate responses to complicated queries and enable users to control the conversation.

    “In contrast to rule-based chatbots, artificial intelligence chatbots can generate responses to complicated queries and enable users to control the conversation [13].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The effectiveness and safety outcomes of chatbots may be different when considering long-term, relative to short-term, findings.

    “The effectiveness and safety outcomes of chatbots may be different when considering long-term, relative to short-term, findings; it is essential to assess long-term outcomes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Although this review found that chatbots may improve depression, distress, stress, and acrophobia, definitive conclusions regarding those results could not be drawn due to high risk of bias, low quality of evidence, lack of studies assessing each outcome, small sample size, and contradictions in results.

    “Although this review found that chatbots may improve depression, distress, stress, and acrophobia, definitive conclusions regarding those results could not be drawn due to the high risk of bias in the included studies, low quality of evidence, lack of studies assessing each outcome, small sample size in the included studies, and contradictions in results of some included studies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Artificial intelligence chatbots can exhibit more empathetic behaviors and humanlike filler language than rule-based chatbots.

    “Artificial intelligence chatbots can exhibit more empathetic behaviors and humanlike filler language than rule-based chatbots [19].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Chatbots may be equivalent to, if not better than, therapist-delivered exposure treatment for phobias.

    “This indicates that chatbots may be equivalent to, if not better, exposure treatment delivered by a therapist in treating phobias.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Effect sizes from different studies and interventions are comparable enough to compare directly.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The intervention of interest in this review was restricted to chatbots that work within standalone software or via a web browser (but not robotics, serious games, SMS, nor telephones).

    “The intervention of interest in this review was restricted to chatbots that work within standalone software or via a web browser (but not robotics, serious games, SMS, nor telephones).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In this review, it was possible to meta-analyze pre-intervention and post-intervention data from one-group trials.

    “In this review, it was possible to meta-analyze pre-intervention and post-intervention data from one-group trials (ie, did not include comparison groups).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    This may make artificial intelligence chatbots more effective in building rapport with users, thereby improving their mental health.

    “This may make artificial intelligence chatbots more effective in building rapport with users, thereby improving their mental health [42].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] source

      Not checked in the free run.

    • Rests on an assumption: Building rapport with users improves mental health.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    To reduce selection bias, two reviewers independently selected studies, extracted data, and assessed the risk of bias in the included studies and quality of the evidence.

    “To reduce selection bias, two reviewers independently selected studies, extracted data, and assessed the risk of bias in the included studies and quality of the evidence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Although the included studies showed that chatbots may be safe and improve depression, distress, stress, and acrophobia, definitive conclusions regarding the effectiveness and safety of chatbots could not be drawn in this review for several reasons.

    “Although the included studies showed that chatbots may be safe and improve depression, distress, stress, and acrophobia, definitive conclusions regarding the effectiveness and safety of chatbots could not be drawn in this review for several reasons.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    It could be argued that artificial intelligence chatbots are more prone to errors than rule-based chatbots, but these errors can be minimized and diminished by extensive training and greater use.

    “It could be argued that artificial intelligence chatbots are more prone to errors than rule-based chatbots, but these errors can be minimized and diminished by extensive training and greater use [49].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] López-Cózar, R, 2011, Conversational Agents and Natural Language Interaction: Techniques and Effective Practices source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Owing to practical constraints, the search was restricted to English studies.

    “Owing to practical constraints, the search was restricted to English studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The risk of bias was high in most included studies, and the quality of the meta-analyzed evidence ranged from very low to low.

    “Second, the risk of bias was high in most included studies, and the quality of the meta-analyzed evidence ranged from very low to low.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Ideally, the difference between pre-intervention and post-intervention data for each group should be used in a meta-analysis.

    “Ideally, the difference between pre-intervention and post-intervention data for each group should be used in a meta-analysis [47].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47] Littell, J, 2008, Systematic reviews and meta-analysis source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In this review, the most popular databases in health and information technology were used to run the most sensitive search possible.

    “In this review, the most popular databases in health and information technology were used to run the most sensitive search possible.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    People in developing countries may be more in need of chatbots than those in developed countries because developing countries have a greater shortage of mental health professionals (0.1 per 1,000,000 vs 9 per 100,000).

    “People in developing countries may be more in need of chatbots than those in developed countries given that developing countries have a greater shortage of mental health professionals than developed countries (0.1 per 1,000,000 people vs 9 per 100,000 people) [7,8].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

    • Rests on an assumption: A shortage of mental health professionals in developing countries means people there are more in need of chatbots.
71 of 71

  • The paper relies on this without support.

    There is a need to critically evaluate and statistically combine findings to inform policy and practice.

    “There is a need to critically evaluate and statistically combine findings to inform policy and practice.”

  • The paper relies on this without support.

    No previous reviews assessed the effectiveness of chatbots in mental health.

    “Given that no reviews assessed the effectiveness of chatbots in mental health, the results were compared with other reviews regarding similar interventions (ie, internet-based psychotherapeutic interventions).”

  • The paper relies on this without support.

    Effect sizes from different studies and interventions are comparable enough to compare directly.

    “The overall effect on depression in this review (–0.55) was comparable to other reviews.”
    “The effect size of chatbots on acrophobia in this RCT [38] was substantially higher than the total effect size of therapist-assisted exposure treatment on phobias reported by a meta-analysis (2.0 versus 1.1) [46].”

  • The paper relies on this without support.

    A shortage of mental health professionals in developing countries means people there are more in need of chatbots.

    “People in developing countries may be more in need of chatbots than those in developed countries given that developing countries have a greater shortage of mental health professionals than developed countries (0.1 per 1,000,000 people vs 9 per 100,000 people) [7,8].”

  • The paper relies on this without support.

    Building rapport with users improves mental health.

    “This may make artificial intelligence chatbots more effective in building rapport with users, thereby improving their mental health [42].”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Overall, our mixed results regarding feasibility, acceptability, and usability are partially aligned with the findings of the existing systematic reviews that reported heterogeneity in these secondary outcome measures and results across studies.

    “Overall, our mixed results regarding feasibility, acceptability, and usability are partially aligned with the findings of the existing systematic reviews that reported heterogeneity in these secondary outcome measures and results across studies [1,2,7,9-11].”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [9]

    The Discussion (Practical and Research Implications; page unavailable) reports heterogeneity in outcome measures and study designs, supporting the general point about variation across studies. But the review says it excluded many studies assessing chatbot usability and acceptance; it does not report heterogeneity in feasibility, acceptability, or usability results specifically. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Chatbot eHealth meta-analyses to date have focused on specific health conditions, resulting in few studies focusing on comparing primary health effects.

    “Chatbot eHealth meta-analyses to-date have focused on specific health conditions resulting in few studies focusing on comparing primary health effects [37, 38].”

    From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [37]

    The Methods (Study Eligibility Criteria; page unavailable) show that this meta-analysis focuses on one health area—mental health—and the Results (Search Results; page unavailable) say only 4 of 12 included studies were meta-analyzed. The paper does not establish that chatbot eHealth meta-analyses generally focus on specific conditions, or that this focus has caused a shortage of studies comparing primary health effects. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • Abd-Alrazaq et al evaluated the effectiveness of CAs in 12 clinical and nonclinical RCTs, finding weak evidence of reduced depression, stress, or agoraphobia rates, but not improved mental well-being.

    “Abd-Alrazaq et al [123] evaluated the effectiveness of CAs in 12 clinical and nonclinical RCTs. Weak evidence of reduced depression, stress, or agoraphobia rates, but not improved mental well-being, was found.”

    From Development and Evaluation of a Smartphone-Based Chatbot Coach to Facilitate a Balanced Lifestyle in Individuals With Headaches (BalanceUP App): Randomized Controlled Trial (2024) · cites this paper as [123]

    The Results (Description of Included Studies; page not provided) identify 12 studies involving clinical and nonclinical samples, but only 6 were RCTs; the other 6 were quasiexperimental. The Discussion (Principal Findings; page not provided) reports weak evidence of reduced depression and stress and no significant improvement in subjective psychological wellbeing. The phobia studied was acrophobia (fear of heights), not agoraphobia. (LLM, conf=0.92)

    Experimental — accuracy not yet measured

  • Of extant systematic reviews on AI chatbots, 6 targeted assessing efficacy in enhancing mental health outcomes, 2 examined feasibility in health care settings, and 1 described technical architectures and characteristics in chronic conditions.

    “Of the extant systematic reviews on AI chatbots, 6 articles targeted at assessing efficacy of AI-chatbots in enhancing mental health outcomes [1,7,9-12], 2 examined the feasibility of AI-chatbots in health care settings [8,13], and 1 described the technical architectures and characteristics of the AI chatbots used in chronic conditions [14].”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [9]

  • This is similar to other findings on chatbot arm clinical interventions that found a lack of evidence that chatbot primary effect is clinically important.

    “This is similar to other findings on chatbot arm clinical interventions that found a lack of evidence that chatbot primary effect is clinically important [37].”

    From Patient Engagement with Conversational Agents in Health Applications 2016–2022: A Systematic Review and Meta-Analysis (2024) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Positive and negative affect is attributed to both two and three studies

    Minor

    The section first says that two studies examined this outcome, then says it was measured in three studies.

  • The description of two-group comparators omits high versus low users

    Minor

    The paper says all two-group trials used no intervention or education as the comparator, but earlier describes a study comparing high users with low users.

  • Sample-size categories account for only 10 of 12 studies

    Minor

    The three sample-size categories contain 6, 3, and 1 studies, totaling 10 rather than the 12 included studies. The table gives no explanation for the two unaccounted-for studies.

    Reported 12 · recomputed 10

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Radziwill, N, 2017, arXiv preprint arXiv, 170404579
  2. Schunemann, H, 2008, Higgins J, Green S. editors, 359
  3. Higgins, JP, 2016, Cochrane Database of Systematic Reviews, 10, 1
  4. Deeks, J, 2008, Higgins J, Green S. editors. Cochrane handbook for systematic reviews of interventions, A
  5. HuangJLiQXueYChengTXuSJiaJTeenchat: a chatterbot system for sensing and releasing adolescents' stresseditors2015International Conference on Health Information ScienceMay 28-30, 2015Melbourne, Australia133145

Cited by

Sorted by citations

Related

Authors
Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M.
Affiliations
  • Hamad bin Khalifa University (QA)
  • University of Warwick (GB)
  • University of Leeds (GB)
Corresponding author
Househ, Mowafa
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2020-07-13
Volume
22
Issue
7
Pages
e16021
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/16021
PMC
PMC7385637
OpenAlex
W3022528244
Semantic Scholar
5c15d6234ac87e69a1cf42c3eba92f566ee4ce30
Cited by
503
References
52
Citations by year
2026: 145 · 2025: 124 · 2024: 99 · 2023: 62 · 2022: 45 · 2021: 23 · 2020: 5
FWCI
5.31
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Warwick Research Archive Portal (University of Warwick) (submittedVersion)
Funders
  • Fonds National de la Recherche Luxembourg
  • Qatar National Research Fund (NPRP12S-0303-190204, NPRP12S-)
  • Qatar National Library
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Social Psychology › Mental Health Treatment and Access
Keywords
chatbots, mental health, depression, anxiety, distress, stress, acrophobia, psychological well-being, positive and negative affect, chatbot safety, digital mental health interventions, mental health workforce shortage
MeSH terms
Mental Disorders, Communication, Humans, Mental Health

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review