CoChat
Sign Up
Cited by 276Digital Mental Health InterventionsSocial Robot Interaction and HRIEmotion and Mood Recognition

Conversational agents in the treatment of mental health problems: mixed-method systematic review

Open access
Gaffney, Hannah, Mansell, Warren, Tai, Sara·2019·JMIR Mental Health
DOI 10.2196/14166PMC PMC6914342

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The use of conversational agent interventions (including chatbots and robots) in mental health is growing at a fast pace. Recent existing reviews have focused exclusively on a subset of embodied conversational agent interventions despite other modalities aiming to achieve the common goal of improved mental health. Objective: This study aimed to review the use of conversational agent interventions in the treatment of mental health problems. Methods: We performed a systematic search using relevant databases (MEDLINE, EMBASE, PsycINFO, Web of Science, and Cochrane library). Studies that reported on an autonomous conversational agent that simulated conversation and reported on a mental health outcome were included. Results: A total of 13 studies were included in the review. Among them, 4 full-scale randomized controlled trials (RCTs) were included. The rest were feasibility, pilot RCTs and quasi-experimental studies. Interventions were diverse in design and targeted a range of mental health problems using a wide variety of therapeutic orientations. All included studies reported reductions in psychological distress postintervention. Furthermore, 5 controlled studies demonstrated significant reductions in psychological distress compared with inactive control groups. In addition, 3 controlled studies comparing interventions with active control groups failed to demonstrate superior effects. Broader utility in promoting well-being in nonclinical populations was unclear. Conclusions: The efficacy and acceptability of conversational agent interventions for mental health problems are promising. However, a more robust experimental design is required to demonstrate efficacy and efficiency. A focus on streamlining interventions, demonstrating equivalence to other treatment modalities, and elucidating mechanisms of action has the potential to increase acceptance by users and clinicians and maximize reach.

AI snapshot

Written from the paper’s full text.

TL;DR

This mixed-method systematic review identified 13 studies of autonomous conversational agents for mental health, including 4 full-scale RCTs and mostly feasibility, pilot RCT, or quasi-experimental designs. Evidence was promising versus inactive controls, but active-control comparisons failed to show superiority and broader well-being utility in nonclinical populations remained unclear.

Key findings

  • Across 13 included studies evaluating 11 different conversational agents, all reported reductions in psychological distress postintervention; 5 controlled studies found significant reductions versus inactive/no-treatment or information controls, with effects ranging from small (d=−0.24) to very large (d=2.0).
  • Three controlled studies comparing conversational agent interventions with active control groups failed to demonstrate superior effects, and 4 controlled studies reported no significant posttreatment differences between intervention and control groups.
  • The review included 1200 participants total; sample sizes ranged from 14 to 454, participants were 16–75 years old, and 70.3% (692/985) were female in the 12 studies reporting gender. Methodological quality on QATSDD ranged from 35% to 88%, with an average of 59%.
  • User experience was generally positive—participants reported satisfaction, accessibility, empathy, interactivity, and relationship-building—but common challenges included repetitive content, agent misunderstanding or inappropriate responses, shallow relationships, and voice quality; broader utility for promoting well-being in nonclinical populations was unclear.

Methods

  • Systematic search of MEDLINE, EMBASE, PsycINFO, Web of Science, and Cochrane library (September 2018, updated January 2019), without publication year or language restriction; reference lists were handsearched and gray literature was included.
  • Inclusion required an autonomous conversational agent simulating 2-way, real-time conversation that could be used independently and reported a mental health outcome; exclusions included Wizard of Oz methods, predetermined output, asynchronous communication, human support, medication/physical health adherence, technical-only reports, and insufficient detail.
  • Because of heterogeneity in designs, outcomes, modalities, durations, and control groups, meta-analysis was not undertaken; data were narratively synthesized, and risk of bias was assessed with the 16-item QATSDD tool.

Limitations

  • Stated by the authors: included studies had variable methodological quality, mainly small self-selected samples, short or absent follow-up, no assessment of psychological comorbidity, safety explicitly evaluated in only 1 study, inconsistent engagement reporting, no comparison of conversational agent design features, no evaluation of therapeutic equivalence or superiority to other treatment modalities such as face-to-face therapy, and eclectic interventions made active ingredients difficult to identify.
  • Review-level limitations stated by the authors: heterogeneity precluded meta-analysis, some potentially relevant conversational agents were excluded for not reporting a mental health outcome, and inconsistent terminology/reporting complicated eligibility assessment.
  • Evident from the text: interrater reliability was not assessed at the title/abstract screening stage, full-text screening was conducted by the first author with only a random 9.8% (26/264) sample independently screened, and methodological quality was assessed by the first author alone.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/14166

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 119 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    All included studies reported reductions in psychological distress postintervention.

    “All included studies reported reductions in psychological distress postintervention.”
    Quote in the paperSubstantial claim
  • Findingown data

    The included studies reported results from a total of 1200 participants.

    “The included studies reported results from a total of 1200 participants.”
    Quote in the paperNot a substantial claim
  • Findingown data

    A large proportion of studies (n=7) were preliminary and included feasibility, pilot RCTs, or nonrandomized trials.

    “A large proportion of studies (n=7) were preliminary and included feasibility [30], pilot RCTs [27,28,31-33], or nonrandomized trials [34].”
    Quote in the paperNot a substantial claim
  • Findingown data

    Five controlled studies demonstrated significant reductions in psychological distress compared with inactive control groups.

    “Furthermore, 5 controlled studies demonstrated significant reductions in psychological distress compared with inactive control groups.”
    Quote in the paperSubstantial claim
  • Findingcitation

    All of the included studies with percentage scores above 70% were RCTs.

    “All of the included studies with percentage scores above 70% were RCTs [23-26].”
    Quote in the paperNot a substantial claim
  • May combine two claims. 53
  • Classified as probably not a substantial claim. 46

  • May combine two claims.

    Findingown data

    The remaining studies recruited self-selected participants from the community through outpatient clinics, universities, online advertisements, and radio advertisements and by downloading the intervention app through the app store.

    “The remaining studies recruited self-selected participants from the community through outpatient clinics [32], universities [24-27,30,31,33], online advertisements [22,33,34], and radio advertisements [23] and by downloading the intervention app through the app store [29].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Cohen kappa was 0.65, indicating substantial interrater agreement.

    “Cohen kappa was 0.65, indicating substantial interrater agreement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The 13 studies identified were conducted between 2013 and 2018 in 4 countries.

    “The 13 studies identified were conducted between 2013 and 2018 in 4 countries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In addition, 5 of the 13 included studies recruited participants who self-reported symptoms of psychological distress to varying degrees.

    “In addition, 5 of the 13 included studies recruited participants who self-reported symptoms of psychological distress to varying degrees.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    One study with 129 participants did not collect data on age or gender, and 1 study recruited only women.

    “One study with 129 participants [29] did not collect data on age or gender, and 1 study recruited only women [32].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Conversational agents used different technologies, with 3 conversational agents accessed on an app, 4 online (5 studies), 3 using an offline computer program (4 studies), and 1 VR program utilizing a VR headset.

    “Conversational agents used different technologies, with 3 conversational agents accessed on an app [25,29,33], 4 online (5 studies) [24,26,27,32,34], 3 using an offline computer program (4 studies) [22,28,30,31], and 1 VR program utilizing a VR headset [23].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Three studies focused on improving mental well-being.

    “Three studies focused on improving mental well-being [32-34].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Two studies used treatment as usual (TAU), which consisted of treatment for depression with a clinician or corresponded to no treatment.

    “Two studies used treatment as usual (TAU), which consisted of treatment for depression with a clinician [28] or corresponded to no treatment [23].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Among the 13 studies, 5 were conducted in the United Kingdom, 6 in the United States, 1 in Sweden, and 1 in Japan.

    “Among them, 5 studies were conducted in the United Kingdom [23,24,27-29], 6 studies in the United States [22,25,26,30-32], 1 study in Sweden [33], and 1 study in Japan [34].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All included studies scored a 2 or 3 for fit between research question and method of analysis and fit between stated research question and method of data collection.

    “All included studies scored a 2 or 3 for fit between research question and method of analysis and fit between stated research question and method of data collection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Study sample sizes ranged from 14 to 454.

    “Study sample sizes ranged from 14 [22] to 454 [34].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The output mainly consisted of questions or written text (6 out of 11 agents, evaluated in 7 studies).

    “The output mainly consisted of questions or written text (6 out of 11 agents, evaluated in 7 studies) [24-27,29,33,34].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The remaining 3 agents took responses from participants using fixed onscreen response options (4 studies).

    “The remaining 3 agents took responses from participants using fixed onscreen response options (4 studies) [22,30-32].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Most studies provided adequate descriptions of procedure for data collection and detailed recruitment data.

    “Most studies provided adequate descriptions of procedure for data collection and detailed recruitment data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    In addition, 2 studies (1 conversational agent) did not specify whether the conversational agent output was written or spoken.

    “In addition, 2 studies (1 conversational agent) [31] did not specify whether the conversational agent output was written or spoken.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Finally, 2 quasi-experimental studies did not use a control group.

    “Finally, 2 quasi-experimental studies did not use a control group [22,29].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The conversational agents provided interventions aimed at reducing symptoms, increasing well-being, or improving self-management.

    “The conversational agents provided interventions aimed at reducing symptoms [22-29], increasing well-being [32-34], or improving self-management [30,31].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    One study used a waitlist control group, and 1 study used a nonrandomized control group of participants who had expressed interest in taking part in the study but could not complete the intervention at that time.

    “One study used a waitlist control group [33], and 1 study used a nonrandomized control group of participants who had expressed interest in taking part in the study but could not complete the intervention at that time [34].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    All included studies received the maximum score of 3 for the criterion statement of aims or objectives in main body of report.

    “All included studies received the maximum score of 3 for the criterion statement of aims or objectives in main body of report.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    A total of 13 articles were included in the review evaluating 11 different conversational agents.

    “A total of 13 studies were included in the review.”
    • Also stated in: § Study Selection · plain
      “A total of 13 articles were included in the review evaluating 11 different conversational agents.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Interventions were diverse in design and targeted a range of mental health problems using a wide variety of therapeutic orientations.

    “Interventions were diverse in design and targeted a range of mental health problems using a wide variety of therapeutic orientations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In addition, 2 studies used quasi-experimental designs, and 4 studies were full-scale RCTs.

    “In addition, 2 studies used quasi-experimental designs [22,29], and 4 studies were full-scale RCTs [23-26].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most studies (n=10) provided discussions of the key strengths and limitations of the study (scoring 2), and 3 studies gave thorough, complete discussions of strengths and limitations, obtaining a maximum score of 3.

    “Most studies (n=10) provided discussions of the key strengths and limitations of the study (scoring 2), and 3 studies gave thorough, complete discussions of strengths and limitations, obtaining a maximum score of 3.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The lowest average scores were found for representative sample of target group of a reasonable size, good justification for analytic method selected, assessment of reliability of analytic process (qualitative only), and evidence of user involvement in design.

    “The lowest average scores were found for representative sample of target group of a reasonable size, good justification for analytic method selected, assessment of reliability of analytic process (qualitative only), and evidence of user involvement in design.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most studies (n=8) used mixed methods, and the majority (n=9) of them reported on both mental health outcomes and user experiences.

    “Most studies (n=8) used mixed methods [22,25,26,28-30,32,33], and the majority (n=9) of them reported on both mental health outcomes and user experiences [22,23,25,26,28-30,32,33].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Furthermore, 4 agents used spoken output.

    “Furthermore, 4 agents used spoken output [22,23,28,32].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Study participants ranged between 16 and 75 years old, and gender prevalence was 70.3% (692/985) female from studies that reported this data (12/13).

    “Study participants ranged between 16 and 75 years old, and gender prevalence was 70.3% (692/985) female from studies that reported this data (12/13).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Ring et al compared groups that used 2 different versions (proactive and passive) of the conversational agent intervention.

    “However, Ring et al [22] compared groups that used 2 different versions (proactive and passive) of the conversational agent intervention.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Significant improvements were observed on measures of depression, psychological distress, anxiety, fear of heights, and positive affect.

    “Significant improvements were observed on measures of depression [25,26,31], psychological distress [34], anxiety [26], fear of heights [23] and positive affect [26,34].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [25] source

      Not checked in the free run.

    • cites [26] source

      Not checked in the free run.

    • cites [31] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

    • Rests on an assumption: The term 'psychological distress' adequately summarizes the varied primary outcome measures used across included studies.

  • May combine two claims.

    Findingown data

    Studies reported differing metrics for engagement, and reporting was inconsistent.

    “Studies reported differing metrics for engagement, and reporting was inconsistent (see Table 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    One study with a high attrition rate (74.1%, 1978/2668) did not report any reasons; however, the majority of the dropouts were from the control condition (1846/2109, 88.3%) compared with 23.6% (132/559) in the intervention condition.

    “One study with a high attrition rate (74.1%, 1978/2668) [34] did not report any reasons; however, it should be noted that the majority of the dropouts were from the control condition (1846/2109, 88.3%) compared with 23.6% (132/559) in the intervention condition.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Over half of the conversational agents (7 out of 11) focused on providing psychoeducation and self-management strategies, 1 agent (evaluated in 2 studies) utilized the principles of MOL therapy in a question-and-answer format, 1 agent offered social companionship, and 1 agent (evaluated in 2 studies) facilitated practice of effective communication with human health care professionals around心理症状.

    “Over half of the conversational agents (7 out of 11) focused on providing psychoeducation and self-management strategies [25,26,28,29,32-34], 1 agent (evaluated in 2 studies) utilized the principles of MOL therapy in a question-and-answer format [24,27], 1 agent offered social companionship [22], and 1 agent (evaluated in 2 studies) facilitated practice of effective communication with human health”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    From short interventions of 1 session (participant-determined length up to 20 min), 3 sessions (unspecified duration, 15-20 min each), and 6 sessions (30 min each through to daily usage over 2 weeks, 4 weeks, or a month).

    “From short interventions of 1 session (participant-determined length [24] up to 20 min [27]), 3 sessions (unspecified duration [31], 15-20 min each [30]), and 6 sessions (30 min each [23] through to daily usage over 2 weeks [25,26,33], 4 weeks [26,28], or a month [32]).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Reasons reported for dropout included difficulties attending the university to take part because of financial difficulties, technical problems, and mental illness.

    “Reasons reported for dropout included difficulties attending the university to take part because of financial difficulties [30], technical problems [22,27], and mental illness [22].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The 2 uncontrolled studies and 2 studies that did not test for between-group effects reported reductions in depression and loneliness postintervention.

    “Finally, the 2 uncontrolled studies included in the review [22,29] and 2 studies that did not test for between-group effects [28,30] reported reductions in depression [28-30] and loneliness [22] postintervention.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [22] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

    • cites [28] source

      Not checked in the free run.

    • cites [30] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Finally, 1 study installed 1 of 2 versions (passive, activated at will, and proactive, activated by a motion sensor) of the same conversational agent into participants' homes for 1 week.

    “Finally, 1 study installed 1 of 2 versions (passive, activated at will, and proactive, activated by a motion sensor) of the same conversational agent into participants’ homes for 1 week.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    In these studies, both intervention and control conditions demonstrated reduced distress or increased uptake of stress management techniques.

    “Furthermore, 4 controlled studies reported no significant posttreatment differences on measures of psychological distress between the intervention and control groups [24,27,32,33] with both intervention and control conditions demonstrating reduced distress [24,27,33] or increased uptake of stress management techniques [32].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [24] source

      Not checked in the free run.

    • cites [27] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • Rests on an assumption: The term 'psychological distress' adequately summarizes the varied primary outcome measures used across included studies.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study only used data from participants who had engaged with the intervention at least every other day (>15 times) over a month.

    “One study only used data from participants who had engaged with the intervention at least every other day (>15 times) over a month [34].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    However, 44 out of 49 (90%) participants completed the intervention, with a mean total intervention time of 124 min.

    “However, 44 out of 49 (90%) participants completed the intervention, with a mean total intervention time of 124 min.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Attrition rates between pre- and postmeasures were reported in 11 studies and varied widely from no attrition to 74.1% (1978/2668) of participants.

    “Attrition rates between pre- and postmeasures were reported in 11 studies and varied widely from no attrition [23,24,33] to 74.1% (1978/2668) of participants [34].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Psychological comorbidity was not assessed in any of the studies despite comorbidities being prevalent in individuals with common mental health problems.

    “Psychological comorbidity was not assessed in any of the studies despite comorbidities being prevalent in individuals with common mental health problems [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    The accessibility of the interventions was a key strength for many participants and where accessibility was limited, participants highlighted this and suggested ways to improve accessibility.

    “The accessibility of the interventions was a key strength for many participants and where accessibility was limited, participants highlighted this and suggested ways to improve accessibility (eg, online access [30]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Participants reported that they found the agent empathic, liked the interactivity, the agent's personality, the agent's ability to form a relationship, and the agent's ability to learn from input.

    “Participants reported that they found the agent empathic [26], that they liked the interactivity [30], the agent’s personality [22,25], the agent’s ability to form a relationship [28,33], and the agent’s ability to learn from input [26].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [26] source

      Not checked in the free run.

    • cites [30] source

      Not checked in the free run.

    • cites [22] source

      Not checked in the free run.

    • cites [25] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Participants reported that they liked the ability to customize the gender and appearance of ECAs and the option to tailor the session length to their own needs.

    “Participants reported that they liked the ability to customize the gender and appearance of ECAs [28] and the option to tailor the session length to their own needs [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    The use of conversational agents for treating mental health problems appears to be limited but is growing quickly, with 5 of the included studies published in 2018 alone.

    “The use of conversational agents for treating mental health problems appears to be limited but is growing quickly, with 5 of the included studies published in 2018 alone [23,24,26,29,34].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [23] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

    • cites [26] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Included interventions were generally brief, allowed participants to control the intensity of intervention, and drew from a wide variety of psychological approaches.

    “Included interventions were generally brief, allowed participants to control the intensity of intervention, and drew from a wide variety of psychological approaches.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The predominant challenges to intervention with a conversational agent included repetitive content, limitations in the agents ability to understand or respond appropriately, a shallow or superficial relationship, the sound and quality of the agents voice, and specific intervention tools or content.

    “The predominant challenges to intervention with a conversational agent included repetitive content [22,25,26,28,29,33], limitations in the agents ability to understand or respond appropriately [22,25,26,29], a shallow or superficial relationship [28,33], the sound and quality of the agents voice [32], and specific intervention tools or content [25,29].”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [22] source

      Not checked in the free run.

    • cites [25] source

      Not checked in the free run.

    • cites [26] source

      Not checked in the free run.

    • cites [28] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Studies managed to recruit participants through several different methods.

    “Studies managed to recruit participants through several different methods.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Engagement with interventions was not reported consistently and appeared highly variable, and the reasons for this remain unexplored.

    “Engagement with interventions was not reported consistently and appeared highly variable, and the reasons for this remain unexplored.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    This is despite inconsistencies in the reporting of interventions which made the process of eligibility assessment more complicated and reflected the heterogeneity and complexity in the field.

    “This is despite inconsistencies in the reporting of interventions which made the process of eligibility assessment more complicated and reflected the heterogeneity and complexity in the field.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Conversational agents hold particular promise compared with other digital mental health interventions as they can provide greater interactivity that emulates therapeutic conversation and provides choice and control over session content and intensity.

    “Conversational agents hold particular promise compared with other digital mental health interventions as they can provide greater interactivity that emulates therapeutic conversation and provides choice and control over session content and intensity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Existing systematic and scoping reviews of conversational agent interventions in mental health have focused on a subset of conversational agents with a visual character (embodied) or are now outdated.

    “Recent existing reviews have focused exclusively on a subset of embodied conversational agent interventions despite other modalities aiming to achieve the common goal of improved mental health.”
    • Also stated in: § Rationale · plain
      “Existing systematic and scoping reviews of conversational agent interventions in the mental health field have focused on a subset of conversational agents with a visual character (embodied) [8-10] or are now outdated [7].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [8] source

      Not checked in the free run.

    • cites [9] source

      Not checked in the free run.

    • cites [10] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Screening procedures were piloted before beginning the screening process.

    “Screening procedures were piloted before beginning the screening process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Data from the included studies were extracted into a prespecified form.

    “Data from the included studies were extracted into a prespecified form”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The search was not restricted by publication year or language.

    “The search was not restricted by publication year or language.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The efficacy and acceptability of conversational agent interventions for mental health problems are promising.

    “The efficacy and acceptability of conversational agent interventions for mental health problems are promising.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The reference lists of all included studies were handsearched to identify all relevant references.

    “The reference lists of all included studies were handsearched to identify all relevant references.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A random, 9.8% (26/264) sample of studies identified for full-text screening were also independently screened by a second reviewer.

    “A random, 9.8% (26/264) sample of studies identified for full-text screening were also independently screened by a second reviewer.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Owing to the diversity in study designs, outcomes measured, intervention modalities, and durations and the varied use of active and inactive control groups, a meta-analysis would not have led to meaningful conclusions and was thus not undertaken.

    “Owing to the diversity in study designs, outcomes measured, intervention modalities, and durations and the varied use of active and inactive control groups, a meta-analysis would not have led to meaningful conclusions and was thus not undertaken.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Risk-of-bias assessment of each study was conducted to ascertain the validity and reliability of the methods and findings to inform the narrative synthesis of the studies.

    “Risk-of-bias assessment of each study was conducted to ascertain the validity and reliability of the methods and findings to inform the narrative synthesis of the studies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Research has demonstrated that users respond and connect to conversational agents in social ways, and they can encourage honest disclosure.

    “Research has demonstrated that users respond and connect to conversational agents in social ways, and they can encourage honest disclosure [14,15].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] source

      Not checked in the free run.

    • cites [15] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The application of conversational agents in mental health is varied and includes diagnostic tools, symptom monitoring, and treatment or intervention.

    “The application of conversational agents in mental health is varied and includes diagnostic tools, symptom monitoring, and treatment or intervention [16].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The protocol was registered prospectively at PROSPERO (registration number: CRD42018106652).

    “The protocol was registered prospectively at PROSPERO (registration number: CRD42018106652).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    We followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines.

    “We followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines [17,18].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [17] source

      Not checked in the free run.

    • cites [18] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    A systematic search of the literature was performed in September 2018 and updated in January 2019 using MEDLINE, EMBASE, PsycINFO, Web of Science, and the Cochrane library.

    “A systematic search of the literature was performed in September 2018 and updated in January 2019 using MEDLINE (1946 to August week 5, 2018), EMBASE (1974 to September 2018), PsycINFO (1806 to September 2018), Web of Science (1900 to September 2018), and the Cochrane library (All to September 2018).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Extracted data were narratively synthesized in line with guidance on the conduct of narrative synthesis in systematic reviews.

    “Instead, extracted data were narratively synthesized in line with guidance on the conduct of narrative synthesis in systematic reviews [20].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] PopayJRobertsHSowdenAPetticrewMAriaLRodgersMBrittenNRoenKDuffySSemantic Scholar20062019-09-17Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product from the ESRC Methods Programme https://pdfs.semanticscholar.org/ed8b/23836338f6fdea0cc55e161b0fc5805f9e27.pdf

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Interrater reliability was assessed at full-text eligibility stage.

    “However, interrater reliability was assessed at full-text eligibility stage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Owing to the large number of articles identified from the initial search and limited researcher resource, interrater reliability was not assessed at the title and abstract stage.

    “Owing to the large number of articles identified from the initial search and limited researcher resource, interrater reliability was not assessed at the title and abstract stage.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Existing digital therapies have suffered from low adherence and concerns about their efficiency without continued human support.

    “Existing digital therapies have suffered from low adherence and concerns about their efficiency without continued human support [12,13].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [12] source

      Not checked in the free run.

    • cites [13] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Gray literature, including conference abstracts or proceedings, and dissertations or theses identified through the database searches were also included for screening.

    “Gray literature, including conference abstracts or proceedings, and dissertations or theses identified through the database searches were also included for screening.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    These studies assessed relative rather than absolute treatment efficacy, and thus, we cannot conclude an absolute lack of treatment efficacy.

    “However, it is important to highlight that these studies assessed relative rather than absolute treatment efficacy, and thus, we cannot conclude an absolute lack of treatment efficacy [35].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [35] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    It is possible that clinician apprehension about digital treatment for mental health problems affected recruitment rates.

    “It is possible that clinician apprehension about digital treatment for mental health problems affected recruitment rates.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [36] source

      Not checked in the free run.

    • cites [37] DunneNElectronic Theses and Dissertations (ETD) Center20172019-09-17Evaluation of Psychology Clinicians’ Attitudes Towards Computerized Cognitive Behavior Therapy, for Use In Their Future Clinical Practice, With Regard to Treating Those Suffering From Anxiety and Depression https://etd.ohiolink.edu/!etd.send_file?accession=antioch1503328670275243&disposition=inline

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The methodological quality of the included studies varied, and sample sizes were mainly small and self-selected, which reduces the ability to draw firm conclusions about the reliability and validity of the findings.

    “The methodological quality of the included studies varied, and sample sizes were mainly small and self-selected, which reduces the ability to draw firm conclusions about the reliability and validity of the findings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Studies have demonstrated that these types of conversational agents are often not able to respond appropriately to risk information such as suicidal ideation and have the potential to result in harm.

    “Studies have demonstrated that these types of conversational agents are often not able to respond appropriately to risk information such as suicidal ideation [40,41] and have the potential to result in harm.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [40] source

      Not checked in the free run.

    • cites [41] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    This perhaps mirrors expectations around core relationship factors such as feeling understood.

    “This perhaps mirrors expectations around core relationship factors such as feeling understood.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Despite the heterogeneity in evaluation methods, there is an increasing emphasis on fully powered RCTs testing efficacy.

    “Furthermore, despite the heterogeneity in evaluation methods, there is an increasing emphasis on fully powered RCTs testing efficacy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We also included a broad range of formats for conversational agents, including VR and embodied and/or text and speech input.

    “We also included a broad range of formats for conversational agents, including VR and embodied and/or text and speech input.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The impact of the design or features of the conversational agents on engagement or outcomes was not explicitly assessed or compared; therefore, conclusions cannot be drawn as to the most effective or acceptable modality.

    “Furthermore, the impact of the design or features of the conversational agents (eg, embodiment and speech or text based) on engagement or outcomes was not explicitly assessed or compared; therefore, conclusions cannot be drawn as to the most effective or acceptable modality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Studies failed to demonstrate superiority when compared with other active, conversational interventions, and their broader utility in promoting well-being in nonclinical populations is unclear.

    “However, studies failed to demonstrate superiority when compared with other active, conversational interventions, and their broader utility in promoting well-being in nonclinical populations is unclear.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Cohen kappa showed substantial agreement in full-text screening, and there was a high percentage of agreement overall.

    “Cohen kappa showed substantial agreement in full-text screening, and there was a high percentage of agreement overall.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Owing to the lack of standardized terminology in this area, we conducted a comprehensive search that prioritized sensitivity over specificity.

    “Owing to the lack of standardized terminology in this area, we conducted a comprehensive search that prioritized sensitivity over specificity.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The review was also registered on PROSPERO before commencing.

    “The review was also registered on PROSPERO before commencing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We also reviewed reference lists for additional papers not identified through the database searches.

    “We also reviewed reference lists for additional papers not identified through the database searches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Users can expect a level of understanding beyond what is currently technologically possible.

    “Furthermore, users can expect a level of understanding beyond what is currently technologically possible [41].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Published abstracts commonly presented in technology conferences were also included as they typically provide enough detail for decisions to be made about inclusion.

    “Published abstracts commonly presented in technology conferences were also included as they typically provide enough detail for decisions to be made about inclusion.”
    Quote in the paperNot a substantial claim
85 of 85

  • The paper relies on this without support.

    Existing digital therapy formats tend to focus on psychoeducation and a modular style of fixed content and duration that is inflexible for users.

    “Existing digital therapy formats tend to focus on psychoeducation and a modular style of fixed content and duration that is inflexible for users.”

  • The paper relies on this without support.

    The narrative synthesis can yield meaningful conclusions despite the considerable heterogeneity across the included studies.

    “Owing to the diversity in study designs, outcomes measured, intervention modalities, and durations and the varied use of active and inactive control groups, a meta-analysis would not have led to meaningful conclusions and was thus not undertaken.”

  • The paper relies on this without support.

    The included studies are representative of the broader field of conversational agent interventions for mental health treatment.

    “A total of 13 articles were included in the review evaluating 11 different conversational agents.”

  • The paper relies on this without support.

    The term 'psychological distress' adequately summarizes the varied primary outcome measures used across included studies.

    “Primary outcome measures were all validated but varied (see Multimedia Appendix 1 for details); therefore, the term psychological distress will be used to facilitate a summary.”

  • The paper relies on this without support.

    Safety is a vital consideration in mental health interventions that use free-text, natural language input either written or spoken.

    “Safety is a vital consideration in mental health interventions that use free-text, natural language input either written [24-27,29,33,34] or spoken [23,28].”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI chatbots can be integrated into embodied functions such as virtual reality that offer additional benefits, such as an immersive experience, which can catalyze the process of health behavior change.

    “AI chatbots can also be integrated into embodied functions (eg, virtual reality) that offer additional benefits, such as an immersive experience, which can catalyze the process of health behavior change [8].”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [8]

    The Introduction (Objectives) includes conversational agents within virtual reality, and the Results (Conversational Agent Interventions) report one program using a VR headset. The cited paper does not say that VR provides an immersive benefit that catalyzes health behavior change. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • A growing number of studies report effectiveness of chatbots in reducing perceived stress.

    “as well as their effectiveness in reducing perceived stress [16-25]”

    From Engagement and effectiveness of a healthy-coping intervention via chatbot for university students during the COVID-19 pandemic: Mixed methods proof-of-concept study (2021) · cites this paper as [21]

    The Results > Psychological Outcomes section (page not provided) reports that five controlled studies found reductions in psychological distress compared with inactive controls. It does not establish reductions in perceived stress specifically, and it also reports no significant advantage in several studies with active controls. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Of extant systematic reviews on AI chatbots, 6 targeted assessing efficacy in enhancing mental health outcomes, 2 examined feasibility in health care settings, and 1 described technical architectures and characteristics in chronic conditions.

    “Of the extant systematic reviews on AI chatbots, 6 articles targeted at assessing efficacy of AI-chatbots in enhancing mental health outcomes [1,7,9-12], 2 examined the feasibility of AI-chatbots in health care settings [8,13], and 1 described the technical architectures and characteristics of the AI chatbots used in chronic conditions [14].”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [8]

  • Use of conversational agents was predominantly investigated in relation to a broad range of mental health problems, or specifically related to cognitive and social abilities, without considering the emotional component of mental health.

    “Moreover, use of CAs was predominantly investigated in relation to a broad range of mental health problems15,16, or specifically related to cognitive and social abilities, without considering the emotional component of mental health24,25.”

    From Use of automated conversational agents in improving young population mental health: a scoping review (2024) · cites this paper as [16]

  • There is a need for relying on unbiased data in mental health chatbot research.

    “the recent review by Gaffney et al [19] has highlighted the need for relying on unbiased data.”

    From A conversational artificial intelligence agent for a mental health care app: Evaluation study of its participatory design (2021) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Recommendation finding cites a review guideline as an included study

    Minor

    The text attributes a participant recommendation finding to study [18], but reference 18 is the PRISMA reporting statement, not a conversational-agent intervention study.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. FastEHorvitzELong-Term Trends in the Public Perception of Artificial IntelligenceProceedings of the Thirty-First AAAI Conference on Artificial Intelligence2017AAAI'17February 4-9, 2017San Francisco, California, USA
  2. 2013, Diagnostic and Statistical Manual of Mental Disorders: DSM-5. Fifth Editiob

Cited by

Sorted by citations

Related

Authors
Gaffney, H., Mansell, W., & Tai, S.
Affiliations
University of Manchester (GB)
Corresponding author
Gaffney, Hannah
ORCID
Venue
JMIR Mental Health
Publisher
JMIR Publications Inc.
Published
2019-10-18
Volume
6
Issue
10
Pages
e14166
ISSN
2368-7959 (ISSN-L)
DOI
10.2196/14166
PMC
PMC6914342
OpenAlex
W2980981514
Semantic Scholar
ccdf220519ea8046b744a50761e1e4ebdead06d8
Cited by
275
References
46
Citations by year
2026: 47 · 2025: 61 · 2024: 50 · 2023: 38 · 2022: 34 · 2021: 26 · 2020: 19
FWCI
2.80
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Mental Health (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Explorer (The University of Manchester) (submittedVersion, other-oa)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
chatbots, psychotherapy, psychological distress, digital mental health interventions, embodied conversational agents, therapeutic interventions, intervention efficacy, acceptability, mechanism of action

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review