CoChat
Sign Up
Cited by 56Digital Mental Health InterventionsArtificial Intelligence in Healthcare and EducationSocial Robot Interaction and HRI

Generative AI Mental Health Chatbots as Therapeutic Tools: Systematic Review and Meta-Analysis of Their Role in Reducing Mental Health Issues

Open access
Zhang, Qiyang, Zhang, Renwen, Xiong, Yiying, Sui, Yuan, Tong, Chang, Lin, Fu-Hung·2025·Journal of Medical Internet Research
DOI 10.2196/78238PMC PMC12707440
Open this paper in CoChat — free account

Abstract

Background: In recent years, artificial intelligence (AI) has driven the rapid development of AI mental health chatbots. Most current reviews investigated the effectiveness of rule-based or retrieval-based chatbots. To date, there is no comprehensive review that systematically synthesizes the effect of generative AI (GenAI) chatbot’s impact on mental health. Objective: This review aims to (1) narratively synthesize existing GenAI mental health chatbots’ technical features, treatment and research designs, and sample characteristics through a systematic review of quantitative studies and (2) quantify the effectiveness and key moderators of these rigorously designed trials on GenAI mental health chatbots through a meta-analysis of only randomized controlled trials (RCTs). Methods: The search strategy includes 11 database searching, backward citation tracking, and a manual ad hoc search to update literature. This thorough literature search, completed in March 2025, returned 5555 records for screening. The systematic review included studies that (1) used generative or hybrid (rule/retrieval-based and generative) AI-based chatbots to deliver interventions and (2) quantitatively measured mental health-related outcomes. The meta-analysis has additional inclusion criteria: (1) studies must be RCTs, (2) must measure negative mental health issues, (3) the comparison group must not have chatbot features, and (4) must provide enough statistics for effect size calculation. We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist and registered the protocol retrospectively during the revision process (September 18, 2025). In meta-regression, data were synthesized in R software using a random-effects model. Results: The narrative synthesis of 26 studies revealed that (1) GenAI chatbot interventions mostly took place in non-WEIRD countries (non-Western, Educated, Industrialized, Rich, and Democratic) and (2) there is a lack of studies focusing on young children and older adults. The meta-analysis of 14 RCTs showed a statistically significant effect (effect size [ES]=0.30, P =.047, N =6314, 95% CI 0.004, 0.59, 95% prediction interval [PI] −0.85, 1.67), which means that GenAI chatbots are, on average, effective in reducing negative mental health issues, such as depression, anxiety, among others. We found that social-oriented chatbots (ie, those that mainly provide social interactions) are more effective than task-oriented programs (ie, those that assist with specific tasks). Risk of bias in the nonrandomized studies and RCTs was assessed using Cochrane ROBINS-I (Risk Of Bias In Non-randomised Studies – of Interventions) and RoB2 (revised Cochrane risk-of-bias tool for randomized trials), respectively, indicating a moderate amount of risk. One main limitation of this meta-analysis is the small number of studies (n=14) included. Conclusions: By identifying research gaps, we suggest that future researchers investigate user groups such as adolescents and older adults, outcomes other than depression and anxiety, cultural adaptations in non-WEIRD countries, ways to streamline chatbots in usual care practices, and explore applications in diverse settings. More importantly, we cannot ignore GenAI chatbots’ risks while acknowledging their promise. This review also emphasized several ethical implications.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review and meta-analysis synthesizes 26 studies of generative AI mental health chatbots and meta-analyzes 14 RCTs, finding a small-to-moderate statistically significant average effect in reducing negative mental health issues. Effects are highly heterogeneous, with social-oriented chatbots outperforming task-oriented ones and depression showing the only statistically significant outcome subgroup.

Key findings

  • The meta-analysis of 14 RCTs (19 treatment-control comparisons, N=6314) found an overall pooled effect size of 0.30 (P=.047, 95% CI 0.004–0.59, 95% PI −0.85 to 1.67), indicating a small-to-moderate average reduction in negative mental health issues.
  • Social-oriented chatbots were more effective than task-oriented chatbots: in the combined moderator model, task-oriented chatbots had a coefficient of −0.76 (P=.04); task-oriented chatbots showed SMD=0.007 (SE=0.06, P=.91) versus social-oriented chatbots SMD=0.77 (SE=0.34, P=.06).
  • Among outcome subgroups, only depression was statistically significant (k=12, ES=0.49, P=.04, 95% CI 0.03–0.96); anxiety (k=19, ES=0.43, P=.15), negative affect/mood (k=8, ES=0.28, P=.39), and stress (k=4, ES=0.10, P=.15) were not.
  • Risk of bias was moderate overall: 66.67% (8/12) of nonrandomized studies were rated serious risk on ROBINS-I, and among RCTs, 64.29% (9/14) had some concerns and 35.71% (5/14) high risk, with no RCT at low risk of bias.

Methods

  • Systematic review searched 11 databases plus backward citation tracking and a manual ad hoc search, completed March 2025, yielding 5555 records; 26 studies met inclusion criteria for narrative synthesis.
  • Meta-analysis included only RCTs measuring negative mental health outcomes, with comparison groups lacking chatbot features and sufficient statistics for effect size calculation, resulting in 14 RCTs.
  • Data were synthesized in R using a multilevel random-effects meta-analysis with REML, HKSJ/SJ inference, and one-moderator meta-regressions for social function, human assistance, and control group type.

Limitations

  • The paper states the meta-analysis included a small number of studies (abstract: n=14; limitation section: 12 studies and 37 effect sizes), reducing statistical power for moderator analyses and precision.
  • The paper reports baseline differences exceeding 0.25 SD in some RCTs (e.g., Jeong 0.30 SD for depression; McFadyen et al 0.26 SD for anxiety; Sabour 0.33 SD for depression) and differential attrition exceeding 15% (e.g., Chen et al 34%; He et al 24%).
  • The paper notes wide prediction intervals, substantial heterogeneity, and risk-of-bias ratings ranging from some concerns to high risk, so results should be interpreted cautiously; the text also makes evident inconsistent counts of included studies/effect sizes across sections and retrospective protocol registration.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/78238

Cites

89 of 89 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

38 of 87 cited sources checked · 19 supported · 3 flagged · accuracy not yet measured

  • Findingcitation

    He et al reported a differential attrition rate of 24%.

    “For instance, Chen et al [50] reported a differential attrition rate of 34% and He et al [54] reported 24 %.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [54] He, Mental health chatbot for young adults with depressive symptoms during the COVID-19 pandemic: single-blind, three-arm randomized controlled trial, J Med Internet Res source

      The Results section, “Adherence and Attrition” (page unavailable), reports attrition of 37% in the control conditions versus 10% in the intervention condition—a difference of 27 percentage points, not 24%. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Over 70% of people with mental disorders receive no treatment from professionals due to stigma, counselors' shortage, and under-resourced care infrastructures.

    “In fact, over 70% of people with mental disorders receive no treatment from professionals due to stigma, counselors’ shortage, and under-resourced care infrastructures [6].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] Henderson, Mental illness stigma, help seeking, and public health programs, Am J Public Health source

      The Abstract reports that globally more than 70% of people with mental illness receive no treatment from health care staff. The section “Discrimination, Stigma, and Mental Health Care Access” identifies stigma and institutional funding and service availability as barriers, but does not identify a shortage of counselors or establish that the listed barriers account for the entire 70% treatment gap. (LLM, conf=0.89)

      Experimental — accuracy not yet measured
  • Premisecitation

    A recent meta-analysis shows that GenAI chatbots outperform rule-based and retrieval-based chatbots in reducing depressive symptoms.

    “A recent meta-analysis shows that GenAI chatbots outperform rule-based and retrieval-based chatbots in reducing depressive symptoms [13].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [13] Li, Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being, NPJ Digit Med source

      The Results of the meta-analysis report that AI-based chatbots reduced depressive symptoms compared with control conditions. They report a larger effect for generative than retrieval-based chatbots on psychological distress, not specifically depression. The Discussion compares AI-based chatbots with rule-based chatbots on psychological distress using results from a different review; it does not establish the claimed three-way comparison for depressive symptoms. Our copy provides section labels but no page numbers for these passages. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    Most current reviews investigated the effectiveness of rule-based or retrieval-based chatbots.

    “However, most of these interventions relied on retrieval-based chatbots, which use predefined responses or static databases and often result in rigid, repetitive interactions [17-19].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [17] Al-Amin M Ali MS Salam A et al. History of generative artificial intelligence (AI) chatbots: past, present, and future development. arXiv. Preprint posted online on Feb 4, 2024. [doi: 10.48550/arXiv.2402.05122] source

      The Introduction (page unavailable) describes rule- and template-based chatbots as using predefined responses that can be predictable and repetitive. It does not establish that most current reviews evaluated such chatbots or that most of the cited interventions used retrieval-based chatbots. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [18] Abd-Alrazaq, An overview of the features of chatbots in mental health: a scoping review, Int J Med Inform source

      Section 3.3 reports that 92.5% of the mental-health chatbot studies reviewed used predefined rules or decision trees, and Section 4.1 says their responses were predetermined and user input was restricted. The paper does not establish that these were specifically retrieval-based chatbots using static databases, or report that their interactions were repetitive. It also describes the interventions in its review, not what most current reviews investigated. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [19] Dsouza R Sahu S Patil R Kalbande DR . Chat with bots intelligently: a critical review & analysis. Presented at: 2019 International Conference on Advances in Computing, Communication and Control (ICAC3); Dec 20-21, 2019; Mumbai, India. 2019.[doi: 10.1109/ICAC347590.2019.9036844] source
  • Premisecitation

    A study found that a mental health chatbot alone did not outperform the traditional bibliotherapy method in developing participants' resilience.

    “a study also found that mental health chatbot alone did not outperform the traditional bibliotherapy method in developing [58] participants’ resilience.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [58] Liu I Chen W Ge Q Song D Ni S . Enhancing psychological resilience with chatbot-based cognitive behavior therapy: a randomized control pilot study. Presented at: Chinese CHI ’22: Proceedings of the Tenth International Symposium of Chinese CHI; Oct 22-23, 2022; Guangzhou, China and Online China. [doi: 10.1145/3565698.3565787] source
  • Classified as probably not a substantial claim. 71
  • May combine two claims. 33

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We adopted HKSJ as it is recommended for meta-analysis with few studies.

    “Since only 14 studies were included in the meta-analysis, we adopted HKSJ as it is recommended for meta-analysis with few studies [38].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [38] Röver, Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies, BMC Med Res Methodol source

      The Abstract Conclusions recommend the modified mKH procedure—not standard HKSJ—especially when few studies have varying precision. The Conclusions say standard HKSJ achieves the intended coverage only when study standard errors are similar; its error probability can exceed the target in unbalanced settings. The paper therefore does not make the unqualified recommendation attributed to it. (LLM, conf=0.85)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    The remainder used quasi-experimental (n=5), mixed-methods (n=3), survey (n=2), and observational design (n=1).

    “while the remainder used quasi-experimental (n=5), mixed-methods (n=3), survey (n=2) [26], and observational design (n=1) [23].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [26] Wang, Tech vs. tradition: ChatGPT and mindfulness in enhancing older adults’ emotional health, Behav Sci (Basel) source

      The citation [23] is attached to the claim that one study used an observational design. Section 2.1, “Study Design,” instead describes a controlled eight-week experiment with participants assigned to ChatGPT or mindfulness interventions and measured before and after. The cited paper therefore does not fit the observational-design category attributed to it. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [23] Habicht J Dina LM Stylianou M Harper R Hauser TU Rollwage M . Generative AI-enabled therapy support tool improves clinical outcomes and patient engagement in NHS talking therapies. PsyArXiv. Preprint posted online on Apr 10, 2024. [doi: 10.31234/osf.io/mj46k] source

  • May combine two claims.

    Premisecitation

    Companion chatbots such as Replika have been found to reduce loneliness and produce outcomes comparable to mindfulness interventions among older adults.

    “Beyond structured therapy, companion chatbots such as Replika have been found to reduce loneliness [25] and produce outcomes comparable to mindfulness interventions among older adults [26].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [25] Liu AR Pataranutaporn P Maes P . Chatbot companionship: a mixed-methods study of companion chatbot usage patterns and their relationship to loneliness in active users. arXiv. Preprint posted online on Oct 28, 2024. [doi: 10.48550/arXiv.2410.21596] source
    • cites [26] Wang, Tech vs. tradition: ChatGPT and mindfulness in enhancing older adults’ emotional health, Behav Sci (Basel) source

  • May combine two claims.

    Premisecitation

    Most of these interventions relied on retrieval-based chatbots, which use predefined responses or static databases and often result in rigid, repetitive interactions.

    “However, most of these interventions relied on retrieval-based chatbots, which use predefined responses or static databases and often result in rigid, repetitive interactions [17-19].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [17] Al-Amin M Ali MS Salam A et al. History of generative artificial intelligence (AI) chatbots: past, present, and future development. arXiv. Preprint posted online on Feb 4, 2024. [doi: 10.48550/arXiv.2402.05122] source

      The Introduction describes template-based chatbots that select predefined responses and says their conversations can be predictable and repetitive. Section 3.10 describes a chatbot retrieving answers from databases. The paper does not establish that most of the interventions in the citing sentence used retrieval-based chatbots. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [18] Abd-Alrazaq, An overview of the features of chatbots in mental health: a scoping review, Int J Med Inform source

      Section 3.3 reports that 92.5% of the reviewed chatbots used predefined rules or decision trees. Section 4.1 says their responses were predetermined and users’ dialogue was restricted. The paper does not identify these chatbots as retrieval-based or report that their interactions were repetitive; it also does not establish that they used static databases. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [19] Dsouza R Sahu S Patil R Kalbande DR . Chat with bots intelligently: a critical review & analysis. Presented at: 2019 International Conference on Advances in Computing, Communication and Control (ICAC3); Dec 20-21, 2019; Mumbai, India. 2019.[doi: 10.1109/ICAC347590.2019.9036844] source

  • May combine two claims.

    Premisecitation

    This capacity may strengthen user engagement, therapeutic alliance, and the sense of being understood.

    “This capacity may strengthen user engagement [20], therapeutic alliance [21], and the sense of being understood [22].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [20] Kumar, Anthropomorphic generative AI chatbots for enhancing customer engagement, experience and recommendation, JCM source
    • cites [21] Siddals, “It happened to be the perfect thing”: experiences of generative AI chatbots for mental health, npj Mental Health Res source
    • cites [22] Yin, AI can help people feel heard, but an AI label diminishes this impact, Proc Natl Acad Sci USA source
    • Rests on an assumption: By tailoring replies to the user’s language, tone, and emotional content, GenAI chatbots enable conversations that are more natural, personalized, and emotionally resonant.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The protocol was registered retrospectively during the revision process (September 18, 2025).

    “We registered the protocol retrospectively during the revision process (September 18, 2025), with Open Science Framework (10.17605/OSF.IO/9DAJ7) [43].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [43] Zhang Q . Generative AI chatbots as therapeutic tools: a systematic review and meta-analysis of their role in mitigating mental health issues. OSF. 2025. URL: https://osf.io/9daj7 [Accessed 08-11-2025] source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    AI chatbot-based interventions offer 24/7 support, enhanced self-management, reduced stigma, and appeal to digital-native users.

    “These AI chatbot-based interventions offer 24/7 support, enhanced self-management [10], reduced stigma, and appeal to digital-native users [11].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [10] Oh, Web-based personalized normative feedback to decrease stigma and increase intentions to seek help, Stigma Health source
    • cites [11] Bond, Digital transformation of mental health services, Npj Ment Health Res source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Emerging primary studies suggest that GenAI mental health chatbots can improve engagement and adherence by supporting between-session cognitive behavioral therapy tasks and delivering positive psychology interventions, such as gratitude or self-reflection exercises.

    “Emerging primary studies suggest that GenAI mental health chatbots can improve engagement and adherence by supporting between-session cognitive behavioral therapy tasks [23] and delivering positive psychology interventions, such as gratitude or self-reflection exercises [24].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [23] Habicht J Dina LM Stylianou M Harper R Hauser TU Rollwage M . Generative AI-enabled therapy support tool improves clinical outcomes and patient engagement in NHS talking therapies. PsyArXiv. Preprint posted online on Apr 10, 2024. [doi: 10.31234/osf.io/mj46k] source
    • cites [24] Liu, Investigating the key success factors of chatbot-based positive psychology intervention with retrieval- and generative pre-trained transformer (GPT)-based chatbots, International Journal of Human–Computer Interaction source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The COVID-19 pandemic has further exacerbated these challenges, highlighting the urgent need for scalable, accessible, and cost-effective interventions.

    “The COVID-19 pandemic has further exacerbated these challenges, highlighting the urgent need for scalable, accessible, and cost-effective interventions [7].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [7] Yusefi, Health human resources challenges during COVID-19 pandemic; evidence of a qualitative study in a developing country, PLOS ONE source

      The Results (Personal challenge) describe increased workload and psychological problems among healthcare personnel during COVID-19, and the Conclusion calls for planning and supportive measures. The paper does not establish a need for interventions that are specifically scalable, accessible, and cost-effective; its findings concern health human resources rather than challenges generally. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    We reported 95% prediction intervals for the true effect.

    “Given that treatment effects over different settings may vary, we also reported 95% prediction intervals (PIs) for the true effect [41].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [41] IntHout, Plea for routinely presenting prediction intervals in meta-analysis, BMJ Open source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Participants' age ranged from 11 to 96 years.

    “The participants’ age ranged from 11 [30] to 96 [49] years.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [30] Zheng, The effects of chatbot use on foreign language reading anxiety and reading performance among Chinese secondary school students, Computers and Education: Artificial Intelligence source
    • cites [49] Carl, Comparing patient’s confidence in clinical capabilities in urology: large language models versus urologists, Eur Urol Open Sci source

      Citation [49] is attached to the upper age of 96 years. Table 1 in Results reports mean ages by group, and Patient enrollment specifies a minimum eligibility age of 18, but neither gives the oldest participant’s age. Searches for “96” and the maximum age found no passage establishing 96 as the upper limit. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The studies' sample size ranged from 12 to 1006 (mean 198.85, SD 210.68).

    “The studies’ sample size ranges all the way from 12 [26] to 1006 [31] (mean 198.85, SD 210.68).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [26] Wang, Tech vs. tradition: ChatGPT and mindfulness in enhancing older adults’ emotional health, Behav Sci (Basel) source
    • cites [31] Maples, Loneliness and suicide mitigation for students using GPT3-enabled chatbots, Npj Ment Health Res source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study drew from positive psychology.

    “one drew from positive psychology [58]”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [58] Liu I Chen W Ge Q Song D Ni S . Enhancing psychological resilience with chatbot-based cognitive behavior therapy: a randomized control pilot study. Presented at: Chinese CHI ’22: Proceedings of the Tenth International Symposium of Chinese CHI; Oct 22-23, 2022; Guangzhou, China and Online China. [doi: 10.1145/3565698.3565787] source

      Section 2.2 describes Philobot’s intervention as CBT, with dialogue written from CBT literature. The Introduction and Measures discuss resilience, well-being, and positive affect, but do not say the study drew from positive psychology. A search for “positive psychology” found no passage establishing that basis; CBT and positive psychology are not necessarily mutually exclusive. (LLM, conf=0.74)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study integrated NLP, BERT, and deep learning.

    “another integrated NLP, BERT, deep learning [58]”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [58] Liu I Chen W Ge Q Song D Ni S . Enhancing psychological resilience with chatbot-based cognitive behavior therapy: a randomized control pilot study. Presented at: Chinese CHI ’22: Proceedings of the Tenth International Symposium of Chinese CHI; Oct 22-23, 2022; Guangzhou, China and Online China. [doi: 10.1145/3565698.3565787] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Based on Beyebach et al's WEIRD versus non-WEIRD framework, 15 out of 26 studies (58%) were conducted in non-WEIRD countries.

    “Based on Beyebach et al’s [33] WEIRD versus non-WEIRD framework, 15 out of 26 studies (58%) were conducted in non-WEIRD countries.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [33] Beyebach, Bibliometric differences between WEIRD and non-WEIRD countries in the outcome research on solution-focused brief therapy, Front Psychol source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study incorporated NLP with dynamic programming (DP).

    “one incorporated NLP with dynamic programming (DP) [54]”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [54] He, Mental health chatbot for young adults with depressive symptoms during the COVID-19 pandemic: single-blind, three-arm randomized controlled trial, J Med Internet Res source

      The Methods > Procedure passage says XiaoE’s technology is rooted in natural language processing and deep learning. It does not say the chatbot incorporated dynamic programming (DP); searches for that term found no supporting passage in our copy. Page numbers were not available. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Only one study focused on adolescents exclusively.

    “Only one study focused on adolescents exclusively [41]”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [41] IntHout, Plea for routinely presenting prediction intervals in meta-analysis, BMJ Open source

      The Discussion and Outlook and Measure passages concern prediction intervals and heterogeneity in meta-analyses, not the ages of participants in individual studies. Searches for “adolescents exclusively” and adolescent age groups found no passage in our copy stating that only one study focused exclusively on adolescents. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Previous literature indicates that social chatbots lead to better consumer satisfaction and social outcomes for older adults.

    “This pattern aligns with previous literature on social chatbots leading to better consumer satisfaction [69] and social outcomes for older adults [70].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [69] Cai, How the communication style of chatbots influences consumers’ satisfaction, trust, and engagement in the context of service failure, Humanit Soc Sci Commun source
    • cites [70] Chattaraman, Should AI-Based, conversational digital assistants employ social- or task-oriented interaction style? A task-competency and reciprocity perspective for older adults, Comput Human Behav source

  • May combine two claims.

    Interpretationcitation

    Social chatbots, by offering empathic, personalized, and emotionally attuned exchanges, can foster trust and disclosure, which may help reduce negative mental health issues.

    “Social chatbots, by offering empathic, personalized, and emotionally attuned exchanges, can foster trust and disclosure, which may help reduce negative mental health issues [69].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [69] Cai, How the communication style of chatbots influences consumers’ satisfaction, trust, and engagement in the context of service failure, Humanit Soc Sci Commun source

      The Discussion reports that social-oriented chatbots increased consumers’ trust through perceived warmth in an online-shopping service-failure setting. The cited paper does not report increased disclosure or reduced mental health problems, and it does not establish the claimed benefit outside that consumer-service setting. Page numbers are unavailable in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Training AI systems with culturally representative data and considering local ethical and regulatory contexts may improve trust, relevance, and uptake.

    “Training AI systems with culturally representative data and considering local ethical and regulatory contexts may improve trust, relevance, and uptake [87].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [87] Farhud, Ethical issues of artificial intelligence in medicine and healthcare, Iran J Public Health source

      The Introduction says practitioners should consider medical ethics before integrating AI into healthcare, and the Privacy and Data Protection section discusses legal safeguards in different jurisdictions (page numbers unavailable). The paper does not say that culturally representative training data or attention to local contexts improves trust, relevance, or uptake. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Twenty-one out of 26 studies (81%) focused on early- and middle-aged adults (18-50 y old).

    “Meanwhile, 21 out of 26 studies (81%) from the systematic review focus on early- and middle-aged adults (18–50 y old).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among the 26 studies, 1 study measured only positive well-being, 1 had chatbot designs in the control group, 3 did not report sufficient data, and 8 were not randomized trials, leaving 14 RCTs eligible for meta-analysis.

    “Among the 26 studies, 1 study measured only positive well-being instead of mental health issues [29], 1 study had chatbot designs in the control group [24], 3 studies did not report sufficient data for calculating pooled effect size, and 8 studies were not randomized trials, leaving a total of 14 RCTs eligible for meta-analysis to estimate the effectiveness of GenAI chatbot on mental health issues”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [29] Vowels LM Vowels MJ Sweeney S Hatch SG Darwiche J . The evaluation of efficacy, feasibility, and technical outcomes of a GPT-4O-based chatbot Amanda for relationship support: a randomized controlled trial. PsyArXiv. Preprint posted online on Sep 20, 2024. [doi: 10.31234/osf.io/q4yh7] source
    • cites [24] Liu, Investigating the key success factors of chatbot-based positive psychology intervention with retrieval- and generative pre-trained transformer (GPT)-based chatbots, International Journal of Human–Computer Interaction source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Eight studies integrated LLM with other AI techniques, such as natural language processing (NLP).

    “In addition, eight studies integrated LLM with other AI techniques, such as natural language processing (NLP).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Ten studies were conducted in China, 6 in the United States, 3 in the United Kingdom, and 1 each in Turkey, Germany, Singapore, Pakistan, Ukraine, Canada, and Japan.

    “with 10 conducted in China, 6 in the United States, 3 in the United Kingdom [263761], and 1 each in Turkey [52], Germany [49], Singapore [62], Pakistan [60], Ukraine [63], Canada [65], and Japan [66].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Six studies used only web-based platforms.

    “6 used only web-based platforms (including two web applications)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The remainder involved general adult participants (n=13).

    “while the remainder involved general adult participants (n=13).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Eleven studies incorporated voice features.

    “with 11 studies incorporating voice features”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Two studies used both smartphone and web-based platforms.

    “2 used both smartphone and web-based platforms”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study incorporated NLP alongside GPT-3.5.

    “one incorporated NLP alongside GPT-3.5 [24]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Liu, Investigating the key success factors of chatbot-based positive psychology intervention with retrieval- and generative pre-trained transformer (GPT)-based chatbots, International Journal of Human–Computer Interaction source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Table 2 presents descriptive statistics of the 14 RCT studies with 19 treatment-control pairs (n=6314) included in the meta-analysis.

    “Table 2 presents the descriptive statistics of the 14 RCT studies with 19 treatment-control pairs (n=6314) included in the meta-analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Seventeen studies were published in 2024 alone.

    “with 17 studies published in 2024 alone”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study used natural language generation (NLG).

    “one used natural language generation (NLG) [57]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57] Kimani E Bickmore T Trinh H Pedrelli P . You’ll be great: virtual agent-based cognitive restructuring to reduce public speaking anxiety. Presented at: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII); Sep 3-6, 2019; Cambridge, United Kingdom. [doi: 10.1109/ACII.2019.8925438] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A few studies (n=5) embedded chatbots into structured health care.

    “A few studies embedded chatbots into structured health care (n=5), including urological counseling for urological surgery [49], preoperative patient education [66], therapy support between sessions [23], knee arthroplasty patients’ consent process [53], assistance to health care professionals in a conventional school nurse hotline [53].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Exactly half of the studies (n=13, including one study recruited partially students and partially nonstudents) focused on student populations.

    “Exactly half of the studies [55] (n=13, including one study recruited partially students and partially nonstudents) focused on student populations”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Heinz MV Mackin D Trudeau B et al. Evaluating Therabot: a randomized control trial investigating the feasibility and effectiveness of a generative AI therapy chatbot for depression, anxiety, and eating disorder symptom treatment. PsyArXiv. Preprint posted online on Jun 14, 2024. [doi: 10.31234/osf.io/pjqmr] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The majority of studies (n=11) required the use of a smartphone only.

    “the majority (n=11) required the use of a smartphone only”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some studies (n=3) embedded chatbots in educational settings.

    “Some studies embedded chatbots in educational settings (n=3), such as oral communication skills courses for university freshmen [52], junior secondary English as a Foreign Language classes’ activities for clarifying reading confusion [30], and online general English-speaking courses [51].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among the 26 studies, 21 implemented AI chatbots with customized features.

    “among the 26 studies, 21 implemented AI chatbots with customized features.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Two studies drew from mindfulness interventions.

    “two from mindfulness interventions [2665]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2665]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Out of the 10 studies conducted in China, 2 used English-language AI platforms because they targeted English learning among Chinese students, and the remaining 8 used Chinese.

    “Out of the 10 studies conducted in China, 2 used English-language AI platforms because they targeted English learning among Chinese students [3067], and the remaining 8 studies used Chinese.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3067]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Six studies further included image-based interactions.

    “six studies further including image-based interactions”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Three studies specifically used GPT-series models.

    “within which three studies specifically used GPT-series models”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Most included studies used a single AI chatbot.

    “Most studies used a single AI chatbot, including ChatGPT (various versions, n=5), Replika (n=4), Limbic care (n=2), and one study each for Elomia [63], Philobot [58], MyAI [62], Virtual Coach Angela [57], Reading Bot [30], Emohaa [64], Amanda [29], XiaoE [54], and Therabot [55].”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Six studies recruited clinical populations, including patients attending urological counseling for elective surgery (n=1), people experiencing symptoms of depression or anxiety (n=3), people with high-risk eating disorders and other mental health conditions (n=1), or knee osteoarthritis patients (n=1).

    “while six recruited clinical populations, including patients attending urological counseling for elective surgery [49] (n=1), people experiencing symptoms of depression or anxiety [162361] (n=3), people with high-risk eating disorders and other mental health conditions (n=1) [55], or knee osteoarthritis patients (n=1) [53].”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among the 11 studies utilizing GenAI, all systems were based on large language model (LLM) architectures.

    “Among the 11 studies utilizing GenAI, all systems were based on large language model (LLM) architectures”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Three studies focused on older adults.

    “three studies focused on older adults [485366].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [485366]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One study used a neural network machine learning model and scripted dialogue content.

    “one study used neural network machine learning model and scripted dialogue content [52]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52] Çakmak, Novitas-ROYAL (Research on Youth and Language)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Among the 26 included studies, 21 used purely GenAI and 5 used hybrid models combining generative and rule or retrieval-based approaches.

    “Among the 26 included studies, 21 used purely GenAI and 5 used hybrid models combining generative and rule or retrieval-based approaches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Three studies explored multiple AI chatbots.

    “Three studies explored multiple AI chatbots, such as ChatGPT, Gemini, and Perplexity [60], or Typebot/ D-ID Agent and EAP Talk [51], or GPT-3.5 Turbo and Baidu UNIT [24] platform chatbot.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Three studies did not specify a chatbot name.

    “Three studies did not specify a chatbot name.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    All studies used nonembodied AI chatbots, except for two that incorporated embodied virtual agents.

    “All studies used nonembodied AI chatbots, except for two that incorporated embodied virtual agents [5157].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5157]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Two studies used long short-term memory (LSTM) models.

    “Two studies used long short-term memory (LSTM) models [2559]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2559]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Publication years ranged from 2019 to early 2025 (March when the search concluded).

    “Publication years ranged from 2019 to early 2025 (March when the search concluded)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The remaining 14 studies did not specify the guiding theoretical model.

    “the rest of the studies (n=14) did not specify the guiding theoretical model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Three studies did not specify the platform.

    “3 studies did not specify the platform”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    All 26 studies used text-based interaction.

    “all 26 studies used text-based interaction”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Depression and anxiety are the most studied outcomes in existing studies of GenAI mental health chatbots.

    “Depression and anxiety are the most studied outcomes in existing studies of GenAI mental health chatbots.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among the outcome subgroups (depression, anxiety, stress, negative moods), effect sizes were positive across all groups, but only the depression subgroup demonstrated a statistically significant effect (ES=.49, P=.041).

    “Among the outcome subgroups (depression, anxiety, stress, negative moods), effect sizes were positive across all groups, but only the depression subgroup demonstrated a statistically significant effect (ES=.49, P=.041).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There was only one effect size for loneliness, so subgroup analysis was not conducted.

    “There is only one effect size on loneliness; therefore, we skipped subgroup analysis on this outcome.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Meta-regression findings indicate that social-oriented chatbots, as opposed to task-oriented ones, demonstrate greater effectiveness, though with wide variability and uncertainty.

    “Meta-regression findings indicate that social-oriented chatbots, as opposed to task-oriented ones, demonstrate greater effectiveness, though with wide variability and uncertainty.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    As a robustness check, we computed cluster-robust (CR2) standard errors with Satterthwaite degrees of freedom, clustering on study.

    “As a robustness check, we computed cluster-robust (CR2) standard errors with Satterthwaite degrees of freedom, clustering on study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The complete dataset and R code are publicly available.

    “In line with open science principles, the complete dataset and R code are publicly available [42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] Generative-AI-mental-health-chatbot. GitHub. 2025. URL: https://github.com/qiyangzh/Generative-AI-Mental-Health-Chatbot [Accessed 08-11-2025]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Cohen κ values were 0.5 and 0.6, indicating fair and substantial agreement, respectively.

    “During the title and abstract screening phase and the full-text review phase, Cohen κ values were 0.5 and 0.6, indicating fair and substantial agreement, respectively.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    We tested three moderators only due to balanced categories, theoretical and practical importance, small sample size (n=14), and the need for a degree of freedom larger than four to ensure enough statistical power.

    “Considering the need for balanced moderator categories, theoretical and practical importance, small sample size (n=14), and the need for a degree of freedom to be larger than four to ensure enough statistical power [47], we tested three moderators only.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [47] Tipton, Small sample adjustments for robust variance estimation with meta-regression, Psychol Methods source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    We fitted a univariate model using the HKSJ adjustment method as a sensitivity analysis.

    “In addition to multivariate meta-analysis, as a sensitivity analysis, we fitted a univariate model using the HKSJ adjustment method.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We followed the PRISMA checklist.

    “We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Two authors coded independently, and a third author resolved discrepancies.

    “Two authors coded (CT, FL) independently, and a third author (QZ) resolved discrepancies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent advances in AI have driven the rapid development of mental health chatbots.

    “Recent advances in artificial intelligence (AI) have driven the rapid development of mental health chatbots [89].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [89] Pani B Crawford J Allen KA . Can generative artificial intelligence foster belongingness, social support, and reduce loneliness? A conceptual analysis. In: Applications of Generative AI. Springer; 2024:261-276. [doi: 10.1007/978-3-031-46238-2_13] source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A number of reviews showed promising effects of AI chatbot interventions on reducing mental health distress and improving quality of life.

    “A number of reviews showed promising effects of AI chatbot interventions on reducing mental health distress and improving quality of life [1012-16undefinedundefinedundefinedundefined] (see Multimedia Appendix 2).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1012-16undefinedundefinedundefinedundefined]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    To evaluate publication bias, selection modeling was used.

    “To evaluate publication bias, selection modeling was used. This approach used a weight function model created by Vevea and Woods [44], implemented through the weightr package.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] Vevea, Publication bias in research synthesis: sensitivity analysis using a priori weight functions, Psychol Methods source

  • May combine two claims.

    Interpretationreasoning

    AI chatbots can assist in-person counseling sessions but might not replace human therapy.

    “It seems like AI chatbots can assist in-person counseling sessions but might not replace human therapy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Task-oriented chatbots often lack the flexibility to respond empathetically, limiting their capacity to generate the relational bonds essential for emotional relief.

    “Task-oriented chatbots, by contrast, often lack the flexibility to respond empathetically, limiting their capacity to generate the relational bonds essential for emotional relief.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The conclusion remained the same when univariate HKSJ-SJ random-effects meta-analysis was applied.

    “we applied univariate HKSJ-SJ random-effects meta-analysis, and the conclusion remains the same”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    GenAI chatbots might act as complementary supports to enhance counselors' efficiency and extend access to care.

    “Therefore, GenAI chatbots might act as complementary supports to enhance counselors’ efficiency and extend access to care.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    GenAI chatbots may serve as valuable adjuncts to care but should not be viewed as standalone solutions for addressing the full spectrum of mental health needs.

    “while GenAI chatbots may serve as valuable adjuncts to care, they should not be viewed as standalone solutions for addressing the full spectrum of mental health needs”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Effective treatment requires careful examination of patients' medical history, symptom trajectory, and sustained therapeutic alliances, which cannot be fully replicated by current GenAI systems.

    “Effective treatment requires careful examination of patients’ medical history, symptom trajectory, and sustained therapeutic alliances, which cannot be fully replicated by current GenAI systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Designing AI interactions that communicate acceptance and genuine care may enhance users' emotional engagement and psychological well-being.

    “Designing AI interactions that communicate acceptance and genuine care may enhance users’ emotional engagement and psychological well-being”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Depression management typically requires long-term care, and approximately half of patients relapse after an initial episode.

    “Depression management typically requires long-term care, and approximately half of patients relapse after an initial episode [81].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [81] Kupfer, J Clin Psychiatry

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    This imbalance reflects a broader gap in the literature, particularly regarding the role of GenAI chatbots in managing more severe or complex mental health conditions.

    “This imbalance reflects a broader gap in the literature, particularly regarding the role of GenAI chatbots in managing more severe or complex mental health conditions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Depression and anxiety are the most prevalent mental disorders and are highly comorbid.

    “both disorders are not only the most prevalent [79] but also highly comorbid [80]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [79] Mental disorders. World Health Organization. 2025. URL: https://www.who.int/news-room/fact-sheets/detail/mental-disorders [Accessed 06-10-2025] source
    • cites [80] Mental health conditions: depression and anxiety. Centers for Disease Control and Prevention. 2023. URL: https://www.cdc.gov/tobacco/campaign/tips/diseases/depression-anxiety.html#three [Accessed 06-10-2025]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    Among the RCTs assessed with Cochrane RoB2, 64.29% (9/14) were rated as having some concerns and 35.71% (5/14) as high risk, with no study judged to be at low risk of bias.

    “The results showed that 64.29% (9/14) of studies were rated as having some concerns and 35.71% (5/14) as high risk, with no study judged to be at low risk of bias.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Task-oriented chatbots, lacking a socio-emotional dimension, primarily provide informational rather than emotional support, limiting their impact on distress.

    “In contrast, task-oriented chatbots, lacking this socio-emotional dimension, primarily provide informational rather than emotional support, limiting their impact on distress.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The role of GenAI chatbots lies in complementing human-delivered services and expanding access to support, especially in contexts where resources are scarce.

    “Rather, their role lies in complementing human-delivered services and expanding access to support, especially in contexts where resources are scarce.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The cross-national observations are descriptive rather than inferential; the study did not test the effects of cultural or regulatory differences statistically.

    “However, these cross-national observations are descriptive rather than inferential; our study did not test the effects of cultural or regulatory differences statistically.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GenAI chatbots have great potential to provide companions for older adults to reduce their sense of loneliness.

    “In the increasingly aging society, GenAI chatbots have great potential to provide companions for the older adults to reduce their sense of loneliness [89].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [89] Pani B Crawford J Allen KA . Can generative artificial intelligence foster belongingness, social support, and reduce loneliness? A conceptual analysis. In: Applications of Generative AI. Springer; 2024:261-276. [doi: 10.1007/978-3-031-46238-2_13] source

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The benefits of these interventions come with risks that cannot be ignored.

    “While these interventions are promising, their benefits come with risks that cannot be ignored.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Users in mental health settings may be especially vulnerable.
    • Rests on an assumption: Generative AI systems can produce responses that are unpredictable, inappropriate, or harmful.

  • May combine two claims.

    Premisecitation

    Prior studies have found a dark side of AI companionship, including emotional dependency, manipulation, privacy violations, and social isolation.

    “News reports of suicide cases linked to interaction with AI companion chatbots [9091] highlight the urgent risks, reinforcing findings from prior studies on the dark side of AI companionship, including emotional dependency, manipulation, privacy violations, and social isolation [9293].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9293]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    A small sample reduces statistical power for performing moderator analyses and the capacity to obtain more precise estimates of the effect size via moderators.

    “Readers should note that a small sample reduces statistical power for performing moderator analyses, and consequently, the capacity to obtain more precise estimates of the effect size via moderators [101].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [101] Hedges, The power of statistical tests for moderators in meta-analysis, Psychol Methods source
87 of 87

  • The paper relies on this without support.

    GenAI chatbots, powered by large language models, generate novel responses in real time.

    “In contrast, GenAI chatbots, powered by large language models (LLMs), such as GPT models, generate novel responses in real time.”

  • The paper relies on this without support.

    By tailoring replies to the user’s language, tone, and emotional content, GenAI chatbots enable conversations that are more natural, personalized, and emotionally resonant.

    “By tailoring replies to the user’s language, tone, and emotional content, they enable conversations that are more natural, personalized, and emotionally resonant.”

  • The authors state this as an assumption.

    The within-study correlation was assumed to be r=0.80.

    “We constructed a block-diagonal sampling variance-covariance matrix V with an assumed within-study correlation r=0.80.”

  • The authors state this as an assumption.

    When studies only provided post-test means and SDs, baseline equivalence was assumed.

    “When studies only provided post-test means and SDs, we assumed baseline equivalence and calculated Hedges g.”

  • The paper relies on this without support.

    Global mental health challenges are increasing in severity and complexity.

    “The increasing severity and complexity of global mental health challenges highlight the limited application of chatbots to severe mental health conditions such as suicidality, schizophrenia, or substance use disorders.”

  • The paper relies on this without support.

    Users in mental health settings may be especially vulnerable.

    “These dangers are particularly acute in mental health settings, where users may be especially vulnerable and the generative nature of AI systems can produce responses that are unpredictable, inappropriate, or even harmful.”

  • The paper relies on this without support.

    Generative AI systems can produce responses that are unpredictable, inappropriate, or harmful.

    “These dangers are particularly acute in mental health settings, where users may be especially vulnerable and the generative nature of AI systems can produce responses that are unpredictable, inappropriate, or even harmful.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • ConsistentF(3, 36) = 3.11reported p = .04 · recomputed p = .038

    “F(3, 36)=3.11, p = .04”

1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Limitation section gives a different meta-analysis study count

    Major

    The limitation section says the meta-analysis included 12 studies, whereas the results repeatedly report 14.

  • Section heading reverses the reported WEIRD-country majority

    Major

    The heading says most interventions took place in WEIRD countries, but the section reports that a majority took place in non-WEIRD countries.

  • Meta-analysis sample exceeds the entire review sample

    Major

    The meta-analysis is a subset of the 26 studies in the systematic review, yet its reported N of 6314 exceeds the reported total of 5469 participants across all 26 studies.

  • Selection-adjusted effect is described as smaller but is larger

    Major

    The reported selection-adjusted pooled effect is 0.54, which is larger than the unadjusted effect of 0.41, not smaller.

  • Table 2 caption conflicts on the number of studies

    Minor

    The caption calls Table 2 a description of 13 studies, while its parenthetical says 14 studies; the results also report 14.

  • Limitation section gives a different effect-size count

    Minor

    The limitation section reports 37 effect sizes, but the results report 44 effects in the meta-analysis.

  • Table 2 caption conflicts on the number used for analysis

    Minor

    The caption says 18 was used for the meta-analysis after stating there were 19 treatments. Elsewhere, the results describe 19 treatment-control comparisons without explaining a reduction to 18.

  • RCT appears in the nonrandomized risk-of-bias table

    Minor

    Table 4 is explicitly designated for nonrandomized studies but includes Vowels et al, which Table 1 identifies as an RCT.

  • Vowels study is described as measuring only positive well-being

    Minor

    The stated reason for excluding Vowels et al is that it focused on positive well-being only, but Table 1 lists distress among its targeted outcomes.

9 of 9

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Popay J Roberts H Sowden A Petticrew M Arai L Rodgers M et al. Guidance on the conduct of narrative synthesis in systematic reviews a product from the ESRC methods programme. Semantic Scholar. 2006. URL: https://www.semanticscholar.org/paper/Guidance-on-the-Conduct-of-Narrative-Synthesis-in-A-Popay-Roberts/ed8b23836338f6fdea0cc55e161b0fc5805f9e27 [Accessed 12-12-2025]
  2. Lipsey MW Wilson DB . Practical Meta-Analysis. Thousand Oaks (CA): Sage Publications; 2001.
  3. Çakmak, Novitas-ROYAL (Research on Youth and Language)
  4. Romanovskyi O Pidbutska N Knysh A . Elomia chatbot: the effectiveness of artificial intelligence in the fight for mental health. Presented at: COLINS-2021: 5th International Conference on Computational Linguistics and Intelligent Systems; Apr 22-23, 2021; Lviv, Ukraine. URL: https://ceur-ws.org/Vol-2870/paper89.pdf [Accessed 08-10-2025]
  5. Roohafza, J Res Med Sci, 19, 944
  6. Kupfer, J Clin Psychiatry, 52 Suppl
  7. Chentsova-Dutton YE Ryder AG . Cultural-clinical psychology: from cultural scripts to contextualized treatments. In: Handbook of Cultural Psychology. 2nd ed. Guilford Press; 2020:732-759.

Cited by

Sorted by citations

Authors
Zhang, Q., Zhang, R., Xiong, Y., Sui, Y., Tong, C., & Lin, F.-H.
Affiliations
  • National University of Singapore (SG)
  • Nanyang Technological University (SG)
  • Johns Hopkins University (US)
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2025-12-16
Volume
27
Pages
e78238-e78238
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/78238
PMC
PMC12707440
OpenAlex
W4417457748
Semantic Scholar
6479fead1928765413454ae4d4fac524831a9a8b
Cited by
56
References
83
Citations by year
2026: 55 · 2025: 1
FWCI
8.66
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • DR-NTU (Nanyang Technological University) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
generative AI chatbots, mental health chatbots, depression, anxiety, mental health interventions, chatbot effectiveness, digital mental health
MeSH terms
Generative Artificial Intelligence, Artificial Intelligence, Mental Disorders, Humans, Mental Health, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review