Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 0

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Engagement, Feasibility, and Perceived Alliance With a Large Language Model–Enabled Mental Health Voice and Chatbot: Exploratory Longitudinal Observational Study

Schäfer, Lea Maria, Krause, Tabea, Köhler, Stephan·2026·JMIR Formative Research
DOI 10.2196/89708
Open this paper in CoChat — free account

Abstract

Background: Conversational AI systems are increasingly used for web-based mental health support, yet evidence on sustained engagement, therapeutic alliance, and longitudinal outcomes remains limited. Baseline data from 527 self-referred users of the large language model–enabled chatbot Clare indicated high psychological distress and early development of a subjective working alliance. Objective: This exploratory study examined the feasibility and descriptive outcome patterns associated with the use of the multimodal conversational agent Clare over 4 and 8 weeks in a real-world setting, focusing on engagement, perceived working alliance, and mental health measures. Methods: We conducted a naturalistic, single-arm, repeated-measures observational study of self-referred, English-speaking adults recruited online via social media and search advertisements. Participants interacted with Clare over 8 weeks. Two analytic samples were defined by engagement and assessment completion: a 4-week sample (active engagement plus completed 4-week assessment) and an 8-week sample (additionally requiring continued engagement and completed 8-week assessment). Outcomes were assessed at baseline, 4, and 8 weeks: engagement metrics (calls, messages, and call duration), alliance measures (Working Alliance Inventory–Short Revised and Mobile Agnew Relationship Measure), and mental health measures (Patient Health Questionnaire–4, Patient Health Questionnaire–Distress, Mini–Social Phobia Inventory, UCLA Loneliness Scale, and Satisfaction With Life Scale). Changes were evaluated using 2-tailed paired-samples t tests with effect sizes; given substantial attrition and reliance on complete-case analyses, linear mixed-effects models were run as sensitivity analyses, and findings warrant cautious interpretation. Results: Of 527 baseline participants, 369 enrolled and 348 actively engaged with Clare; 53 out of 348 (15.2%) met the 4-week criteria and 21 out of 348 (6.0%) met the 8-week criteria. Engagement peaked in week 1 (calls: mean 1.77, SD 1.52; messages: mean 10.02, SD 16.68) and declined thereafter. Among T1-T2 complete pairs (n=52), mean Patient Health Questionnaire–Distress scores decreased by 1.13 (95% CI 0.25 to 2.02) points, Patient Health Questionnaire–4 scores decreased by 0.69 (95% CI −0.01 to 1.40) points, Mini–Social Phobia Inventory scores decreased by 1.73 (95% CI 0.88 to 2.58) points, Satisfaction With Life Scale scores increased by 6.23 (95% CI 4.70 to 7.76) points, and UCLA scores increased by 0.48 (95% CI 0.00 to 0.96) points. Among participants with Working Alliance Inventory–Short Revised data (n=53), total scores increased by 0.20 points from T1b to T2 (95% CI −0.01 to 0.42); among the 8-week retained subgroup (n=21), Mobile Agnew Relationship Measure scores increased by 0.22 points from T2 to T3 (95% CI 0.03 to 0.42). Conclusions: Sustained engagement was limited, and outcome estimates were obtained from small, highly selected follow-up samples. The observed changes cannot be attributed to Clare because the study had no control group and substantial, nonrandom attrition.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/89708

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 67

  1. Waumans, Barriers and facilitators for treatment-seeking in adults with a depressive or anxiety disorder in a Western-European health care setting: a qualitative study, BMC Psychiatry, 22, 10.1186/s12888-022-03806-5 · DOI
  2. Cuijpers, How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence, World Psychiatry, 15, 245, 10.1002/wps.20346 · DOI · Paper page
  3. Cuijpers, Cognitive behavior therapy vs. control conditions, other psychotherapies, pharmacotherapies and combined treatment for depression: a comprehensive meta‐analysis including 409 trials with 52,702 patients, World Psychiatry, 22, 105, 10.1002/wps.21069 · DOI · Paper page
  4. Schaeuffele, A systematic review and meta-analysis of transdiagnostic cognitive behavioural therapies for emotional disorders, Nat Hum Behav, 8, 493, 10.1038/s41562-023-01787-3 · DOI · Paper page
  5. Simon, Management of depression in adults: a review, JAMA, 332, 141, 10.1001/jama.2024.5756 · DOI · Paper page
  6. Edge, The efficacy of self-guided internet and mobile-based interventions for preventing anxiety and depression - a systematic review and meta-analysis, Behav Res Ther, 164, 10.1016/j.brat.2023.104292 · DOI · Paper page
  7. He, Conversational agent interventions for mental health problems: systematic review and meta-analysis of randomized controlled trials, J Med Internet Res, 25, 10.2196/43862 · DOI · Paper page
  8. Li, Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being, NPJ Digit Med, 6, 10.1038/s41746-023-00979-5 · DOI · Paper page
  9. Diaz-Asper, A framework for language technologies in behavioral research and clinical applications: ethical challenges, implications, and solutions, Am Psychol, 79, 79, 10.1037/amp0001195 · DOI · Paper page
  10. Deng Y Liao L Zheng Z Yang GH Chua TS . Towards human-centered proactive conversational agents. Presented at: SIGIR ’24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval; Jul 14-18, 2024; Washington, DC, USA. [doi: 10.1145/3626772.3657843] · DOI
  11. Salah, The good, the bad, and the GPT: reviewing the impact of generative artificial intelligence on psychology, Curr Opin Psychol, 59, 10.1016/j.copsyc.2024.101872 · DOI
  12. Torous, Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis, J Affect Disord, 263, 10.1016/j.jad.2019.11.167 · DOI · Paper page
  13. Dosovitsky, Bonding with bot: user feedback on a chatbot for social isolation, Front Digit Health, 3, 10.3389/fdgth.2021.735053 · DOI
  14. Schäfer, Exploring user characteristics, motives, and expectations and the therapeutic alliance in the mental health conversational AI Clare: a baseline study, Front Digit Health, 7, 10.3389/fdgth.2025.1576135 · DOI
  15. Starke, Risks and protective measures for synthetic relationships, Nat Hum Behav, 8, 1834, 10.1038/s41562-024-02005-4 · DOI · Paper page
  16. Baier, Therapeutic alliance as a mediator of change: a systematic review and evaluation of research, Clin Psychol Rev, 82, 10.1016/j.cpr.2020.101921 · DOI · Paper page
  17. Herbener, A theoretical framework of the processes of change in mental health interventions delivered by artificial therapists, J Psychol AI, 1, 2590495, 10.1080/29974100.2025.2590495 · DOI
  18. Chiauzzi, Demographic and clinical characteristics associated with anxiety and depressive symptom outcomes in users of a digital mental health intervention incorporating a relational agent, BMC Psychiatry, 24, 10.1186/s12888-024-05532-6 · DOI
  19. Kim, Therapeutic potential of social chatbots in alleviating loneliness and social anxiety: quasi-experimental mixed methods study, J Med Internet Res, 27, 10.2196/65589 · DOI · Paper page
  20. Sobowale, Evaluating the quality of psychotherapy conversational agents: framework development and cross-sectional study, JMIR Form Res, 9, 10.2196/65605 · DOI · Paper page
  21. Heng, Attachment anxiety and problematic use of conversational artificial intelligence: mediation of emotional attachment and moderation of anthropomorphic tendencies, Psychol Res Behav Manag, 18, 10.2147/PRBM.S531805 · DOI · Paper page
  22. Wu, Trust, anxious attachment, and conversational AI adoption intentions in digital counseling: a preliminary cross-sectional questionnaire study, JMIR AI, 4, 10.2196/68960 · DOI
  23. Schäfer, Verhaltensther Psychosoz Prax, 57
  24. Leslie D . Understanding artificial intelligence ethics and safety: a guide for the responsible design and implementation of AI systems in the public sector. SSRN. Preprint posted online on Jun 10, 2019. [doi: 10.2139/ssrn.3403301] · DOI · Paper page
  25. Stade, Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation, Npj Mental Health Res, 3, 12, 10.1038/s44184-024-00056-z · DOI · Paper page
  26. De la Barrera, Using ecological momentary assessment and machine learning techniques to predict depressive symptoms in emerging adults, Psychiatry Res, 332, 10.1016/j.psychres.2023.115710 · DOI
  27. Shiffman, Ecological momentary assessment, Annu Rev Clin Psychol, 4, 10.1146/annurev.clinpsy.3.022806.091415 · DOI · Paper page
  28. Dosovitsky, Artificial intelligence chatbot for depression: descriptive study of usage, JMIR Form Res, 4, 10.2196/17065 · DOI · Paper page
  29. Martinengo, Evaluation of chatbot-delivered interventions for self-management of depression: content analysis, J Affect Disord, 319, 10.1016/j.jad.2022.09.028 · DOI · Paper page
  30. Hatcher, Development and validation of a revised short version of the Working Alliance Inventory, Psychother Res, 16, 12, 10.1080/10503300500352500 · DOI · Paper page
  31. Clarke, Therapeutic alliance with a fully automated mobile phone and web-based intervention: secondary analysis of a randomized controlled trial, JMIR Ment Health, 3, 10.2196/mental.4656 · DOI · Paper page
  32. Kay-Lambkin, Acceptability of a clinician-assisted computerized psychological intervention for comorbid mental health and substance use problems: treatment adherence data from a randomized controlled trial, J Med Internet Res, 13, 10.2196/jmir.1522 · DOI · Paper page
  33. Grodniewicz, Waiting for a digital therapist: three challenges on the path to psychotherapy delivered by artificial intelligence, Front Psychiatry, 14, 10.3389/fpsyt.2023.1190084 · DOI · Paper page
  34. Kroenke, An ultra-brief screening scale for anxiety and depression: the PHQ–4, Psychosomatics, 50, 613, 10.1176/appi.psy.50.6.613 · DOI · Paper page
  35. Connor, Psychometric properties of the Social Phobia Inventory (SPIN). New self-rating scale, Br J Psychiatry, 176, 10.1192/bjp.176.4.379 · DOI · Paper page
  36. Hughes, A short scale for measuring loneliness in large surveys, Res Aging, 26, 655, 10.1177/0164027504268574 · DOI · Paper page
  37. Diener, The Satisfaction with Life Scale, J Pers Assess, 49, 71, 10.1207/s15327752jpa4901_13 · DOI · Paper page
  38. Jasper, The working alliance in a randomized controlled trial comparing Internet-based self-help and face-to-face cognitive behavior therapy for chronic tinnitus, Internet Interv, 1, 49, 10.1016/j.invent.2014.04.002 · DOI · Paper page
  39. Jabir, Attrition in conversational agent–delivered mental health interventions: systematic review and meta-analysis, J Med Internet Res, 26, 10.2196/48168 · DOI
  40. Lee, Designing a chatbot as a mediator for promoting deep self-disclosure to a real mental health professional, Proc ACM Hum Comput Interact, 4, 10.1145/3392836 · DOI
  41. Nwosu, Digital therapeutics for mental health: is attrition the Achilles heel?, Front Psychiatry, 13, 10.3389/fpsyt.2022.900615 · DOI · Paper page
  42. Fernandez, Meta-analysis of dropout from cognitive behavioral therapy: magnitude, timing, and moderators, J Consult Clin Psychol, 83, 1108, 10.1037/ccp0000044 · DOI · Paper page
  43. Kumar, Large language model agents for improving engagement with behavior change interventions: application to digital mindfulness, Proc ACM Hum Comput Interact, 9, 1, 10.1145/3757619 · DOI
  44. Xue F Fu Y Zhou W Zheng Z You Y . To repeat or not to repeat: insights from scaling LLM under token-crisis. Presented at: NIPS ’23: Proceedings of the 37th International Conference on Neural Information Processing Systems; Dec 10-16, 2023; New Orleans, Louisiana, USA. [doi: 10.5555/3666122.3668712] · doi:10.5555/3666122.3668712
  45. Yao J Yang S Xu J Hu L Li M Wang D . Understanding the repeat curse in large language models from a feature perspective. In: Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics; 2025:7787-7815. [doi: 10.18653/v1/2025.findings-acl.406] · DOI
  46. Linardon, Attrition and adherence in smartphone-delivered interventions for mental health problems: a systematic and meta-analytic review, J Consult Clin Psychol, 88, 1, 10.1037/ccp0000459 · DOI · Paper page
  47. Lee YK Suh J Zhan H Li JJ Ong DC . Large language models produce responses perceived to be empathic. Presented at: 2024 12th International Conference on Affective Computing and Intelligent Interaction (ACII); Sep 15-18, 2024; Glasgow, United Kingdom. [doi: 10.1109/ACII63134.2024.00012] · DOI
  48. Tkalac Verčič, Situational dependence of internal communication channels, Public Relat Rev, 51, 102608, 10.1016/j.pubrev.2025.102608 · DOI
  49. Zhang L Yu J Zhang S et al. Unveiling the impact of multi-modal interactions on user engagement: a comprehensive evaluation in AI-driven conversations. arXiv. Preprint posted online on Jun 21, 2024. [doi: 10.48550/arXiv.2406.15000] · DOI
  50. Smith, Engagement and attrition in digital mental health: current challenges and potential solutions, NPJ Digit Med, 8, 10.1038/s41746-025-01778-w · DOI
  51. Feng, The effectiveness of AI chatbots in alleviating mental distress and promoting health behaviors among adolescents and young adults: systematic review and meta-analysis, J Med Internet Res, 27, 10.2196/79850 · DOI · Paper page
  52. Cross, The digital cumulative complexity model: a framework for improving engagement in digital mental health interventions, Front Psychiatry, 15, 10.3389/fpsyt.2024.1382726 · DOI
  53. Zantvoort K . Machine learning dropout predictions for personalizing digital mental health interventions [Dissertation]. Leuphana Universität Lüneburg; 2025. [doi: 10.48548/pubdata-1596] · DOI
  54. Stieglitz, Design principles for conversational agents to support Emergency Management Agencies, Int J Inf Manage, 63, 10.1016/j.ijinfomgt.2021.102469 · DOI
  55. Sung J Christensen HI Grinter RE . Robots in the wild: understanding long-term use. Presented at: HRI ’09: Proceedings of the 4th ACM/IEEE international conference on Human robot interaction; Mar 9-13, 2009; La Jolla, CA, USA. [doi: 10.1145/1514095.1514106] · DOI
  56. Prusiński, Factors promoting alliance quality: differentiation of therapeutic alliance according to the formal aspects of the psychotherapeutic process and demographic variables, Eur J Psychiatry, 37, 190, 10.1016/j.ejpsy.2022.11.002 · DOI
  57. Bordin, The generalizability of the psychoanalytic concept of the working alliance, Psychother Theory Res Pract, 16, 252, 10.1037/h0085885 · DOI · Paper page
  58. Xu, The digital therapeutic alliance with mental health chatbots: diary study and thematic analysis, JMIR Ment Health, 12, 10.2196/76642 · DOI
  59. Beatty, Evaluating the therapeutic alliance with a free-text CBT conversational agent (Wysa): a mixed-methods study, Front Digit Health, 4, 10.3389/fdgth.2022.847991 · DOI · Paper page
  60. Darcy, Evidence of human-level bonds established with a digital conversational agent: cross-sectional, retrospective observational study, JMIR Form Res, 5, 10.2196/27868 · DOI · Paper page
  61. Hoffman, User engagement clusters of an 8-week digital mental health intervention guided by a relational agent (Woebot): exploratory study, J Med Internet Res, 25, 10.2196/47198 · DOI
  62. Ebner P Szczuka J . Predicting romantic human-chatbot relationships: a mixed-method study on the key psychological factors. arXiv. Preprint posted online on Feb 28, 2025. [doi: 10.48550/arXiv.2503.00195] · DOI · Paper page
  63. Jagoe, Depersonalized intimacy: the cases of Sherry Turkle and Spike Jonze, Engl Stud Can, 42, 155, 10.1353/esc.2016.0004 · DOI
  64. Diebel, When AI-based agents are proactive: implications for competence and system satisfaction in human–AI collaboration, Bus Inf Syst Eng, 68, 259, 10.1007/s12599-024-00918-y · DOI
  65. Dwyer, Suitability of text-based communications for the delivery of psychological therapeutic services to rural and remote communities: scoping review, JMIR Ment Health, 8, 10.2196/19478 · DOI
  66. Islam R Bae SW . Revolutionizing mental health support: an innovative affective mobile framework for dynamic, proactive, and context-adaptive conversational agents. arXiv. Preprint posted online on Jun 22, 2024. [doi: 10.48550/arXiv.2406.15942] · DOI
  67. Kozlov, Assessing the care modality preferences and predictors for digital mental health treatment seekers in a technology-enabled stepped care delivery system: cross-sectional study, JMIR Form Res, 5, 10.2196/30162 · DOI

Metadata

Authors
Schäfer, L. M., Krause, T., & Köhler, S.
Venue
JMIR Formative Research
Publisher
JMIR Publications Inc.
Published
2026-11-09
Volume
10
Pages
e89708-e89708
ISSN
2561-326X
DOI
10.2196/89708
Cited by
0
References
4
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.