CoChat
Sign Up
PreprintCited by 40Digital Mental Health InterventionsSocial Robot Interaction and HRIEthics and Social Impacts of AI

Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support

Open access
Ma, Zilin, Mei, Yiyang, Su, Zhaoyuan·2023·arXiv
DOI 10.48550/arxiv.2307.15810arXiv 2307.15810PMC PMC10785945
Open this paper in CoChat — free accountOpen PDF

Abstract

Conversational agents powered by large language models (LLM) have increasingly been utilized in the realm of mental well-being support. However, the implications and outcomes associated with their usage in such a critical field remain somewhat ambiguous and unexplored. We conducted a qualitative analysis of 120 posts, encompassing 2917 user comments, drawn from the most popular subreddit focused on mental health support applications powered by large language models (u/Replika). This exploration aimed to shed light on the advantages and potential pitfalls associated with the integration of these sophisticated models in conversational agents intended for mental health support. We found the app (Replika) beneficial in offering on-demand, non-judgmental support, boosting user confidence, and aiding self-discovery. Yet, it faced challenges in filtering harmful content, sustaining consistent communication, remembering new information, and mitigating users' overdependence. The stigma attached further risked isolating users socially. We strongly assert that future researchers and designers must thoroughly evaluate the appropriateness of employing LLMs for mental well-being support, ensuring their responsible and effective application.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.15810

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Companion AIs may provide harmful content and reinforce dependence.

    “Meanwhile, surveys and forum analyses on companion AI use found that usage was generally associated with negative mental health81 and that companion AIs may provide harmful content and reinforce dependence82.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [82]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. Caldeira C, Chen Y, Chan L, Pham V, Chen Y, Zheng K. Mobile apps for mood tracking: an analysis of features and user reviews. AMIA Annu Symp Proc AMIA Symp. 2017;2017:495-504.
  2. Qu C, Sas C, Doherty G. Exploring and Designing for Memory Impairments in Depression. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems [Internet]. Glasgow Scotland Uk: ACM; 2019 [cited 2021 Nov 10]. p. 1-15. Available from: https://dl.acm.org/doi/10.1145/3290605.3300740 · DOI
  3. Su Z, Schneider JA, Young SD. The Role of Conversational Agents for Substance Use Disorder in Social Distancing Contexts. Subst Use Misuse [Internet]. 2021 Sep 19 [cited 2023 Mar 14];56(11):1732-5. Available from: https://www.tandfonline.com/doi/full/10.1080/10826084.2021.1949609 · DOI
  4. Huang M, Zhu X, Gao J. Challenges in Building Intelligent Open-domain Dialog Systems. ACM Trans Inf Syst [Internet]. 2020 Jul 31 [cited 2023 Mar 14];38(3):1-32. Available from: https://dl.acm.org/doi/10.1145/3383123 · DOI · Paper page
  5. Su Z, Figueiredo MC, Jo J, Zheng K, Chen Y. Analyzing Description, User Understanding and Expectations of AI in Mobile Health Applications. AMIA Annu Symp Proc AMIA Symp. 2020;2020:1170-9.
  6. Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput Surv [Internet]. 2023 Sep 30 [cited 2023 Mar 14];55(9):1-35. Available from: https://dl.acm.org/doi/10.1145/3560815 · DOI · Paper page
  7. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language Models are Few-Shot Learners. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, editors. Advances in Neural Information Processing Systems [Internet].
  8. Curran Associates, Inc.; 2020. p. 1877-901. Available from: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  9. Pilault J, Li R, Subramanian S, Pal C. On Extractive and Abstractive Neural Document Summarization with Transformer Language Models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) [Internet]. Online: Association for Computational Linguistics; 2020 [cited 2023 Mar 14]. p. 9308-19. Available from: https://www.aclweb.org/anthology/2020.emnlp-main.748
  10. Chen M, Tworek J, Jun H, Yuan Q, Pinto HP de O, Kaplan J, et al. Evaluating Large Language Models Trained on Code [Internet]. arXiv; 2021 [cited 2023 Mar 14]. Available from: http://arxiv.org/abs/2107.03374 · arXiv
  11. Crasto R, Dias L, Miranda D, Kayande D. CareBot: A Mental Health ChatBot. In: 2021 2nd International Conference for Emerging Technology (INCET) [Internet]. Belagavi, India: IEEE; 2021 [cited 2023 Mar 14]. p. 1-5. Available from: https://ieeexplore.ieee.org/document/9456326/
  12. O'Leary K. Human-AI collaboration boosts mental health support. Nat Med [Internet]. 2023 Feb 27 [cited 2023 Mar 14];d41591-023-00022-w. Available from: https://www.nature.com/articles/d41591-023-00022-w
  13. Bender EM, Gebru T, McMillan-Major A, Shmitchell S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency [Internet]. Virtual Event Canada: ACM; 2021 [cited 2023 Mar 14]. p. 610-23. Available from: https://dl.acm.org/doi/10.1145/3442188.3445922 · DOI · Paper page
  14. Hussain MI, Figueiredo MC, Tran BD, Su Z, Molldrem S, Eikey EV, et al. A scoping review of qualitative research in JAMIA: past contributions and opportunities for future work. J Am Med Inform Assoc [Internet]. 2021 Feb 15 [cited 2023 Mar 14];28(2):402-13. Available from: https://academic.oup.com/jamia/article/28/2/402/5998477
  15. Ancker JS, Benda NC, Reddy M, Unertl KM, Veinot T. Guidance for publishing qualitative research in informatics. J Am Med Inform Assoc [Internet]. 2021 Nov 25 [cited 2023 Mar 20];28(12):2743-8. Available from: https://academic.oup.com/jamia/article/28/12/2743/6372394
  16. Reynolds TL, Zhang J, Zheng K, Chen Y. Unpacking the Use of Laboratory Test Results in an Online Health Community throughout the Medical Care Trajectory. Proc ACM Hum-Comput Interact [Internet]. 2022 Nov 7 [cited 2023 Mar 14];6(CSCW2):1-32. Available from: https://dl.acm.org/doi/10.1145/3555086 · DOI
  17. Chikersal P, Belgrave D, Doherty G, Enrique A, Palacios JE, Richards D, et al. Understanding Client Support Strategies to Improve Clinical Outcomes in an Online Mental Health Intervention. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems [Internet]. Honolulu HI USA: ACM; 2020 [cited 2021 Nov 10]. p. 1-16. Available from: https://dl.acm.org/doi/10.1145/3313831.3376341 · DOI
  18. Williams M, Moser T. The Art of Coding and Thematic Exploration in Qualitative Research. Int Manag Rev [Internet]. 2019;15(1):45-55,71-72. Available from: http://www.library.hbs.edu/intra/go/abi.html?url=http://search.proquest.com/scholarly-journals/art-coding-them atic-exploration-qualitative/docview/2210886420/se-2?accountid=34924
  19. Prepare for truly useful large language models. Nat Biomed Eng [Internet]. 2023 Feb 1;7(2):85-6. Available from: https://doi.org/10.1038/s41551-023-01012-6 · DOI
  20. Cheng X, Zhang X, Cohen J, Mou J. Human vs. AI: Understanding the impact of anthropomorphism on consumer response to chatbots from the perspective of trust and relationship norms. Inf Process Manag [Internet]. 2022 May [cited 2023 Mar 20];59(3):102940. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0306457322000620
  21. Youn S, Jin SV. "In A.I. we trust?" The effects of parasocial interaction and technopian versus luddite ideological views on chatbot-based customer relationship management in the emerging "feeling economy." Comput Hum Behav [Internet]. 2021 Jun [cited 2023 Mar 20];119:106721. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0747563221000431
  22. Knaak S, Mantler E, Szeto A. Mental illness-related stigma in healthcare: Barriers to access and care and evidence-based solutions. Healthc Manage Forum. 2017 Mar;30(2):111-6.
  23. National Academies of Sciences E and Medicine. Ending Discrimination Against People with Mental and Substance Use Disorders: The Evidence for Stigma Change [Internet]. Washington (DC): National Academies Press (US); 2016. 4 p. Available from: https://www.ncbi.nlm.nih.gov/books/NBK384914/
  24. Jo E, Epstein DA, Jung H, Kim YH. Understanding the Benefts and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention. Proc 2023 CHI Conf Hum Factors Comput Syst. 2023 Mar 20;16.
  25. Veinot TC, Ancker JS, Cole-Lewis H, Mynatt ED, Parker AG, Siek KA, et al. Leveling Up: On the Potential of Upstream Health Informatics Interventions to Enhance Health Equity. Med Care [Internet]. 2019 Jun [cited 2023 Mar 14];57(Suppl 2):S108-14. Available from: https://journals.lww.com/00005650-201906001-00005
  26. Su Z, He L, Jariwala SP, Zheng K, Chen Y. "What is Your Envisioned Future?": Toward Human-AI Enrichment in Data Work of Asthma Care. Proc ACM Hum-Comput Interact [Internet]. 2022 Nov 7 [cited 2023 Mar 14];6(CSCW2):1-28. Available from: https://dl.acm.org/doi/10.1145/3555157 · DOI
  27. Stowell E, Lyson MC, Saksono H, Wurth RC, Jimison H, Pavel M, et al. Designing and Evaluating mHealth Interventions for Vulnerable Populations: A Systematic Review. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems [Internet]. Montreal QC Canada: ACM; 2018 [cited 2023 Mar 14]. p. 1-17. Available from: https://dl.acm.org/doi/10.1145/3173574.3173589 · DOI · Paper page
  28. Veinot TC, Mitchell H, Ancker JS. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc JAMIA. 2018 Aug 1;25(8):1080-8.

Cited by

Sorted by citations

Authors
Ma, Z., Mei, Y., & Su, Z.
Affiliations
  • Harvard University (US)
  • Emory University (US)
  • University of California, Irvine (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
2023
Pages
1105-1114
DOI
10.48550/arxiv.2307.15810
arXiv
2307.15810
PMC
PMC10785945
OpenAlex
W4385473492
Cited by
40
References
4
Citations by year
2026: 2 · 2025: 19 · 2024: 18 · 2023: 1
FWCI
10.31
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, chatbots, Replika, mental health apps, online mental health communities, user experience, overdependence, stigma, content filtering, non-judgmental support, social isolation
MeSH terms
Communication, Humans, Language, Mental Health

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review