CoChat
Sign Up
PreprintCited by 78Digital Mental Health InterventionsArtificial Intelligence in Healthcare and EducationEthics and Social Impacts of AI

Benefits and Harms of Large Language Models in Digital Mental Health

Open access
De Choudhury, Munmun, Pendse, Sachin R, Kumar, Neha·2023
DOI 10.31234/osf.io/y8ax9
Open this paper in CoChat — free accountOpen PDF

Abstract

The past decade has been transformative for mental health research and practice. The ability to harness large repositories of data, whether from electronic health records (EHR), mobile devices, or social media, has revealed a potential for valuable insights into patient experiences, promising early, proactive interventions, as well as personalized treatment plans. Recent developments in generative artificial intelligence, particularly large language models (LLMs), show promise in leading digital mental health to uncharted territory. Patients are arriving at doctors' appointments with information sourced from chatbots, state-of-the-art LLMs are being incorporated in medical software and EHR systems, and chatbots from an ever-increasing number of startups promise to serve as AI companions, friends, and partners. This article presents contemporary perspectives on the opportunities and risks posed by LLMs in the design, development, and implementation of digital mental health tools. We adopt an ecological framework and draw on the affordances offered by LLMs to discuss four application areas---care-seeking behaviors from individuals in need of care, community care provision, institutional and medical care provision, and larger care ecologies at the societal level. We engage in a thoughtful consideration of whether and how LLM-based technologies could or should be employed for enhancing mental health. The benefits and harms our article surfaces could serve to help shape future research, advocacy, and regulatory efforts focused on creating more responsible, user-friendly, equitable, and secure LLM-based tools for mental health treatment and intervention.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31234/osf.io/y8ax9

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hallucinations may be especially harmful in psychiatric crisis situations and for non-English-speaking users, due to less non-English training data.

    “Hallucinations may be especially harmful in psychiatric crisis situations and for non-English-speaking users, due to less non-English training data50.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [50]

    Section 4.2, Potential Harms (page unavailable), warns that hallucinated crisis responses could harm callers. Section 5.2, Potential Harms (page unavailable), says hallucinated information in non-English health contexts disadvantages non-English-speaking patients. The passages do not attribute that problem to a smaller amount of non-English training data. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Biases can occur via underrepresentation of certain groups in training data, training data including stigmatizing representations, and sycophantic reflection of users' expressed biases.

    “Specifically, biases can occur via underrepresentation of certain groups in training data, training data including stigmatizing representations, and sycophantic reflection of users’ expressed biases50,51.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [50]

    The “Demographic and Representational Biases” section (page unavailable) says that skewed demographic representation and stigmatizing language in training data can produce biased outputs. It also describes LLMs perpetuating patients’ or providers’ pre-existing biases through “echo chambers,” but does not specifically describe sycophantic responses to biases users express during an interaction. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • One article suggested that LLM-CB use for mental health by patients may harm existing patient-psychotherapist relationships, for example, if therapists and LLM-CBs disagree or if their advice diverges.

    “It has been suggested by one article that LLM-CB use for mental health by patients may harm existing patient-psychotherapist relationships, for example, if therapists and LLM-CBs disagree or if their advice diverges50.”

    From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [50]

    The section “An Erosion of the Therapeutic Alliance” says LLM chatbots risk taking away the human element of psychotherapy and discusses the importance of trust between therapists and clients. It does not specifically say that patients’ chatbot use may damage an existing patient–psychotherapist relationship when the chatbot and therapist disagree or give divergent advice. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 47

  1. Moura, I., Teles, A., Viana, D., Marques, J., Coutinho, L. & Silva, F. Digital phenotyping of mental health using multimodal sensing of multiple situations of interest: A systematic literature review. Journal of Biomedical Informatics, 104278 (2022).
  2. De Choudhury, M., Counts, S. & Horvitz, E. Predicting postpartum changes in emotion and behavior via social media in Proceedings of the SIGCHI conference on human factors in computing systems (2013), 3267-3276.
  3. De Choudhury, M., Kiciman, E., Dredze, M., Coppersmith, G. & Kumar, M. Discovering shifts to suicidal ideation from mental health content in social media in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (2016), 2098-2110.
  4. Bossewitch, J., Brown, L. X., Gooding, P. M., Harris, L., Horton, J., Katterl, S., Myrick, K., Ubozoh, K. & Vasquez Encalada, A. Digital Futures in Mind: Reflecting on Technological Experiments in Mental Health & Crisis Support. Available at SSRN 4215994 (2022).
  5. Thieme, A., Belgrave, D. & Doherty, G. Machine learning in mental health: A systematic review of the HCI literature to support the development of effective and implementable ML systems. ACM Transactions on Computer-Human Interaction (TOCHI) 27, 1-53 (2020).
  6. Crasto, R., Dias, L., Miranda, D. & Kayande, D. CareBot: A Mental Health ChatBot in 2021 2nd International Conference for Emerging Technology (INCET) [Internet]. [cited 2023 Mar (IEEE, Belagavi, India, 2021), 1-5.
  7. O'Leary, K. Human-AI collaboration boosts mental health support. Nat Med. https : //www.nature.com/articles/d41591-023-00022-w (2023).
  8. Zack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. Coding Inequity: Assessing GPT- 4's Potential for Perpetuating Racial and Gender Biases in Healthcare. medRxiv, 2023-07 (2023).
  9. Omiye, J. A., Lester, J., Spichak, S., Rotemberg, V. & Daneshjou, R. Beyond the hype: large language models propagate race-based medicine. medRxiv, 2023-07 (2023).
  10. Chen, Y., Mendes, E., Das, S., Xu, W. & Ritter, A. Can Language Models be Instructed to Protect Personal Information? arXiv preprint arXiv:2310.02224 (2023). · arXiv
  11. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? in Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (2021), 610-623.
  12. Colby, K. M., Watt, J. B. & Gilbert, J. P. A computer method of psychotherapy: Preliminary communication. The Journal of Nervous and Mental Disease 142, 148-152 (1966).
  13. Ma, Z., Mei, Y. & Su, Z. Understanding the benefits and challenges of using large lan- guage model-based conversational agents for mental well-being support. arXiv preprint arXiv:2307.15810 (2023). · arXiv
  14. Vallée, J., Shareck, M., Kestens, Y. & Frohlich, K. L. Everyday geography and service ac- cessibility: the contours of disadvantage in relation to mental health. Annals of the American Association of Geographers 112, 931-947 (2022).
  15. Rochford, B., Pendse, S., Kumar, N. & De Choudhury, M. Leveraging symptom search data to understand disparities in US mental health care: demographic analysis of search engine trace data. JMIR Mental Health 10, e43253 (2023).
  16. Scrutton, A. P. in The Routledge handbook of epistemic injustice 347-355 (Routledge, 2017).
  17. Pendse, S. R., Kumar, N. & De Choudhury, M. Marginalization and the Construction of Mental Illness Narratives Online: Foregrounding Institutions in Technology-Mediated Care. Proceedings of the ACM on Human-Computer Interaction 7, 1-30 (2023).
  18. Hsu, S.-L., Shah, R. S., Senthil, P., Ashktorab, Z., Dugan, C., Geyer, W. & Yang, D. Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback. arXiv preprint arXiv:2305.08982 (2023). · arXiv
  19. Sharma, A., Rushton, K., Lin, I. W., Wadden, D., Lucas, K. G., Miner, A. S., Nguyen, T. & Althoff, T. Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction. arXiv preprint arXiv:2305.02466 (2023). · arXiv
  20. Ziems, C., Li, M., Zhang, A. & Yang, D. Inducing positive perspectives with text reframing. arXiv preprint arXiv:2204.02952 (2022). · arXiv
  21. Butler, A. C., Chapman, J. E., Forman, E. M. & Beck, A. T. The empirical status of cognitive- behavioral therapy: A review of meta-analyses. Clinical psychology review 26, 17-31 (2006).
  22. Black, V. E. De-humanizing Care: An Ethnography of Mental Health Artificial Intelligence PhD thesis (University of California, Berkeley, 2023).
  23. Stanhope, V., Barrenger, S. L., Salzer, M. S. & Marcus, S. C. Examining the relationship between choice, therapeutic alliance and outcomes in mental health services. Journal of Personalized Medicine 3, 191-202 (2013).
  24. Lambert, M. J. & Barley, D. E. Research summary on the therapeutic relationship and psy- chotherapy outcome. Psychotherapy: Theory, research, practice, training 38, 357 (2001).
  25. Federal Trade Commission. FTC to Ban BetterHelp from Revealing Consumers' Data, In- cluding Sensitive Mental Health Information, to Facebook and Others for Targeted Adver- tising. Press Release, Federal Trade Commission (2023).
  26. De Freitas, J., Uguralp, A. K., Oguz-Uguralp, Z. & Puntoni, S. Chatbots and Mental Health: Insights into the Safety of Generative AI. Journal of Consumer Psychology (2022).
  27. Pendse, S. R., Sharma, A., Vashistha, A., De Choudhury, M. & Kumar, N. "Can I not be suicidal on a Sunday?": understanding technology-mediated pathways to mental health support in Proceedings of the 2021 CHI Conference on Human Factors in Computing Sys- tems (2021), 1-16.
  28. Matthews, S., Cantor, J. H., Holliday, S. B., Bialas, A., Eberhart, N. K., Breslau, J. & McBain, R. K. National preparedness for 988-the new mental health emergency hotline in the United States. Preventive medicine reports 33, 102208 (2023).
  29. Skorburg, J. A. & Friesen, P. Mind the Gaps: Ethical and Epistemic Issues in the Digital Mental Health Response to Covid-19. Hastings Center Report 51, 23 (2021).
  30. Hoffberg, A. S., Stearns-Yoder, K. A. & Brenner, L. A. The effectiveness of crisis line services: a systematic review. Frontiers in public health 7, 399 (2020).
  31. Pendse, S. R., Lalani, F. M., De Choudhury, M., Sharma, A. & Kumar, N. " Like Shock Ab- sorbers": understanding the human infrastructures of technology-mediated mental health support in Proceedings of the 2020 CHI Conference on Human Factors in Computing Sys- tems (2020), 1-14.
  32. Chakrabarty, T., Hidey, C. & Muresan, S. ENTRUST: Argument reframing with language models and entailment. arXiv preprint arXiv:2103.06758 (2021). · arXiv
  33. Maddela, M., Ung, M., Xu, J., Madotto, A., Foran, H. & Boureau, Y.-L. Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts. arXiv preprint arXiv:2307.02768 (2023). · arXiv
  34. Sharma, E. & De Choudhury, M. Mental Health Support and its Relationship to Linguistic Accommodation in Online Communities in CHI (2018).
  35. Cheng, Y. & Jiang, H. AI-Powered mental health chatbots: Examining users' motivations, active communicative action and engagement after mass-shooting disasters. Journal of Con- tingencies and Crisis Management 28, 339-354 (2020).
  36. Ehsan, U., Wintersberger, P., Liao, Q. V., Watkins, E. A., Manger, C., Daumé III, H., Riener, A. & Riedl, M. O. Human-Centered Explainable AI (HCXAI): beyond opening the black- box of AI in CHI conference on human factors in computing systems extended abstracts (2022), 1-7.
  37. Turkington, R., Mulvenna, M. D., Bond, R. R., O'Neill, S., Potts, C., Armour, C., Ennis, E. & Millman, C. Why do people call crisis helplines? Identifying taxonomies of presenting reasons and discovering associations between these reasons. Health informatics journal 26, 2597-2613 (2020).
  38. Nissenbaum, H. Privacy as contextual integrity. Wash. L. Rev. 79, 119 (2004).
  39. Li, H., Vincent, N., Chancellor, S. & Hecht, B. The Dimensions of Data Labor: A Road Map for Researchers, Activists, and Policymakers to Empower Data Producers in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023), 1151- 1161.
  40. Yoo, D. W., Woo, H., Pendse, S. R., Lu, N., Birnbaum, M. L., Abowd, G. & De Choudhury, M. Missed Opportunities for Human-Centered AI Research: Understanding Stakeholder Collaboration in Mental Health AI Research. Proceedings of the ACM on Human-Computer Interaction (2024).
  41. Saha, K., Torous, J., Kiciman, E., De Choudhury, M., et al. Understanding side effects of antidepressants: Large-scale longitudinal study on social media data. JMIR mental health 8, e26589 (2021).
  42. Ayvaz, S., Horn, J., Hassanzadeh, O., Zhu, Q., Stan, J., Tatonetti, N. P., Vilar, S., Brochhausen, M., Samwald, M., Rastegar-Mojarad, M., et al. Toward a complete dataset of drug-drug in- teraction information from publicly available sources. Journal of biomedical informatics 55, 206-217 (2015).
  43. Chancellor, S., Nitzburg, G., Hu, A., Zampieri, F. & De Choudhury, M. Discovering Alter- native Treatments for Opioid Use Recovery Using Social Media in Proc. CHI (2019). 118. ElSherief, M., Sumner, S. A., Jones, C. M., Law, R. K., Kacha-Ochana, A., Shieber, L., Cordier, L., Holton, K. & De Choudhury, M. Characterizing and identifying the prevalence of web-based misinformation relating to medication for opioid use disorder: Machine learn- ing approach. Journal of medical Internet research 23, e30753 (2021). 119.
  44. Yoo, D. W., Ernala, S. K., Saket, B., Weir, D., Arenare, E., Ali, A. F., Van Meter, A. R., Birn- baum, M. L., Abowd, G. D. & De Choudhury, M. Clinician perspectives on using compu- tational mental health insights from patients' social media activities: design and qualitative evaluation of a prototype. JMIR Mental Health 8, e25455 (2021). 123. Duffourc, M. & Gerke, S. Generative AI in health care and liability risks for physicians and safety concerns for patients. Jama (2023). 124.
  45. Saenz, A. D., Harned, Z., Banerjee, O., Abràmoff, M. D. & Rajpurkar, P. Autonomous AI systems in the face of liability, regulations and costs. NPJ digital medicine 6, 185 (2023). 125. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021). 126. Halleck, S. L. & Halleck, S. L. The Psychiatrist's Liability for Negligent Diagnosis. Law in the Practice of Psychiatry: A Handbook for Clinicians, 65-82 (1980). 127. Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M. & Zubizarreta, J. R. Suicide prediction models: a critical review of recent research with recommendations for the way forward. Molecular psychiatry 25, 168-179 (2020). 128. Large, M. M. The role of prediction in suicide prevention. Dialogues in clinical neuro- science (2018). 129. · arXiv
  46. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D. & Craw- ford, K. Datasheets for datasets. Communications of the ACM 64, 86-92 (2021). 161. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. & Gebru, T. Model cards for model reporting in Proceedings of the conference on fairness, accountability, and transparency (2019), 220-229. 162.
  47. Ajmani, L. H., Chancellor, S., Mehta, B., Fiesler, C., Zimmer, M. & De Choudhury, M. A Systematic Review of Ethics Disclosures in Predictive Mental Health Research in Proceed- ings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023), 1311-1323. 163. Guaglione, S. The case for and against open-source large language models for use in newsrooms -digiday.com https : / / digiday . com / media / the -case -for - and-against-open-source-large-language-models-for-use-in- newsrooms/. [Accessed 05-11-2023]. 164. Harrer, S. Council Post: From Boring And Safe To Exciting And Dangerous: Why Large Language Models Need To Be Regulated -forbes.com https://www.forbes.com/ sites/forbestechcouncil/2023/03/22/from-boring-and-safe-to- exciting-and-dangerous-why-large-language-models-need-to-be- regulated/. [Accessed 05-11-2023]. 165. Joint Statement on AI Safety and Openness -open.mozilla.org https://open.mozilla. org/letter/. [Accessed 05-11-2023]. 166. Biden, J. Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence The White House. Accessed: 2023-11-03. 2023. https : / / www.whitehouse.gov/briefing-room/presidential-actions/2023/ 10/30/executive-order-on-the-safe-secure-and-trustworthy- development-and-use-of-artificial-intelligence/. 167. Press corner -ec.europa.eu https://ec.europa.eu/commission/presscorner/ detail/en/ip_23_5379. [Accessed 06-11-2023]. 168. Meskó, B. & Topol, E. J. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. npj Digital Medicine 6, 120 (2023). 169. Empatica. Empatica's Platform Receives New FDA Clearance for Cardiac Digital Biomark- ers 2023. https : / / www . prnewswire . com / news -releases / empaticas - platform -receives -new -fda -clearance -for -cardiac -digital - biomarkers-301975974.html. 170. Empatica. Empatica Receives New FDA Clearance for Its Health Monitoring Platform and Announces Series B Financing 2022. https : / / www . prnewswire . com / news - releases/empatica-receives-new-fda-clearance-for-its-health- monitoring-platform-and-announces-series-b-financing-301685344. html. 171. Kleinman, A. The soul of care: the moral education of a husband and a doctor (Penguin, 2020).

Cited by

Sorted by citations

Authors
De Choudhury, M., Pendse, S. R., & Kumar, N.
Affiliations
  • Georgia Institute of Technology
  • Georgia Institute of Technology (US)
ORCID
Publisher
Center for Open Science
Published
2023-11-15
DOI
10.31234/osf.io/y8ax9
OpenAlex
W4388733340
Cited by
78
References
28
Citations by year
2026: 35 · 2025: 28 · 2024: 15
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • American Foundation for Suicide Prevention
  • National Institute of Mental Health (R01MH117172, P50MH115838)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, digital mental health, generative AI, chatbots, care-seeking behavior, community-based care, institutional care, ecological model, affordances, electronic health records, AI companions, benefits and harms, AI regulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review