CoChat
Sign Up
PreprintCited by 39Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationDigital Mental Health Interventions

Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation

Open access
Grabb, Declan, Lamparth, Max, Vasan, Nina·2024·arXiv
DOI 10.48550/arxiv.2406.11852arXiv 2406.11852
Open this paper in CoChat — free accountOpen PDF

Abstract

Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue and proposes a structured framework that delineates levels of autonomy, outlines ethical requirements, and defines beneficial default behaviors for AI agents in the context of mental health support. We also evaluate fourteen state-of-the-art language models (ten off-the-shelf, four fine-tuned) using 16 mental health-related questionnaires designed to reflect various mental health conditions, such as psychosis, mania, depression, suicidal thoughts, and homicidal tendencies. The questionnaire design and response evaluations were conducted by mental health clinicians (M.D.s). We find that existing language models are insufficient to match the standard provided by human professionals who can navigate nuances and appreciate context. This is due to a range of issues, including overly cautious or sycophantic responses and the absence of necessary safeguards. Alarmingly, we find that most of the tested models could cause harm if accessed in mental health emergencies, failing to protect users and potentially exacerbating existing symptoms. We explore solutions to enhance the safety of current models. Before the release of increasingly task-autonomous AI systems in mental health, it is crucial to ensure that these models can reliably detect and manage symptoms of common psychiatric disorders to prevent harm to users. This involves aligning with the ethical framework and default behaviors outlined in our study. We contend that model developers are responsible for refining their systems per these guidelines to safeguard against the risks posed by current AI technologies to user mental health and safety. Trigger warning: Contains and discusses examples of sensitive mental health topics, including suicide and self-harm.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.11852

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 45 · Cited by 39

Research works printed but not linked: 52

  1. American Medical Association. Doctor Shortages Are Here and They'll Get Worse if We Don't Act. https://www.ama-assn.org/practice-management/sustainability/ doctor-shortages-are-here-and-they-ll-get-worse-if-we-don-t-act, 2022. [Online; accessed 26-March-2024].
  2. American Psychiatric Association. DSM-5. American Psychiatric Association, 2022. Diag- nostic and Statistical Manual of Mental Disorders, Fifth Edition.
  3. Lora Aroyo, Alex Taylor, Mark Diaz, Christopher M Homan, Alicia Parrish, Greg Serapio- Garcia, Vinodkumar Prabhakaran, and Ding Wang. DICES dataset: Diversity in conversa- tional AI evaluation for safety. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
  4. H. Ceren Ates, Cihan Ates, and Can Dincer. Stress monitoring with wearable technology and AI. Nature Electronics, pp. 1-2, 2024.
  5. C. Beard et al. Validation of the PHQ-9 in a psychiatric sample. Journal of Affective Disorders, 193:267-273, 2016.
  6. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations, 2024.
  7. Ryan Carey and Tom Everitt. Human control: definitions and algorithms. Uncertainty in Artificial Intelligence, pp. 271-281, 2023.
  8. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  9. Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Ben- jamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. Black-box access is insufficient for rigorous ai audits. In Proceedings of the 2024 ACM Con- ference on Fairness, Accountability, and Transparency, FAccT '24, pp. 2254-2272. Association for Computing Machinery, 2024.
  10. Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj. Harms from increasingly agentic algorithmic systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT '23, pp. 651-666. Association for Computing Machinery, 2023.
  11. Tracey A Davenport, Haley M LaMonica, Shelley Rowe, Julie Sturgess, Elizabeth M Scott, Grace Lee, Vanessa Wan Sze Cheng, and Ian B Hickie. Innovative preclinic triage system to guide Australians to the right mental health care first time. Australian Health Review, 45 (5):578-583, 2021.
  12. Frederico Soares Falcetta, Fernando Kude de Almeida, Janaína Conceic ¸ão Sutil Lemos, José Roberto Goldim, and Cristiano André da Costa. Automatic documentation of professional health interactions: A systematic review. Artif. Intell. Med., 137(C), mar 2023.
  13. R. Gillon. Medical ethics: Four principles plus attention to scope. https://www.bmj.com/ content/309/6948/184, 1994. [Online; accessed 26-March-2024].
  14. Johanna Habicht, Sruthi Viswanathan, Ben Carrington, Tobias U. Hauser, Ross Harper, and Max Rollwage. Closing the accessibility gap to mental health treatment with a personalized self-referral Chatbot. Nature Medicine, pp. 1-8, 2024.
  15. Salah Hamdoun, Rebecca Monteleone, Terri Bookman, and Katina Michael. AI-based and digital mental health apps: Balancing need and risk. IEEE Technology and Society Magazine, 42(1):25-36, 2023.
  16. Oliver Higgins, Brooke L Short, Stephan K Chalup, and Rhonda L Wilson. Artificial intelligence (AI) and machine learning (ML) based decision support systems in mental health: An integrative review. International Journal of Mental Health Nursing, 32(4):966-978, 2023.
  17. Theresa J Hoeft, John C Fortney, Vikram Patel, and J ürgen Un ützer. Task-sharing approaches to improve mental health care in rural and other low-resource settings: a systematic review. The Journal of Rural Health, 34(1):48-62, 2018.
  18. Balaram Yadav Kasula. Ethical Considerations in the Adoption of Artificial Intelligence for Mental Health Diagnosis. International Journal of Creative Research In Computer Technology and Design, 5(5):1-7, 2023.
  19. Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored language models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, pp. 2362-2373. Association for Computing Machinery, 2024.
  20. Han Li, Renwen Zhang, Yi-Chieh Lee, Robert E. Kraut, and David C. Mohr. Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being. NPJ Digital Medicine, 6(1):236, 2023.
  21. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav San- thanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
  22. Baihan Lin, Djallel Bouneffouf, Guillermo Cecchi, and Kush Varshney. Towards healthy AI: Large language models need therapists too. In Proceedings of the 4th Workshop on Trustwor- thy Natural Language Processing (TrustNLP 2024), pp. 61-70. Association for Computational Linguistics, 2024.
  23. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations, 2024.
  24. Alex Mei, Sharon Levy, and William Wang. ASSERT: Automated safety scenario red teaming for evaluating the robustness of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5831-5847. Association for Computational Linguistics, 2023.
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In 36th Conference on Neural Information Processing Systems, 2022. URL https://proceedings.neurips.cc/paper/2022.
  26. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5336-5358. Association for Computational Linguistics, 2022.
  27. Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST '23, New York, NY, USA, 2023. Association for Computing Machinery.
  28. Pat Pataranutaporn, Ruby Liu, Ed Finn, and Pattie Maes. Influencing human-AI interac- tion by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence, 5(10):1076-1086, 2023.
  29. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419-3448. Association for Computational Linguistics, 2022.
  30. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387-13434. Association for Computational Linguistics, 2023.
  31. Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient benchmarking (of language models). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 2519-2536. Association for Computational Linguistics, 2024.
  32. S. Pfohl et al. A toolbox for surfacing health equity harms and biases in large language models. arXiv preprint arXiv:2403.12025, 2024. · arXiv
  33. Barbara Plank. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10671- 10682, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  34. Yue Qi. Pilot Quasi-Experimental Research on the Effectiveness of the Woebot AI Chat- bot for Reducing Mild Depression Symptoms among Athletes. International Journal of Human-Computer Interaction, pp. 1-8, 2024.
  35. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  36. Costina R. Poetar Raluca Balan, Anca Dobrean. Use of automated conversational agents in improving young population mental health: a scoping review. npj Digital Medicine, 7(1): 75, 2024.
  37. Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. Escalation risks from language models in military and diplomatic decision-making. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, pp. 836-898. Association for Computing Machinery, 2024.
  38. Paul R öttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5377-5400. Association for Computational Linguistics, 2024.
  39. Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. Human-AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1):46-57, 2023.
  40. Jacqueline Sin. An AI chatbot for talking therapy referrals. Nature Medicine, pp. 1-2, 2024.
  41. Dara H Sorkin, Emily A Janio, Elizabeth V Eikey, Margaret Schneider, Katelyn Davis, Stephen M Schueller, Nicole A Stadnick, Kai Zheng, Martha Neary, David Safani, and Dana B Mukamel. Rise in use of digital mental health tools and technologies in the United States during the COVID-19 pandemic: survey study. Journal of Medical Internet Research, 23(4):e26994, 2021.
  42. Betsy Stade, johannes C. Eichstaedt, Jane P. Kim, and Shannon Wiltsey Stirman. Readiness for AI deployment and implementation (READI): A proposed framework for the evaluation of AI-mental health applications. PsyArXiv preprint 10.31234, 2024a.
  43. Elizabeth C. Stade, Shannon Wiltsey Stirman, Lyle H. Ungar, Cody L. Boland, H. Andrew Schwartz, David B. Yaden, João Sedoc, Robert J. DeRubeis, Robb Willer, and Johannes C. Eichstaedt. Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation. npj Mental Health Research, 3(1): 1-12, 2024b.
  44. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  45. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021.
  46. Anja Thieme, Maryann Hanratty, Maria Lyons, Jorge Palacios, Rita Faia Marques, Cecily Morrison, and Gavin Doherty. Designing human-centered AI for mental health: Devel- oping clinically relevant applications for online CBT treatment. ACM Transactions on Computer-Human Interaction, 30(2):1-50, 2023.
  47. Emma L van der Schyff, Brad Ridout, Krestina L Amon, Rowena Forsyth, and Andrew J Campbell. Providing Self-Led Mental Health Support Through an Artificial Intelli- gence-Powered Chat Bot (Leora) to Meet the Demand of Mental Health Care. Journal of Medical Internet Research, 25:e46448, 2023.
  48. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehensive assessment of trustworthiness in GPT models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
  49. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024.
  50. Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ana- niadou. Mentallama: Interpretable mental health analysis on social media with large language models. In Proceedings of the ACM on Web Conference 2024, WWW '24, pp. 4489-4500. Association for Computing Machinery, 2024.
  51. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045, 2023. · arXiv
  52. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. · arXiv

Authors
Grabb, D., Lamparth, M., & Vasan, N.
Affiliations
  • Northwestern University (US)
  • Stanford University (US)
Corresponding author
Lamparth, Max
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2406.11852
arXiv
2406.11852
OpenAlex
W4394567566
Cited by
39
References
45
Citations by year
2026: 24 · 2025: 13 · 2024: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • medRxiv (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Center for International Security and Cooperation, Stanford University
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
language models, AI ethics, task autonomy, AI safety, mental illness, suicidal ideation, depression, psychosis, mania, default behavior, psychiatric emergencies, self-harm

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review