Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue and proposes a structured framework that delineates levels of autonomy, outlines ethical requirements, and defines beneficial default behaviors for AI agents in the context of mental health support. We also evaluate ten state-of-the-art language models using 16 mental health-related questions designed to reflect various mental health conditions, such as psychosis, mania, depression, suicidal thoughts, and homicidal tendencies. The question design and response evaluations were conducted by mental health clinicians (M.D.s). We find that existing language models are insufficient to match the standard provided by human professionals who can navigate nuances and appreciate context. This is due to a range of issues, including overly cautious or sycophantic responses and the absence of necessary safeguards. Alarmingly, we find that most of the tested models could cause harm if accessed in mental health emergencies, failing to protect users and potentially exacerbating existing symptoms. We explore solutions to enhance the safety of current models. Before the release of increasingly task-autonomous AI systems in mental health, it is crucial to ensure that these models can reliably detect and manage symptoms of common psychiatric disorders to prevent harm to users. This involves aligning with the ethical framework and default behaviors outlined in our study. We contend that model developers are responsible for refining their systems per these guidelines to safeguard against the risks posed by current AI technologies to user mental health and safety. Trigger warning: Contains and discusses examples of sensitive mental health topics, including suicide and self-harm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-CBs generally do not sufficiently adhere to professional guidelines or standards65,71–73 and, according to health professionals’ analyses on chatbot responses, show special harmful potential in the context of crisis situations71.”
From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [65]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“Other work studied using language models as agents in environments directly, for example Park et al. (2023); Wang et al. (2023b); Zhu et al. (2023); Liu et al. (2023a).”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Medical ethics: four principles plus attention to scope
Gillon, R. (1994) · BMJ · doi:10.1136/bmj.309.6948.184 · Cited by 1,107
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Hoeft, T. J., Fortney, J. C., Patel, V., & Unützer, J. (2018) · The Journal of Rural Health · doi:10.1111/jrh.12229 · Cited by 619 · cited 1×
“…health tools even further (Sorkin et al., 2021) and their ability to address this growing crisis is also included in the Presidential Executive Order that provides support for the ethical utilization of AI to improve healthcare delivery and outcomes (The White House, 2023; Hoeft et al., 2018).”
BackgroundLi, H., Zhang, R., Lee, Y.-C., Kraut, R. E., & Mohr, D. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00979-5 · Cited by 598 · cited 1×
“…(Higgins et al., 2023), deliver treatment in the form of increasingly automated psychotherapy or conversational agent (Thieme et al., 2023; Li et al., 2023; Balan et al., 2024), aid in psychiatric diagnosis (Kasula, 2023), and investigated the use of AI to analyze biometric data in the…”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
Validation of the PHQ-9 in a psychiatric sample
Beard, C., Hsu, K. J., Rifkin, L. S., Busch, A. B., & Björgvinsson, T. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.12.075 · Cited by 467
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · cited 1×
“We tested Mistral-instruct-7b (Jiang et al., 2023), gpt-3.”
PMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 296
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
Bianchi, F., Süzgün, M., Attanasio, G., Röttger, P., Jurafsky, D., Hashimoto, T., & Zou, J. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.07875 · Cited by 266
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Long Range Arena: A Benchmark for Efficient Transformers
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2011.04006 · Cited by 241 · cited 1×
“Also, safe TAIMH deployment would require long-range treatment accuracy tests, similar to long-range attention testing (Tay et al., 2021), simulation environments to find rare failure modes, similar to high-stakes conflict decision-making in Rivera et al. (2024), and comparisons of human experts to…”
BackgroundVoyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · cited 1×
“Other work studied using language models as agents in environments directly, for example Park et al. (2023); Wang et al. (2023b); Zhu et al. (2023); Liu et al. (2023a).”
BackgroundDiscovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.847 · Cited by 194
Pataranutaporn, P., Liu, R., Finn, E., & Maes, P. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00720-7 · Cited by 192
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Sifre, L., Rivière, M., Kale, M., Love, J., Tafti, P. D., et al. (2024) · Kaggle · doi:10.34740/kaggle/m/3301 · Cited by 175
Thieme, A., Hanratty, M., Lyons, M., Palacios, J., Marques, R. F., Morrison, C., & Doherty, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3564752 · Cited by 171
Higgins, O., Short, B. L., Chalup, S. K., & Wilson, R. L. (2023) · International Journal of Mental Health Nursing · doi:10.1111/inm.13114 · Cited by 150
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149 · cited 1×
“Bommasani et al. (2022); Chan et al. (2023); Mei et al. (2023); Zhang et al. (2023); Bianchi et al. (2024); Pfohl et al. (2024) only partially touch on the topic of mental health treatment, evaluate the model on knowledge about mental health without studying related harms to users with mental…”
BackgroundClosing the accessibility gap to mental health treatment with a personalized self-referral chatbot
Habicht, J., Viswanathan, S., Carrington, B., Hauser, T. U., Harper, R., & Rollwage, M. (2024) · Nature Medicine · doi:10.1038/s41591-023-02766-x · Cited by 132 · cited 1×
“…publish on AI’s ability to influence mental health-care delivery in separate areas, including its ability to automate therapy referrals (Sin, 2024; Habicht et al., 2024), empower peer support specialists (Sharma et al., 2023), and how the technology is viewed by patients (Pataranutaporn et al.,…”
BackgroundRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · cited 1×
“Other work evaluates physical/medical-related safety concerns (Ganguli et al., 2022; Wang et al., 2023a; Liu et al., 2023b), but does not study mental health-related issues.”
BackgroundMEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 1×
“Our code is available at github.com/maxlampe/taimh eval (MIT License) and the full, unredacted data set is available upon request due to the harmful content contained.”
MethodThe “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation
Plank, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.731 · Cited by 121
Automatic documentation of professional health interactions: A systematic review
Falcetta, F. S., de Almeida, F. K., Lemos, J., Goldim, J. R., & da Costa, C. A. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102487 · Cited by 116
A toolbox for surfacing health equity harms and biases in large language models
Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03258-2 · Cited by 113 · cited 1×
“Bommasani et al. (2022); Chan et al. (2023); Mei et al. (2023); Zhang et al. (2023); Bianchi et al. (2024); Pfohl et al. (2024) only partially touch on the topic of mental health treatment, evaluate the model on knowledge about mental health without studying related harms to users with mental…”
BackgroundDecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11698 · Cited by 110
van der Schyff, E. L., Ridout, B., Amon, K. L., Forsyth, R., & Campbell, A. J. (2023) · Journal of Medical Internet Research · doi:10.2196/46448 · Cited by 109 · cited 1×
“Current use of AI in digital mental health interventions centers on personalized chat bot-related support (Qi, 2024; van der Schyff et al., 2023; Reardon, 2023) or high-level concepts (Kellogg & Sadeh-Sharvit, 2022).”
BackgroundSorkin, D. H., Janio, E. A., Eikey, E. V., Schneider, M., Davis, K., Schueller, S. M., Stadnick, N. A., Zheng, K., Neary, M., Safani, D., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26994 · Cited by 108 · cited 1×
“The COVID-19 pandemic expanded the general use of digital mental health tools even further (Sorkin et al., 2021) and their ability to address this growing crisis is also included in the Presidential Executive Order that provides support for the ethical utilization of AI to improve healthcare…”
BackgroundDecodingTrust: A comprehensive assessment of trustworthiness in GPT models
Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1361 · Cited by 107
AI-Based and Digital Mental Health Apps: Balancing Need and Risk
Hamdoun, S., Monteleone, R., Bookman, T., & Michael, K. (2023) · IEEE Technology and Society Magazine · doi:10.1109/mts.2023.3241309 · Cited by 103
Sun, C.-F., Correll, C. U., Trestman, R. L., Lin, Y., Xie, H., Hankey, M. S., Uymatiao, R. P., Patel, R. T., Metsutnan, V. L., McDaid, E. C., et al. (2023) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2023.05.012 · Cited by 100
Trojaning Language Models for Fun and Profit
Zhang, X., Zhang, Z., Ji, S., & Wang, T. (2021) · 2021 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp51992.2021.00022 · Cited by 99
Escalation Risks from Language Models in Military and Diplomatic Decision-Making
Rivera, J.-P., Mukobi, G., Reuel, A., Lamparth, M., Smith, C., & Schneider, J. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658942 · Cited by 77
Self-citationXSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
Röttger, P., Kirk, H., Vidgen, B., Attanasio, G., Bianchi, F., & Hovy, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.301 · Cited by 77
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O'Keefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Cass-Beggs, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03718 · Cited by 77
Ates, H. C., & Dincer, C. (2023) · Nature Reviews Bioengineering · doi:10.1038/s44222-022-00011-7 · Cited by 67
Use of automated conversational agents in improving young population mental health: a scoping review
Balan, R., Dobrean, A., & Poetar, C. R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01072-1 · Cited by 56 · cited 1×
“…et al., 2023), deliver treatment in the form of increasingly automated psychotherapy or conversational agent (Thieme et al., 2023; Li et al., 2023; Balan et al., 2024), aid in psychiatric diagnosis (Kasula, 2023), and investigated the use of AI to analyze biometric data in the setting of stress…”
BackgroundKellogg, K. C., & Sadeh‐Sharvit, S. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.990370 · Cited by 54
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51
Fatal self-injury in the United States, 1999–2018: Unmasking a national mental health crisis
Rockett, I. R. H., Caine, E. D., Banerjee, A., Ali, B., Miller, T. R., Connery, H. S., Lulla, V. O., Nolte, K. B., Larkin, G. L., Stack, S. J., et al. (2021) · EClinicalMedicine · doi:10.1016/j.eclinm.2021.100741 · Cited by 43
Qi, Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2301256 · Cited by 39
Lamparth, M., Corso, A., Ganz, J., Mastro, O. S., Schneider, J., & Trinkunas, H. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31681 · Cited by 39
Self-citationASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models
Mei, A., Levy, S., & Wang, W. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.388 · Cited by 38
Deducing health cues from biometric data
Ross, A. A., Banerjee, S., & Chowdhury, A. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103438 · Cited by 37
Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17144 · Cited by 33 · cited 1×
“Other work studied using language models as agents in environments directly, for example Park et al. (2023); Wang et al. (2023b); Zhu et al. (2023); Liu et al. (2023a).”
BackgroundTowards Healthy AI: Large Language Models Need Therapists Too
Lin, B., Bouneffouf, D., Cecchi, G., & Varshney, K. (2024) · Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024) · doi:10.18653/v1/2024.trustnlp-1.6 · Cited by 28
SafetyBench: Evaluating the Safety of Large Language Models
Zhang, Z., Lei, L., Wu, L., Sun, R., Huang, Y., Long, C., Liu, X., Lei, X., Tang, J., & Huang, M. (2023) · arXiv · doi:10.48550/arxiv.2309.07045 · Cited by 19
Lamparth, M., Corso, A., Ganz, J., Mastro, O. S., Schneider, J., & Trinkunas, H. (2024) · arXiv · doi:10.48550/arxiv.2403.03407 · Cited by 15
Self-citationAn AI chatbot for talking therapy referrals
Sin, J. (2024) · Nature Medicine · doi:10.1038/s41591-023-02773-y · Cited by 13
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.08608 · Cited by 10
Innovative preclinic triage system to guide Australians to the right mental health care first time
Davenport, T. A., LaMonica, H. M., Rowe, S. C., Sturgess, J., Scott, E. M., Lee, G. Y., Cheng, V. W. S., & Hickie, I. B. (2021) · Australian Health Review · doi:10.1071/ah20233 · Cited by 9
Human Control: Definitions and AlgorithmsPreprint
Carey, R., & Everitt, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.19861 · Cited by 9
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., & Hovy, D. (2023) · arXiv · doi:10.48550/arxiv.2308.01263 · Cited by 6 · cited 1×
“In a way, our analysis complements evaluations on over-exaggerated safety responses, e.g., (R¨ottger et al., 2023), as we are looking for under-exaggerated safety responses.”
MethodHabicht, J., Viswanathan, S., Carrington, B., et al. (2023) · medRxiv · doi:10.1101/2023.04.29.23289204 · Cited by 4
Towards Healthy AI: Large Language Models Need Therapists Too
Lin, B., Bouneffouf, D., Cecchi, G., & Varshney, K. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00416
Research works printed but not linked: 20
- American Medical Association. Doctor Shortages Are Here and They'll Get Worse if We Don't Act. https://www.ama-assn.org/practice-management/sustainability/ doctor-shortages-are-here-and-they-ll-get-worse-if-we-don-t-act. [Online; ac- cessed 26-March-2024].
- A. Andy Zou et al. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. · arXiv
- L. Aroyo et al. Dices dataset: Diversity in conversational ai evaluation for safety. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 53330-53342.
- Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper files/paper/2023/ file/a74b697bce4cac6c91896372abaa8863-Paper-Datasets and Benchmarks.pdf.
- Y. Bai et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022a. · arXiv
- Y. Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022b. · arXiv
- N. Carlini et al. Are aligned neural networks adversarially aligned? In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- S. Casper, C. Ezell, et al. Black-box access is insufficient for rigorous ai audits. arXiv preprint arXiv:2401.14446, 2024. · arXiv
- K. Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- DSM. DSM-5. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition.
- N. Kandpal et al. Trojaning language models for fun and profit. arXiv preprint arXiv:2307.14692, 2023. · arXiv
- B. Y. Kasula. Ethical Considerations in the Adoption of Artificial Intelligence for Mental Health Diagnosis. International Journal of Creative Research In Computer Technology and Design, 5(5):1-7, 2023.
- M. Lamparth and A. Reuel. Analyzing and editing inner mechanisms of backdoored language models. arXiv preprint arXiv:2302.12461, 2023. · arXiv
- X. Liu et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023a. · arXiv
- X. Liu et al. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. arXiv preprint arXiv:2311.17600, 2023b. · arXiv
- OpenAI et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- Y. Perlitz et al. Efficient benchmarking of language models. arXiv preprint arXiv:2308.11696, 2024. · arXiv
- D. Rein et al. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. · arXiv
- Gemini Team et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
- H. Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
Cited by
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
Self-citationEvaluating Generative AI Psychotherapy Chatbots Used by Youth: Cross-Sectional Study
Sobowale, K., Humphrey, D. K., & Zhao, S. Y. (2025) · JMIR Mental Health · doi:10.2196/79838 · Cited by 52
Training large language models on narrow tasks can lead to broad misalignment
Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., Srivastava, M., Labenz, N., & Evans, O. (2026) · Nature · doi:10.1038/s41586-025-09937-5 · Cited by 31
“It is well-known that present-day language models can exhibit a wide range of potentially harmful behaviour in response to benign queries from users, from generating insecure code to encouraging self-harm 29,30 .”
Latt, P. M., Aung, E. T., Htaik, K., Soe, N. N., Lee, D. M., King, A. J., Fortune, R., Ong, J. J., Chow, E. P. F., Bradshaw, C. S., et al. (2025) · BMC Public Health · doi:10.1186/s12889-025-22933-8 · Cited by 40
Voultsiou, E., & Moussiades, L. (2026) · Electronics · doi:10.3390/electronics15030524 · Cited by 13
Metadata
- Authors
- Grabb, D., Lamparth, M., & Vasan, N.
- Affiliations
- Northwestern University (US)
- Stanford University (US)
- Corresponding author
- Lamparth, Max
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-04-08
- OpenAlex
- W4394567566
- Cited by
- 40
- References
- 45
- Citations by year
- 2026: 24 · 2025: 13 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- medRxiv (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Center for International Security and Cooperation, Stanford University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- language models, AI ethics, task autonomy, AI safety, mental illness, suicidal ideation, depression, psychosis, mania, default behavior, psychiatric emergencies, self-harm
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.