The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Open accessAbstract
Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue and proposes a structured framework that delineates levels of autonomy, outlines ethical requirements, and defines beneficial default behaviors for AI agents in the context of mental health support. We also evaluate ten state-of-the-art language models using 16 mental health-related questions designed to reflect various mental health conditions, such as psychosis, mania, depression, suicidal thoughts, and homicidal tendencies. The question design and response evaluations were conducted by mental health clinicians (M.D.s). We find that existing language models are insufficient to match the standard provided by human professionals who can navigate nuances and appreciate context. This is due to a range of issues, including overly cautious or sycophantic responses and the absence of necessary safeguards. Alarmingly, we find that most of the tested models could cause harm if accessed in mental health emergencies, failing to protect users and potentially exacerbating existing symptoms. We explore solutions to enhance the safety of current models. Before the release of increasingly task-autonomous AI systems in mental health, it is crucial to ensure that these models can reliably detect and manage symptoms of common psychiatric disorders to prevent harm to users. This involves aligning with the ethical framework and default behaviors outlined in our study. We contend that model developers are responsible for refining their systems per these guidelines to safeguard against the risks posed by current AI technologies to user mental health and safety. Trigger warning: Contains and discusses examples of sensitive mental health topics, including suicide and self-harm.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLM-CBs generally do not sufficiently adhere to professional guidelines or standards65,71–73 and, according to health professionals’ analyses on chatbot responses, show special harmful potential in the context of crisis situations71.”
From A scoping review on the mental health harms of LLM-based chatbots (2026) · cites this paper as [65]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“Other work studied using language models as agents in environments directly, for example Park et al. (2023); Wang et al. (2023b); Zhu et al. (2023); Liu et al. (2023a).”
BackgroundGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Medical ethics: four principles plus attention to scope
Gillon, R. (1994) · BMJ · doi:10.1136/bmj.309.6948.184 · Cited by 1,107
Hoeft, T. J., Fortney, J. C., Patel, V., & Unützer, J. (2018) · The Journal of Rural Health · doi:10.1111/jrh.12229 · Cited by 619
Li, H., Zhang, R., Lee, Y.-C., Kraut, R. E., & Mohr, D. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00979-5 · Cited by 598 · cited 1×
“…(Higgins et al., 2023), deliver treatment in the form of increasingly automated psychotherapy or conversational agent (Thieme et al., 2023; Li et al., 2023; Balan et al., 2024), aid in psychiatric diagnosis (Kasula, 2023), and investigated the use of AI to analyze biometric data in the…”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500
Bianchi, F., Suzgun, M., Attanasio, G., Röttger, P., Jurafsky, D., Hashimoto, T. B., & Zou, J. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.07875 · Cited by 479
Validation of the PHQ-9 in a psychiatric sample
Beard, C., Hsu, K. J., Rifkin, L. S., Busch, A. B., & Björgvinsson, T. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.12.075 · Cited by 467
Research works printed but not linked: 23
- American Medical Association. Doctor Shortages Are Here and They’ll Get Worse if We Don’t Act. https://www.ama-assn.org/practice-management/sustainability/doctor-shortages-are-here-and-they-ll-get-worse-if-we-don-t-act. [Online; accessed 26-March-2024].
- M. Anderljung et al. Frontier ai regulation: Managing emerging risks to public safety. arXiv preprint arXiv:2307.03718, 2023. · arXiv
- A. Andy Zou et al. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. · arXiv
- 2023, Advances in Neural Information Processing Systems, 36, 53330
- Y. Bai et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022a. · arXiv
- Y. Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022b. · arXiv
- N. Carlini et al. Are aligned neural networks adversarially aligned? In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- S. Casper , C. Ezell , et al. Black-box access is insufficient for rigorous ai audits. arXiv preprint arXiv:2401.14446, 2024. · arXiv
- K. Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- DSM. DSM-5. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition.
- N. Kandpal et al. Trojaning language models for fun and profit. arXiv preprint arXiv:2307.14692, 2023. · arXiv
- 2023, Ethical Considerations in the Adoption of Artificial Intelligence for Mental Health Diagnosis, International Journal of Creative Research In Computer Technology and Design, 5, 1
- M. Lamparth and A. Reuel . Analyzing and editing inner mechanisms of backdoored language models. arXiv preprint arXiv:2302.12461, 2023. · arXiv
- M. Lamparth et al. Human vs. machine: Language models and wargames. arXiv preprint arXiv:2403.03407, 2024. · arXiv
- X. Liu et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023a. · arXiv
- X. Liu et al. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. arXiv preprint arXiv:2311.17600, 2023b. · arXiv
- OpenAI et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- Y. Perlitz et al . Efficient benchmarking of language models. arXiv preprint arXiv:2308.11696, 2024. · arXiv
- D. Rein et al. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. · arXiv
- Gemini Team et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
- H. Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
- Z. Zhang et al. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045, 2023. · arXiv
- X. Zhu et al. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023. · arXiv
Cited by
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
Self-citationEvaluating Generative AI Psychotherapy Chatbots Used by Youth: Cross-Sectional Study
Sobowale, K., Humphrey, D. K., & Zhao, S. Y. (2025) · JMIR Mental Health · doi:10.2196/79838 · Cited by 45
Training large language models on narrow tasks can lead to broad misalignment
Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., Srivastava, M., Labenz, N., & Evans, O. (2026) · Nature · doi:10.1038/s41586-025-09937-5 · Cited by 31
“It is well-known that present-day language models can exhibit a wide range of potentially harmful behaviour in response to benign queries from users, from generating insecure code to encouraging self-harm 29,30 .”
Latt, P. M., Aung, E. T., Htaik, K., Soe, N. N., Lee, D. M., King, A. J., Fortune, R., Ong, J. J., Chow, E. P. F., Bradshaw, C. S., et al. (2025) · BMC Public Health · doi:10.1186/s12889-025-22933-8 · Cited by 40
Voultsiou, E., & Moussiades, L. (2026) · Electronics · doi:10.3390/electronics15030524 · Cited by 27
Metadata
- Authors
- Grabb, D., Lamparth, M., & Vasan, N.
- Affiliations
- Northwestern University (US)
- Stanford University (US)
- Corresponding author
- Lamparth, Max
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-04-08
- OpenAlex
- W4394567566
- Cited by
- 40
- References
- 45
- Citations by year
- 2026: 24 · 2025: 13 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- medRxiv (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Center for International Security and Cooperation, Stanford University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Keywords
- language models, AI ethics, task autonomy, AI safety, mental illness, suicidal ideation, depression, psychosis, mania, default behavior, psychiatric emergencies, self-harm
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation. https://app.cochat.ai/papers/10.1101/2024.04.07.24305462
- The paper
- Grabb, D., Lamparth, M., & Vasan, N. (2024). Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation. medRxiv. https://doi.org/10.1101/2024.04.07.24305462
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.