Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.

Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation

Open access

Abstract

Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue and proposes a structured framework that delineates levels of autonomy, outlines ethical requirements, and defines beneficial default behaviors for AI agents in the context of mental health support. We also evaluate ten state-of-the-art language models using 16 mental health-related questions designed to reflect various mental health conditions, such as psychosis, mania, depression, suicidal thoughts, and homicidal tendencies. The question design and response evaluations were conducted by mental health clinicians (M.D.s). We find that existing language models are insufficient to match the standard provided by human professionals who can navigate nuances and appreciate context. This is due to a range of issues, including overly cautious or sycophantic responses and the absence of necessary safeguards. Alarmingly, we find that most of the tested models could cause harm if accessed in mental health emergencies, failing to protect users and potentially exacerbating existing symptoms. We explore solutions to enhance the safety of current models. Before the release of increasingly task-autonomous AI systems in mental health, it is crucial to ensure that these models can reliably detect and manage symptoms of common psychiatric disorders to prevent harm to users. This involves aligning with the ethical framework and default behaviors outlined in our study. We contend that model developers are responsible for refining their systems per these guidelines to safeguard against the risks posed by current AI technologies to user mental health and safety. Trigger warning: Contains and discusses examples of sensitive mental health topics, including suicide and self-harm.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.04.07.24305462

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 45 references

Research works printed but not linked: 23

  1. American Medical Association. Doctor Shortages Are Here and They’ll Get Worse if We Don’t Act. https://www.ama-assn.org/practice-management/sustainability/doctor-shortages-are-here-and-they-ll-get-worse-if-we-don-t-act. [Online; accessed 26-March-2024].
  2. M. Anderljung et al. Frontier ai regulation: Managing emerging risks to public safety. arXiv preprint arXiv:2307.03718, 2023. · arXiv
  3. A. Andy Zou et al. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. · arXiv
  4. 2023, Advances in Neural Information Processing Systems, 36, 53330
  5. Y. Bai et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022a. · arXiv
  6. Y. Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022b. · arXiv
  7. N. Carlini et al. Are aligned neural networks adversarially aligned? In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  8. S. Casper , C. Ezell , et al. Black-box access is insufficient for rigorous ai audits. arXiv preprint arXiv:2401.14446, 2024. · arXiv
  9. K. Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  10. DSM. DSM-5. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition.
  11. N. Kandpal et al. Trojaning language models for fun and profit. arXiv preprint arXiv:2307.14692, 2023. · arXiv
  12. 2023, Ethical Considerations in the Adoption of Artificial Intelligence for Mental Health Diagnosis, International Journal of Creative Research In Computer Technology and Design, 5, 1
  13. M. Lamparth and A. Reuel . Analyzing and editing inner mechanisms of backdoored language models. arXiv preprint arXiv:2302.12461, 2023. · arXiv
  14. M. Lamparth et al. Human vs. machine: Language models and wargames. arXiv preprint arXiv:2403.03407, 2024. · arXiv
  15. X. Liu et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023a. · arXiv
  16. X. Liu et al. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. arXiv preprint arXiv:2311.17600, 2023b. · arXiv
  17. OpenAI et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  18. Y. Perlitz et al . Efficient benchmarking of language models. arXiv preprint arXiv:2308.11696, 2024. · arXiv
  19. D. Rein et al. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. · arXiv
  20. Gemini Team et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
  21. H. Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  22. Z. Zhang et al. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045, 2023. · arXiv
  23. X. Zhu et al. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023. · arXiv

Cited by

See all 40

Metadata

Authors
Grabb, D., Lamparth, M., & Vasan, N.
Affiliations
  • Northwestern University (US)
  • Stanford University (US)
Corresponding author
Lamparth, Max
Venue
medRxiv
Publisher
openRxiv
Published
2024-04-08
OpenAlex
W4394567566
Cited by
40
References
45
Citations by year
2026: 24 · 2025: 13 · 2024: 3
Type
posted content
Language
en
Open access
Licence
cc-by
Locations
  • medRxiv (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Center for International Security and Cooperation, Stanford University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
language models, AI ethics, task autonomy, AI safety, mental illness, suicidal ideation, depression, psychosis, mania, default behavior, psychiatric emergencies, self-harm

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation. https://app.cochat.ai/papers/10.1101/2024.04.07.24305462
The paper
Grabb, D., Lamparth, M., & Vasan, N. (2024). Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation. medRxiv. https://doi.org/10.1101/2024.04.07.24305462

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.