Safety of Large Language Models in Addressing Depression
Open accessAbstract
Background Generative artificial intelligence (AI) models, exemplified by systems such as ChatGPT, Bard, and Anthropic, are currently under intense investigation for their potential to address existing gaps in mental health support. One implementation of these large language models involves the development of mental health-focused conversational agents, which utilize pre-structured prompts to facilitate user interaction without requiring specialized knowledge in prompt engineering. However, uncertainties persist regarding the safety and efficacy of these agents in recognizing severe depression and suicidal tendencies. Given the well-established correlation between the severity of depression and the risk of suicide, improperly calibrated conversational agents may inadequately identify and respond to crises. Consequently, it is crucial to investigate whether publicly accessible repositories of mental health-focused conversational agents can consistently and safely address crisis scenarios before considering their adoption in clinical settings. This study assesses the safety of publicly available ChatGPT-3.5 conversational agents by evaluating their responses to a patient simulation indicating worsening depression and suicidality. Methodology This study evaluated ChatGPT-3.5 conversational agents on a publicly available repository specifically designed for mental health counseling. Each conversational agent was evaluated twice by a highly structured patient simulation. First, the simulation indicated escalating suicide risk based on the Patient Health Questionnaire (PHQ-9). For the second patient simulation, the escalating risk was presented in a more generalized manner not associated with an existing risk scale to assess the more generalized ability of the conversational agent to recognize suicidality. Each simulation recorded the exact point at which the conversational agent recommended human support. Then, the simulation continued until the conversational agent stopped entirely and shut down completely, insisting on human intervention. Results All 25 agents available on the public repository FlowGPT.com were evaluated. The point at which the conversational agents referred to a human occurred around the mid-point of the simulation, and definitive shutdown predominantly only happened at the highest risk levels. For the PHQ-9 simulation, the average initial referral and shutdown aligned with PHQ-9 scores of 12 (moderate depression) and 25 (severe depression). Few agents included crisis resources - only two referenced suicide hotlines. Despite the conversational agents insisting on human intervention, 22 out of 25 agents would eventually resume the dialogue if the simulation reverted to a lower risk level. Conclusions Current generative AI-based conversational agents are slow to escalate mental health risk scenarios, postponing referral to a human to potentially dangerous levels. More rigorous testing and oversight of conversational agents are needed before deployment in mental healthcare settings. Additionally, further investigation should explore if sustained engagement worsens outcomes and whether enhanced accessibility outweighs the risks of improper escalation. Advancing AI safety in mental health remains imperative as these technologies continue rapidly advancing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Given the unpredictable 'black box' nature of generative AI, and the existence of at least one tragic example of early generative AI chatbots supporting users in dying by suicide, current literature advocates that when users display signs of crisis, models revert to scripted responses that signpost towards human support.
“Given the unpredictable “black box” nature of generative AI43, and the existence of at least one tragic example of early generative AI chatbots supporting users in dying by suicide55, current literature advocates that when users display signs of crisis, models revert to scripted responses that signpost towards human support42,51.”
From “It happened to be the perfect thing”: experiences of generative AI chatbots for mental health (2024) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910 · see citing works
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · see citing works
Estimating the true global burden of mental illness
Vigo, D., Thornicroft, G., & Atun, R. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00505-2 · Cited by 2,494 · see citing works
Prevalence and Treatment of Mental Disorders, 1990 to 2003
Kessler, R. C., Demler, O., Frank, R. G., Olfson, M., Pincus, H. A., Walters, E. E., Wang, P., Wells, K. B., & Zaslavsky, A. M. (2005) · New England Journal of Medicine · doi:10.1056/nejmsa043266 · Cited by 1,910 · see citing works
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
Barriers to mental health treatment: results from the WHO World Mental Health surveys
Andrade, L. H., Alonso, J., Mneimneh, Z., Wells, J. E., Al-Hamzawi, A., Borges, G., Bromet, E., Bruffaerts, R., de Girolamo, G., de Graaf, R., et al. (2014) · Psychological Medicine · doi:10.1017/s0033291713001943 · Cited by 1,289 · see citing works
Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020) · Journal of Medical Internet Research · doi:10.2196/16021 · Cited by 503 · cited 1× · see citing works
“The safety of using LLM-based conversational agents to deliver mental health services is not established [17].”
BackgroundSharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00593-2 · Cited by 500 · see citing works
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · see citing works
A Review of ChatGPT AI's Impact on Several Business Sectors
George, A. S., George, A. S. H., & Martin, A. S. G. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7644359 · Cited by 443 · see citing works
Kazdin, A. E. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12937 · Cited by 274 · see citing works
Prompt Engineering in Medical Education
Heston, T. F., & Khun, C. (2023) · International Medical Education · doi:10.3390/ime2030019 · Cited by 232 · see citing works
Self-citationXie, Y., Seth, I., Hunter-Smith, D. J., Rozen, W. M., Ross, R., & Lee, M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03338-7 · Cited by 198 · see citing works
Towards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155 · see citing works
ChatGPT broke the Turing test — the race is on for new ways to assess AI
Biever, C. (2023) · Nature · doi:10.1038/d41586-023-02361-7 · Cited by 153 · cited 1× · see citing works
“In 2020, OpenAI released ChatGPT-3, a landmark in online chatbot technology, known for its ability to mimic human-like text and pass traditional benchmarks such as the Turing test, which assesses a machine’s ability to exhibit intelligent behavior indistinguishable from that of a human [9].”
BackgroundRibino, P. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10540-1 · Cited by 105 · see citing works
ChatGPT performance in the medical specialty exam: An observational study
Oztermeli, A. D., & Öztermeli, A. (2023) · Medicine · doi:10.1097/md.0000000000034673 · Cited by 71 · see citing works
Walker, J., Hansen, C. H., Hodges, L. J., Thekkumpurath, P., O’Connor, M. J., Sharma, N., Kleiboer, A. M., Murray, G. D., Kroenke, K., & Sharpe, M. (2010) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2009.11.011 · Cited by 59 · see citing works
The Capability of Large Language Models to Measure Psychiatric FunctioningPreprint
Galatzer-Levy, I., McDuff, D. J., Natarajan, V., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.01834 · Cited by 54 · see citing works
Sarkar, S., Gaur, M., Chen, L. K., Garg, M., & Srivastava, B. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1229805 · Cited by 38 · see citing works
Cardiac Risk Stratification in Renal Transplantation Using a Form of Artificial Intelligence
Heston, T. F., Norman, D. J., Barry, J. M., et al. (1997) · The American Journal of Cardiology · doi:10.1016/s0002-9149(96)00778-3 · Cited by 30 · see citing works
Self-citationSharma, A. K., Lin, I. W., Miner, A. S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.15144 · Cited by 23 · see citing works
Adding guardrails to advanced chatbotsPreprint
Wang, Y., & Singh, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07500 · Cited by 6 · see citing works
AI in Primary Care, Preventative Medicine, and Triage
Abedin, Y., Ahmad, O. F., & Bajwa, J. (2023) · AI in Clinical Medicine · doi:10.1002/9781119790686.ch9 · Cited by 5 · see citing works
Evaluating Risk Progression in Mental Health Chatbots with Escalating Prompts Dataset
Heston, T. F. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.8332778 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [7]Attention is all you need NIPS Vaswani A Shazeer N Parmar N 1 11 30 2017 https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
- [16]Towards interpretable mental health analysis with ChatGPT arXiv Yang K Ji S Zhang T Xie Q Kuang Z Ananiadou S 2023
Cited by
Sorted by citations
Large Language Models for Mental Health Applications: Systematic Review
Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024) · JMIR Mental Health · doi:10.2196/57400 · Cited by 269 · see citing works
“It happened to be the perfect thing”: experiences of generative AI chatbots for mental health
Siddals, S., Torous, J., & Coxon, A. (2024) · npj Mental Health Research · doi:10.1038/s44184-024-00097-4 · Cited by 241 · see citing works
Hua, Y., Siddals, S., Ma, Z., Galatzer‐Levy, I., Xia, W., Hau, C., Na, H., Flathers, M., Linardon, J., Ayubcha, C., et al. (2025) · World Psychiatry · doi:10.1002/wps.21352 · Cited by 117 · see citing works
Li, L., Kong, S., Zhao, H., Li, C., Teng, Y., & Wang, Y. (2025) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13781 · Cited by 10 · see citing works
Oo, C. T. L., Wider, W., Pang, N. T. P., Koh, E. B. Y., Vasanthi, R. K., Thet, K. Z. Z., Ramalho, R., Özdemir, B. N., & Mahboob, K. (2026) · International Journal of Qualitative Studies on Health and Well-being · doi:10.1080/17482631.2026.2621802 · Cited by 5 · see citing works
“In mental health contexts, simple, clear, and responsive conversational systems help create a sense of comfort and psychological safety for both clinicians and patients (Heston, 2023).”
- Authors
- Heston, T. F.
- Affiliations
- Washington State University Spokane (US)
- Corresponding author
- Heston, Thomas F
- ORCID
- Heston, Thomas F
- Venue
- Cureus
- Published
- 2023-12-18
- Volume
- 15
- Issue
- 12
- Pages
- e50729-e50729
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC10727113
- OpenAlex
- W4389923012
- Cited by
- 61
- References
- 26
- Citations by year
- 2026: 19 · 2025: 27 · 2024: 15
- FWCI
- 15.75
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ResearchWorks at the University of Washington (University of Washington) (submittedVersion, cc-by-nc)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- large language models, ChatGPT-3.5, chatbots, depression, suicidality, PHQ-9, AI safety, mental health chatbots, suicide hotline, patient simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
