Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Background: ChatGPT, a linguistic artificial intelligence (AI) model engineered by OpenAI, offers prospective contributions to mental health professionals. Although having significant theoretical implications, ChatGPT’s practical capabilities, particularly regarding suicide prevention, have not yet been substantiated. Objective: The study’s aim was to evaluate ChatGPT’s ability to assess suicide risk, taking into consideration 2 discernable factors—perceived burdensomeness and thwarted belongingness—over a 2-month period. In addition, we evaluated whether ChatGPT-4 more accurately evaluated suicide risk than did ChatGPT-3.5. Methods: ChatGPT was tasked with assessing a vignette that depicted a hypothetical patient exhibiting differing degrees of perceived burdensomeness and thwarted belongingness. The assessments generated by ChatGPT were subsequently contrasted with standard evaluations rendered by mental health professionals. Using both ChatGPT-3.5 and ChatGPT-4 (May 24, 2023), we executed 3 evaluative procedures in June and July 2023. Our intent was to scrutinize ChatGPT-4’s proficiency in assessing various facets of suicide risk in relation to the evaluative abilities of both mental health professionals and an earlier version of ChatGPT-3.5 (March 14 version). Results: During the period of June and July 2023, we found that the likelihood of suicide attempts as evaluated by ChatGPT-4 was similar to the norms of mental health professionals (n=379) under all conditions (average Z score of 0.01). Nonetheless, a pronounced discrepancy was observed regarding the assessments performed by ChatGPT-3.5 (May version), which markedly underestimated the potential for suicide attempts, in comparison to the assessments carried out by the mental health professionals (average Z score of –0.83). The empirical evidence suggests that ChatGPT-4’s evaluation of the incidence of suicidal ideation and psychache was higher than that of the mental health professionals (average Z score of 0.47 and 1.00, respectively). Conversely, the level of resilience as assessed by both ChatGPT-4 and ChatGPT-3.5 (both versions) was observed to be lower in comparison to the assessments offered by mental health professionals (average Z score of –0.89 and –0.90, respectively). Conclusions: The findings suggest that ChatGPT-4 estimates the likelihood of suicide attempts in a manner akin to evaluations provided by professionals. In terms of recognizing suicidal ideation, ChatGPT-4 appears to be more precise. However, regarding psychache, there was an observed overestimation by ChatGPT-4, indicating a need for further research. These results have implications regarding ChatGPT-4’s potential to support gatekeepers, patients, and even mental health professionals’ decision-making. Despite the clinical potential, intensive follow-up studies are necessary to establish the use of ChatGPT-4’s capabilities in clinical practice. The finding that ChatGPT-3.5 frequently underestimates suicide risk, especially in severe cases, is particularly troubling. It indicates that ChatGPT may downplay one’s actual suicide risk level.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017) · Psychological Bulletin · doi:10.1037/bul0000084 · Cited by 3,784
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1×
“A recent review scrutinized the benefits conferred by ChatGPT and analogous LLMs in enhancing medical education, refining clinical decision-making processes, and propelling superior patient outcomes [41].”
BackgroundBenefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Van Orden, K. A., Witte, T. K., Gordon, K. H., Bender, T. W., & Joiner, T. E. (2008) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.76.1.72 · Cited by 1,126 · cited 2×
“5 and ChatGPT-4 (May 24 versions) to identify the risk of suicide based on the interpersonal theory of suicide (ITS), an established and empirically backed theoretical model proposed by Joiner et al [34,35] for assessing suicide risk and identifying associated factors.”
BackgroundJoiner, T. E., Van Orden, K. A., Witte, T. K., Selby, E. A., Ribeiro, J. D., Lewis, R., & Rudd, M. D. (2009) · Journal of Abnormal Psychology · doi:10.1037/a0016500 · Cited by 814 · cited 2×
“5 and ChatGPT-4 (May 24 versions) to identify the risk of suicide based on the interpersonal theory of suicide (ITS), an established and empirically backed theoretical model proposed by Joiner et al [34,35] for assessing suicide risk and identifying associated factors.”
BackgroundCapabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
Patient apprehensions about the use of artificial intelligence in healthcare
Richardson, J. P., Smith, C., Curtis, S., Watson, S., Zhu, X., Barry, B., & Sharp, R. R. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00509-1 · Cited by 424
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363
Shahsavar, Y., & Choudhury, A. (2023) · JMIR Human Factors · doi:10.2196/47564 · Cited by 341 · cited 1×
“In a cross-sectional investigation, it was ascertained that 78.4% of the respondents exhibited a willingness to use ChatGPT for the purpose of self-diagnosis [39].”
MethodInfluentialChatGPT outperforms humans in emotional awareness evaluations
Elyoseph, Z., Hadar-Shoval, D., Asraf, K., & Lvovsky, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1199058 · Cited by 324
Self-citation
Cited by
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738
“Within this review, only one study conducted a financial impact or cost-effectiveness analysis.”
BackgroundTesting and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2025) · JAMA · doi:10.1001/jama.2024.21700 · Cited by 662
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283
Large Language Models for Mental Health Applications: Systematic Review
Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024) · JMIR Mental Health · doi:10.2196/57400 · Cited by 269
“While they can generally capture extreme emotions and recognize scenarios requiring urgent action, such as suicide ideation [54,65,70,72,78], they often fail to provide direct, practical measures, typically only advising users to seek professional help [96].”
Background(2023) · doi:10.2196/54369 · Cited by 127
Metadata
- Authors
- Levkovich, I., & Elyoseph, Z.
- Venue
- JMIR Mental Health
- Publisher
- JMIR Publications Inc.
- Published
- 2023-09-20
- Volume
- 10
- Pages
- e51232
- ISSN
- 2368-7959 (ISSN-L)
- OpenAlex
- W4386887838
- Cited by
- 118
- References
- 21
- FWCI
- 32.36
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study. https://app.cochat.ai/papers/10.2196/51232
- The paper
- Levkovich, I., & Elyoseph, Z. (2023). Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study. JMIR Mental Health. https://doi.org/10.2196/51232
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.