Reliability of Suicide Risk Estimates: A Vignette Study
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: Suicide risk assessments are obligatory when patients express a death wish in clinical practice. Yet, suicide risk estimates based on unguided risk assessments have been shown to be of low reliability. Since generalizability of previous studies is limited, the current study aimed to assess inter-rater and intra-rater reliability of risk estimates conducted by psychotherapists and psychology students using written case vignettes. METHOD: = 256 participants (psychology students, psychotherapists) were presented with 24 case vignettes describing patients at either low, moderate, severe or extreme risk of suicide. Participants were asked to assign a level of risk to each single vignette at a baseline assessment and again at a follow-up assessment two weeks later. RESULTS: = .47). In general, intra- and intra-rater reliability were highest for vignettes displaying "low" and "extreme" risk. CONCLUSIONS: The results highlight that the reliability of unguided suicide risk assessments is questionable. Standardized risk assessment protocols are therefore recommended. Nonetheless, even reliable risk estimation does not imply predictive validity of risk estimates for future suicidal behavior.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Various studies have demonstrated that the reliability of risk assessments is moderate at best (9), and that the prediction of suicidal behavior, is currently insufficient for clinically useful individual-level prediction (10).”
From Suicide risk assessment: clinical implications of the unpredictability of suicidal behavior (2026) · cites this paper as [9]
Citations
Cites
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · cited 3×
“However, only the AC 1 -confidence intervals of the therapists’ ratings of the “low” and the “extreme” risk categories as well as the students’ ratings of the “low” risk category exceeded the lower limit of .666 allowing for “cautious” conclusions (cf. Krippendorff, 2019).”
InfluentialVan Orden, K. A., Witte, T. K., Gordon, K. H., Bender, T. W., & Joiner, T. E. (2008) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.76.1.72 · Cited by 1,126 · cited 3×
“…risk assessments found support for sufficient inter-rater as well as re-test reliability of derived risk estimates (Alphs et al., 2020; Gallyer et al., 2020; Van Orden et al., 2008), structured suicide risk assessments are preferable to unstructured suicide risk assessments in clinical practice.”
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-61 · Cited by 1,041 · cited 1×
“The AC 1 statistic offers an alternative chance-corrected agreement measure, that has mathematically and empirically been shown to be robust against the marginal frequency paradoxes unlike traditional Kappa statistics (Wongpakaran et al., 2013).”
Evans, S. C., Roberts, M. C., Keeley, J. W., Blossom, J. B., Amaro, C. M., Garcia, A. M., Stough, C. O., Canter, K. S., Robles, R., & Reed, G. M. (2015) · International Journal of Clinical and Health Psychology · doi:10.1016/j.ijchp.2014.12.001 · Cited by 757
The interpersonal theory of suicide: Guidance for working with suicidal clients.
Joiner, T. E., Van Orden, K. A., Witte, T. K., & Rudd, M. D. (2009) · American Psychological Association eBooks · doi:10.1037/11869-000 · Cited by 441 · cited 1×
“…their consent. which variables should be assessed as part of a risk assessment (e.g., Chu et al., 2015) and (b) which constellations of risk and protective factors constitute a mild, moderate, high, or extreme risk for suicidal behavior (Chu et al., 2015; Joiner et al., 2009; Rudd et al., 2001).”
Routinized Assessment of Suicide Risk in Clinical Practice: An Empirically Informed Update
Chu, C., Klein, K. M., Buchman‐Schmitt, J. M., Hom, M. A., Hagan, C. R., & Joiner, T. E. (2015) · Journal of Clinical Psychology · doi:10.1002/jclp.22210 · Cited by 269 · cited 12×
“Furthermore, there was comparatively little agreement with the correct risk categories according to Chu et al. (2015) – particularly in the “moderate” and “severe” risk categories.”
InfluentialHawton, K., Lascelles, K., Pitman, A., Gilbert, S., & Silverman, M. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(22)00232-2 · Cited by 196 · cited 2×
“Instead, various authors advocate for a needs-based approach rather than a risk-based approach in the treatment of suicidal patients (Hawton et al., 2022; Ryan et al., 2010).”
(2001) · Cited by 150
Reformulating Suicide Risk Formulation: From Prediction to Prevention
Pisani, A. R., Murrie, D. C., & Silverman, M. M. (2016) · Academic Psychiatry · doi:10.1007/s40596-015-0434-6 · Cited by 143 · cited 1×
“In addition, contextual effects (e.g., professional field of work, frequency of working with suicidal patients) might also influence the assessment of risk (cf. Pisani et al., 2016).”
The role of prediction in suicide prevention
Large, M. M. (2018) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2018.20.3/mlarge · Cited by 99 · cited 1×
“Even if psychotherapists were largely consistent in their risk assessment of suicidal patients, various findings uniformly indicate that respective risk categories lack predictive validity regarding future suicidal behavior (Bryan, 2021; Hawton et al., 2022; Large, 2018).”
Research works printed but not linked: 2
- Gwet, K. L. (2021). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters - Volume 1: Analysis of categorical ratings (4th ed., Vol. 1). Advances Analytics LLC.
- 10.3109/10398562.2010.50781 · doi:10.3109/10398562.2010.50781
Cited by
Suicide- and crisis-risk detection using large language models in mental-health chatbotsPreprint
Weber, S., Klebe, D., Wolf, L., Aeberli, C., Homan, S., Psathakis, N., Casanova, A., Macias, J. A. G., Sykstus, J., Li, J., et al. (2026) · medRxiv · doi:10.64898/2026.01.12.26343914 · Cited by 12
Suicide risk assessment: clinical implications of the unpredictability of suicidal behavior
Teismann, T., Janssen, W. C., & Heering, H. D. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1844322 · Cited by 3
“Various studies have demonstrated that the reliability of risk assessments is moderate at best (9), and that the prediction of suicidal behavior, is currently insuf fi cient for clinically useful individual-level prediction (10).”
Self-citationKumari, D. P., Gardner, K. J., & Taylor, P. J. (2025) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph22101563 · Cited by 2
“This mirrors findings from Cahill & Rakow [70] and Kolochowski et al. [71], who emphasised that clinician bias and the variability of clinicians’ interpretations of intent can lead to inconsistencies in treatment decisions.”
Landau, C., Getty, P., Gruler, C., Stadje, R., Arampatzi, S., Mandal, A., Goel, A., Gurevych, I., Reif, A., & Grimm, O. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1616955 · Cited by 2
“Unstructured suicide risk assessment shows substantially lower concordance: psychotherapists demonstrate Gwet’s AC1 = 0.44 with only 58% identical risk classifications, while psychology students achieve AC1 = 0.35 (65).”
InfluentialSuicide-risk implications in legalising medical assistance in dying for mental disorders
Gay, M. (2025) · Mortality · doi:10.1080/13576275.2025.2481265 · Cited by 2
“Current psychometric tools show poor reliability with varying degrees of specificity and sensitivity (T. Carter et al., 2018; Kolochowski et al., 2024; Saab et al., 2021).”
Metadata
- Authors
- Kolochowski, F. D., Kreckeler, N., Forkmann, T., & Teismann, T.
- Affiliations
- Ruhr University Bochum (DE)
- Corresponding author
- Teismann, Tobias
- Venue
- Archives of Suicide Research
- Publisher
- Informa UK Limited
- Published
- 2024-07-24
- Volume
- 29
- Issue
- 2
- Pages
- 544-555
- ISSN
- 1381-1118 (ISSN-L)
- 1543-6136
- OpenAlex
- W4400955811
- Cited by
- 8
- References
- 29
- Citations by year
- 2026: 6 · 2025: 2
- FWCI
- 3.99
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Archives of Suicide Research (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Psychology › Clinical Psychology › Psychopathy, Forensic Psychiatry, Sexual Offending
- Keywords
- suicide risk assessment, inter-rater reliability, intra-rater reliability, case vignettes, psychotherapists, psychology students
- MeSH terms
- Psychotherapists, Suicide Prevention, Adult, Female, Humans, Male, Middle Aged, Students, Suicide, Reproducibility of Results, Observer Variation, Risk Assessment, Young Adult, Suicidal Ideation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reliability of Suicide Risk Estimates: A Vignette Study. https://app.cochat.ai/papers/10.1080/13811118.2024.2382709
- The paper
- Kolochowski, F. D., Kreckeler, N., Forkmann, T., & Teismann, T. (2025). Reliability of Suicide Risk Estimates: A Vignette Study. Archives of Suicide Research. https://doi.org/10.1080/13811118.2024.2382709
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.