CoChat
Sign Up
Cited by 87Mental Health via WritingSuicide and Self-Harm StudiesMachine Learning in Healthcare

Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models

Open access
Levis, Maxwell, Leonard Westgate, Christine, Gui, Jiang, Watts, Bradley V., Shiner, Brian·2021·Psychological Medicine
DOI 10.1017/s0033291720000173PMC PMC8920410
Open this paper in CoChat — free account

Abstract

Background: This study evaluated whether natural language processing (NLP) of psychotherapy note text provides additional accuracy over and above currently used suicide prediction models. Methods: We used a cohort of Veterans Health Administration (VHA) users diagnosed with post-traumatic stress disorder (PTSD) between 2004–2013. Using a case-control design, cases (those that died by suicide during the year following diagnosis) were matched to controls (those that remained alive). After selecting conditional matches based on having shared mental health providers, we chose controls using a 5:1 nearest-neighbor propensity match based on the VHA's structured Electronic Medical Records (EMR)-based suicide prediction model. For cases, psychotherapist notes were collected from diagnosis until death. For controls, psychotherapist notes were collected from diagnosis until matched case's date of death. After ensuring similar numbers of notes, the final sample included 246 cases and 986 controls. Notes were analyzed using Sentiment Analysis and Cognition Engine, a Python-based NLP package. The output was evaluated using machine-learning algorithms. The area under the curve (AUC) was calculated to determine models' predictive accuracy. Results: NLP derived variables offered small but significant predictive improvement (AUC = 0.58) for patients that had longer treatment duration. A small sample size limited predictive accuracy. Conclusions: Study identifies a novel method for measuring suicide risk over time and potentially categorizing patient subgroups with distinct risk sensitivities. Findings suggest leveraging NLP derived variables from psychotherapy notes offers an additional predictive value over and above the VHA's state-of-the-art structured EMR-based suicide prediction model. Replication with a larger non-PTSD specific sample is required.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1017/s0033291720000173

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Advancements in automated detection of SI, SB, and SA from clinical notes have shown varied success.

    “Advancements in automated detection of SI, SB, and SA from clinical notes have shown varied success17,18,19,20,21,22,23,24,25,26.”

    From Deep learning for identifying personal and family history of suicidal thoughts and behaviors from EHRs (2024) · cites this paper as [18]

    The Results (page unavailable) report varied predictive performance from NLP analysis of clinical notes: little to no improvement above chance in most cohorts, but a small improvement in the 12-month cohort. The paper studies prediction of death by suicide, not automated detection of suicidal ideation (SI), suicidal behavior (SB), and suicide attempts (SA) individually. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Prior automated approaches primarily focus on identifying suicidal tendencies without considering their temporal aspects, often conflating current and historical events.

    “These approaches, ranging from rule-based algorithms to ML techniques, primarily focus on identifying suicidal tendencies without considering their temporal aspects, often conflating current and historical events.”

    From Deep learning for identifying personal and family history of suicidal thoughts and behaviors from EHRs (2024) · cites this paper as [18]

    The Discussion (page unavailable) says REACH VET variables are often static and contrasts them with a method that monitors how predictors change over time. This supports the narrower point that an existing automated model may miss temporal changes. The paper does not say prior approaches generally conflate current and historical events, or establish that limitation for rule-based approaches. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Levis et al. observed an 8% increase in predictability in the 12-month study cohort compared to more advanced available methods for suicide risk.

    “Specifically, an 8% increase in predictability was observed in the 12-month study cohort compared to more advanced available methods.”

    From Artificial intelligence in healthcare: transforming patient safety with intelligent systems—A systematic review (2025) · cites this paper as [76]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 72 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Lasswell HD, & Namenwirth JZ (1969). The Lasswell value dictionary. New Haven, CT: Yale Press.
  2. Stone PJ, Dunphy DC, Smith MS, & Ogilvie DM (1966). The general inquirer: A computer approach to content analysis: Studies in psychology, sociology, anthropology, and political science. Cambridge, MA: MIT Press.

Cited by

Sorted by citations

Related

Authors
Levis, M., Leonard Westgate, C., Gui, J., Watts, B. V., & Shiner, B.
Affiliations
Dartmouth College (US)
Corresponding author
Levis, Maxwell
ORCID
Venue
Psychological Medicine
Publisher
Cambridge University Press (CUP)
Published
2020-02-17
Volume
51
Issue
8
Pages
1382-1391
ISSN
  • 0033-2917 (ISSN-L)
  • 1469-8978
DOI
10.1017/s0033291720000173
PMC
PMC8920410
OpenAlex
W3008864515
Cited by
87
References
72
Citations by year
2026: 7 · 2025: 17 · 2024: 15 · 2023: 19 · 2022: 22 · 2021: 5 · 2020: 2
FWCI
13.12
Type
journal article
Language
en
Open access
Licence
www.cambridge.org/core/terms (unspecified)
Locations
  • Psychological Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
U.S. Department of Veterans Affairs (CDA11-263, IK2 HX000854)
Topics
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
  • Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
natural language processing, suicide risk prediction, post-traumatic stress disorder, veterans, electronic health records, sentiment analysis, treatment duration
MeSH terms
Machine Learning, Suicide Prevention, Algorithms, Humans, Mental Health, Natural Language Processing, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review