Improving Prediction of Suicide and Accidental Death After Discharge From General Hospitals With Natural Language Processing
Open accessAbstract
IMPORTANCE: Suicide represents the 10th leading cause of death across age groups in the United States (12.6 cases per 100 000) and remains challenging to predict. While many individuals who die by suicide are seen by physicians before their attempt, they may not seek psychiatric care. OBJECTIVE: To determine the extent to which incorporating natural language processing of narrative discharge notes improves stratification of risk for death by suicide after medical or surgical hospital discharge. DESIGN, SETTING, AND PARTICIPANTS: In this retrospective health care use study, clinical data were analyzed from individuals with discharges from 2 large academic medical centers between January 1, 2005, and December 31, 2013. MAIN OUTCOMES AND MEASURES: The primary outcome was suicide as a reported cause of death based on Massachusetts Department of Public Health records. Regression models for prediction of death by suicide or accidental death were compared relying solely on coded clinical data and those using natural language processing of hospital discharge notes. RESULTS: There were 845 417 hospital discharges represented in the cohort, including 458 053 unique individuals. Overall, all-cause mortality was 18% during 9 years, and the median follow-up was 5.2 years. The cohort included 235 (0.1%) who died by suicide during 2.4 million patient-years of follow-up. Positive valence reflected in narrative notes was associated with a 30% reduction in risk for suicide in models adjusted for coded sociodemographic and clinical features (hazard ratio, 0.70; 95% CI, 0.58-0.85; P < .001) and improved model fit (χ22 = 14.843, P < .001 by log-likelihood test). The C statistic was 0.741 (95% CI, 0.738-0.744) for models of suicide with or without inclusion of accidental death. CONCLUSIONS AND RELEVANCE: Multiple clinical features available at hospital discharge identified a cohort of individuals at substantially increased risk for suicide. Greater positive valence expressed in narrative discharge summaries was associated with substantially diminished risk. Automated tools to aid clinicians in evaluating these risks may assist in identifying high-risk individuals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors' prior study found that among patients identified through an NLP-based approach as having PHQ-9 scores, 31% showed scores suggesting major depressive disorder, yet lacked a corresponding structured ICD or SNOMED diagnosis code.
“Our prior study found that among patients identified through NLP-based approach as having PHQ-9 scores—a clinical instrument measuring depression severity—31% showed scores suggesting major depressive disorder, yet lacked a corresponding structured ICD or SNOMED diagnosis code39.”
From Deep learning for identifying personal and family history of suicidal thoughts and behaviors from EHRs (2024) · cites this paper as [39]
The Discussion reports that 48.9% of patients who died by suicide had no coded psychiatric ICD-9 diagnosis, but it does not report the claimed PHQ-9 finding. Searches for PHQ-9, major depressive disorder, SNOMED, and 31% found no passage in our copy addressing that result. The available Results instead concern NLP-derived language features and suicide-risk prediction; page numbers were not available. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 40 · Cited by 130
Research works printed but not linked: 26
- [2]Xu J, Murphy SL, Kochanek KD, Bastian BA. Deaths: final data for 2013. Natl Vital Stat Rep. 2016; 64(2): 1– 119. 26905861
- [6]Niculescu AB, Levey D, Le-Niculescu H, Niculescu E, Kurian SM, Salomon D. Psychiatric blood biomarkers: avoiding jumping to premature negative or positive conclusions. Mol Psychiatry. 2015; 20(3): 286– 288. 25582618 PMC4357859
- [7]Motto JA, Heilbron DC, Juster RP. Development of a clinical instrument to estimate suicide risk. Am J Psychiatry. 1985; 142(6): 680– 686. 4003585
- [9]Dobscha SK, Denneson LM, Kovas AE, Correlates of suicide among veterans treated in primary care: case-control study of a nationally representative sample. J Gen Intern Med. 2014; 29 (suppl 4): 853– 860. 25355088 PMC4239287
- [11]Hirschfeld RM, Russell JM. Assessment and treatment of suicidal patients. N Engl J Med. 1997; 337(13): 910– 915. 9302306
- [13]Tidemalm D, Långström N, Lichtenstein P, Runeson B. Risk of suicide after suicide attempt according to coexisting psychiatric disorder: Swedish cohort study with long term follow-up. BMJ. 2008; 337(7682): 1328– 1333. PMC2590902 19018040
- [14]Luoma JB, Martin CE, Pearson JL. Contact withmental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry. 2002; 159(6): 909– 916. 12042175 PMC5072576
- [15]McCoy TH, Castro VM, Cagan A, Roberson AM, Kohane IS, Perlis RH. Sentiment measured in hospital discharge notes is associated with readmission and mortality risk: an electronic health record study. PLoS One. 2015; 10(8): e0136341. doi: 10.1371/journal.pone.0136341. 26302085 PMC4547711 · DOI
- [16]Bourne LE. Stimulus selection and memorystrategies in concept problems. Mem Cognit. 1974; 2 (2): 270– 274. 24214753
- [17]Murphy SN, Mendis M, Hackett K,. Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside. AMIA Annu Symp Proc. 2007: 548– 552. 18693896 PMC2655844
- [18]Murphy SN, Weber G, Mendis M, Serving the enterprise and beyond with Informatics for Integrating Biology and the Bedside (i2b2). J Am Med Inform Assoc. 2010; 17(2): 124– 130. 20190053 PMC3000779
- [19]Murphy S, Churchill S, Bry L,. Instrumenting the health care enterprise for discovery research in the genomic era. Genome Res. 2009; 19(9): 1675– 1681. 19602638 PMC2752136
- [20]Castro VM, McCoy TH, Cagan A,. Stratification of risk for hospital admissions for injury related to fall: cohort study. BMJ. 2014; 349: g5863. 25954985 PMC4208628
- [21]De Smedt T, Daelemans W. Pattern for Python. J Mach Learn Res. 2012; 13: 2063– 2067.
- [23]Pang B, Lee L, Vaithyanathan S. Thumbs up?sentiment classification using machine learning techniques. In: Proceedings from the Association for Computational Linguistics’ Empirical Methods in Natural Language Processing; July 7–12, 2002; Philadelphia, PA.
- [24]Mitchell L, Frank MR, Harris KD, Dodds PS, Danforth CM. The geography of happiness: connecting Twitter sentiment and expression, demographics, and objective characteristics of place. PLoS One. 2013; 8(5): e64417. doi: 10.1371/journal.pone.0064417. 23734200 PMC3667195 · DOI
- [25]Therneau T, Grambsch P, Pankratz V. Penalizedsurvival models and frailty. J Comput Graph Stat. 2003; 12(1): 156– 175.
- [27]Perlis RH, Iosifescu DV, Castro VM, Using electronic medical records to enable large-scale studies in psychiatry: treatment resistant depression as a model. Psychol Med. 2012; 42(1): 4150. PMC3837420 21682950
- [29]Pencina MJ, D’Agostino RB, Pencina KM, Janssens AC, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012; 176(6): 473– 481. 22875755 PMC3530349
- [30]Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006; 26(6): 565– 574. 17099194 PMC2577036
- [31]Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008; 8(1): 53. 19036144 PMC2611975
- [32]Rumshisky A, Ghassemi M, Naumann TJ,. Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Transl Psychiatry. In press. PMC5315537 27754482
- [33]McCoy TH, Castro VM, Rosenfield HR, Cagan A, Kohane IS, Perlis RH. A clinical perspective on the relevance of research domain criteria in electronic health records. Am J Psychiatry. 2015; 172(4): 316– 320. 25827030 PMC9980718
- [34]Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014; 35(29): 1925– 1931. 24898551 PMC4155437
- [35]Steyerberg EW, Vickers AJ, Cook NR,. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010; 21(1): 128– 138. 20010215 PMC3575184
- [36]Clark IA, Cowden WB, Butcher GA, Hunt NH. Possible roles of tumor necrosis factor in the pathology of malaria. Am J Pathol. 1987; 129(1): 192199. PMC1899696 3661678
- Authors
- McCoy, T. H., Castro, V. M., Roberson, A. M., Snapper, L. A., & Perlis, R. H.
- Affiliations
- Massachusetts General Hospital (US)
- Corresponding author
- Perlis, Roy H.
- Venue
- JAMA Psychiatry
- Publisher
- American Medical Association (AMA)
- Published
- 2016-10-01
- Volume
- 73
- Issue
- 10
- Pages
- 1064
- ISSN
- 2168-622X (ISSN-L)
- PMC
- PMC9980717
- OpenAlex
- W2518663325
- Cited by
- 130
- References
- 40
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 10 · 2023: 13 · 2022: 20 · 2021: 15 · 2020: 12 · 2019: 26 · 2018: 10 · 2017: 9
- FWCI
- 20.08
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- JAMA Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Stanley Center for Psychiatric Research, Broad Institute
- National Institute of Mental Health (R25MH094612, R01MH106577, P50MH106933)
- National Human Genome Research Institute
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- suicide risk prediction, accidental death, natural language processing, narrative text analysis, suicide risk stratification, positive valence, post-discharge mortality, automated risk assessment
- MeSH terms
- Academic Medical Centers, Accidents, Adult, Aged, Aged, 80 and over, Cause of Death, Female, Follow-Up Studies, Hospitals, General, Humans, Male, Massachusetts, Middle Aged, Natural Language Processing, Patient Discharge, Regression Analysis, Retrospective Studies, Suicide, Cohort Studies, Proportional Hazards Models, Risk Assessment, Narration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
