CoChat
Sign Up
Cited by 78Suicide and Self-Harm StudiesMachine Learning in HealthcareElectronic Health Records Systems

Prospective Validation of an Electronic Health Record–Based, Real-Time Suicide Risk Model

Open access
Walsh, Colin G., Johnson, Kevin B., Ripperger, Michael, Sperry, Sarah, Harris, Joyce, Clark, Nathaniel, Fielstein, Elliot, Novak, Laurie +2 more·2021·JAMA Network Open
DOI 10.1001/jamanetworkopen.2021.1428PMC PMC7955273
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Numerous prognostic models of suicide risk have been published, but few have been implemented outside of integrated managed care systems. Objective: To evaluate performance of a suicide attempt risk prediction model implemented in a vendor-supplied electronic health record to predict subsequent (1) suicidal ideation and (2) suicide attempt. Design, Setting, and Participants: This observational cohort study evaluated implementation of a suicide attempt prediction model in live clinical systems without alerting. The cohort comprised patients seen for any reason in adult inpatient, emergency department, and ambulatory surgery settings at an academic medical center in the mid-South from June 2019 to April 2020. Main Outcomes and Measures: Primary measures assessed external, prospective, and concurrent validity. Manual medical record validation of coded suicide attempts confirmed incident behaviors with intent to die. Subgroup analyses were performed based on demographic characteristics, relevant clinical context/setting, and presence or absence of universal screening. Performance was evaluated using discrimination (number needed to screen, C statistics, positive/negative predictive values) and calibration (Spiegelhalter z statistic). Recalibration was performed with logistic calibration. Results: The system generated 115 905 predictions for 77 973 patients (42 490 [54%] men, 35 404 [45%] women, 60 586 [78%] White, 12 620 [16%] Black). Numbers needed to screen in highest risk quantiles were 23 and 271 for suicidal ideation and attempt, respectively. Performance was maintained across demographic subgroups. Numbers needed to screen for suicide attempt by sex were 256 for men and 323 for women; and by race: 373, 176, and 407 for White, Black, and non-White/non-Black patients, respectively. Model C statistics were, across the health system: 0.836 (95% CI, 0.836-0.837); adult hospital: 0.77 (95% CI, 0.77-0.772); emergency department: 0.778 (95% CI, 0.777-0.778); psychiatry inpatient settings: 0.634 (95% CI, 0.633-0.636). Predictions were initially miscalibrated (Spiegelhalter z = -3.1; P = .001) with improvement after recalibration (Spiegelhalter z = 1.1; P = .26). Conclusions and Relevance: In this study, this real-time predictive model of suicide attempt risk showed reasonable numbers needed to screen in nonpsychiatric specialty settings in a large clinical system. Assuming that research-valid models will translate without performing this type of analysis risks inaccuracy in clinical practice, misclassification of risk, wasted effort, and missed opportunity to correct and prevent such problems. The next step is careful pairing with low-cost, low-harm preventive strategies in a pragmatic trial of effectiveness in preventing future suicidality.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2021.1428

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior automated approaches primarily focus on identifying suicidal tendencies without considering their temporal aspects, often conflating current and historical events.

    “These approaches, ranging from rule-based algorithms to ML techniques, primarily focus on identifying suicidal tendencies without considering their temporal aspects, often conflating current and historical events.”

    From Deep learning for identifying personal and family history of suicidal thoughts and behaviors from EHRs (2024) · cites this paper as [17]

    The Methods > Real-Time Prediction section describes an automated model that explicitly separates five years of historical predictors from suicide attempts or suicidal ideation in the subsequent 30 days. Methods > Study Outcomes also specifies a 30-day prediction target. The paper therefore describes an approach that considers timing rather than conflating historical and future events; it does not establish how common that problem is among other approaches. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Advancements in automated detection of SI, SB, and SA from clinical notes have shown varied success.

    “Advancements in automated detection of SI, SB, and SA from clinical notes have shown varied success17,18,19,20,21,22,23,24,25,26.”

    From Deep learning for identifying personal and family history of suicidal thoughts and behaviors from EHRs (2024) · cites this paper as [17]

    The Discussion’s Limitations section says this model used structured EHR data and did not capture information in unstructured clinical notes. Although the paper evaluates automated prediction of suicidal ideation and suicide attempts, it does not describe detecting them from clinical notes or compare the success of note-based detection methods. No page number is available in our copy. (LLM, conf=0.87)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. [2]Gunnell D, Appleby L, Arensman E,; COVID-19 Suicide Prevention Research Collaboration. Suicide risk and prevention during the COVID-19 pandemic. Lancet Psychiatry. 2020; 7(6): 468- 471. doi: 10.1016/S2215-0366(20)30171-1 32330430 PMC7173821 · DOI · Paper page
  2. [3]Kawohl W, Nordt C. COVID-19, unemployment, and suicide. Lancet Psychiatry. 2020; 7(5): 389- 390. doi: 10.1016/S2215-0366(20)30141-3 32353269 PMC7185950 · DOI
  3. [4]Reger MA, Stanley IH, Joiner TE. Suicide mortality and coronavirus disease 2019—a perfect storm? JAMA Psychiatry. 2020. doi: 10.1001/jamapsychiatry.2020.1060 32275300 · DOI · Paper page
  4. [7]Kline-Simon AH, Sterling S, Young-Wolff K,. Estimates of workload associated with suicide risk alerts after implementation of risk-prediction model. JAMA Netw Open. 2020; 3(10): e2021189. doi: 10.1001/jamanetworkopen.2020.21189 33084896 PMC7578770 · DOI
  5. [8]Miller IW, Camargo CA Jr, Arias SA,; ED-SAFE Investigators. Suicide prevention in an emergency department population: the ED-SAFE study. JAMA Psychiatry. 2017; 74(6): 563- 570. doi: 10.1001/jamapsychiatry.2017.0678 28456130 PMC5539839 · DOI · Paper page
  6. [9]Vannoy SD, Robins LS. Suicide-related discussions with depressed primary care patients in the USA: gender and quality gaps. a mixed methods analysis. BMJ Open. 2011; 1(2): e000198. doi: 10.1136/bmjopen-2011-000198 22021884 PMC3191598 · DOI
  7. [10]Cox DW, Ogrodniczuk JS, Oliffe JL, Kealy D, Rice SM, Kahn JH. Distress concealment and depression symptoms in a national sample of Canadian men: feeling understood and loneliness as sequential mediators. J Nerv Ment Dis. 2020; 208(6): 510- 513. doi: 10.1097/NMD.0000000000001153 32472812 · DOI
  8. [11]Laanani M, Imbaud C, Tuppin P,. Contacts with health services during the year prior to suicide death and prevalent conditions: a nationwide study. J Affect Disord. 2020; 274: 174- 182. doi: 10.1016/j.jad.2020.05.071 32469801 · DOI
  9. [12]Ahmedani BK, Simon GE, Stewart C,. Health care contacts in the year before suicide death. J Gen Intern Med. 2014; 29(6): 870- 877. doi: 10.1007/s11606-014-2767-3 24567199 PMC4026491 · DOI · Paper page
  10. [13]Ahmedani BK, Westphal J, Autio K,. Variation in patterns of health care before suicide: a population case-control study. Prev Med. 2019; 127: 105796. doi: 10.1016/j.ypmed.2019.105796 31400374 PMC6744956 · DOI · Paper page
  11. [20]Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science. 2017; 5(3): 457- 469. doi: 10.1177/2167702617691560 · DOI
  12. [21]Gradus JL, Rosellini AJ, Horváth-Puhó E,. Prediction of sex-specific suicide risk using machine learning and single-payer health care registry data from Denmark. JAMA Psychiatry. 2020; 77(1): 25- 34. doi: 10.1001/jamapsychiatry.2019.2905 31642880 PMC6813578 · DOI
  13. [22]Berrouiguet S, Barrigón ML, Castroman JL, Courtet P, Artés-Rodríguez A, Baca-García E. Combining mobile-health (mHealth) and artificial intelligence (AI) methods to avoid suicide attempts: the Smartcrises study protocol. BMC Psychiatry. 2019; 19(1): 277. doi: 10.1186/s12888-019-2260-y 31493783 PMC6731613 · DOI · Paper page
  14. [23]Chen Q, Zhang-James Y, Barnett EJ,. Predicting suicide attempt or suicide death following a visit to psychiatric specialty care: a machine learning study using Swedish national registry data. PLoS Med. 2020; 17(11): e1003416. doi: 10.1371/journal.pmed.1003416 33156863 PMC7647056 · DOI
  15. [24]Lindsell CJ, Stead WW, Johnson KB. Action-informed artificial intelligence—matching the algorithm to the problem. JAMA. 2020; 323(21): 2141- 2142. doi: 10.1001/jama.2020.5035 32356878 · DOI
  16. [25]McKernan LC, Lenert MC, Crofford LJ, Walsh CG. Outpatient engagement and predicted risk of suicide attempts in fibromyalgia. Arthritis Care Res (Hoboken). 2019; 71(9): 1255- 1263. doi: 10.1002/acr.23748 30192068 PMC6405325 · DOI
  17. [26]Isaacman DJ, Burke BL. Utility of the serum C-reactive protein for detection of occult bacterial infection in children. Arch Pediatr Adolesc Med. 2002; 156(9): 905- 909. doi: 10.1001/archpedi.156.9.905 12197798 · DOI
  18. [27]Lenert MC, Matheny ME, Walsh CG. Prognostic models will be victims of their own success, unless…. J Am Med Inform Assoc. 2019; 26(12): 1645- 1650. doi: 10.1093/jamia/ocz145 31504588 PMC6857506 · DOI
  19. [29]Hedegaard H, Schoenbaum M, Claassen C, Crosby A, Holland K, Proescholdbell S. Issues in developing a surveillance case definition for nonfatal suicide attempt and intentional self-harm using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) coded data. Natl Health Stat Report. 2018;(108): 1- 19. 29616901
  20. [30]Swain RS, Taylor LG, Braver ER, Liu W, Pinheiro SP, Mosholder AD. A systematic review of validated suicide outcome classification in observational studies. Int J Epidemiol. 2019; 48(5): 1636- 1649. doi: 10.1093/ije/dyz038 30907424 · DOI
  21. [31]Roden DM, Pulley JM, Basford MA,. Development of a large-scale de-identified DNA biobank to enable personalized medicine. Clin Pharmacol Ther. 2008; 84(3): 362- 369. doi: 10.1038/clpt.2008.89 18500243 PMC3763939 · DOI
  22. [32]Smith GCS, Seaman SR, Wood AM, Royston P, White IR. Correcting for optimistic prediction in small data sets. Am J Epidemiol. 2014; 180(3): 318- 324. doi: 10.1093/aje/kwu140 24966219 PMC4108045 · DOI
  23. [33]Singh GK. Area deprivation and widening inequalities in US mortality, 1969-1998. Am J Public Health. 2003; 93(7): 1137- 1143. doi: 10.2105/AJPH.93.7.1137 12835199 PMC1447923 · DOI
  24. [34]Holmberg L, Vickers A. Evaluation of prediction models for decision-making: beyond calibration and discrimination. PLoS Med. 2013; 10(7): e1001491. doi: 10.1371/journal.pmed.1001491 23935462 PMC3728013 · DOI
  25. [35]Steyerberg EW, Borsboom GJJM, van Houwelingen HC, Eijkemans MJ, Habbema JDF. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med. 2004; 23(16): 2567- 2586. doi: 10.1002/sim.1844 15287085 · DOI · Paper page
  26. [36]Elias J, Heuschmann PU, Schmitt C,. Prevalence dependent calibration of a predictive model for nasal carriage of methicillin-resistant Staphylococcus aureus. BMC Infect Dis. 2013; 13(1): 111. doi: 10.1186/1471-2334-13-111 23448529 PMC3599956 · DOI
  27. [37]Spiegelhalter DJ. Probabilistic prediction in patient management and clinical trials. Stat Med. 1986; 5(5): 421- 433. doi: 10.1002/sim.4780050506 3786996 · DOI
  28. [38]Walsh C, Hripcsak G. The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions. J Biomed Inform. 2014; 52: 418- 426. doi: 10.1016/j.jbi.2014.08.006 25182868 PMC4261028 · DOI · Paper page
  29. [39]Rembold CM. Number needed to screen: development of a statistic for disease screening. BMJ. 1998; 317(7154): 307- 312. doi: 10.1136/bmj.317.7154.307 9685274 PMC28622 · DOI
  30. [40]Passos IC, Ballester P. Positive predictive values and potential success of suicide prediction models. JAMA Psychiatry. 2019; 76(8): 869. doi: 10.1001/jamapsychiatry.2019.1507 31241727 · DOI
  31. [42]Davis SE, Greevy RA, Fonnesbeck C, Lasko TA, Walsh CG, Matheny ME. A nonparametric updating method to correct clinical prediction model drift. J Am Med Inform Assoc. 2019; 26(12): 1448- 1457. doi: 10.1093/jamia/ocz127 31397478 PMC6857513 · DOI
  32. [43]Institute of Medicine (US) Roundtable on Evidence-Based Medicine. The Learning Healthcare System: Workshop Summary. National Academies Press; 2007. doi: 10.17226/11903 21452449 · DOI
  33. [44]Dickerman BA, Hernán MA. Counterfactual prediction is not only for causal inference. Eur J Epidemiol. 2020; 35(7): 615- 617. doi: 10.1007/s10654-020-00659-8 32623620 PMC7393612 · DOI
  34. [45]Doshi R, Aseltine RH, Wang F, Schwartz H, Rogers S, Chen K. Illustrating the role of health information exchange in a learning health system: improving the identification and management of suicide risk. Connecticut Medicine. 2018; 82(6): 327- 333.
  35. [46]Puchi C, Tyndall B, McPheeters M, Walsh CG. Testing the feasibility of an academic-state partnership to combat the opioid epidemic in Tennessee through predictive analytics. In: Proceedings of the AMIA Annual Symposium; 2019.

Cited by

Sorted by citations

Authors
Walsh, C. G., Johnson, K. B., Ripperger, M., Sperry, S., Harris, J., Clark, N., Fielstein, E., Novak, L., Robinson, K., & Stead, W. W.
Affiliations
Vanderbilt University Medical Center (US)
Corresponding author
Walsh, Colin G.
ORCID
Venue
JAMA Network Open
Publisher
American Medical Association (AMA)
Published
2021-03-12
Volume
4
Issue
3
Pages
e211428
ISSN
2574-3805 (ISSN-L)
DOI
10.1001/jamanetworkopen.2021.1428
PMC
PMC7955273
OpenAlex
W3134494703
Cited by
78
References
9
Citations by year
2026: 16 · 2025: 17 · 2024: 15 · 2023: 15 · 2022: 13 · 2021: 2
FWCI
7.67
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
National Institute of Mental Health (R01MH121455, R01MH116269)
Topics
  • Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
suicide risk prediction, suicide attempts, suicidal ideation, electronic health records, model validation, model calibration, emergency department, psychiatric inpatient care
MeSH terms
Adult, Computer Systems, Female, Humans, Male, Middle Aged, Predictive Value of Tests, Suicide, Attempted, Models, Statistical, Cohort Studies, Risk Assessment, Electronic Health Records, Suicidal Ideation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review