The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Predicting Suicide Attempts and Suicide Deaths Following Outpatient Visits Using Electronic Health Records
Open accessAbstract
OBJECTIVE: The authors sought to develop and validate models using electronic health records to predict suicide attempt and suicide death following an outpatient visit. METHOD: Across seven health systems, 2,960,929 patients age 13 or older (mean age, 46 years; 62% female) made 10,275,853 specialty mental health visits and 9,685,206 primary care visits with mental health diagnoses between Jan. 1, 2009, and June 30, 2015. Health system records and state death certificate data identified suicide attempts (N=24,133) and suicide deaths (N=1,240) over 90 days following each visit. Potential predictors included 313 demographic and clinical characteristics extracted from records for up to 5 years before each visit: prior suicide attempts, mental health and substance use diagnoses, medical diagnoses, psychiatric medications dispensed, inpatient or emergency department care, and routinely administered depression questionnaires. Logistic regression models predicting suicide attempt and death were developed using penalized LASSO (least absolute shrinkage and selection operator) variable selection in a random sample of 65% of the visits and validated in the remaining 35%. RESULTS: Mental health specialty visits with risk scores in the top 5% accounted for 43% of subsequent suicide attempts and 48% of suicide deaths. Of patients scoring in the top 5%, 5.4% attempted suicide and 0.26% died by suicide within 90 days. C-statistics (equivalent to area under the curve) for prediction of suicide attempt and suicide death were 0.851 (95% CI=0.848, 0.853) and 0.861 (95% CI=0.848, 0.875), respectively. Primary care visits with scores in the top 5% accounted for 48% of subsequent suicide attempts and 43% of suicide deaths. C-statistics for prediction of suicide attempt and suicide death were 0.853 (95% CI=0.849, 0.857) and 0.833 (95% CI=0.813, 0.853), respectively. CONCLUSIONS: Prediction models incorporating both health record data and responses to self-report questionnaires substantially outperform existing suicide risk prediction tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One study of machine learning of health records for the prediction of suicide and suicide attempts examined the medical and psychiatric records of ~3 million patients with at least one mental health diagnosis across seven different health care systems 222 .”
From Suicide and suicide risk (2019) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this study, an algorithm was developed that could identify 5% of the population that accounted for 43-48% of the suicides and suicide attempts, and was able to predict suicide and suicide attempts in the next 90 days with a specificity of ~70% and a sensitivity of ~80% in both specialty mental health and primary care settings 222 .”
From Suicide and suicide risk (2019) · cites this paper as [222]
Experimental — accuracy not yet measuredIn the citing paper
“multiple research groups have developed suicide risk prediction models using electronic health record data.10,11,12,13”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“and 2% to 6%11,12 for predicting suicide attempts”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“The corresponding PPVs were 0.1% to 0.3% for predicting suicide death12,13”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“48%, from a model developed in a multistate collaboration.12”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“even higher PPVs (>10%)12 could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified.”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Validation of a combined comorbidity index
Charlson, M., Szatrowski, T. P., Peterson, J., & Gold, J. (1994) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)90129-5 · Cited by 6,587
Posner, K., Brown, G. K., Stanley, B., Brent, D. A., Yershova, K. V., Oquendo, M. A., Currier, G. W., Melvin, G. A., Greenhill, L., Shen, S., et al. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.10111704 · Cited by 5,157
Kroenke, K., Spitzer, R. L., Williams, J. B. W., & Löwe, B. (2010) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2010.03.006 · Cited by 4,057
Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017) · Psychological Bulletin · doi:10.1037/bul0000084 · Cited by 3,784
Research works printed but not linked: 1
- Ross TR, 2014, EGEMS (Wash DC), 2, 1049
Cited by
Sorted by citations
Turecki, G., Brent, D. A., Gunnell, D., O’Connor, R. C., Oquendo, M. A., Pirkis, J., & Stanley, B. H. (2019) · Nature Reviews Disease Primers · doi:10.1038/s41572-019-0121-0 · Cited by 1,074
Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0174 · Cited by 540
Reynolds, C. F., Jeste, D. V., Sachdev, P. S., & Blazer, D. G. (2022) · World Psychiatry · doi:10.1002/wps.20996 · Cited by 465
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
“In addition, risk assessment mainly depends on patient-reported data [37] and, and patients may be prone to hide suicidal notions [38].”
BackgroundUnderstanding Links among Opioid Use, Overdose, and Suicide
Bohnert, A. S. B., & Ilgen, M. A. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1802148 · Cited by 333
Metadata
- Authors
- Simon, G. E., Johnson, E., Lawrence, J. M., Rossom, R. C., Ahmedani, B., Lynch, F. L., Beck, A., Waitzfelder, B., Ziebell, R., Penfold, R. B., & Shortreed, S. M.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- ORCID
- Venue
- American Journal of Psychiatry
- Published
- 2018-10-01
- Volume
- 175
- Issue
- 10
- Pages
- 951-960
- ISSN
- 0002-953X (ISSN-L)
- 1535-7228
- PMC
- PMC6167136
- OpenAlex
- W2804266670
- Cited by
- 400
- References
- 46
- Citations by year
- 2026: 42 · 2025: 57 · 2024: 37 · 2023: 56 · 2022: 53 · 2021: 49 · 2020: 55 · 2019: 44 · 2018: 7
- FWCI
- 45.53
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (U19MH092201)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- suicide attempts, suicide mortality, suicide risk prediction, outpatient mental health visits, primary care visits, electronic health records, LASSO feature selection, risk score, substance use, psychiatric medication, mental health diagnoses, prediction model validation, death certificates, emergency department care, inpatient care, self-report questionnaire
- MeSH terms
- Adolescent, Adult, Aged, Female, Humans, Male, Middle Aged, Models, Psychological, Outpatients, Surveys and Questionnaires, Risk Factors, Suicide, Attempted, Young Adult, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
