The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Predicting Suicide Attempts and Suicide Deaths Following Outpatient Visits Using Electronic Health Records
Open accessAbstract
OBJECTIVE: The authors sought to develop and validate models using electronic health records to predict suicide attempt and suicide death following an outpatient visit. METHOD: Across seven health systems, 2,960,929 patients age 13 or older (mean age, 46 years; 62% female) made 10,275,853 specialty mental health visits and 9,685,206 primary care visits with mental health diagnoses between Jan. 1, 2009, and June 30, 2015. Health system records and state death certificate data identified suicide attempts (N=24,133) and suicide deaths (N=1,240) over 90 days following each visit. Potential predictors included 313 demographic and clinical characteristics extracted from records for up to 5 years before each visit: prior suicide attempts, mental health and substance use diagnoses, medical diagnoses, psychiatric medications dispensed, inpatient or emergency department care, and routinely administered depression questionnaires. Logistic regression models predicting suicide attempt and death were developed using penalized LASSO (least absolute shrinkage and selection operator) variable selection in a random sample of 65% of the visits and validated in the remaining 35%. RESULTS: Mental health specialty visits with risk scores in the top 5% accounted for 43% of subsequent suicide attempts and 48% of suicide deaths. Of patients scoring in the top 5%, 5.4% attempted suicide and 0.26% died by suicide within 90 days. C-statistics (equivalent to area under the curve) for prediction of suicide attempt and suicide death were 0.851 (95% CI=0.848, 0.853) and 0.861 (95% CI=0.848, 0.875), respectively. Primary care visits with scores in the top 5% accounted for 48% of subsequent suicide attempts and 43% of suicide deaths. C-statistics for prediction of suicide attempt and suicide death were 0.853 (95% CI=0.849, 0.857) and 0.833 (95% CI=0.813, 0.853), respectively. CONCLUSIONS: Prediction models incorporating both health record data and responses to self-report questionnaires substantially outperform existing suicide risk prediction tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One study of machine learning of health records for the prediction of suicide and suicide attempts examined the medical and psychiatric records of ~3 million patients with at least one mental health diagnosis across seven different health care systems 222 .”
From Suicide and suicide risk (2019) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this study, an algorithm was developed that could identify 5% of the population that accounted for 43-48% of the suicides and suicide attempts, and was able to predict suicide and suicide attempts in the next 90 days with a specificity of ~70% and a sensitivity of ~80% in both specialty mental health and primary care settings 222 .”
From Suicide and suicide risk (2019) · cites this paper as [222]
Experimental — accuracy not yet measuredIn the citing paper
“multiple research groups have developed suicide risk prediction models using electronic health record data.10,11,12,13”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“and 2% to 6%11,12 for predicting suicide attempts”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“The corresponding PPVs were 0.1% to 0.3% for predicting suicide death12,13”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“48%, from a model developed in a multistate collaboration.12”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“even higher PPVs (>10%)12 could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified.”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Validation of a combined comorbidity index
Charlson, M., Szatrowski, T. P., Peterson, J., & Gold, J. (1994) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)90129-5 · Cited by 6,587
Posner, K., Brown, G. K., Stanley, B., Brent, D. A., Yershova, K. V., Oquendo, M. A., Currier, G. W., Melvin, G. A., Greenhill, L., Shen, S., et al. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.10111704 · Cited by 5,157
Kroenke, K., Spitzer, R. L., Williams, J. B. W., & Löwe, B. (2010) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2010.03.006 · Cited by 4,057
Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017) · Psychological Bulletin · doi:10.1037/bul0000084 · Cited by 3,784
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Logistic regression and Artificial Neural Network Classification models: A methodology review
Dreiseitl, S., & Ohno-Machado, L. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00034-0 · Cited by 2,177
Signal detection theory and ROC analysis
Egan, J. P. (1975) · Academic Press eBooks · Cited by 1,659
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066
Cognitive Therapy for the Prevention of Suicide Attempts: A Randomized Controlled Trial
Brown, G. K., Ten Have, T., Henriques, G. R., Xie, S. X., Hollander, J. E., & Beck, A. T. (2005) · JAMA · doi:10.1001/jama.294.5.563 · Cited by 1,045
Self-citationHealth care contacts in the year before suicide death.
Ahmedani, B. K., Simon, G. E., Stewart, C., Beck, A., Waitzfelder, B. E., Rossom, R., Lynch, F., Owen-Smith, A., Hunkeler, E. M., Whiteside, U., et al. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-2767-3 · Cited by 671
Self-citationPredicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 571
Taylor, R. A., Pare, J. R., Venkatesh, A. K., Mowafi, H. O., Melnick, E. R., Fleischman, W. M., & Hall, M. K. (2015) · Academic Emergency Medicine · doi:10.1111/acem.12876 · Cited by 513
Kessler, R. C., Warner, C. H., Ivany, C., Petukhova, M. V., Rose, S., Bromet, E. J., Brown, M., Cai, T., Colpe, L. J., Cox, K. L., et al. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.1754 · Cited by 447
Suicide Prevention in an Emergency Department Population: The ED-SAFE Study
Miller, I. W., Camargo, C. A., Arias, S. A., Sullivan, A. F., Allen, M. H., Goldstein, A. B., Manton, A. P., Espinola, J. A., Jones, R., Hasegawa, K., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0678 · Cited by 418
The HMO Research Network Virtual Data Warehouse: A Public Data Model to Support Collaboration
Ross, T. R., Ng, D., Brown, J. S., Pardee, R., Hornbrook, M., Hart, G., & Steiner, J. (2014) · eGEMs · doi:10.13063/2327-9214.1049 · Cited by 400
Frizzell, J. D., Liang, L., Schulte, P. J., Yancy, C. W., Heidenreich, P. A., Hernandez, A. F., Bhatt, D. L., Fonarow, G. C., & Laskey, W. K. (2017) · JAMA Cardiology · doi:10.1001/jamacardio.2016.3956 · Cited by 361
Predicting suicidal behavior from longitudinal electronic health records.
Barak-Corren, Y., Castro, V. M., Javitt, S., Hoffnagle, A. G., Dai, Y., Perlis, R. H., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.16010077 · Cited by 339
Rana, J. S., Tabada, G. H., Solomon, M. D., Lo, J. C.-M., Jaffe, M. G., Sung, S. H., Ballantyne, C. M., & Go, A. S. (2016) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2016.02.055 · Cited by 299
McCarthy, J. F., Bossarte, R. M., Katz, I. R., Thompson, C., Kemp, J., Hannemann, C. M., Nielson, C., & Schoenbaum, M. (2015) · American Journal of Public Health · doi:10.2105/ajph.2015.302737 · Cited by 251
Kessler, R. C., Hwang, I., Hoffmire, C. A., McCarthy, J. F., Petukhova, M. V., Rosellini, A. J., Sampson, N. A., Schneider, A. L., Bradley, P. A., Katz, I. R., et al. (2017) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1575 · Cited by 209
Simon, G. E., Coleman, K. J., Rossom, R. C., Beck, A., Oliver, M., Johnson, E., Whiteside, U., Operskalski, B., Penfold, R. B., Shortreed, S. M., et al. (2016) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.15m09776 · Cited by 194
Self-citationKessler, R. C., Stein, M. B., Petukhova, M. V., Bliese, P., Bossarte, R. M., Bromet, E. J., Fullerton, C. S., Gilman, S. E., Ivany, C., Lewandowski-Romps, L., et al. (2017) · Molecular Psychiatry · doi:10.1038/mp.2016.110 · Cited by 163
Suicide prevention: an emerging priority for health care
Hogan, M. F., & Grumet, J. G. (2016) · Health Affairs · doi:10.1377/hlthaff.2015.1672 · Cited by 159
Mortality in the United States, 2016.
Kochanek, K. D., Murphy, S. L., Xu, J., & Arias, E. (2017) · PubMed · Cited by 155
Detecting and treating suicide ideation in all settings.
(2016) · PubMed · Cited by 143
Racial/Ethnic Differences in Health Care Visits Made Before Suicide Attempt Across the United States
Ahmedani, B. K., Stewart, C., Simon, G. E., Lynch, F., Lu, C. Y., Waitzfelder, B. E., Solberg, L. I., Owen-Smith, A. A., Beck, A., Copeland, L. A., et al. (2015) · Medical Care · doi:10.1097/mlr.0000000000000335 · Cited by 141
Self-citationPsychosocial treatments of suicidal behaviors: A practice‐friendly review
Comtois, K. A., & Linehan, M. M. (2006) · Journal of Clinical Psychology · doi:10.1002/jclp.20220 · Cited by 139
McCoy, T. H., Castro, V. M., Roberson, A. M., Snapper, L. A., & Perlis, R. H. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.2172 · Cited by 130
Focusing Suicide Prevention on Periods of High Risk
Olfson, M., Marcus, S. C., & Bridge, J. A. (2014) · JAMA · doi:10.1001/jama.2014.501 · Cited by 106
The accuracy of suicide statistics: are true suicide deaths misclassified?
Bakst, S. S., Braun, T., Zucker, I., Amitai, Z., & Shohat, T. (2016) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-015-1119-x · Cited by 91
Machine Learning and Electronic Health Records: A Paradigm Shift
Adkins, D. E. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.16101169 · Cited by 58
Frost, D., Vembu, S., Wang, J., Tu, K., Morris, Q., & Abrams, H. B. (2017) · The American Journal of Medicine · doi:10.1016/j.amjmed.2016.12.008 · Cited by 51
Changes in Coding of Suicide Attempts or Self-Harm With Transition From ICD-9 to ICD-10
Stewart, C. C., Crawford, P. M., & Simon, G. E. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201600450 · Cited by 49
Self-citationMental Health Treatment Patterns Among Adults With Recent Suicide Attempts in the United States
Han, B., Compton, W. M., Gfroerer, J., & McKeon, R. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.302163 · Cited by 39
How complete are E‐codes in commercial plan claims databases?
Lu, C. Y., Stewart, C. C., Ahmed, A. T., Ahmedani, B. K., Coleman, K. J., Copeland, L. A., Hunkeler, E. M., Lakoma, M. D., Madden, J. M., Penfold, R. B., et al. (2014) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3551 · Cited by 36
Self-citationCan We Predict Stroke in Atrial Fibrillation?
Lip, G. Y. H. (2012) · Clinical Cardiology · doi:10.1002/clc.20969 · Cited by 35
Between-visit changes in suicidal ideation and risk of subsequent suicide attempt
Simon, G. E., Shortreed, S. M., Johnson, E., Beck, A., Coleman, K. J., Rossom, R. C., Whiteside, U. S., Operskalski, B. H., & Penfold, R. B. (2017) · Depression and Anxiety · doi:10.1002/da.22623 · Cited by 34
Self-citationFacilitating Action for Suicide Prevention by Learning Health Care Systems
Rossom, R. C., Simon, G. E., Beck, A. L., Ahmedani, B. K., Steinfeld, B. I., Trangle, M. A., & Solberg, L. I. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201600068 · Cited by 30
Self-citationCox, K. L., Nock, M. K., Biggs, Q. M., Bornemann, J., Colpe, L. J., Dempsey, C. L., Heeringa, S. G., McCarroll, J. E., Ng, T. H. H., Schoenbaum, M., et al. (2016) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12280 · Cited by 27
Flagged first, then by citations
Research works printed but not linked: 1
- Ross TR, 2014, EGEMS (Wash DC), 2, 1049
Cited by
Sorted by citations
Turecki, G., Brent, D. A., Gunnell, D., O’Connor, R. C., Oquendo, M. A., Pirkis, J., & Stanley, B. H. (2019) · Nature Reviews Disease Primers · doi:10.1038/s41572-019-0121-0 · Cited by 1,074
Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0174 · Cited by 540
Reynolds, C. F., Jeste, D. V., Sachdev, P. S., & Blazer, D. G. (2022) · World Psychiatry · doi:10.1002/wps.20996 · Cited by 465
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
“In addition, risk assessment mainly depends on patient-reported data [37] and, and patients may be prone to hide suicidal notions [38].”
BackgroundUnderstanding Links among Opioid Use, Overdose, and Suicide
Bohnert, A. S. B., & Ilgen, M. A. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1802148 · Cited by 333
Metadata
- Authors
- Simon, G. E., Johnson, E., Lawrence, J. M., Rossom, R. C., Ahmedani, B., Lynch, F. L., Beck, A., Waitzfelder, B., Ziebell, R., Penfold, R. B., & Shortreed, S. M.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- ORCID
- Venue
- American Journal of Psychiatry
- Published
- 2018-10-01
- Volume
- 175
- Issue
- 10
- Pages
- 951-960
- ISSN
- 0002-953X (ISSN-L)
- 1535-7228
- PMC
- PMC6167136
- OpenAlex
- W2804266670
- Cited by
- 400
- References
- 46
- Citations by year
- 2026: 42 · 2025: 57 · 2024: 37 · 2023: 56 · 2022: 53 · 2021: 49 · 2020: 55 · 2019: 44 · 2018: 7
- FWCI
- 45.53
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (U19MH092201)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- suicide attempts, suicide mortality, suicide risk prediction, outpatient mental health visits, primary care visits, electronic health records, LASSO feature selection, risk score, substance use, psychiatric medication, mental health diagnoses, prediction model validation, death certificates, emergency department care, inpatient care, self-report questionnaire
- MeSH terms
- Adolescent, Adult, Aged, Female, Humans, Male, Middle Aged, Models, Psychological, Outpatients, Surveys and Questionnaires, Risk Factors, Suicide, Attempted, Young Adult, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
