The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Predicting Suicide Attempts and Suicide Deaths Following Outpatient Visits Using Electronic Health Records
Open accessAbstract
OBJECTIVE: The authors sought to develop and validate models using electronic health records to predict suicide attempt and suicide death following an outpatient visit. METHOD: Across seven health systems, 2,960,929 patients age 13 or older (mean age, 46 years; 62% female) made 10,275,853 specialty mental health visits and 9,685,206 primary care visits with mental health diagnoses between Jan. 1, 2009, and June 30, 2015. Health system records and state death certificate data identified suicide attempts (N=24,133) and suicide deaths (N=1,240) over 90 days following each visit. Potential predictors included 313 demographic and clinical characteristics extracted from records for up to 5 years before each visit: prior suicide attempts, mental health and substance use diagnoses, medical diagnoses, psychiatric medications dispensed, inpatient or emergency department care, and routinely administered depression questionnaires. Logistic regression models predicting suicide attempt and death were developed using penalized LASSO (least absolute shrinkage and selection operator) variable selection in a random sample of 65% of the visits and validated in the remaining 35%. RESULTS: Mental health specialty visits with risk scores in the top 5% accounted for 43% of subsequent suicide attempts and 48% of suicide deaths. Of patients scoring in the top 5%, 5.4% attempted suicide and 0.26% died by suicide within 90 days. C-statistics (equivalent to area under the curve) for prediction of suicide attempt and suicide death were 0.851 (95% CI=0.848, 0.853) and 0.861 (95% CI=0.848, 0.875), respectively. Primary care visits with scores in the top 5% accounted for 48% of subsequent suicide attempts and 43% of suicide deaths. C-statistics for prediction of suicide attempt and suicide death were 0.853 (95% CI=0.849, 0.857) and 0.833 (95% CI=0.813, 0.853), respectively. CONCLUSIONS: Prediction models incorporating both health record data and responses to self-report questionnaires substantially outperform existing suicide risk prediction tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One study of machine learning of health records for the prediction of suicide and suicide attempts examined the medical and psychiatric records of ~3 million patients with at least one mental health diagnosis across seven different health care systems 222 .”
From Suicide and suicide risk (2019) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this study, an algorithm was developed that could identify 5% of the population that accounted for 43-48% of the suicides and suicide attempts, and was able to predict suicide and suicide attempts in the next 90 days with a specificity of ~70% and a sensitivity of ~80% in both specialty mental health and primary care settings 222 .”
From Suicide and suicide risk (2019) · cites this paper as [222]
Experimental — accuracy not yet measuredIn the citing paper
“multiple research groups have developed suicide risk prediction models using electronic health record data.10,11,12,13”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“and 2% to 6%11,12 for predicting suicide attempts”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“The corresponding PPVs were 0.1% to 0.3% for predicting suicide death12,13”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“48%, from a model developed in a multistate collaboration.12”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
In the citing paper
“even higher PPVs (>10%)12 could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified.”
From Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US (2021) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Validation of a combined comorbidity index
Charlson, M., Szatrowski, T. P., Peterson, J., & Gold, J. (1994) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)90129-5 · Cited by 6,587
Posner, K., Brown, G. K., Stanley, B., Brent, D. A., Yershova, K. V., Oquendo, M. A., Currier, G. W., Melvin, G. A., Greenhill, L., Shen, S., et al. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.10111704 · Cited by 5,157
Kroenke, K., Spitzer, R. L., Williams, J. B. W., & Löwe, B. (2010) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2010.03.006 · Cited by 4,057
Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017) · Psychological Bulletin · doi:10.1037/bul0000084 · Cited by 3,784
Flagged first, then by citations
Research works printed but not linked: 1
- Ross TR, 2014, EGEMS (Wash DC), 2, 1049
Cited by
Sorted by citations
Turecki, G., Brent, D. A., Gunnell, D., O’Connor, R. C., Oquendo, M. A., Pirkis, J., & Stanley, B. H. (2019) · Nature Reviews Disease Primers · doi:10.1038/s41572-019-0121-0 · Cited by 1,074
Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0174 · Cited by 540
Reynolds, C. F., Jeste, D. V., Sachdev, P. S., & Blazer, D. G. (2022) · World Psychiatry · doi:10.1002/wps.20996 · Cited by 465
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
“In addition, risk assessment mainly depends on patient-reported data [37] and, and patients may be prone to hide suicidal notions [38].”
BackgroundUnderstanding Links among Opioid Use, Overdose, and Suicide
Bohnert, A. S. B., & Ilgen, M. A. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1802148 · Cited by 333
Machine learning model to predict mental health crises from electronic health records
Garriga, R., Mas, J., Abraha, S., Nolan, J., Harrison, O., Tadros, G., & Matic, A. (2022) · Nature Medicine · doi:10.1038/s41591-022-01811-5 · Cited by 229
Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R. (2020) · Molecular Psychiatry · doi:10.1038/s41380-019-0531-0 · Cited by 234
de Pablo, G. S., Studerus, E., Vaquerizo‐Serrano, J., Irving, J., Catalán, A., Oliver, D., Baldwin, H., Danese, A., Fazel, S., Steyerberg, E. W., et al. (2020) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa120 · Cited by 232
Richards, J. E., Whiteside, U., Ludman, E. J., Pabiniak, C., Kirlin, B., Hidalgo, R., & Simon, G. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201800342 · Cited by 167
Yang, Z., Mitra, A., Liu, W., Berlowitz, D., & Yu, H. (2023) · Nature Communications · doi:10.1038/s41467-023-43715-z · Cited by 158
“Third, while we followed the previous studies to use ICD-10-CM codes to identify intentional self-harm 44,48,49 , we acknowledge that the ICD-10-CM representation of self-harm would miss some, although a small percentage of patients who conducted self-harm 50 .”
InfluentialSu, C., Aseltine, R. H., Doshi, R., Chen, K., Rogers, S. C., & Wang, F. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-01100-0 · Cited by 163
Variation in patterns of health care before suicide: a population case-control study.
Ahmedani, B. K., Westphal, J., Autio, K., Elsiss, F., Peterson, E. L., Beck, A., Waitzfelder, B. E., Rossom, R. C., Owen-Smith, A. A., Lynch, F., et al. (2019) · Preventive Medicine · doi:10.1016/j.ypmed.2019.105796 · Cited by 146
Self-citationMachine and deep learning for longitudinal biomedical data: a review of methods and applications
Cascarano, A., Mur-Petit, J., Hernández-González, J., Camacho, M., Eadie, N. D. T., Gkontra, P., Chadeau‐Hyam, M., Vitria, J. M., & Lekadir, K. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10561-w · Cited by 147
The Lancet Commission on self-harm
Moran, P., Chandler, A., Dudgeon, P., Kirtley, O. J., Knipe, D. W., Pirkis, J., Sinyor, M., Allister, R., Ansloos, J. P., Ball, M. A., et al. (2024) · The Lancet · doi:10.1016/s0140-6736(24)01121-8 · Cited by 143
Barak‐Corren, Y., Castro, V. M., Nock, M. K., Mandl, K. D., Madsen, E. M., Seiger, A. N., Adams, W. G., Applegate, R. J., Bernstam, E. V., Klann, J. G., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.1262 · Cited by 102
“In recent years, other reports 7,8,18-20 of EHR-based suicide risk prediction have appeared, using a range of statistical and machine learning methods.”
BackgroundMethodInfluentialTsui, F. R., Shi, L., Ruiz, V., Ryan, N. D., Biernesser, C. L., Iyengar, S., Walsh, C. G., & Brent, D. A. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab011 · Cited by 105
“While EXGB was the most accurate of the four ML models, its results are not transparent, and a slightly less accurate but more readily understood algorithm, such as one based on LASSO regression, may be more clinically useful.(8) The left censoring of data prior to 2005 may have led to mislabeling of cases and control…”
BackgroundMethodInfluentialBjureberg, J., Dahlin, M., Carlborg, A., Edberg, H., Haglund, A., & Runeson, B. (2021) · Psychological Medicine · doi:10.1017/s0033291721000751 · Cited by 111
“This is within the lower range usually found in the evaluation of suicide prediction models (Barak-Corren et al., 2017; Belsher et al., 2019; Runeson et al., 2017; Simon et al., 2018).”
BackgroundMcCarthy, J. F., Cooper, S. A., Dent, K. R., Eagan, A. E., Matarazzo, B. B., Hannemann, C. M., Reger, M. A., Landes, S. J., Trafton, J. A., Schoenbaum, M., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.29900 · Cited by 98
Translating promise into practice: a review of machine learning in suicide research and prevention
Kirtley, O. J., van Mens, K., Hoogendoorn, M., Kapur, N., & de Beurs, D. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00254-6 · Cited by 100
Illuminating the Black Box: Interpreting Deep Neural Network Models for Psychiatric Research
Sheu, Y.-H. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.551299 · Cited by 101
“…subjects)—such as interview records in the forms of text of videos or comprehensive questionnaires—to those that collect data from a larger group of subjects but possibly less in-depth per subject, such as electronic health records (EHR), health insurance claims databases, and cohort data (42–45).”
BackgroundSheu, Y.-H., Magdamo, C., Miller, M. R., Das, S., Blacker, D., & Smoller, J. W. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00817-8 · Cited by 90
Haines-Delmont, A., Chahal, G., Bruen, A. J., Wall, A., Khan, C. T., Sadashiv, R., & Fearnley, D. (2020) · JMIR mhealth and uhealth · doi:10.2196/15901 · Cited by 87
“Although previous studies have utilized electronic health care record data to create an actuarial model of suicide risk [30,34,35,37,63-65] or focused on a single aspect of user input such as language [29,30,36], this study adds to the literature by introducing external, user-generated input, and smartphone data and c…”
BackgroundMethodColey, R. Y., Johnson, E., Simon, G. E., Cruz, M., & Shortreed, S. M. (2021) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2021.0493 · Cited by 100
Self-citationOn the genetic and environmental relationship between suicide attempt and death by suicide
Edwards, A. C., Ohlsson, H., Mościcki, E., Crump, C., Sundquist, J., Lichtenstein, P., Kendler, K. S., & Sundquist, K. (2021) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2020.20121705 · Cited by 86
Kusuma, K., Larsen, M. E., Quiroz, J. C., Gillies, M. B., Burnett, A., Qian, J., & Torok, M. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.050 · Cited by 94
Chen, Q., Zhang‐James, Y., Barnett, E. J., Lichtenstein, P., Jokinen, J., D'Onofrio, B. M., Faraone, S. V., Larsson, H., & Fazel, S. (2020) · PLoS Medicine · doi:10.1371/journal.pmed.1003416 · Cited by 86
“When selecting potential predictors, we took into consideration previous studies on suicidal behavior [7,16,18,19] and the availability and quality of the information in the Swedish national registers.”
MethodResultProspective Validation of an Electronic Health Record–Based, Real-Time Suicide Risk Model
Walsh, C. G., Johnson, K. B., Ripperger, M., Sperry, S., Harris, J., Clark, N., Fielstein, E., Novak, L., Robinson, K., & Stead, W. W. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.1428 · Cited by 78
Simon, G. E., Shortreed, S. M., Rossom, R. C., Beck, A. L., Clarke, G. N., Whiteside, U. S., Richards, J. A., Penfold, R. B., Boggs, J. M., & Smith, J. (2022) · JAMA · doi:10.1001/jama.2022.0423 · Cited by 78
Self-citationNashwan, A. J., Gharib, S., Alhadidi, M. M. B., El‐Ashry, A. M., Alamgir, A., Al-Hassan, M., Khedr, M. A., Dawood, S. S., & Abufarsakh, B. (2023) · Issues in Mental Health Nursing · doi:10.1080/01612840.2023.2263579 · Cited by 87
“An AI-enhanced telemedicine platform also proved effective at identifying patients in danger of suicide (Simon et al., 2018).”
BackgroundThe Occupational Depression Inventory: A new tool for clinicians and epidemiologists
Bianchi, R., & Schonfeld, I. S. (2020) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2020.110249 · Cited by 97
Cusick, M. M., Adekkanattu, P. M., Campion, T. R., Sholle, E., Myers, A., Banerjee, S., Alexopoulos, G. S., Wang, Y., & Pathak, J. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.01.052 · Cited by 80
Boudreaux, E. D., Rundensteiner, E. A., Liu, F., Wang, B., Larkin, C., Agu, E., Ghosh, S., Semeter, J., Simon, G. E., & Davis-Martin, R. E. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.707916 · Cited by 79
“Variants that have been explored for suicide prediction include Lasso (16) and Elastic Net (17).”
InfluentialSelf-citationSuicidal behaviour prediction models using machine learning techniques: A systematic review
Nordin, N., Zainol, Z. B., Noor, M. H. M., & Chan, L. F. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102395 · Cited by 84
Identifying and classifying opioid‐related overdoses: A validation study
Green, C. A., Perrin, N., Hazlehurst, B. L., Janoff, S. L., DeVeaugh‐Geiss, A., Carrell, D., Grijalva, C. G., Liang, C., Enger, C. L., & COPLAN, P. M. (2019) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4772 · Cited by 69
“32 We created a binary variable if any suicide ‐ related codes in Table 4 were present, binary variables for single or multiple episode depression diagnoses, and interaction terms for suicide and recurrent depression.”
MethodMiché, M., Studerus, E., Meyer, A. H., Gloster, A. T., Beesdo‐Baum, K., Wïttchen, H., & Lieb, R. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.11.093 · Cited by 69
Shortreed, S. M., Cook, A. J., Coley, R. Y., Bobb, J. F., & Nelson, J. C. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwy292 · Cited by 74
Self-citationRoss, E. L., Zuromski, K. L., Reis, B. Y., Nock, M. K., Kessler, R. C., & Smoller, J. W. (2021) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2021.0089 · Cited by 65
Suicide prevention research priorities in health care.
Gordon, J. A., Avenevoli, S., & Pearson, J. L. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2020.1042 · Cited by 65
Selective serotonin reuptake inhibitors and suicidal behaviour: a population-based cohort study
Lagerberg, T., Fazel, S., Sjölander, A., Hellner, C., Lichtenstein, P., & Chang, Z. (2022) · Neuropsychopharmacology · doi:10.1038/s41386-021-01179-z · Cited by 61
Can machine-learning methods really help predict suicide?
McHugh, C. M., & Large, M. M. B. (2020) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000609 · Cited by 67
“In recent years, there has been growing interest in (Table 1) [20 && ,21 & ,22,23,24 & ].”
BackgroundSimpson, S. A., Goans, C., Loh, R. M., Ryall, K. A., Middleton, M., & Dalton, A. (2020) · Academic Emergency Medicine · doi:10.1111/acem.14198 · Cited by 58
Meta-analysis of the strength of exploratory suicide prediction models; from clinicians to computers
Corke, M., Mullin, K., Angel-Scott, H. N., Xia, S., & Large, M. M. B. (2021) · BJPsych Open · doi:10.1192/bjo.2020.162 · Cited by 61
“In addition to examining a large number of potential suicide risk factors increasing the possibility of chance associations, 46,59,97 some machine-learning studies examined clinical contacts rather than people, resulting in possible oversampling of suicides if they were more frequent attenders.”
Machine learning-based prediction for self-harm and suicide attempts in adolescents
Su, R., John, J. R., & Lin, P. (2023) · Psychiatry Research · doi:10.1016/j.psychres.2023.115446 · Cited by 72
“2 health records (Barak-Corren et al., 2017; Simon et al., 2018; Su et al., 2020), social media (Fodeh et al.”
Wilimitis, D., Turer, R. W., Ripperger, M. A., McCoy, A. B., Sperry, S. H., Fielstein, E. M., Kurz, T. L., & Walsh, C. G. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.12095 · Cited by 51
Kessler, R. C., Bauer, M. S., Bishop, T. M., Demler, O. V., Dobscha, S. K., Gildea, S. M., Goulet, J. L., Karras, E., Kreyenbuhl, J., Landes, S. J., et al. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00390 · Cited by 52
A Comprehensive Survey on Rare Event Prediction
Shyalika, C., Wickramarachchi, R., & Sheth, A. (2024) · ACM Computing Surveys · doi:10.1145/3699955 · Cited by 52
“…claims data, state mortality records from government websites, outpatient visit details from electronic health records, and responses to patient-reported measures like the Patient Health Questionnaire (PHQ-9) have provided valuable information in predicting rare medical incidents [93, 128].”
McKernan, L. C., Clayton, E. W., & Walsh, C. G. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00650 · Cited by 52
“Multiple accurate and large-scale prediction models have been recently published in both civilian and military settings (4–7) The emphasis remains on novel prediction, however, and not on the literacy, numeracy, and education necessary to integrate AI technologies (and medical innovation in…”
BackgroundMartínez‐Alés, G., Hernández‐Calle, D., Khauli, N., & Keyes, K. M. (2020) · Current topics in behavioral neurosciences · doi:10.1007/7854_2020_158 · Cited by 59
Patel, J. S., Su, C., Téllez, M., Albandar, J. M., Rao, R., Iyer, V., Shi, E., & Wu, H. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.979525 · Cited by 49
“For instance, Simon et al. (2018) developed a prediction model for suicide attempts and suicide deaths using electronic health record (EHR) data.”
InfluentialSuicide Attempts Among a Cohort of Transgender and Gender Diverse People
Mak, J., Shires, D. A., Zhang, Q., Prieto, L., Ahmedani, B. K., Kattari, L., Becerra-Culqui, T. A., Bradlyn, A. S., Flanders, W. D., Getahun, D., et al. (2020) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2020.03.026 · Cited by 52
Self-citationHaroz, E. E., Kitchen, C., Nestadt, P. S., Wilcox, H. C., DeVylder, J., & Kharrazi, H. H. K. (2021) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12800 · Cited by 48
Reconciling Statistical and Clinicians’ Predictions of Suicide Risk
Simon, G. E., Matarazzo, B. B., Walsh, C. G., Smoller, J. W., Boudreaux, E. D., Yarborough, B. J. H., Shortreed, S. M., Coley, R. Y., Ahmedani, B. K., Doshi, R., et al. (2021) · Psychiatric Services · doi:10.1176/appi.ps.202000214 · Cited by 44
Self-citationShortreed, S. M., Walker, R. L., Johnson, E., Wellman, R., Cruz, M., Ziebell, R. A., Coley, R. Y., Yaseen, Z. S., Dharmarajan, S. H., Penfold, R. B., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00772-4 · Cited by 46
“…we compare a well-performing previously developed logistic regression model predicting suicidal behavior following an outpatient mental health visit 15 , which uses relatively simple temporal predictors extracted from clinical data, to newly developed models that used over 1400 predictors…”
MethodResultInfluentialSelf-citationAn awakening in medicine: the partnership of humanity and intelligent machines
Celi, L. A., Fine, B., & Stone, D. J. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30127-x · Cited by 45
Digital devices and continuous telemetry: opportunities for aligning psychiatry and neuroscience
Baker, J. T., Germine, L. T., Ressler, K. J., Rauch, S. L., & Carlezon, W. A. (2018) · Neuropsychopharmacology · doi:10.1038/s41386-018-0172-z · Cited by 45
Sperl‐Hillen, J. M., Rossom, R. C., Kharbanda, E. O., Gold, R. B., Geissal, E. D., Elliott, T. E., DESAI, J. R., Rindal, D. B., Saman, D. M., Waring, S. C., et al. (2019) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.284 · Cited by 45
Self-citationTormohlen, K. N., Mojtabai, R., Seiwell, A., McGinty, E. E., Stuart, E. A., Tobin, K. E., & Troiani, V. (2021) · Journal of Dual Diagnosis · doi:10.1080/15504263.2021.1979349 · Cited by 48
“Inclusion of ICD codes was based on previous literature (Boscarino et al., 2016; Fiest et al., 2014; Hedegaard et al., 2017; Ingram et al., 2020; Krawczyk et al., 2020; McGinty et al., 2018; Simon et al., 2018; Tian et al., 2013; Tonelli et al., 2016; Walkup et al., 2012).”
MethodAgne, N. A., Tisott, C. G., Ballester, P. L., Passos, I. C., & Ferrão, Y. A. (2020) · Psychological Medicine · doi:10.1017/s0033291720002329 · Cited by 46
Iorfino, F., Ho, N., Carpenter, J. S., Cross, S. P. M., Davenport, T. A., Hermens, D. F., Yee, H. F., Nichles, A., Zmicerevska, N., Guastella, A. J., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243467 · Cited by 44
Accuracy of ICD-10-CM encounter diagnoses from health records for identifying self-harm events
Simon, G. E., Shortreed, S. M., Boggs, J. M., Clarke, G. N., Rossom, R. C., Richards, J. A., Beck, A. L., Ahmedani, B. K., Coleman, K. J., Bhakta, B. B., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac144 · Cited by 42
Self-citationZhang‐James, Y., Chen, Q., Kuja‐Halkola, R., Lichtenstein, P., Larsson, H., & Faraone, S. V. (2020) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13226 · Cited by 47
Šagud, M., Tudor, L., Šimunić, L., Jezernik, D., Madžarac, Z., Jakšić, N., Peleš, A. M., Vuksan‐Ćusa, B., Filipčić, I. Š., Stefanović, I., et al. (2020) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12724 · Cited by 41
Danielsen, A. A., Fenger, M. H. J., Østergaard, S. D., Nielbo, K. L., & Mors, O. (2019) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13061 · Cited by 38
“This approach has been used before (27) and enables modeling of changes in risk over time (new predicted risk for each included admission) within patients.”
MethodKern, D. M., Canuso, C. M., Daly, E., Johnson, J., Fu, D., Doherty, T., Blauer-Peterson, C. J., & Cepeda, M. S. (2023) · Brain and Behavior · doi:10.1002/brb3.3171 · Cited by 44
Machine learning for suicidology: A practical review of exploratory and hypothesis-driven approaches
Cox, C. R., Moscardini, E. H., Cohen, A. S., & Tucker, R. P. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101940 · Cited by 45
Assessment of a Risk Index for Suicide Attempts Among US Army Soldiers With Suicide Ideation
Zuromski, K. L., Bernecker, S. L., Gutierrez, P. M., Joiner, T. E., King, A. J., Liu, H., Naifeh, J. A., Nock, M. K., Sampson, N. A., Zaslavsky, A. M., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0766 · Cited by 38
“5% male; median [interquartile range] age as of December 2014, 29 [25-36] years [range, 18-55 years]; 69.”
Simon, G. E., Yarborough, B. J. H., Rossom, R. C., Lawrence, J. M., Lynch, F. L., Waitzfelder, B. E., Ahmedani, B. K., & Shortreed, S. M. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201800381 · Cited by 37
Self-citationSenior, M., Fanshawe, T. R., Fazel, M. S., & Fazel, S. (2021) · JCPP Advances · doi:10.1002/jcv2.12034 · Cited by 37
Damschroder, L. J., Knighton, A. J., Griese, E. R., Greene, S. M., Lozano, P., Kilbourne, A. M., Buist, D. S. M., Crotty, K., Elwy, A. R., Fleisher, L. A., et al. (2021) · Healthcare · doi:10.1016/j.hjdsi.2020.100455 · Cited by 39
Martínez‐Alés, G., & Keyes, K. M. (2019) · Current Psychiatry Reports · doi:10.1007/s11920-019-1080-6 · Cited by 38
Bentley, K. H., Zuromski, K. L., Fortgang, R. G., Madsen, E., Kessler, D. T., Lee, H., Nock, M. K., Reis, B. Y., Castro, V. M., & Smoller, J. W. (2022) · JMIR Formative Research · doi:10.2196/30946 · Cited by 33
“In the future, in addition to the clinical questionnaire data that some models incorporate [38], combining EHR-based data with fine-grained real-time data (eg, from smartphones or wearable devices [39]) that may capture more acute increases in risk may help meet this need.”
BackgroundWhat health records data are required for accurate prediction of suicidal behavior?
Simon, G. E., Shortreed, S. M., Johnson, E., Rossom, R. C., Lynch, F. L., Ziebell, R. A., & Penfold, A. R. B. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz136 · Cited by 35
“The study sample and methods for development and validation of risk prediction models are described in detail elsewhere 16 and summarized here.”
BackgroundMethodSelf-citationEdgcomb, J. B., Thiruvalluru, R. K., Pathak, J., & Brooks, J. O. (2021) · Medical Care · doi:10.1097/mlr.0000000000001467 · Cited by 32
Yarborough, B. J. H., & Stumbo, S. P. (2021) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2021.02.008 · Cited by 35
Development of a Suicide Prediction Model for the Elderly Using Health Screening Data
Cho, S., Geem, Z. W., & Na, K. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph181910150 · Cited by 37
“A previous study on the suicide prediction model [17], which included 2,960,929 subjects from 2009 to 2015, was limited in that only outpatients from the department of psychiatric medicine were included.”
BackgroundSpittal, M. J., Guo, X. A., Kang, L., Kirtley, O. J., Clapperton, A., Hawton, K. E., Kapur, N., Pirkis, J., & Carter, G. L. (2025) · PLoS Medicine · doi:10.1371/journal.pmed.1004581 · Cited by 41
“One illustrative study found that among those with mental health speciality visits, those in the top 5% of risk accounted for 43% of suicide attempts and 48% of suicides over a 90-day prediction window [71].”
Predicting suicide attempts and suicide deaths among adolescents following outpatient visits
Penfold, R. B., Johnson, E., Shortreed, S. M., Ziebell, R. A., Lynch, F. L., Clarke, G. N., Coleman, K. J., Waitzfelder, B. E., Beck, A. L., Rossom, R. C., et al. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.06.057 · Cited by 45
Self-citationLee, J., Jang, H. G., Kim, J.-K., & Min, S. (2019) · Scientific Reports · doi:10.1038/s41598-019-38886-z · Cited by 35
Detection of Suicide Attempters among Suicide Ideators Using Machine Learning
Ryu, S., Lee, H., Lee, D.-K., Kim, S., & Kim, C.-E. (2019) · Psychiatry Investigation · doi:10.30773/pi.2019.06.19 · Cited by 32
Garriga, R., Buda, T. S., Guerreiro, J., Iglesias, J. O., Aguerri, I. E., & Matić, A. (2023) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101260 · Cited by 36
Machine learning discovery of longitudinal patterns of depression and suicidal ideation
Gong, J., Simon, G. E., & Liu, S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0222665 · Cited by 29
“There are several recent studies aim to predict suicide and examine suicidal behaviors using large-scale EHR data ranging from thousands to millions of patients [1,3,32,33].”
BackgroundMethodSelf-citationGradus, J. L., Rosellini, A. J., Horváth–Puhó, E., Jiang, T., Street, A. E., Galatzer‐Levy, I. R., Lash, T. L., & Sørensen, H. T. (2021) · American Journal of Epidemiology · doi:10.1093/aje/kwab112 · Cited by 33
A Critical Review of Text Mining Applications for Suicide Research
Boggs, J. M., & Kafka, J. M. (2022) · Current Epidemiology Reports · doi:10.1007/s40471-022-00293-w · Cited by 32
“Prediction models for suicidal behavior that use health records vary considerably in their accuracy [24–28].”
BackgroundLaber, E. B., Meyer, N. J., Reich, B. J., Pacifici, K., Collazo, J. A., & Drake, J. M. (2018) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/rssc.12266 · Cited by 31
“We are approaching an era where algorithms are routinely used to decide patients’ treatment, though most are static in time (e.g. the software platforms used to determine warfarin dosing, or a recently developed suicide risk predictor (Simon et al., 2018)).”
How Can Digital Mental Health Enhance Psychiatry?
Stern, É., Micoulaud‐Franchi, J., Dumas, G., Moreira, J., Mouchabac, S., Maruani, J., Philip, P., Lejoyeux, M., & Geoffroy, P. A. (2022) · The Neuroscientist · doi:10.1177/10738584221098603 · Cited by 31
“For instance, it was found that it is possible to predict—and thus prevent—suicide attempts by using factors such as the history of self-harm, diagnoses, substance use, and use of emergency and inpatient care (Simon and others 2018).”
BackgroundClinical implementation of suicide risk prediction models in healthcare: a qualitative study
Yarborough, B. J. H., Stumbo, S. P., Schneider, J. L., Richards, J. A., Hooker, S. A., & Rossom, R. C. (2022) · BMC Psychiatry · doi:10.1186/s12888-022-04400-5 · Cited by 30
“The model predicts risk of suicide attempt within 90 days following an outpatient visit in primary care (in which a mental health diagnosis was made) or a mental health specialty clinic [11].”
BackgroundSelf-citationColey, R. Y., Liao, Q., Simon, N., & Shortreed, S. M. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01844-5 · Cited by 30
Self-citationLagerberg, T., Matthews, A., Zhu, N., Fazel, S., Carrero, J. J., & Chang, Z. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01676-3 · Cited by 37
“Our fi ndings further highlight that individuals with a history of suicidal behaviour have an elevated risk of further suicidal behaviour as compared to individuals with no such history, which corresponds to prior evidence [1, 22, 33].”
BackgroundResultBig Data From Health Records in Mental Health Care
Simon, G. E. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.4510 · Cited by 30
Self-citationAllowing for Nondisclosure in High Suicide Risk Groups
Podlogar, M. C., & Joiner, T. E. (2019) · Assessment · doi:10.1177/1073191119845495 · Cited by 29
“…are beginning to be researched, including through the use of blood biomarkers, implicit behavioral task indicators, personal digital devices, and big data algorithms (Cha, Najmi, Park, Finn, & Nock, 2010; Glenn & Monteith, 2014; Niculescu et al., 2015; Nock et al., 2010; Simon et al., 2018).”
BackgroundMethodApplying machine learning on health record data from general practitioners to predict suicidality
van Mens, K., Elzinga, E., Nielen, M. M. J., Lokkerbol, J., Poortvliet, R., Donker, G., Heins, M. J., Korevaar, J. C., Dückers, M., Aussems, C., et al. (2020) · Internet Interventions · doi:10.1016/j.invent.2020.100337 · Cited by 31
Parghi, N., Chennapragada, L., Barzilay, S., Newkirk, S., Ahmedani, B. K., Lok, B. C. L., & GALYNKER, I. I. (2020) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1863 · Cited by 32
“…in the analysis of electronic medical records of approximately 3 million patients; here, short ‐ term SB following mental health specialty visits and primary care visits were retrospectively predicted with seemingly greater ability than extant suicide risk assessment tools (Simon et al., 2018).”
BackgroundInfluentialSelf-citationPredicting outcomes of psychotherapy for depression with electronic health record data
Coley, R. Y., Boggs, J. M., Beck, A., & Simon, G. E. (2021) · Journal of Affective Disorders Reports · doi:10.1016/j.jadr.2021.100198 · Cited by 29
“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript 85% compared to 57–61% discrimination we observed for prediction of psychotherapy outcome (Simon et al., 2018).”
BackgroundInfluentialSelf-citationAlcohol use disorder and non‐fatal suicide attempt: findings from a Swedish National Cohort Study
Edwards, A. C., Ohlsson, H., Mościcki, E. K., Crump, C., Sundquist, J., Kendler, K. S., & Sundquist, K. (2021) · Addiction · doi:10.1111/add.15621 · Cited by 34
Digital Technology for Suicide Prevention
Braciszewski, J. M. (2021) · Advances in Psychiatry and Behavioral Health · doi:10.1016/j.ypsc.2021.05.008 · Cited by 28
Bryan, C. J., Allen, M. H., Thomsen, C. J., May, A. M., Baker, J. C., Bryan, A. O., Harris, J. A., Cunningham, C. A., Taylor, K. B., Wine, M. D., et al. (2021) · The Annals of Family Medicine · doi:10.1370/afm.2729 · Cited by 28
Hopkins, D., Rickwood, D., Hallford, D. J., & Watsford, C. R. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.945006 · Cited by 29
Gunlicks‐Stoessel, M., Klimes‐Dougan, B., VanZomeren‐Dohm, A., & Ma, S. (2020) · Translational Psychiatry · doi:10.1038/s41398-020-01005-y · Cited by 27
Temporally informed random forests for suicide risk prediction
Bayramli, I., Castro, V. M., Barak‐Corren, Y., Madsen, E. M., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab225 · Cited by 27
Positive Predictive Values and Potential Success of Suicide Prediction Models
Simon, G. E., Shortreed, S. M., & Coley, R. Y. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1516 · Cited by 26
Self-citation
Metadata
- Authors
- Simon, G. E., Johnson, E., Lawrence, J. M., Rossom, R. C., Ahmedani, B., Lynch, F. L., Beck, A., Waitzfelder, B., Ziebell, R., Penfold, R. B., & Shortreed, S. M.
- Affiliations
- Kaiser Permanente Washington Health Research Institute (US)
- ORCID
- Venue
- American Journal of Psychiatry
- Published
- 2018-10-01
- Volume
- 175
- Issue
- 10
- Pages
- 951-960
- ISSN
- 0002-953X (ISSN-L)
- 1535-7228
- PMC
- PMC6167136
- OpenAlex
- W2804266670
- Cited by
- 400
- References
- 46
- Citations by year
- 2026: 42 · 2025: 57 · 2024: 37 · 2023: 56 · 2022: 53 · 2021: 49 · 2020: 55 · 2019: 44 · 2018: 7
- FWCI
- 45.53
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- American Journal of Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (U19MH092201)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- suicide attempts, suicide mortality, suicide risk prediction, outpatient mental health visits, primary care visits, electronic health records, LASSO feature selection, risk score, substance use, psychiatric medication, mental health diagnoses, prediction model validation, death certificates, emergency department care, inpatient care, self-report questionnaire
- MeSH terms
- Adolescent, Adult, Aged, Female, Humans, Male, Middle Aged, Models, Psychological, Outpatients, Surveys and Questionnaires, Risk Factors, Suicide, Attempted, Young Adult, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
