Natural language processing and machine learning of electronic health records for prediction of first-time suicide attempts
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Objective: Limited research exists in predicting first-time suicide attempts that account for two-thirds of suicide decedents. We aimed to predict first-time suicide attempts using a large data-driven approach that applies natural language processing (NLP) and machine learning (ML) to unstructured (narrative) clinical notes and structured electronic health record (EHR) data. Methods: This case-control study included patients aged 10–75 years who were seen between 2007 and 2016 from emergency departments and inpatient units. Cases were first-time suicide attempts from coded diagnosis; controls were randomly selected without suicide attempts regardless of demographics, following a ratio of nine controls per case. Four data-driven ML models were evaluated using 2-year historical EHR data prior to suicide attempt or control index visits, with prediction windows from 7 to 730 days. Patients without any historical notes were excluded. Model evaluation on accuracy and robustness was performed on a blind dataset (30% cohort). Results: The study cohort included 45 238 patients (5099 cases, 40 139 controls) comprising 54 651 variables from 5.7 million structured records and 798 665 notes. Using both unstructured and structured data resulted in significantly greater accuracy compared to structured data alone (area-under-the-curve [AUC]: 0.932 vs. 0.901 P < .001). The best-predicting model utilized 1726 variables with AUC = 0.932 (95% CI, 0.922–0.941). The model was robust across multiple prediction windows and subgroups by demographics, points of historical most recent clinical contact, and depression diagnosis history. Conclusions: Our large data-driven approach using both structured and unstructured EHR data demonstrated accurate and robust first-time suicide attempt prediction, and has the potential to be deployed across various populations and clinical settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Rising morbidity and mortality in midlife among white non-Hispanic Americans in the 21st century
Case, A., & Deaton, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1518393112 · Cited by 2,759
Savova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · cited 1×
“The cTAKES tool can process a variety of narrative notes, such as history and physical, progress, discharge summary, and radiology notes.(31,33,34) This process consists of extracting clinical concepts that are then annotated with Concept Unique Identifiers (CUIs) from the Unified Medical Language System (UMLS).”
MethodEpidemiology of suicide and the psychiatric perspective
Bachmann, S. (2018) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph15071425 · Cited by 1,377 · cited 1×
“First-time suicide attempt was chosen as a primary outcome because of its increased correlation with suicide death and future suicide attempts.(26) Thus, we excluded suicide attempts where patients had any previous suicide attempt diagnosis from historical encounters within the UPMC hospital network as far back as Jan…”
MethodHealth care contacts in the year before suicide death.
Ahmedani, B. K., Simon, G. E., Stewart, C., Beck, A., Waitzfelder, B. E., Rossom, R., Lynch, F., Owen-Smith, A., Hunkeler, E. M., Whiteside, U., et al. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-2767-3 · Cited by 671
Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0174 · Cited by 540 · cited 2×
“suicide attempt prevalence in predictive modeling literature ranged from 36 to 62 900 per 100 000 individuals.(7) Outpatient visits were excluded from case and control index visits because the vast majority (98%) of suicide attempts were found in either emergency department or inpatients settings in our database.”
BackgroundSimon, G. E., Johnson, E., Lawrence, J. M., Rossom, R. C., Ahmedani, B., Lynch, F. L., Beck, A., Waitzfelder, B., Ziebell, R., Penfold, R. B., et al. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17101167 · Cited by 400 · cited 4×
“While EXGB was the most accurate of the four ML models, its results are not transparent, and a slightly less accurate but more readily understood algorithm, such as one based on LASSO regression, may be more clinically useful.(8) The left censoring of data prior to 2005 may have led to mislabeling of cases and control…”
BackgroundMethodInfluentialPredicting suicidal behavior from longitudinal electronic health records.
Barak-Corren, Y., Castro, V. M., Javitt, S., Hoffnagle, A. G., Dai, Y., Perlis, R. H., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.16010077 · Cited by 339 · cited 3×
“First, suicidal behavior is relatively rare and predictive models often require large population samples.7 Second, risk assessment relies heavily on patient self-report, yet patients may be motivated to conceal their suicidal intentions.8,9 Third, prior to suicide attempts, the last point of clinical contact of patien…”
BackgroundMethodInfluentialImproving the Short-Term Prediction of Suicidal Behavior
Glenn, C. R., & Nock, M. K. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.06.004 · Cited by 328
What Does the Odds Ratio Estimate in a Case-Control Study?
PEARCE, N. (1993) · International Journal of Epidemiology · doi:10.1093/ije/22.6.1189 · Cited by 266
Predicting suicide attempts in adolescents with longitudinal clinical data and machine learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12916 · Cited by 264 · cited 4×
“Second, risk assessment relies heavily on patient self-report, yet patients may be motivated to conceal their suicidal intentions.(8,9) Third, prior to suicide attempts, the last point of clinical contact of patients who die by suicide commonly involves providers with varying levels of suicidal-risk assessment trainin…”
BackgroundMethodInfluentialSelf-citationKessler, R. C., Hwang, I., Hoffmire, C. A., McCarthy, J. F., Petukhova, M. V., Rosellini, A. J., Sampson, N. A., Schneider, A. L., Bradley, P. A., Katz, I. R., et al. (2017) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1575 · Cited by 209 · cited 1×
“While EHRs contain extensive longitudinal data on a large number of individuals, ML approaches have shown success to handle the challenges inherent to EHR data, such as variable collinearity, highdimensionality, nonlinear interactions, and missing data.(14,15) There have been promising ML applications to predict the r…”
MethodVariation in patterns of health care before suicide: a population case-control study.
Ahmedani, B. K., Westphal, J., Autio, K., Elsiss, F., Peterson, E. L., Beck, A., Waitzfelder, B. E., Rossom, R. C., Owen-Smith, A. A., Lynch, F., et al. (2019) · Preventive Medicine · doi:10.1016/j.ypmed.2019.105796 · Cited by 146
McCoy, T. H., Castro, V. M., Roberson, A. M., Snapper, L. A., & Perlis, R. H. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.2172 · Cited by 130
Schaffer, A., Sinyor, M., Kurdyak, P., Vigod, S., Sareen, J., Reis, C., Green, D., Bolton, J., Rhodes, A., Grigoriadis, S., et al. (2016) · World Psychiatry · doi:10.1002/wps.20321 · Cited by 118 · cited 2×
“Third, prior to suicide attempts, the last point of clinical contact of patients who die by suicide commonly involves providers with varying levels of suicidal-risk assessment training.(10,11) The increasing ubiquity of electronic health record (EHR) data and advances in the use of machine learning (ML) present an opp…”
BackgroundPractical challenges in integrating genomic data into the electronic health record
Kho, A. N., Rasmussen, L. V., Connolly, J. J., Peissig, P. L., Starren, J., Hakonarson, H., & Hayes, M. G. (2013) · Genetics in Medicine · doi:10.1038/gim.2013.131 · Cited by 108 · cited 1×
“The use of unstructured data can be of value to predict suicide risk,(17) especially because approximately 80% of EHR data are locked in narrative form.(18) NLP can identify suicide attempt predictors from clinical notes,(19,20) such as clinician positive valence assessments(21) and social determinants of health (SDOH…”
BackgroundSteeg, S., Quinlivan, L., Nowland, R., Carroll, R., Casey, D., Clements, C., Cooper, J., Davies, L., Knipe, D., Ness, J., et al. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1693-z · Cited by 91
López Pineda, A., Ye, Y., Visweswaran, S., Cooper, G. F., Wagner, M. M., & Tsui, F. (. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.08.019 · Cited by 88 · cited 2×
“operator (LASSO) regression, random forest (RF), and the ensemble of extreme gradient boosting (EXGB).(15,37,38) All the models estimated the posterior probability of suicide attempt based on historical data extracted within 2 years prior to the index visit, that is, P(suicide attempt j2-year EHR data prior to the ind…”
MethodComparison of MetaMap and cTAKES for entity extraction in clinical notes
Reátegui, R., & Ratté, S. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0654-2 · Cited by 83
Cited by
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407
“EHRs [44] Clinical Notes χ χ χ χ χ EHRs [39] Clinical Notes χ χ χ χ χ”
BackgroundMethodInfluentialYang, Z., Mitra, A., Liu, W., Berlowitz, D., & Yu, H. (2023) · Nature Communications · doi:10.1038/s41467-023-43715-z · Cited by 158
“Our work is also related to predictive model studies focused on intentional self-harm 40 – 44 .”
Bartal, A., Jagodnik, K. M., Chan, S. J., Babu, M. S., & Dekel, S. (2023) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2022.100834 · Cited by 44
Maarseveen, T. D., Glas, H. K., Veris-van Dieren, J., van den Akker, E., & Knevel, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01495-4 · Cited by 17
Adekkanattu, P., Furmanchuk, A., Wu, Y., Pathak, A., Patra, B. G., Bost, S., Morrow, D., Wang, G. H.-M., Yang, Y., Forrest, N. J., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01266-7 · Cited by 12
Metadata
- Authors
- Tsui, F. R., Shi, L., Ruiz, V., Ryan, N. D., Biernesser, C., Iyengar, S., Walsh, C. G., & Brent, D. A.
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-01-01
- Volume
- 4
- Issue
- 1
- Pages
- ooab011-ooab011
- ISSN
- 2574-2531 (ISSN-L)
- OpenAlex
- W3136738164
- Cited by
- 103
- References
- 19
- FWCI
- 4.59
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.