CoChat
Sign Up
Cited by 12Machine Learning in HealthcareMental Health via WritingSuicide and Self-Harm Studies

Deep learning for identifying personal and family history of suicidal thoughts and behaviors from EHRs

Open access
Adekkanattu, Prakash, Furmanchuk, Al’ona, Wu, Yonghui, Pathak, Aman, Patra, Braja Gopal, Bost, Sarah, Morrow, Destinee, Wang, Grace Hsin-Min +11 more·2024·npj Digital Medicine
DOI 10.1038/s41746-024-01266-7PMC PMC10980141

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Personal and family history of suicidal thoughts and behaviors (PSH and FSH, respectively) are significant risk factors associated with future suicide events. These are often captured in narrative clinical notes in electronic health records (EHRs). Collaboratively, Weill Cornell Medicine (WCM), Northwestern Medicine (NM), and the University of Florida (UF) developed and validated deep learning (DL)-based natural language processing (NLP) tools to detect PSH and FSH from such notes. The tool's performance was further benchmarked against a method relying exclusively on ICD-9/10 diagnosis codes. Materials and Methods: We developed DL-based NLP tools utilizing pre-trained transformer models Bio_ClinicalBERT and GatorTron, and compared them with expert-informed, rule-based methods. The tools were initially developed and validated using manually annotated clinical notes at WCM. Their portability and performance were further evaluated using clinical notes at NM and UF. Results: The DL tools outperformed the rule-based NLP tool in identifying PSH and FHS. For detecting PSH, the rule-based system obtained an F1-score of 0.75 ± 0.07, while the Bio_ClinicalBERT and GatorTron DL tools scored 0.83 ± 0.09 and 0.84 ± 0.07, respectively. For detecting FSH, the rule-based NLP tool's F1-score was 0.69 ± 0.11, compared to 0.89 ± 0.10 for Bio_ClinicalBERT and 0.92 ± 0.07 for GatorTron. For the gold standard corpora across the three sites, only 2.2% (WCM), 9.3% (NM), and 7.8% (UF) of patients reported to have an ICD-9/10 diagnosis code for suicidal thoughts and behaviors prior to the clinical notes report date. The best performing GatorTron DL tool identified 93.0% (WCM), 80.4% (NM), and 89.0% (UF) of patients with documented PSH, and 85.0%(WCM), 89.5%(NM), and 100%(UF) of patients with documented FSH in their notes. Discussion: While PSH and FSH are significant risk factors for future suicide events, little effort has been made previously to identify individuals with these history. To address this, we developed a transformer based DL method and compared with conventional rule-based NLP approach. The varying effectiveness of the rule-based tools across sites suggests a need for improvement in its dictionary-based approach. In contrast, the performances of the DL tools were higher and comparable across sites. Furthermore, DL tools were fine-tuned using only small number of annotated notes at each site, underscores its greater adaptability to local documentation practices and lexical variations. Conclusion: Variations in local documentation practices across health care systems pose challenges to rule-based NLP tools. In contrast, the developed DL tools can effectively extract PSH and FSH information from unstructured clinical notes. These tools will provide clinicians with crucial information for assessing and treating patients at elevated risk for suicide who are rarely been diagnosed.

AI snapshot

Written from the paper’s full text.

TL;DR

This multi-site study developed and validated deep learning NLP tools to detect personal and family history of suicidal thoughts and behaviors from clinical notes, benchmarking them against rule-based NLP and ICD codes. Transformer models outperformed rule-based NLP and identified many cases missed by diagnosis codes.

Key findings

  • For personal history (PSH), the rule-based tool obtained an F1-score of 0.75 ± 0.07, while Bio_ClinicalBERT and GatorTron scored 0.83 ± 0.09 and 0.84 ± 0.07, respectively. For family history (FSH), rule-based F1 was 0.69 ± 0.11, compared with 0.89 ± 0.10 for Bio_ClinicalBERT and 0.92 ± 0.07 for GatorTron.
  • Site-specific GatorTron F1-scores were 0.92 (WCM), 0.78 (NM), and 0.83 (UF) for PSH, and 0.88 (WCM), 0.90 (NM), and 1.00 (UF) for FSH. Rule-based macro-average F1 ranged from 0.81 to 0.63 for PSH and 0.80 to 0.58 for FSH across sites.
  • Only 2.2% (WCM), 9.3% (NM), and 7.8% (UF) of patients with documented PSH had an ICD-9/10 diagnosis code prior to the clinical note date. The best-performing GatorTron tool identified 93.0% (WCM), 80.4% (NM), and 89.0% (UF) of patients with documented PSH, and 85.0% (WCM), 89.5% (NM), and 100% (UF) with documented FSH.
  • Across all sites, more than 90% of patients had no recorded STB diagnosis by ICD-9/10 codes, and no ICD code exists for FSH.

Methods

  • Multi-site development and validation at Weill Cornell Medicine (WCM), Northwestern Medicine (NM), and University of Florida Health (UF). WCM training data included more than 13.8 million clinical notes from 177,993 patients prescribed antidepressants or diagnosed with mental health conditions between 2000–2020; gold-standard corpora comprised 301 notes/134 patients at WCM, 400 notes/313 patients at NM, and 400 notes/341 patients at UF.
  • Rule-based NLP used the Leo framework’s SuicideHistoryExtractor with dual lexicons and ConText negation. Deep learning classifiers used Bio_ClinicalBERT and GatorTron, fine-tuned with context windows of 8, 16, 24, or 32 words, batch size 16, learning rate 1e-05, dropout 0.3, AdamW, 5 epochs, and 10-fold cross-validation; performance was measured by weighted precision, recall, and F1.
  • Manual annotation followed shared guidelines with majority polling and last-instance rules; inter-rater Cohen’s kappa was 0.89 for PSH and 0.89 for FSH at WCM, 0.75 and 0.85 at NM, and 0.89 combined at UF. Results were benchmarked against ICD-9/10 diagnosis codes.

Limitations

  • Stated: the STB and history-modifier lexicon was restricted, leading to missed site-specific keywords and moderate external-site performance, especially for self-harm behaviors with diverse operational definitions; local lexicon customization could improve accuracy.
  • Stated: the tools were tested at only three academic medical institutions, which may not reflect the broader healthcare system, so broader testing is needed.
  • Stated: templating varied across sites, and the rule-based NLP algorithm did not account for templating structure, possibly hurting performance. Evident: the DL tools were fine-tuned on relatively small site-specific annotated corpora, and direct comparison with prior STB detection tools was not feasible because prior work did not focus on the historical aspect of suicidality.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01266-7

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 14 cited sources checked · 0 supported · 2 flagged · accuracy not yet measured

  • Premisecitation

    Prior automated approaches primarily focus on identifying suicidal tendencies without considering their temporal aspects, often conflating current and historical events.

    “These approaches, ranging from rule-based algorithms to ML techniques, primarily focus on identifying suicidal tendencies without considering their temporal aspects, often conflating current and historical events.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [17] Walsh, C. G. et al. Prospective validation of an electronic health record–based, real-time suicide risk model. JAMA Netw. Open 4, e211428 (2021). source

      The Methods > Real-Time Prediction section describes an automated model that explicitly separates five years of historical predictors from suicide attempts or suicidal ideation in the subsequent 30 days. Methods > Study Outcomes also specifies a 30-day prediction target. The paper therefore describes an approach that considers timing rather than conflating historical and future events; it does not establish how common that problem is among other approaches. (LLM, conf=0.84)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [18] Levis, M., Westgate, C. L., Gui, J., Watts, B. V. & Shiner, B. Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models. Psychol. Med. 51, 1382–1391 (2021). source

      The Discussion (page unavailable) says REACH VET variables are often static and contrasts them with a method that monitors how predictors change over time. This supports the narrower point that an existing automated model may miss temporal changes. The paper does not say prior approaches generally conflate current and historical events, or establish that limitation for rule-based approaches. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [19] Pathak, J., Kho, A. N. & Denny, J. C. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc. 20, e206–e211 (2013). source

      The paper’s discussion of EHR phenotyping describes rule-based and machine-learning approaches, supporting that part of the sentence. It does not discuss identifying suicidal tendencies or conflating current and historical suicidal events; instead, it explicitly lists temporal relations among the problems clinical NLP techniques address. No page number is available in our copy. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Premisecitation

    Personal and family history of suicidal thoughts and behaviors (PSH and FSH, respectively) are significant risk factors associated with future suicide events.

    “Personal and family history of suicidal thoughts and behaviors (PSH and FSH, respectively) are significant risk factors associated with future suicide events.”
    • Also stated in: § Introduction · plain
      “Personal suicide history and a family suicide history (PSH and FSH, respectively) significantly increase future suicide risk, emphasizing the importance of identifying these factors8,9.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [8] Probert-Lindström, S., Berge, J., Westrin, Å., Öjehagen, A. & Pavulans, K. S. Long-term risk factors for suicide in suicide attempters examined at a medical emergency in patient unit: results from a 32-year follow-up study. BMJ Open 10, e038794 (2020). source

      The Results table (Risk factors at long term) identifies previous suicide attempts as a significant predictor of later death by suicide, supporting the personal-history component for attempts. The paper does not establish family history as a risk factor, and its finding about prior attempts does not establish the same association for a personal history of suicidal thoughts or for future nonfatal suicidal events. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [9] Brent, D. A. & Melhem, N. Familial transmission of suicidal behavior. Psychiatr. Clin. North Am. 31, 157–177 (2008). source
  • Premisecitation

    The authors' prior study found that among patients identified through an NLP-based approach as having PHQ-9 scores, 31% showed scores suggesting major depressive disorder, yet lacked a corresponding structured ICD or SNOMED diagnosis code.

    “Our prior study found that among patients identified through NLP-based approach as having PHQ-9 scores—a clinical instrument measuring depression severity—31% showed scores suggesting major depressive disorder, yet lacked a corresponding structured ICD or SNOMED diagnosis code39.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [39] McCoy, T. H., Castro, V. M., Roberson, A. M., Snapper, L. A. & Perlis, R. H. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry 73, 1064–1071 (2016). source

      The Discussion reports that 48.9% of patients who died by suicide had no coded psychiatric ICD-9 diagnosis, but it does not report the claimed PHQ-9 finding. Searches for PHQ-9, major depressive disorder, SNOMED, and 31% found no passage in our copy addressing that result. The available Results instead concern NLP-derived language features and suicide-risk prediction; page numbers were not available. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • Premisecitation

    The sole use of ICD-9/10 diagnosis codes in EHRs to document STB has proven insufficient.

    “While International Classification of Disease (ICD-9/10) diagnosis codes in EHRs are used to document STB, their sole use has proven insufficient14.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [14] Carter, G. et al. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br. J. Psychiatry 210, 387–395 (2017). source

      The paper’s CUI+ and clinical-assessment sections evaluate the accuracy of suicide-risk scales for predicting future behaviour. They do not address whether ICD-9/10 codes alone adequately document suicidal thoughts and behaviours in EHRs; searches for ICD-9, diagnosis codes, and electronic health records found no passage on that point. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
  • Findingown data

    GatorTron-based DL reported slightly higher FSH F1-scores of 0.88, 0.90, 1.00 at WCM, NM and UF, respectively.

    “while GatorTron-based DL reported slightly higher scores of 0.88, 0.90, 1.00 at WCM, NM and UF, respectively.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 16
  • May combine two claims. 24

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Advancements in automated detection of SI, SB, and SA from clinical notes have shown varied success.

    “Advancements in automated detection of SI, SB, and SA from clinical notes have shown varied success17,18,19,20,21,22,23,24,25,26.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [17] Walsh, C. G. et al. Prospective validation of an electronic health record–based, real-time suicide risk model. JAMA Netw. Open 4, e211428 (2021). source

      The Discussion’s Limitations section says this model used structured EHR data and did not capture information in unstructured clinical notes. Although the paper evaluates automated prediction of suicidal ideation and suicide attempts, it does not describe detecting them from clinical notes or compare the success of note-based detection methods. No page number is available in our copy. (LLM, conf=0.87)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [18] Levis, M., Westgate, C. L., Gui, J., Watts, B. V. & Shiner, B. Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models. Psychol. Med. 51, 1382–1391 (2021). source

      The Results (page unavailable) report varied predictive performance from NLP analysis of clinical notes: little to no improvement above chance in most cohorts, but a small improvement in the 12-month cohort. The paper studies prediction of death by suicide, not automated detection of suicidal ideation (SI), suicidal behavior (SB), and suicide attempts (SA) individually. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [19] Pathak, J., Kho, A. N. & Denny, J. C. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc. 20, e206–e211 (2013). source

      The main discussion describes automated extraction from clinical notes and notes challenges in achieving high accuracy; the Summary of Papers in the Special Issue also describes differing performance across phenotyping applications (page numbers unavailable). But our copy does not report detection results for suicidal ideation, suicidal behavior, or suicide attempts, so it does not support the claim’s specific application to SI, SB, and SA. (LLM, conf=0.81)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    BERT models effectively capture the interaction between a key clinical concept and their surrounding context, and have shown superior performance in various NLP tasks including text classification.

    “In particular, the BERT (Bidirectional Encoder Representations from Transformers) models effectively capture the interaction between a key clinical concept and their surrounding context, and have shown superior performance in various NLP tasks including text classification16.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [16] Kessler, R. C. et al. Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration. Int J. Methods Psychiatr. Res. 26, e1575 (2017). source

      The Analysis methods (§2.3) and Results (§3.2) discuss BART—Bayesian additive regression trees—for predicting suicide risk, not BERT. Searches for BERT, transformers, clinical-context interactions, and NLP text classification found no passage addressing the attributed claim. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    The majority of suicide decedents had a physician visit in the year prior to their death and 45% had a visit in the last month of life.

    “Notably, the majority of suicide decedents had a physician visit in the year prior to their death and 45% had a visit in the last month of life, highlighting the potential of EHR data in suicide risk prediction and prevention13.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [13] Velupillai, S. et al. Risk assessment tools and data-driven approaches for predicting and preventing suicidal behavior. Front. Psychiatry 10, 36 (2019). source

      The Targeted Suicide Prevention and Data-Driven Approaches sections discuss health-service data and EHR-based prediction, but neither reports physician visits among suicide decedents. Searches for visits in the year and month before death, including the claimed 45%, found no matching passage in our copy. The paper’s separate figure of 33–45% in the Data-Driven Approaches table refers to prediction sensitivity, not physician visits; page numbers were unavailable. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    For FSH, the lowest performance occurred at NM, mainly because of a low recall of 0.44.

    “Similarly, for FSH, the lowest performance occurred at NM, mainly because of a low recall of 0.44.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    WCM (32.8%) and UF (33.4%) had a higher representation of older patients (aged 60 or older) compared to NM (17.9%).

    “Both WCM (32.8%) and UF (33.4%) had a higher representation of older patients (aged 60 or older) compared to NM (17.9%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The DL-based approach achieves a higher and more robust performance across diverse health care systems compared to rule-based NLP.

    “The DL-based approach achieves a higher and more robust performance across diverse health care systems compared to rule-based NLP.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Demographic variations were evident across the sites, particularly in age, race, and ethnicity.

    “Notably, demographic variations (Table 1) were evident across the sites, particularly in age, race, and ethnicity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For FSH, both methods showed comparable effectiveness: Bio_ClinicalBERT-based DL achieved F1-scores of 0.88, 0.81, and 1.00 at WCM, NM and UF, respectively.

    “For FSH, both methods showed comparable effectiveness: Bio_ClinicalBERT-based DL achieved F1-score of 0.88, 0.81, and 1.00 at WCM, NM and UF, respectively”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Notes from WCM and UF are mostly in free-text format with little or no templating components, while those from NM exhibit a mix of semi-structured and unstructured formats varying with the type of note.

    “While notes from WCM and UF are mostly in free-text format with little or no templating components, those from NM exhibit a mix of semi-structured and unstructured formats varying with the type of note.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GatorTron successfully identified 57 (89.0%) of the PSH cases and 12 (100%) of the FSH cases in the UF cohort.

    “GatorTron, successfully identified 57 (89.0%) of the PSH cases and 12 (100%) of the FSH cases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Traditional methods of structured information collection might miss up to 90% of patients exhibiting suicidal thoughts or behaviors, as these are often only mentioned in unstructured clinical notes.

    “Our findings indicate that traditional methods of structured information collection might miss up to 90% of patients exhibiting suicidal thoughts or behaviors, as these are often only mentioned in unstructured clinical notes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The patient cohorts are predominantly women across all three study sites: 60.4% at WCM, 63.3% at NM, and 67.7% at UF.

    “The patient cohorts are predominantly women across all three study sites: 60.4% at WCM, 63.3% at NM, and 67.7% at UF.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The DL-based GatorTron tool accurately detected 42 (93.3%) of the PSH cases and 12 (85.7%) of the FSH cases from patient notes in the WCM cohort.

    “Conversely, the DL-based GatorTron tool accurately detected 42 (93.3%) of the PSH cases and 12 (85.7%) of the FSH cases from patient notes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A larger proportion of non-Hispanic patients was observed at UF when compared to WCM and NM sites (96.2% vs 89.6% and 81.5%).

    “Additionally, a larger proportion of non-Hispanic patients was observed at UF when compared to WCM and NM sites (96.2% vs 89.6% and 81.5%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the WCM cohort of 134 patients, manual annotation identified 45 (33.6%) patients with a PSH and 14 (10.4%) patients with a FSH.

    “In the WCM cohort of 134 patients, while manual annotation identified 45 (33.6%) patients with a PSH and 14(10.4%) patients with a FSH”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The study involved examining 301 notes from 134 patients at WCM, 400 notes from 313 patients at NM, and 400 notes from 341 patients at UF.

    “The study involved examining 301 notes from 134 patients at WCM, 400 notes from 313 patients at NM, and 400 notes from 341 patients at UF.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GatorTron successfully identified 78 (80.4%) of the PSH cases and 51 (89.5%) of the FSH cases from notes in the NM cohort.

    “GatorTron, however, successfully identified 78(80.4%) of the PSH cases and 51 (89.5%) of the FSH cases from notes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At NM, notes with suicidal events documented using a timestamp accounted for the majority of false negative cases and contributed towards the low recall.

    “For instance, at NM notes suicidal events documented using a timestamp accounted for the majority of false negative cases and contributed towards the low recall.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The best performing GatorTron DL tool identified 93.0% (WCM), 80.4% (NM), and 89.0% (UF) of patients with documented PSH in their notes.

    “The best performing GatorTron DL tool identified 93.0% (WCM), 80.4% (NM), and 89.0% (UF) of patients with documented PSH, and 85.0%(WCM), 89.5%(NM), and 100%(UF) of patients with documented FSH in their notes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the WCM cohort, only 1 (2.2%) PSH patient and 1 (7.1%) FSH patient had relevant ICD codes on or before the notes report date.

    “only 1(2.2%) PSH patient and 1(7.1%) FSH patient had relevant ICD codes on or before the notes report date.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the NM cohort of 313 patients, manual annotation identified 97 (31.0%) patients with PSH and 57 (18.2%) with FSH.

    “In the NM cohort of 313 patients, while manual annotation identified 97 (31.0%) patients with PSH and 57 (18.2%) with FSH”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The UF cohort (36.4%) had a higher representation of Black patients when compared to WCM (7.5%) and NM (14.1%) cohorts.

    “The UF cohort (36.4%) had a higher representation of Black patients when compared to WCM (7.5%) and NM (14.1%) cohorts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The current rule-based tool demonstrates comparable performance at the development site (WCM), but its performance at external sites (NM and UF) was lower.

    “While the current rule-based tool demonstrates comparable performance at the development site (WCM), its performance at external sites (NM and UF) was lower, suggesting room for further improvement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Transformer-based BERT model has comparable performances at both the development sites and external sites, making it a better choice for detecting history of STBs.

    “The Transformer-based BERT model, on the other hand, has comparable performances at both the development sites and external sites making it a better choice for detecting history of STBs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Since the sample cohort was unbalanced for PSH and FSH across all sites, the authors used the weighted average from scikit learn python library to compute precision, recall and F1-score.

    “Since the sample cohort was unbalanced for PSH and FSH across all sites, we used the weighted average from scikit learn python library to compute these measures36.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] Adamou, M. et al. Mining free-text medical notes for suicide risk assessment. In: Proceedings of the 10th hellenic conference on artificial intelligence 1–8 (Association for Computing Machinery, 2018). source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The varying effectiveness of the rule-based tools across sites suggests a need for improvement in its dictionary-based approach.

    “The varying effectiveness of the rule-based tools across sites suggests a need for improvement in its dictionary-based approach.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The performances of the DL tools were higher and comparable across sites.

    “In contrast, the performances of the DL tools were higher and comparable across sites.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Previous research indicates that mental health conditions were inadequately recorded as structured ICD or SNOMED codes in EHRs, but were more likely to be documented in patient notes.

    “Previous research indicates that mental health conditions were inadequately recorded as structured ICD or SNOMED codes in EHRs, but were more likely to be documented in patient notes38.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38] Ben-Ari, A. & Hammond, K. Text mining the EMR for modeling and predicting suicidal behavior among US veterans of the 1991 Persian Gulf War. In: 2015 48th Hawaii International Conference on System Sciences (eds. Bui, T. & Sprague, Jr., R.) 3168–3175 (IEEE Computer Society, 2015). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The multi-site study was conducted with the approval of the Institutional Review Boards at each participating site and by a central IRB.

    “This multi-site study was conducted with the approval of the Institutional Review Boards (IRBs) at each participating site and by a central IRB, ensuring adherence to ethical standards and patient privacy regulations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    At NM, before the final voting process, Cohen's Kappa coefficients were 0.75 for PSH and 0.85 for FSH.

    “Before the final voting process among all three reviewers at NM, the Cohen’s Kappa coefficient were 0.75 and 0.85 for PSH and FSH, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All annotators were given the same annotation guidelines.

    “All annotators were given the same annotation guidelines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The DL-based tools were fine-tuned on a relatively small number of notes (gold standard corpus) from each site, which reduces the overall development effort required.

    “Moreover, the DL-based tools were fine-tuned on relatively small number of notes (gold standard corpus) from each site which reduces the overall development effort required.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Disagreements between annotators were resolved through joint sessions to produce a final gold standard annotated document set.

    “After both annotators completed annotations on the documents, any disagreements between the two sets were resolved through joint sessions of annotations, giving us a final gold standard annotated document set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each model was evaluated via 10-fold cross-validation by randomly training using 80% of the data while keeping out 20% for testing to avoid overfitting.

    “Each model was evaluated via 10-fold cross-validation by randomly training using 80% of the data while keeping out 20% for testing to avoid overfitting.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    At WCM, Cohen's Kappa was 0.89 for PSH and 0.89 for FSH.

    “At WCM, the Cohen’s Kappa measured was 0.89 for PSH and 0.89 for FSH.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The rule-based NLP and DL tools were tested across three academic medical institutions, which may not reflect the broader healthcare system.

    “Secondly, the rule-based NLP and DL tools were tested across three academic medical institutions, which may not reflect the broader healthcare system.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The rule-based NLP algorithm did not account for possible templating structure and may have impacted negatively on the performance.

    “Our rule-based NLP algorithm did not account for possible templating structure and may have impacted negatively on the performance.”
    Quote in the paperNot a substantial claim
37 of 37

  • The paper relies on this without support.

    Patients at risk of suicide are often underdiagnosed.

    “However, patients at risk of suicide are often underdiagnosed for a variety of reasons such as stigma associated with mental health issues and the limitations of diagnostic codes in capturing the complexity of mental health conditions.”

  • The paper relies on this without support.

    Details like personal and family history of STB are often only recorded in clinical notes instead of ICD-9/10 diagnoses codes.

    “This inadequacy is partly because details like personal and family history of STB are often only recorded in clinical notes instead of ICD-9/10 diagnoses codes.”

  • The paper relies on this without support.

    The ICD-9 had no defined diagnosis code to document the historical aspect of STB.

    “The ICD-9 had no defined diagnosis code to document the historical aspect of STB.”

  • The paper relies on this without support.

    Personal history of self-harm was introduced only in 2016 when ICD-10 was formally adopted by EHR systems.

    “Personal history of self-harm was introduced only in 2016 when ICD-10 was formally adopted by EHR systems.”

  • The paper relies on this without support.

    There has been a lack of emphasis on distinguishing personal and family histories of STB.

    “Notably, there has been a lack of emphasis on distinguishing personal and family histories of STB, despite the critical but different insights each of these factors offer.”

  • The paper relies on this without support.

    Patients with pain conditions or those prescribed opioids are at an increased risk of STBs.

    “Patients with pain conditions or those prescribed opioids are at an increased risk of STBs.”

  • The paper relies on this without support.

    None of the existing methods reported for the detection of STB specifically looked at the temporal or historical nature of these events.

    “However, none of the existing methods reported for the detection of STB specifically looked at the temporal or historical nature of these events.”

  • The paper relies on this without support.

    Individuals with a personal or family history of suicidal thoughts and behaviors have an increased risk for future suicide events.

    “Individuals with a PSH or a FSH have an increased risk for future suicide events.”

  • The paper relies on this without support.

    No ICD code exists for family history of suicidal thoughts and behaviors.

    “No ICD code exists for FSH, precluding direct comparison of underdiagnosis in EHRs.”

  • The paper relies on this without support.

    Modern EHR systems use pre-defined templating components in organizing and documenting clinical notes.

    “Thirdly, modern EHR systems use pre-defined templating component in organizing and documenting clinical notes.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • UF psychiatric inpatient-note percentage is incorrect

    Minor

    Eight of 400 UF notes is 2.0%, not 2.1%.

    Reported 2.1 · recomputed 2

  • NM progress-note percentage uses the wrong denominator

    Minor

    The paper reports 400 NM validation notes, but 167 progress notes would be 41.75% of them, not 53.4%. The reported percentage instead corresponds approximately to 167 divided by the 313 patients.

    Reported 53.4 · recomputed 41.75

  • Ten-fold cross-validation conflicts with the stated test split

    Minor

    In ten-fold cross-validation, each fold holds out one tenth of the data. The stated procedure instead holds out 20% for testing.

  • UF other-note percentage is incorrect

    Minor

    Thirty-three of 400 UF notes is 8.25%, not 8.4%.

    Reported 8.4 · recomputed 8.25

  • NM specialist counts do not cover 400 notes

    Minor

    The two specialist counts total 313, although the passage describes the authors of the 400 NM validation notes.

    Reported 400 · recomputed 313

  • NM note types total 338, not 400

    Minor

    The listed NM note-type counts sum to 338, leaving 62 of the stated 400 validation notes unaccounted for.

    Reported 400 · recomputed 338

6 of 6

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Garrido-Merchan, E. C., Gozalo-Brizuela, R. & Gonzalez-Carvajal, S. Comparing BERT against traditional machinelearning models in text classification. J Comput Cogn Eng. 2, 352–356 (2023).

Cited by

Sorted by citations

Related

Authors
Adekkanattu, P., Furmanchuk, A., Wu, Y., Pathak, A., Patra, B. G., Bost, S., Morrow, D., Wang, G. H.-M., Yang, Y., Forrest, N. J., Luo, Y., Walunas, T. L., Lo-Ciganic, W., Gelad, W., Bian, J., Bao, Y., Weiner, M., Oslin, D., & Pathak, J.
Affiliations
  • Weill Cornell Medicine (US)
  • Northwestern University (US)
  • University of Florida Health (US)
  • Lawrence Berkeley National Laboratory (US)
  • Philadelphia VA Medical Center (US)
Corresponding author
Adekkanattu, Prakash
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-09-28
Volume
7
Issue
1
Pages
260-260
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-024-01266-7
PMC
PMC10980141
OpenAlex
W4402945049
Semantic Scholar
9baead57e7cda73ba0d024a7f9894f61b81712b1
Cited by
12
References
49
Citations by year
2026: 5 · 2025: 7
FWCI
1.77
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
  • PubMed Central (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Health & Human Services | NIH | National Institute on Drug Abuse (1R01DA050676)
  • U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (5R01MH121907-04)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
  • Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
Keywords
suicidal ideation, suicidal behavior, family history of suicide, suicide risk, clinical notes, electronic health records, clinical text mining, transformer models, rule-based natural language processing, suicide risk identification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review