CoChat
Sign Up
Cited by 65Suicide and Self-Harm StudiesHealth Systems, Economic Evaluations, Quality of LifeMental Health Treatment and Access

Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US

Open access
Ross, Eric L., Zuromski, Kelly L., Reis, Ben Y., Nock, Matthew K., Kessler, Ronald C., Smoller, Jordan W.·2021·JAMA Psychiatry
DOI 10.1001/jamapsychiatry.2021.0089PMC PMC7970389
Open this paper in CoChat — free account

Abstract

Importance: Several statistical models for predicting suicide risk have been developed, but how accurate such models must be to warrant implementation in clinical practice is not known. Objective: To identify threshold values of sensitivity, specificity, and positive predictive value that a suicide risk prediction method must attain to cost-effectively target a suicide risk reduction intervention to high-risk individuals. Design, Setting, and Participants: This economic evaluation incorporated published data on suicide epidemiology, the health care and societal costs of suicide, and the costs and efficacy of suicide risk reduction interventions into a novel decision analytic model. The model projected suicide-related health economic outcomes over a lifetime horizon among a population of US adults with a primary care physician. Data analysis was performed from September 19, 2019, to July 5, 2020. Interventions: Two possible interventions were delivered to individuals at high predicted risk: active contact and follow-up (ACF; relative risk of suicide attempt, 0.83; annual health care cost, $96) and cognitive behavioral therapy (CBT; relative risk of suicide attempt, 0.47; annual health care cost, $1088). Main Outcomes and Measures: Fatal and nonfatal suicide attempts, quality-adjusted life-years (QALYs), health care sector costs and societal costs (in 2016 US dollars), and incremental cost-effectiveness ratios (ICERs) (with ICERs ≤$150 000 per QALY designated cost-effective). Results: With a specificity of 95% and a sensitivity of 25%, primary care-based suicide risk prediction could reduce suicide death rates by 0.5 per 100 000 person-years (if used to target ACF) or 1.6 per 100 000 person-years (if used to target CBT) from a baseline of 15.3 per 100 000 person-years. To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method would need to have a sensitivity of 17.0% or greater (95% CI, 7.4%-37.3%) if used to target ACF and 35.7% or greater (95% CI, 23.1%-60.3%) if used to target CBT. To achieve cost-effectiveness, ACF required positive predictive values of 0.8% for predicting suicide attempt and 0.07% for predicting suicide death; CBT required values of 1.7% for suicide attempt and 0.2% for suicide death. Conclusions and Relevance: These findings suggest that with sufficient accuracy, statistical suicide risk prediction models can provide good health economic value in the US. Several existing suicide risk prediction models exceed the accuracy thresholds identified in this analysis and thus may warrant pilot implementation in US health care systems.

AI snapshot

Written from the paper’s full text.

TL;DR

This economic evaluation estimates accuracy thresholds at which suicide risk prediction would be cost-effective for targeting two interventions in US primary care. It finds that at 95% specificity, active contact and follow-up requires sensitivity ≥17% and cognitive behavioral therapy requires ≥36%, with corresponding positive predictive value thresholds around 1% and 2%.

Key findings

  • At 95% specificity, risk prediction followed by active contact and follow-up (ACF) needed sensitivity ≥17.0% (95% CI, 7.4%-37.3%) to be cost-effective from a health care sector perspective (ICER ≤$150,000/QALY); cognitive behavioral therapy (CBT) needed sensitivity ≥35.7% (95% CI, 23.1%-60.3%).
  • Required positive predictive values were 0.8% for predicting suicide attempt and 0.07% for predicting suicide death with ACF; CBT required 1.7% for suicide attempt and 0.2% for suicide death.
  • With 95% specificity and 25% sensitivity, risk prediction reduced suicide death rates by 0.52 per 100,000 person-years with ACF and 1.56 with CBT from a baseline of 15.34; suicide attempt rates fell by 5.90 and 17.76 per 100,000 person-years from 174.15. ICERs were $107,000/QALY (ACF) and $221,000/QALY (CBT) from a health care sector perspective, and $57,000/QALY and $224,000/QALY from a societal perspective.
  • Existing models at 95% specificity had sensitivities of 26% (Veterans Health Administration), 33% (Massachusetts academic medical centers), and 48% (multistate collaboration), with PPVs of 0.1%-0.3% for suicide death and 2%-6% for suicide attempts, suggesting some may meet the thresholds.

Methods

  • Decision analytic state-transition model simulating a US primary care population over a lifetime horizon, calibrated to suicide attempt and suicide death rates of 175 and 15 per 100,000 person-years, respectively, and an 8.81% probability of death per suicide attempt; mean age was 48.8 (SD 17.2) years.
  • The model varied sensitivity and specificity of risk prediction to target ACF (relative risk of suicide attempt, 0.83; annual health care cost, $96) or CBT (relative risk, 0.47; annual health care cost, $1088), tracking fatal and nonfatal attempts, QALYs, and health care sector and societal costs in 2016 US dollars, discounted at 3% annually; ICERs ≤$150,000/QALY were designated cost-effective.
  • Inputs came from published epidemiology, cost, and meta-analytic efficacy data; probabilistic sensitivity analyses used 1,000 model runs, with additional univariate sensitivity analyses.

Limitations

  • The paper states that most intervention efficacy data come from emergency department or help-seeking populations after a suicide attempt, not proactively identified primary care outpatients; no evidence exists on intervention efficacy in an outpatient population, so the analysis relies on extrapolation and large-scale implementation would be premature.
  • The paper states that modeling elides real-world complexities (e.g., heterogeneity of suicide case-fatality by method and demographic factors) and requires assumptions about model structure and inputs; conservative assumptions, such as a 12-month duration of intervention efficacy, may make threshold accuracy estimates more stringent.
  • The paper states uncertainty around primary and secondary benefits and harms, intervention heterogeneity preventing a single generalizable cost estimate, no primary care data on secondary effects, and no accounting for implementation costs such as staff training, workflow integration, or merging statistical prediction with practitioner assessment. Evident from the text: the model directly specifies predictive accuracy rather than simulating clinical and demographic predictors, so it does not capture how real-world predictor inputs affect calibration.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1001/jamapsychiatry.2021.0089

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 70 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § Abstract

    To achieve cost-effectiveness, ACF required a positive predictive value of 0.8% for predicting suicide attempt.

    “To achieve cost-effectiveness, ACF required positive predictive values of 0.8% for predicting suicide attempt and 0.07% for predicting suicide death; CBT required values of 1.7% for suicide attempt and 0.2% for suicide death.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reduction in suicide risk with any intervention would last for 12 months.
  • Findingown data
    § Results

    With a specificity of 95% and a sensitivity of 25%, mean QALYs increased by 0.0016 with ACF.

    “Mean QALYs increased by 0.0016 with ACF and 0.0046 with CBT.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reduction in suicide risk with any intervention would last for 12 months.
  • Findingown data
    § Results

    With a specificity of 95% and a sensitivity of 25%, primary care-based suicide risk prediction targeting ACF reduces the suicide death rate by 0.52 per 100,000 person-years from a baseline of 15.34 per 100,000 person-years.

    “If used to target ACF, this risk prediction method reduced the population’s suicide death rate by 0.52 and suicide attempt rate by 5.90 per 100 000 person-years”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reduction in suicide risk with any intervention would last for 12 months.
  • Findingown data
    § Abstract

    To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method targeting ACF would need a sensitivity of 17.0% or greater (95% CI, 7.4%-37.3%).

    “To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method would need to have a sensitivity of 17.0% or greater (95% CI, 7.4%-37.3%) if used to target ACF and 35.7% or greater (95% CI, 23.1%-60.3%) if used to target CBT.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reduction in suicide risk with any intervention would last for 12 months.
  • Findingown data
    § Abstract

    To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method targeting CBT would need a sensitivity of 35.7% or greater (95% CI, 23.1%-60.3%).

    “To be cost-effective from a health care sector perspective at a specificity of 95%, a risk prediction method would need to have a sensitivity of 17.0% or greater (95% CI, 7.4%-37.3%) if used to target ACF and 35.7% or greater (95% CI, 23.1%-60.3%) if used to target CBT.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reduction in suicide risk with any intervention would last for 12 months.
  • The quote only partly states this claim. 1
  • May combine two claims. 12
  • Classified as probably not a substantial claim. 24

  • The quote only partly states this claim.

    Findingown data
    § Results

    With a specificity of 95% and a sensitivity of 25%, primary care-based suicide risk prediction targeting CBT reduces the suicide death rate by 1.56 per 100,000 person-years from a baseline of 15.34 per 100,000 person-years.

    “if used to target CBT, risk prediction reduced the population's suicide death rate by 1.56 and suicide attempt rate by 17.76 per 100 000 person-years”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The reduction in suicide risk with any intervention would last for 12 months.

  • May combine two claims.

    Findingown data
    § Sensitivity Analyses

    For CBT, threshold PPV predicting suicide attempt ranged from 0.9% to 3.4% (base case 1.7%), whereas threshold PPV predicting suicide death ranged from 0.1% to 0.3% (base case, 0.2%).

    “For CBT, threshold PPV predicting suicide attempt ranged from 0.9% to 3.4% (base case 1.7%), whereas threshold PPV predicting suicide death ranged from 0.1% to 0.3% (base case, 0.2%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Sensitivity Analyses

    For ACF, threshold PPV predicting suicide attempt ranged from 0.5% to 1.6% (base case, 0.8%), whereas threshold PPV predicting suicide death ranged from 0.05% to 0.14% (base case 0.07%).

    “For ACF, threshold PPV predicting suicide attempt ranged from 0.5% to 1.6% (base case, 0.8%), whereas threshold PPV predicting suicide death ranged from 0.05% to 0.14% (base case 0.07%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Sensitivity Analyses

    Threshold PPV was quite stable to changes in population suicide risk; for example, for CBT, threshold PPV predicting suicide death varied from 0.14% to 0.15% when population risk was halved or doubled.

    “In contrast, threshold PPV was quite stable to changes in population suicide risk; for example, for CBT, threshold PPV predicting suicide death varied from 0.14% to 0.15% when population risk was halved or doubled.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § Introduction

    The health economic value of risk prediction depends not only on its accuracy but also on the consequences of the condition it predicts and the efficacy of interventions for that condition.

    “the health economic value of risk prediction depends not only on its accuracy but also on the consequences of the condition it predicts and the efficacy of interventions for that condition”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Kessler RC. Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go. JAMA Psychiatry. 2019;76(8):777-778. doi: 10.1001/jamapsychiatry.2019.1238 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § Introduction

    Most suicide risk prediction models had a positive predictive value less than 1% for predicting suicide mortality.

    “most models had a PPV <1% for predicting suicide mortality”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Belsher BE, Smolenski DJ, Pruitt LD, et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76(6):642-651. doi: 10.1001/jamapsychiatry.2019.0174 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § Introduction

    Multiple interventions have evidence of efficacy in reducing suicide risk, including cognitive behavioral therapy, dialectical behavioral therapy, and active contact and follow-up interventions.

    “Multiple interventions have evidence of efficacy in reducing suicide risk, including cognitive behavioral therapy,3,4 dialectical behavioral therapy,3 and active contact and follow-up interventions.5,6”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [3] D’Anci KE, Uhl S, Giradi G, Martin C. Treatments for the prevention and management of suicide. Ann Intern Med. 2019;171(5):334-342. doi: 10.7326/M19-0869 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [4] Gøtzsche PC, Gøtzsche PK. Cognitive behavioural therapy halves the risk of repeated suicide attempts: systematic review. J R Soc Med. 2017;110(10):404-410. doi: 10.1177/0141076817731904 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [5] Inagaki M, Kawashima Y, Kawanishi C, et al. Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis. J Affect Disord. 2015;175:66-78. doi: 10.1016/j.jad.2014.12.048 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [6] Riblet NBV, Shiner B, Young-Xu Y, Watts BV. Strategies to prevent death by suicide: meta-analysis of randomised controlled trials. Br J Psychiatry. 2017;210(6):396-402. doi: 10.1192/bjp.bp.116.187799 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Introduction

    A recent systematic review identified 17 suicide risk prediction models.

    “A recent systematic review10 identified 17 such predictive models that have been described in the literature.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Belsher BE, Smolenski DJ, Pruitt LD, et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76(6):642-651. doi: 10.1001/jamapsychiatry.2019.0174 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § Introduction

    Multiple research groups have developed suicide risk prediction models using electronic health record data.

    “multiple research groups have developed suicide risk prediction models using electronic health record data.10,11,12,13”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [10] Belsher BE, Smolenski DJ, Pruitt LD, et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76(6):642-651. doi: 10.1001/jamapsychiatry.2019.0174 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [11] Barak-Corren Y, Castro VM, Javitt S, et al. Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry. 2017;174(2):154-162. doi: 10.1176/appi.ajp.2016.16010077 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [12] Simon GE, Johnson E, Lawrence JM, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951-960. doi: 10.1176/appi.ajp.2018.17101167 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [13] Kessler RC, Hwang I, Hoffmire CA, et al. Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration. Int J Methods Psychiatr Res. 2017;26(3):1-7. doi: 10.1002/mpr.1575 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § Methods

    Uptake of the CBT intervention was 89.9%.

    “yielding an uptake of 89.9%”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] Rudd MD, Bryan CJ, Wertenberger EG, et al. Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up. Am J Psychiatry. 2015;172(5):441-449. doi: 10.1176/appi.ajp.2014.14070843 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Methods

    Several univariate sensitivity analyses were performed to assess robustness to alternative assumptions.

    “we also performed several univariate sensitivity analyses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    US guidelines recommend offering endocrine therapy to women whose annual risk of breast cancer exceeds 0.3%.

    “offering endocrine therapy to women whose annual risk of breast cancer exceeds 0.3%.42,43”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [42] Owens DK, Davidson KW, Krist AH, et al. ; US Preventive Services Task Force . Medication use to reduce risk of breast cancer: US Preventive Services Task Force recommendation statement. JAMA. 2019;322(9):857-867. doi: 10.1001/jama.2019.11885 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [43] Visvanathan K, Fabian CJ, Bantug E, et al. Use of endocrine therapy for breast cancer risk reduction: ASCO clinical practice guideline update. J Clin Oncol. 2019;37(33):3152-3165. doi: 10.1200/JCO.19.01472 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Discussion

    Higher PPVs (>10%) could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified.

    “even higher PPVs (>10%)12 could be attained by increasing specificity above 99%, although with a consequent decrease in the proportion of cases identified.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Simon GE, Johnson E, Lawrence JM, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951-960. doi: 10.1176/appi.ajp.2018.17101167 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    US guidelines recommend offering anticoagulants to patients with atrial fibrillation whose annual risk of stroke exceeds 0.6% to 2%.

    “US guidelines recommend offering anticoagulants to patients with atrial fibrillation whose annual risk of stroke exceeds 0.6% to 2%38,39”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [38] January CT, Wann LS, Calkins H, et al. 2019 AHA/ACC/HRS focused update of the 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart R. J Am Coll Cardiol. 2019;74(1):104-132. doi: 10.1016/j.jacc.2019.01.011 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [39] Friberg L, Rosenqvist M, Lip GYH. Net clinical benefit of warfarin in patients with atrial fibrillation: a report from the Swedish atrial fibrillation cohort study. Circulation. 2012;125(19):2298-2307. doi: 10.1161/CIRCULATIONAHA.111.055079 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Methods

    Annual background health care costs range from $4016 to $11,740 depending on age.

    “Annual background health care costs, derived from the 2017 Medical Expenditure Panel Survey, ranged from $4016 to $11 740, depending on age.26”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Agency for Healthcare Research and Quality . Mean expenditure per person with expense (standard errors) by age groups, United States, 2017, Medical Expenditure Panel Survey. Accessed March 19, 2020. https://meps.ahrq.gov/mepstrends/hc_use/ source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § Methods

    Probabilistic sensitivity analyses were performed with 1000 individual model runs using parameter values drawn at random from distributions reflecting their uncertainty.

    “we performed probabilistic sensitivity analyses. In this process, 1000 individual model runs were performed using parameter values drawn at random from distributions reflecting their uncertainty.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § Discussion

    Until such data are available, the current analysis must rely on extrapolation.

    “Until such data are available, the current analysis must rely on extrapolation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The efficacy of risk reduction interventions observed in emergency department populations can be extrapolated to proactively identified primary care outpatients.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    US guidelines recommend offering bisphosphonates to patients with low bone density whose annual risk of hip fracture or major osteoporotic fracture exceeds 0.3% or 2%, respectively.

    “offering bisphosphonates to patients with low bone density whose annual risk of hip fracture or major osteoporotic fracture exceeds 0.3% or 2%, respectively40,41”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [40] Cosman F, de Beur SJ, LeBoff MS, et al. ; National Osteoporosis Foundation . Clinician’s guide to prevention and treatment of osteoporosis. Osteoporos Int. 2014;25(10):2359-2381. doi: 10.1007/s00198-014-2794-2 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [41] Tosteson ANA, Melton LJ III, Dawson-Hughes B, et al. ; National Osteoporosis Foundation Guide Committee . Cost-effective osteoporosis treatment thresholds: the United States perspective. Osteoporos Int. 2008;19(4):437-447. doi: 10.1007/s00198-007-0550-6 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § Methods

    The study adhered to the 2013 Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline.

    “we adhere to the 2013 Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline.19”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Husereau D, Drummond M, Petrou S, et al. ; ISPOR Health Economic Evaluation Publication Guidelines-CHEERS Good Reporting Practices Task Force . Consolidated Health Economic Evaluation Reporting Standards (CHEERS): explanation and elaboration: a report of the ISPOR Health Economic Evaluations Publication Good Reporting Practices Task Force. Value Health. 2013;16(2):231-250. doi: 10.1016/j.jval.2013.02.002 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § Introduction

    What constitutes adequate accuracy for suicide risk prediction is unknown.

    “what constitutes an adequate accuracy for suicide risk prediction is unknown.14”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Kessler RC. Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go. JAMA Psychiatry. 2019;76(8):777-778. doi: 10.1001/jamapsychiatry.2019.1238 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Methods

    Initial uptake of the ACF intervention was 99.4%.

    “initial uptake of the intervention was 99.4%”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Stanley B, Brown GK, Brenner LA, et al. Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry. 2018;75(9):894-900. doi: 10.1001/jamapsychiatry.2018.1776 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation
    § Methods

    The population suicide risk distribution was calibrated to reproduce suicide attempt and suicide death rates of 175 per 100,000 person-years and 15 per 100,000 person-years, and a fraction of suicide attempts preceded by a prior attempt of 54%.

    “We calibrated the population suicide risk distribution to reproduce the following outcomes: suicide attempt and suicide death rates of 175 per 100 000 person-years and 15 per 100 000 person-years, derived from individuals 15 years or older in national emergency department and inpatient databases,20 and fraction of suicide attempts preceded by a prior attempt of 54%, drawn from the National Epidemi”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [20] Conner A, Azrael D, Miller M. Suicide case-fatality rates in the United States, 2007 to 2014: a nationwide population-based study. Ann Intern Med. 2019;171(12):885-895. doi: 10.7326/M19-1324 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [21] Olfson M, Blanco C, Wall M, et al. National trends in suicide attempts among adults in the United States. JAMA Psychiatry. 2017;74(11):1095-1103. doi: 10.1001/jamapsychiatry.2017.2582 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationcitation
    § Discussion

    The low PPV thresholds reflect the immense societal and individual consequences of suicide.

    “However, that these values are so low reflects the immense societal and individual consequences of suicide.14,25”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] Kessler RC. Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go. JAMA Psychiatry. 2019;76(8):777-778. doi: 10.1001/jamapsychiatry.2019.1238 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [25] Centers for Disease Control and Prevention . WISQARS cost of injury reports. Accessed March 19, 2020. https://wisqars.cdc.gov:8443/costT/ source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § Discussion

    Using the current findings to guide large-scale implementation of any specific suicide risk prediction model would be premature.

    “For this reason, using the current findings to guide large-scale implementation of any specific suicide risk prediction model would be premature”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The efficacy of risk reduction interventions observed in emergency department populations can be extrapolated to proactively identified primary care outpatients.

  • May combine two claims.

    Premisecitation
    § Discussion

    Currently, no evidence is available on the efficacy of interventions in a proactively identified outpatient population, although a randomized clinical trial in this population is underway.

    “Currently, no evidence is available on the efficacy of interventions in a proactively identified outpatient population, although a randomized clinical trial in this population is underway.44”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [44] Simon GE, Beck A, Rossom R, et al. Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial. Trials. 2016;17(1):452. doi: 10.1186/s13063-016-1566-z [DOI] [PMC free article] [PubMed] [Google Scholar]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § Discussion

    Overall, this study's benchmarks suggest that any of these models could be adequate for targeting an ACF intervention, and some may be accurate enough to use for targeting CBT.

    “Overall, this study’s benchmarks suggest that any of these models could be adequate for targeting an ACF intervention, and some may be accurate enough to use for targeting CBT.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The efficacy of risk reduction interventions observed in emergency department populations can be extrapolated to proactively identified primary care outpatients.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § Limitations

    These factors were not relevant to this analysis given the primary objective of establishing accuracy benchmarks.

    “These factors were not relevant to this analysis given the primary objective of establishing accuracy benchmarks”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § Limitations

    This issue was addressed by performing a thorough cost accounting from 2 specific exemplar interventions.

    “this issue was addressed by instead performing a thorough cost accounting from 2 specific exemplar interventions.30,31”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [30] Stanley B, Brown GK, Brenner LA, et al. Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry. 2018;75(9):894-900. doi: 10.1001/jamapsychiatry.2018.1776 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [31] Rudd MD, Bryan CJ, Wertenberger EG, et al. Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up. Am J Psychiatry. 2015;172(5):441-449. doi: 10.1176/appi.ajp.2014.14070843 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § Limitations

    Because these secondary benefits were observed in selected patient populations (eg, after a suicide attempt or psychiatric hospitalization), the most conservative approach was to not assume equivalent benefits in primary care patients.

    “Because these secondary benefits were observed in selected patient populations (eg, after a suicide attempt or psychiatric hospitalization), the most conservative approach was to not assume equivalent benefits in primary care patients”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There are no data from primary care populations to inform whether risk reduction interventions would produce benefits or harms.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § Limitations

    Where possible, this study made conservative assumptions about the benefits of risk prediction and intervention (eg, assuming a 12-month duration of intervention efficacy), which may result in more stringent estimates of threshold accuracy values.

    “Where possible, this study made conservative assumptions about the benefits of risk prediction and intervention (eg, assuming a 12-month duration of intervention efficacy), which may result in more stringent estimates of threshold accuracy values.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § Limitations

    Previous meta-analyses have estimated varying effect sizes for primary benefits of risk reduction interventions, with not all showing statistically significant benefits.

    “previous meta-analyses4,5,6,45 have estimated varying effect sizes, with not all showing statistically significant benefits”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [4] Gøtzsche PC, Gøtzsche PK. Cognitive behavioural therapy halves the risk of repeated suicide attempts: systematic review. J R Soc Med. 2017;110(10):404-410. doi: 10.1177/0141076817731904 [DOI] [PMC free article] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [5] Inagaki M, Kawashima Y, Kawanishi C, et al. Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis. J Affect Disord. 2015;175:66-78. doi: 10.1016/j.jad.2014.12.048 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [6] Riblet NBV, Shiner B, Young-Xu Y, Watts BV. Strategies to prevent death by suicide: meta-analysis of randomised controlled trials. Br J Psychiatry. 2017;210(6):396-402. doi: 10.1192/bjp.bp.116.187799 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

    • cites [45] Milner AJ, Carter G, Pirkis J, Robinson J, Spittal MJ. Letters, green cards, telephone calls and postcards: systematic and meta-analytic review of brief contact interventions for reducing self-harm, suicide attempts and suicide. Br J Psychiatry. 2015;206(3):184-190. doi: 10.1192/bjp.bp.114.147819 [DOI] [PubMed] [Google Scholar]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § Limitations

    As a modeling analysis, the work elides some real-world complexities (eg, heterogeneity of suicide case-fatality rates by method and demographic factors) and requires assumptions about model structure and input data.

    “As a modeling analysis, the work elides some real-world complexities (eg, heterogeneity of suicide case-fatality rates by method and demographic factors20) and requires assumptions about model structure and input data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] Conner A, Azrael D, Miller M. Suicide case-fatality rates in the United States, 2007 to 2014: a nationwide population-based study. Ann Intern Med. 2019;171(12):885-895. doi: 10.7326/M19-1324 [DOI] [PubMed] [Google Scholar] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § Limitations

    This choice could underestimate or overestimate the value of risk reduction interventions to some extent.

    “however, this choice could underestimate or overestimate the value of risk reduction interventions to some extent.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: There are no data from primary care populations to inform whether risk reduction interventions would produce benefits or harms.

  • May combine two claims.

    Interpretationreasoning
    § Conclusions

    Although results do not conclusively demonstrate that any specific suicide risk prediction model is cost-effective, they suggest that current risk prediction models have achieved sufficient accuracy for health systems to move forward with pilot implementation projects.

    “Although the results do not conclusively demonstrate that any specific suicide risk prediction model is cost-effective, they suggest that current risk prediction models have achieved sufficient accuracy for health systems to move forward with pilot implementation projects.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The efficacy of risk reduction interventions observed in emergency department populations can be extrapolated to proactively identified primary care outpatients.
34 of 34

  • The authors state this as an assumption.

    Any patient with high predicted suicide risk would require in-person evaluation consisting of an added or extended primary care appointment.

    “We assumed that any patient with high predicted suicide risk would require in-person evaluation that consisted of an added or extended primary care appointment.”

  • The authors state this as an assumption.

    The reduction in suicide risk with any intervention would last for 12 months.

    “We assumed the reduction in suicide risk with any intervention would last for 12 months”

  • The authors state this as an assumption.

    Initial safety planning would be incorporated into the baseline assessment.

    “We assumed that initial safety planning would be incorporated into the baseline assessment described above.”

  • The authors state this as an assumption.

    Each follow-up call would average 20 minutes of patient time.

    “we assumed that each call would average 20 minutes of patient time”

  • The authors state this as an assumption.

    Each CBT session would require a mean of 2 hours of patient time.

    “we assumed each session would require a mean of 2 hours of patient time”

  • The paper relies on this without support.

    The efficacy of risk reduction interventions observed in emergency department populations can be extrapolated to proactively identified primary care outpatients.

    “Until such data are available, the current analysis must rely on extrapolation.”

  • The paper relies on this without support.

    There are no data from primary care populations to inform whether risk reduction interventions would produce benefits or harms.

    “With respect to secondary effects, there are no data from primary care populations to inform whether risk reduction interventions would produce benefits (eg, improvements in depression) or harms (eg, increased stigma).”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

3 of the 55 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Ross, E. L., Zuromski, K. L., Reis, B. Y., Nock, M. K., Kessler, R. C., & Smoller, J. W.
Affiliations
  • Harvard University (US)
  • Boston Children's Hospital (US)
  • Massachusetts General Hospital (US)
Corresponding author
Ross, Eric L.
ORCID
Venue
JAMA Psychiatry
Publisher
American Medical Association (AMA)
Published
2021-06-01
Volume
78
Issue
6
Pages
642
ISSN
2168-622X (ISSN-L)
DOI
10.1001/jamapsychiatry.2021.0089
PMC
PMC7970389
OpenAlex
W3137644897
Semantic Scholar
b9711145f52634899aa8bd290dcd65a9f327a2bf
Cited by
65
References
55
Citations by year
2026: 12 · 2025: 12 · 2024: 10 · 2023: 16 · 2022: 11 · 2021: 4
FWCI
6.99
Type
journal article
Language
en
Open access
Locations
  • JAMA Psychiatry (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
National Institute of Mental Health (K23MH120439, R25MH094612, R01MH117599)
Topics
  • Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
  • Social Sciences › Psychology › Social Psychology › Mental Health Treatment and Access
Keywords
suicide risk prediction, primary care, cost-effectiveness, positive predictive value, cognitive behavioral therapy, suicide attempts, suicide mortality, quality-adjusted life years, incremental cost-effectiveness ratio, decision-analytic model, United States adults, health care costs, quality of life
MeSH terms
Adult, Aftercare, Aged, Cost-Benefit Analysis, Female, Humans, Male, Middle Aged, Primary Health Care, Sensitivity and Specificity, Suicide, Attempted, United States, Models, Statistical, Cognitive Behavioral Therapy, Risk Assessment, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review