Which PHQ-9 Items Can Effectively Screen for Suicide? Machine Learning Approaches
Open accessThe DOI matches its Crossref record; 54 of the 56 references it cites were found in a registry or an index.
Abstract
(1) Background: The Patient Health Questionnaire-9 (PHQ-9) is a tool that screens patients for depression in primary care settings. In this study, we evaluated the efficacy of PHQ-9 in evaluating suicidal ideation (2) Methods: A total of 8760 completed questionnaires collected from college students were analyzed. The PHQ-9 was scored in combination with and evaluated against four categories (PHQ-2, PHQ-8, PHQ-9, and PHQ-10). Suicidal ideations were evaluated using the Mini-International Neuropsychiatric Interview suicidality module. Analyses used suicide ideation as the dependent variable, and machine learning (ML) algorithms, k-nearest neighbors, linear discriminant analysis (LDA), and random forest. (3) Results: Random forest application using the nine items of the PHQ-9 revealed an excellent area under the curve with a value of 0.841, with 94.3% accuracy. The positive and negative predictive values were 84.95% (95% CI = 76.03–91.52) and 95.54% (95% CI = 94.42–96.48), respectively. (4) Conclusion: This study confirmed that ML algorithms using PHQ-9 in the primary care field are reliably accurate in screening individuals with suicidal ideation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also, the role and efficacy of pharmacological antidepressant treatment among patients with cancer with depressive symptoms have not been well studied, particularly in elderly patients 56,57 .”
From Suicide risk and mortality among patients with cancer (2022) · cites this paper as [57]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
(1998) · The Journal of Clinical Psychiatry · Cited by 19,139
(1998) · PubMed · Cited by 10,216
Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017) · Psychological Bulletin · doi:10.1037/bul0000084 · Cited by 3,784 · cited 2×
“These techniques will accurately predict suicidal behavior and actual suicide through longitudinal records of patients [31,33,34], socio-demographic information, psychopathological factors, and control for risk factors [49].”
MethodSuicide Prevention Strategies: A Systematic Review
Mann, J. J., Apter, A., Bertolote, J., Beautrais, A., Currier, D., Haas, A., Hegerl, U., Lonnqvist, J., Malone, K., Marusic, A., et al. (2005) · JAMA · doi:10.1001/jama.294.16.2064 · Cited by 2,852 · cited 1×
“Justification for population-based screening and prevention would require both evidence that the screening predicts suicidal risk in the general population and evidence that some feasible intervention can reduce that risk [53,54].”
BackgroundHand, D. J., Mannila, H., & Smyth, P. (2001) · Cited by 2,581
The diagnostic odds ratio: a single indicator of test performance
Glas, A. S., Lijmer, J. G., Prins, M. H., Bonsel, G. J., & Bossuyt, P. M. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00177-x · Cited by 2,261 · cited 1×
“We used the subtotal score of PHQ-9 items, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUC) for comparison with the clinical diagnosis of suicide ideation using 95% confidence intervals for each measure [43].”
MethodContact With Mental Health and Primary Care Providers Before Suicide: A Review of the Evidence
Luoma, J. B., Martin, C. E., & Pearson, J. L. (2002) · American Journal of Psychiatry · doi:10.1176/appi.ajp.159.6.909 · Cited by 1,576 · cited 1×
“Most individuals who die by suicide have visited mental health services providers within one year before death (rates of contact average 32%, range = 16–46%) [1,2].”
BackgroundLichtman, J. H., Bigger, J. T., Blumenthal, J. A., Frasure-Smith, N., Kaufmann, P. G., Lespérance, F., Mark, D. B., Sheps, D. S., Taylor, C. B., & Froelicher, E. S. (2008) · Circulation · doi:10.1161/circulationaha.108.190769 · Cited by 1,482 · cited 1×
“The American Heart Association Science Advisory recommends that all patients who answer “yes” to item 9 of PHQ-9 be immediately assessed for suicide [25].”
BackgroundRibeiro, J. D., Franklin, J. C., Fox, K. R., Bentley, K. H., Kleiman, E. M., Chang, B. P., & Nock, M. K. (2016) · Psychological Medicine · doi:10.1017/s0033291715001804 · Cited by 1,346
Risk factors for suicide in psychiatric outpatients: A 20-year prospective study.
Brown, G. K., Beck, A. T., Steer, R. A., & Grisham, J. R. (2000) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.68.3.371 · Cited by 1,317
Gilbody, S., Richards, D., Brealey, S., & Hewitt, C. (2007) · Journal of General Internal Medicine · doi:10.1007/s11606-007-0333-y · Cited by 1,290
Validation of PHQ-2 and PHQ-9 to Screen for Major Depression in the Primary Care Population
Arroll, B., Goodyear-Smith, F., Crengle, S., Gunn, J., Kerse, N., Fishman, T., Falloon, K., & Hatcher, S. (2010) · The Annals of Family Medicine · doi:10.1370/afm.1139 · Cited by 1,279 · cited 1×
“This tool has been verified for screening for depression in the general population and various groups outside the primary care environment [11,12].”
MethodDiscriminant Analysis and Statistical Pattern Recognition
McLachlan, G. J. (1992) · Wiley Series in Probability and Statistics · doi:10.1002/0471725293 · Cited by 1,267
Pignone, M. P., Gaynes, B. N., Rushton, J. L., Burchell, C. M., Orleans, C. T., Mulrow, C. D., & Lohr, K. N. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-136-10-200205210-00013 · Cited by 1,086
Hand, D. J. (2007) · Drug Safety · doi:10.2165/00002018-200730070-00010 · Cited by 1,044
Ribeiro, J. D., Huang, X., Fox, K. R., & Franklin, J. C. (2018) · British Journal of Psychiatry · doi:10.1192/bjp.2018.27 · Cited by 894
Simon, G. E., Rutter, C. M., Peterson, D., Oliver, M., Whiteside, U., Operskalski, B., & Ludman, E. J. (2013) · Psychiatric Services · doi:10.1176/appi.ps.201200587 · Cited by 575
Predicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 567 · cited 2×
“These techniques will accurately predict suicidal behavior and actual suicide through longitudinal records of patients [31,33,34], socio-demographic information, psychopathological factors, and control for risk factors [49].”
BackgroundMethodDiagnostic accuracy of the mood module of the patient health questionnaire: a systematic review
Wittkampf, K. A., Naeije, L., Schene, A. H., Huyser, J., & van Weert, H. C. (2007) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2007.06.004 · Cited by 493
Kessler, R. C., Warner, C. H., Ivany, C., Petukhova, M. V., Rose, S., Bromet, E. J., Brown, M., Cai, T., Colpe, L. J., Cox, K. L., et al. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.1754 · Cited by 447 · cited 1×
“These techniques will accurately predict suicidal behavior and actual suicide through longitudinal records of patients [31,33,34], socio-demographic information, psychopathological factors, and control for risk factors [49].”
MethodPredicting suicidal behavior from longitudinal electronic health records.
Barak-Corren, Y., Castro, V. M., Javitt, S., Hoffnagle, A. G., Dai, Y., Perlis, R. H., Nock, M. K., Smoller, J. W., & Reis, B. Y. (2017) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2016.16010077 · Cited by 339
Does Suicidal Ideation as Measured by the PHQ-9 Predict Suicide Among VA Patients?
Louzon, S. A., Bossarte, R., McCarthy, J. F., & Katz, I. R. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201500149 · Cited by 291 · cited 3×
“Higher scores in question 9 are better at predicting suicide risk and mortality [20].”
BackgroundResultAttendance at the accident and emergency department in the year before suicide: retrospective study.
Gairin, I., House, A., & Owens, D. (2003) · British Journal of Psychiatry · doi:10.1192/bjp.183.1.28 · Cited by 263 · cited 1×
“Most individuals who die by suicide have visited mental health services providers within one year before death (rates of contact average 32%, range = 16–46%) [1,2].”
BackgroundNa, P. J., Yaramala, S. R., Kim, J. A., Kim, H., Goes, F. S., Zandi, P. P., Vande Voort, J. L., Sutor, B., Croarkin, P., & Bobo, W. V. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.02.045 · Cited by 239
Corson, K., Gerrity, M., & Dobscha, S. K. (2004) · PubMed · Cited by 214
van Steenbergen-Weijenburg, K. M., de Vroege, L., Ploeger, R. R., Brals, J. W., Vloedbeld, M. G., Veneman, T. F., Hakkaart-van Roijen, L., Rutten, F. F., Beekman, A. T., & van der Feltz-Cornelis, C. M. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-235 · Cited by 200
Better Off Dead: Suicidal Thoughts in Cancer Patients
Walker, J., Waters, R. A., Murray, G., Swanson, H., Hibberd, C. J., Rush, R. W., Storey, D. J., Strong, V. A., Fallon, M. T., Wall, L. R., et al. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2007.11.8844 · Cited by 195
Razykov, I., Ziegelstein, R. C., Whooley, M. A., & Thombs, B. D. (2012) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2012.06.001 · Cited by 188 · cited 2×
“Therefore, studies are being conducted on the predictive value of item 9 of PHQ-9 for suicide or depression [19,26].”
BackgroundSuicide screening in schools, primary care and emergency departments
Horowitz, L. M., Ballard, E. D., & Pao, M. (2009) · Current Opinion in Pediatrics · doi:10.1097/mop.0b013e3283307a89 · Cited by 181 · cited 1×
“Furthermore, it should be highly sensitive to confidently rule out patients who do not show any visible risk [52].”
BackgroundGaynes, B. N., West, S. L., Ford, C. A., Frame, P., Klein, J., & Lohr, K. N. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-10-200405180-00015 · Cited by 175 · cited 1×
“Justification for population-based screening and prevention would require both evidence that the screening predicts suicidal risk in the general population and evidence that some feasible intervention can reduce that risk [53,54].”
BackgroundMachine learning in suicide science: Applications and ethics
Linthicum, K. P., Schafer, K. M., & Ribeiro, J. D. (2019) · Behavioral Sciences & the Law · doi:10.1002/bsl.2392 · Cited by 163 · cited 2×
“Further research is required to assess the correct therapeutic approach for those at risk [29].”
BackgroundSuicidal Ideation and Attempts in Adult Survivors of Childhood Cancer
Recklitis, C. J., Lockwood, R. A., Rothwell, M. A., & Diller, L. R. (2006) · Journal of Clinical Oncology · doi:10.1200/jco.2006.06.5409 · Cited by 138
Bauer, A. M., Chan, Y.-F., Huang, H. Y., Vannoy, S. D., & Unützer, J. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2194-2 · Cited by 123 · cited 1×
“Additionally, some studies have argued that item 9 responses yield a high false-positive ratio by evaluating both the thought of passive death and desire for self-harm in a single-response question [13,16,20,23,24,26].”
BackgroundYawn, B. P., Pace, W., Wollan, P. C., Bertram, S., Kurland, M., Graham, D., & Dietrich, A. (2009) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2009.05.080155 · Cited by 121
Rates and Predictors of Suicidal Ideation During the First Year After Traumatic Brain Injury
Mackelprang, J. L., Bombardier, C. H., Fann, J. R., Temkin, N. R., Barber, J. K., & Dikmen, S. S. (2014) · American Journal of Public Health · doi:10.2105/ajph.2013.301794 · Cited by 121
Standardized Screening for Suicidal Adolescents in Primary Care
Wintersteen, M. B. (2010) · Pediatrics · doi:10.1542/peds.2009-2458 · Cited by 121 · cited 1×
“These medical visits are an opportunity for clinicians to identify and refer patients at risk of suicide; however, patients often talk about their physical problems without discussing suicidal ideations and plans unless asked directly [3,4].”
BackgroundUebelacker, L. A., German, N. M., Gaudiano, B. A., & Miller, I. W. (2011) · The Primary Care Companion For CNS Disorders · doi:10.4088/pcc.10m01027 · Cited by 114 · cited 1×
“Additionally, some studies have argued that item 9 responses yield a high false-positive ratio by evaluating both the thought of passive death and desire for self-harm in a single-response question [13,16,20,23,24,26].”
BackgroundWalker, J., Hansen, C. H., Butcher, I., Sharma, N., Wall, L. R., Murray, G. D., & Sharpe, M. (2011) · Psychosomatics · doi:10.1016/j.psym.2011.02.003 · Cited by 104 · cited 2×
“Most people with suicidal ideations or self-harm respond positively to this question [16,44].”
BackgroundSuicidal Ideation Among Older Primary Care Patients
Callahan, C. M., Hendrie, H. C., Nienaber, N. A., & Tierney, W. M. (1996) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.1996.tb01370.x · Cited by 102 · cited 1×
“In addition, the cost-effectiveness of the suicide risk assessment scale as a first-line screening strategy is limited because of the relatively low prevalence of suicide among primary care patients [6,7].”
BackgroundDemographics as predictors of suicidal thoughts and behaviors: A meta-analysis
Huang, X., Ribeiro, J. D., Musacchio, K. M., & Franklin, J. C. (2017) · PLOS ONE · doi:10.1371/journal.pone.0180793 · Cited by 94
Ryu, S., Lee, H., Lee, D.-K., & Park, K. (2018) · Psychiatry Investigation · doi:10.30773/pi.2018.08.27 · Cited by 79 · cited 1×
“Other ML studies have shown suicidal ideation as a better potential predictor of suicidal risk than suicidal attempts [50].”
BackgroundSelf-citationWarning signs for suicide within a week of healthcare contact in Veteran decedents
Britton, P. C., Ilgen, M. A., Rudd, M. D., & Conner, K. R. (2012) · Psychiatry Research · doi:10.1016/j.psychres.2012.06.036 · Cited by 75
Oh, J., Yun, K., Hwang, J.-H., & Chae, J.-H. (2017) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2017.00192 · Cited by 65
Chronic Medical Conditions and Wishes to Die among Older Primary Care Patients
Kim, Y. A., Bogner, H. R., Brown, G. K., & Gallo, J. J. (2006) · The International Journal of Psychiatry in Medicine · doi:10.2190/3qxd-ur0h-k8fh-2cu8 · Cited by 58 · cited 1×
“In addition, the cost-effectiveness of the suicide risk assessment scale as a first-line screening strategy is limited because of the relatively low prevalence of suicide among primary care patients [6,7].”
BackgroundGradus, J. L., King, M. W., Galatzer‐Levy, I., & Street, A. E. (2017) · Journal of Traumatic Stress · doi:10.1002/jts.22210 · Cited by 56
Valuck, R. J., Anderson, H. O., Libby, A. M., Brandt, E., Bryan, C., Allen, R. R., Staton, E. W., West, D. R., & Pace, W. D. (2012) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2012.05.110053 · Cited by 54
The recognition of diagnosable psychiatric disorders in suicide cases' last medical contacts
Pan, Y., Lee, M., Chiang, H.-C., & Liao, S. (2009) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2008.12.010 · Cited by 46
Razykov, I., Hudson, M., Baron, M., Thombs, B. D., & Canadian Scleroderma Research Group (2013) · Arthritis Care & Research · doi:10.1002/acr.21894 · Cited by 41
Machine Learning for Suicide Research–Can It Improve Risk Factor Identification?
Fazel, S., & O’Reilly, L. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.2896 · Cited by 35 · cited 2×
“Machine learning algorithms have been proposed to improve diagnostic and prognostic accuracy and determine treatment options [27].”
BackgroundSuicide: Epidemiology, Prevention, and Treatment.
Kachur, S. P., Potter, L. B., Powell, K. E., & Rosenberg, M. L. (1995) · PubMed · Cited by 35
Denneson, L. M., Corson, K., Helmer, D. A., Bair, M. J., & Dobscha, S. K. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.03.017 · Cited by 30
The relationship between circadian typology and lifetime experiences of hypomanic symptoms
Lee, K., Lee, H.-K., Kim, S., & Kim, S. H. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.113788 · Cited by 7 · cited 2×
“In total, 8760 responses (4354 males, 4406 females) were analyzed [34].”
MethodSelf-citation
Research works printed but not linked: 3
- Yoo, 2006, Validity of Korean version of the mini-international neuropsychiatric interview, Anxiety Mood, 2, 50
- Stork, D.G., Duda, R.O., Hart, P.E., and Stork, D. (2001). Pattern Classification, Wiley.
- Zhu, W., Zeng, N., and Wang, N. (2010, January 14–17). Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. Proceedings of the NESUG proceedings: Health care and life sciences, Baltimore, MD, USA.
Cited by
Suicide risk and mortality among patients with cancer
Heinrich, M., Hofmann, L., Baurecht, H., Kreuzer, P. M., Knüttel, H., Leitzmann, M. F., & Seliger, C. (2022) · Nature Medicine · doi:10.1038/s41591-022-01745-y · Cited by 136
“57 ).”
Dananjaya, V., Chevali, V. S., Dear, J. P., Potluri, P., & Abeykoon, C. (2024) · Progress in Materials Science · doi:10.1016/j.pmatsci.2024.101336 · Cited by 160
Spottswood, M., Lim, C. T., Davydow, D., & Huang, H. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.892205 · Cited by 40
“Evidence exists for machine learning algorithms in the EMR paired with the PHQ-9 question 9 resulting in effective screening for suicide in a college student population (38).”
MethodAssociation between long-term sedentary behavior and depressive symptoms in U.S. adults
Guo, Y., Li, K., Zhao, Y., Wang, C., Mo, H., & Li, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-55898-6 · Cited by 50
“, was used to evaluate participants’ self-injury tendencies 31 .”
MethodPotential paths to suicidal ideation and suicide attempts among high-risk women
Weiss, S. J., Simeonova, D. I., Koleva, H. K., Muzik, M., Clark, K. D., Özerdem, A., Cooper, B. A., & Ammerman, R. T. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.033 · Cited by 45
Metadata
- Authors
- Kim, S., Lee, H.-K., & Lee, K.
- Affiliations
- Hanyang University Seoul Hospital (KR)
- Kongju National University (KR)
- Corresponding author
- Lee, Kounseok
- Venue
- International Journal of Environmental Research and Public Health
- Publisher
- MDPI AG
- Published
- 2021-03-24
- Volume
- 18
- Issue
- 7
- Pages
- 3339
- ISSN
- 1660-4601 (ISSN-L)
- PMC
- PMC8036742
- OpenAlex
- W3145871323
- Cited by
- 99
- References
- 56
- Citations by year
- 2026: 11 · 2025: 21 · 2024: 26 · 2023: 15 · 2022: 20 · 2021: 6
- FWCI
- 11.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- International Journal of Environmental Research and Public Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Research Foundation of Korea (2018R1D1A1B07050245)
- Ministry of Trade, Industry and Energy (20012931)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Keywords
- PHQ-9, suicidal ideation, suicide risk screening, depression screening, PHQ-2, PHQ-8, university students, random forest, linear discriminant analysis, k-nearest neighbors, primary care
- MeSH terms
- Machine Learning, Patient Health Questionnaire, Depression, Humans, Mass Screening, Psychiatric Status Rating Scales, Surveys and Questionnaires, Suicidal Ideation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
