Which PHQ-9 Items Can Effectively Screen for Suicide? Machine Learning Approaches
Open accessThe DOI matches its Crossref record; 54 of the 56 references it cites were found in a registry or an index.
Abstract
(1) Background: The Patient Health Questionnaire-9 (PHQ-9) is a tool that screens patients for depression in primary care settings. In this study, we evaluated the efficacy of PHQ-9 in evaluating suicidal ideation (2) Methods: A total of 8760 completed questionnaires collected from college students were analyzed. The PHQ-9 was scored in combination with and evaluated against four categories (PHQ-2, PHQ-8, PHQ-9, and PHQ-10). Suicidal ideations were evaluated using the Mini-International Neuropsychiatric Interview suicidality module. Analyses used suicide ideation as the dependent variable, and machine learning (ML) algorithms, k-nearest neighbors, linear discriminant analysis (LDA), and random forest. (3) Results: Random forest application using the nine items of the PHQ-9 revealed an excellent area under the curve with a value of 0.841, with 94.3% accuracy. The positive and negative predictive values were 84.95% (95% CI = 76.03–91.52) and 95.54% (95% CI = 94.42–96.48), respectively. (4) Conclusion: This study confirmed that ML algorithms using PHQ-9 in the primary care field are reliably accurate in screening individuals with suicidal ideation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Also, the role and efficacy of pharmacological antidepressant treatment among patients with cancer with depressive symptoms have not been well studied, particularly in elderly patients 56,57 .”
From Suicide risk and mortality among patients with cancer (2022) · cites this paper as [57]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
(1998) · The Journal of Clinical Psychiatry · Cited by 19,139
(1998) · PubMed · Cited by 10,216
Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research
Franklin, J. C., Ribeiro, J. D., Fox, K. R., Bentley, K. H., Kleiman, E. M., Huang, X., Musacchio, K. M., Jaroszewski, A. C., Chang, B. P., & Nock, M. K. (2017) · Psychological Bulletin · doi:10.1037/bul0000084 · Cited by 3,784 · cited 2×
“These techniques will accurately predict suicidal behavior and actual suicide through longitudinal records of patients [31,33,34], socio-demographic information, psychopathological factors, and control for risk factors [49].”
MethodSuicide Prevention Strategies: A Systematic Review
Mann, J. J., Apter, A., Bertolote, J., Beautrais, A., Currier, D., Haas, A., Hegerl, U., Lonnqvist, J., Malone, K., Marusic, A., et al. (2005) · JAMA · doi:10.1001/jama.294.16.2064 · Cited by 2,852 · cited 1×
“Justification for population-based screening and prevention would require both evidence that the screening predicts suicidal risk in the general population and evidence that some feasible intervention can reduce that risk [53,54].”
BackgroundHand, D. J., Mannila, H., & Smyth, P. (2001) · Cited by 2,581
The diagnostic odds ratio: a single indicator of test performance
Glas, A. S., Lijmer, J. G., Prins, M. H., Bonsel, G. J., & Bossuyt, P. M. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00177-x · Cited by 2,261 · cited 1×
“We used the subtotal score of PHQ-9 items, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUC) for comparison with the clinical diagnosis of suicide ideation using 95% confidence intervals for each measure [43].”
MethodContact With Mental Health and Primary Care Providers Before Suicide: A Review of the Evidence
Luoma, J. B., Martin, C. E., & Pearson, J. L. (2002) · American Journal of Psychiatry · doi:10.1176/appi.ajp.159.6.909 · Cited by 1,576 · cited 1×
“Most individuals who die by suicide have visited mental health services providers within one year before death (rates of contact average 32%, range = 16–46%) [1,2].”
BackgroundLichtman, J. H., Bigger, J. T., Blumenthal, J. A., Frasure-Smith, N., Kaufmann, P. G., Lespérance, F., Mark, D. B., Sheps, D. S., Taylor, C. B., & Froelicher, E. S. (2008) · Circulation · doi:10.1161/circulationaha.108.190769 · Cited by 1,482 · cited 1×
“The American Heart Association Science Advisory recommends that all patients who answer “yes” to item 9 of PHQ-9 be immediately assessed for suicide [25].”
Background
Research works printed but not linked: 3
- Yoo, 2006, Validity of Korean version of the mini-international neuropsychiatric interview, Anxiety Mood, 2, 50
- Stork, D.G., Duda, R.O., Hart, P.E., and Stork, D. (2001). Pattern Classification, Wiley.
- Zhu, W., Zeng, N., and Wang, N. (2010, January 14–17). Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. Proceedings of the NESUG proceedings: Health care and life sciences, Baltimore, MD, USA.
Cited by
Suicide risk and mortality among patients with cancer
Heinrich, M., Hofmann, L., Baurecht, H., Kreuzer, P. M., Knüttel, H., Leitzmann, M. F., & Seliger, C. (2022) · Nature Medicine · doi:10.1038/s41591-022-01745-y · Cited by 136
“57 ).”
Dananjaya, V., Chevali, V. S., Dear, J. P., Potluri, P., & Abeykoon, C. (2024) · Progress in Materials Science · doi:10.1016/j.pmatsci.2024.101336 · Cited by 160
Spottswood, M., Lim, C. T., Davydow, D., & Huang, H. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.892205 · Cited by 40
“Evidence exists for machine learning algorithms in the EMR paired with the PHQ-9 question 9 resulting in effective screening for suicide in a college student population (38).”
MethodAssociation between long-term sedentary behavior and depressive symptoms in U.S. adults
Guo, Y., Li, K., Zhao, Y., Wang, C., Mo, H., & Li, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-55898-6 · Cited by 50
“, was used to evaluate participants’ self-injury tendencies 31 .”
MethodPotential paths to suicidal ideation and suicide attempts among high-risk women
Weiss, S. J., Simeonova, D. I., Koleva, H. K., Muzik, M., Clark, K. D., Özerdem, A., Cooper, B. A., & Ammerman, R. T. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.033 · Cited by 45
Metadata
- Authors
- Kim, S., Lee, H.-K., & Lee, K.
- Affiliations
- Hanyang University Seoul Hospital (KR)
- Kongju National University (KR)
- Corresponding author
- Lee, Kounseok
- Venue
- International Journal of Environmental Research and Public Health
- Publisher
- MDPI AG
- Published
- 2021-03-24
- Volume
- 18
- Issue
- 7
- Pages
- 3339
- ISSN
- 1660-4601 (ISSN-L)
- PMC
- PMC8036742
- OpenAlex
- W3145871323
- Cited by
- 99
- References
- 56
- Citations by year
- 2026: 11 · 2025: 21 · 2024: 26 · 2023: 15 · 2022: 20 · 2021: 6
- FWCI
- 11.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- International Journal of Environmental Research and Public Health (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Research Foundation of Korea (2018R1D1A1B07050245)
- Ministry of Trade, Industry and Energy (20012931)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Suicide and Self-Harm Studies
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Keywords
- PHQ-9, suicidal ideation, suicide risk screening, depression screening, PHQ-2, PHQ-8, university students, random forest, linear discriminant analysis, k-nearest neighbors, primary care
- MeSH terms
- Machine Learning, Patient Health Questionnaire, Depression, Humans, Mass Screening, Psychiatric Status Rating Scales, Surveys and Questionnaires, Suicidal Ideation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Which PHQ-9 Items Can Effectively Screen for Suicide? Machine Learning Approaches. https://app.cochat.ai/papers/10.3390/ijerph18073339
- The paper
- Kim, S., Lee, H.-K., & Lee, K. (2021). Which PHQ-9 Items Can Effectively Screen for Suicide? Machine Learning Approaches. International Journal of Environmental Research and Public Health. https://doi.org/10.3390/ijerph18073339
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
