Pairing Machine Learning and Clinical Psychology: How You Evaluate Predictive Performance Matters

R: A Language and Environment for Statistical Computing

R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works

Classification and Regression Trees.

Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works

Regularization and Variable Selection Via the Elastic Net

Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957 · see citing works

A decision-theoretic generalization of on-line learning and an application to boosting

Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466 · see citing works

Regularization Paths for Generalized Linear Models via Coordinate Descent

Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works

Classification and Regression by randomForest

Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594 · see citing works

Lifetime and 12-Month Prevalence of DSM-III-R Psychiatric Disorders in the United States: Results From the National Comorbidity Survey

Kessler, R. C. (1994) · Archives of General Psychiatry · doi:10.1001/archpsyc.1994.03950010008002 · Cited by 12,370 · see citing works

Building Predictive Models in R Using the caret Package

Kühn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 9,986 · see citing works

Estimating the reproducibility of psychological science

Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works

Stochastic gradient boosting

Friedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067 · see citing works

Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis

Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878 · see citing works

Applied Predictive Modeling

Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · see citing works

The relationship between Precision-Recall and ROC curves

Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works

A working guide to boosted regression trees

Elith, J., Leathwick, J. R., & Hastie, T. (2008) · Journal of Animal Ecology · doi:10.1111/j.1365-2656.2008.01390.x · Cited by 6,484 · see citing works

Untitled work

(2010) · University of Twente Research Information · Cited by 6,037 · see citing works

The Elements of Statistical Learning

Hastie, T. J., Tibshirani, R. J., & Friedman, J. H. (2009) · Springer series in statistics · doi:10.1007/b94608 · Cited by 4,111 · see citing works

A few useful things to know about machine learning

Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326 · see citing works

An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests.

Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583 · see citing works

Estimating the error rate of a prediction rule: improvement on cross-validation

Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · see citing works

A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models

Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · see citing works

Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning

Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,930 · see citing works

Improvements on Cross-Validation: The .632+ Bootstrap Method

Efron, B., & Tibshirani, R. J. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,638 · see citing works

Problems in the Analysis of Survey Data, and a Proposal

Morgan, J. N., & Sonquist, J. A. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500855 · Cited by 1,224 · see citing works

Cross-validation pitfalls when selecting and assessing regression and classification models

Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014) · Journal of Cheminformatics · doi:10.1186/1758-2946-6-10 · Cited by 1,090 · see citing works

Machine Learning Approaches for Clinical Psychology and Psychiatry

Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966 · see citing works

WHY SUMMARIES OF RESEARCH ON PSYCHOLOGICAL THEORIES ARE OFTEN UNINTERPRETABLE

Meehl, P. E. (1990) · Psychological Reports · doi:10.2466/pr0.1990.66.1.195 · Cited by 901 · see citing works

Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints

van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-137 · Cited by 809 · see citing works

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

Steyerberg, E. W., Bleeker, S. E., Moll, H. A., Grobbee, D. E., & Moons, K. G. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00047-7 · Cited by 611 · see citing works

gbm: Generalized Boosted Regression Models

Ridgeway, G. K., & Developers, G. (2003) · doi:10.32614/cran.package.gbm · Cited by 580 · see citing works

An Empirical Comparison of Model Validation Techniques for Defect Prediction Models

Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574 · see citing works

Predicting Risk of Suicide Attempts Over Time Through Machine Learning

Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 571 · see citing works

Toward a Philosophical Structure for Psychiatry

Kendler, K. S. (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.3.433 · Cited by 563 · see citing works

False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2015) · American Psychological Association eBooks · doi:10.1037/14805-033 · Cited by 516 · see citing works

Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods

Borra, S., & Ciaccio, A. D. (2010) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.03.004 · Cited by 236 · see citing works

The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: A systematic review

Burke, T. A., Ammerman, B. A., & Jacobucci, R. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.11.073 · Cited by 202 · see citing works

Studying depression using imaging and machine learning methods

Patel, M. J., Khalaf, A. M., & Aizenstein, H. J. (2015) · NeuroImage Clinical · doi:10.1016/j.nicl.2015.11.003 · Cited by 179 · see citing works

The Random Forests statistical technique: An examination of its value for the study of reading

Matsuki, K., Kuperman, V., & Dyke, J. A. V. (2016) · Scientific Studies of Reading · doi:10.1080/10888438.2015.1107073 · Cited by 175 · see citing works

Machine Learning and Psychological Research: The Unexplored Effect of Measurement

Jacobucci, R., & Grimm, K. J. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620902467 · Cited by 160 · see citing works

Bridging a translational gap: using machine learning to improve the prediction of PTSD

For members of the Jerusalem Trauma Outreach and Prevention Study (J-TOPS) group, Karstoft, K., Galatzer‐Levy, I. R., Statnikov, A., Li, Z., & Shalev, A. Y. (2015) · BMC Psychiatry · doi:10.1186/s12888-015-0399-8 · Cited by 151 · see citing works

From Many to One to Many—the Search for Causes of Psychiatric Illness

Kendler, K. S. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1200 · Cited by 129 · see citing works

A Comparison between Decision Tree and Random Forest in Determining the Risk Factors Associated with Type 2 Diabetes.

Esmaily, H., Tayefi, M., Doosti, H., Ghayour‐Mobarhan, M., Nezami, H., & Amirabadizadeh, A. (2018) · PubMed Central · Cited by 85 · see citing works

Regression trees for predicting mortality in patients with cardiovascular disease: What improvement is achieved by using ensemble‐based methods?

Austin, P. C., Lee, D. S., Steyerberg, E. W., & Tu, J. V. (2012) · Biometrical Journal · doi:10.1002/bimj.201100251 · Cited by 80 · see citing works

Predicting imminent suicidal thoughts and nonfatal attempts: The role of complexity

Ribeiro, J. D., Huang, X., Fox, K. R., Walsh, C. G., & Linthicum, K. P. (2019) · Clinical Psychological Science · doi:10.1177/2167702619838464 · Cited by 75 · see citing works

The differences between suicide ideators and suicide attempters: simple, complicated, or complex

Huang, X., Ribeiro, J. D., & Franklin, J. C. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000498 · Cited by 50 · see citing works

Model complexity improves the prediction of nonsuicidal self-injury.

Fox, K., Huang, X., Linthicum, K. P., Wang, S. B., Franklin, J., & Ribeiro, J. D. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000421 · Cited by 49 · see citing works

Psychological primitives can make sense of biopsychosocial factor complexity in psychopathology

Franklin, J. C. (2019) · BMC Medicine · doi:10.1186/s12916-019-1435-1 · Cited by 28 · see citing works

Translating promise into practice: a review of machine learning in suicide research and prevention

Kirtley, O. J., van Mens, K., Hoogendoorn, M., Kapur, N., & de Beurs, D. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00254-6 · Cited by 100 · see citing works

Predicting acute suicidal ideation on Instagram using ensemble machine learning models

Lekkas, D., Klein, R. J., & Jacobson, N. C. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100424 · Cited by 59 · see citing works

The Differences Between Individuals Engaging in Nonsuicidal Self-Injury and Suicide Attempt Are Complex (vs. Complicated or Simple)

Huang, X., Ribeiro, J. D., & Franklin, J. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00239 · Cited by 49 · see citing works

Irremediability in psychiatric euthanasia: examining the objective standard

Nicolini, M. E., Jardas, E., Zarate, C. A., Gastmans, C., & Kim, S. Y. H. (2023) · Psychological Medicine · doi:10.1017/s0033291722002951 · Cited by 31 · see citing works

Predicting Mental Health Treatment Access Among Adolescents With Elevated Depressive Symptoms: Machine Learning Approaches

Dobias, M., Sugarman, M. B., Mullarkey, M. C., & Schleider, J. L. (2021) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-021-01146-2 · Cited by 22 · see citing works

Machine Learning to Classify Suicidal Thoughts and Behaviors: Implementation Within the Common Data Elements Used by the Military Suicide Research Consortium

Littlefield, A. K., Cooke, J. T., Bagge, C. L., Glenn, C. R., Kleiman, E. M., Jacobucci, R., Millner, A. J., & Steinley, D. (2021) · Clinical Psychological Science · doi:10.1177/2167702620961067 · Cited by 14 · see citing works

Review on Psychology Research Based on Artificial Intelligence Methodologies

Dave, R., Sargeant, K., Vanamala, M., & Seliya, N. (2022) · Journal of Computer and Communications · doi:10.4236/jcc.2022.105008 · Cited by 7 · see citing works

Predicting non-improvement of symptoms in daily mental healthcare practice using routinely collected patient-level data: a machine learning approach

Franken, K., ten Klooster, P. M., Bohlmeijer, E. T., Westerhof, G. J., & Kraiss, J. T. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1236551 · Cited by 7 · see citing works

A Systematic Literature Review of How to Treat Cognitive Psychology with Artificial Intelligence

Saeed, S., Hussain, M., Naqvi, M., & Jabbar, K. A. (2023) · Advances in intelligent systems and computing · doi:10.1007/978-981-99-3611-3_29 · Cited by 3 · see citing works

Predicting mental health treatment access in high-symptom adolescents: Machine learning approachesPreprint

Dobias, M., Sugarman, M. B., Mullarkey, M. C., & Schleider, J. L. (2019) · doi:10.31234/osf.io/294ty

MACHINE LEARNING TO CREATE DECISION TREE MODEL TO PREDICT OUTCOME OF ENTERPRENEURSHIP PSYCHOLOGICAL READINESS (EPR)

Syafitri, N., Farradinna, S., Jayanti, W., & Arta, Y. (2023) · Jurnal Teknik Informatika (Jutif) · doi:10.52436/1.jutif.2023.4.2.590

Predicting adolescent suicidal tendency in Chinese secondary school students: a machine learning approach with XGBoost and SHAP interpretation

Yu, R., Chen, J., Wang, Y., Guan, L., Xia, H., Chen, H., Tan, T., Chi, Y., Yang, P., Li, J., et al. (2026) · BMC Public Health · doi:10.1186/s12889-026-27943-8

Evidence of Inflated Prediction Performance: A Commentary on Machine Learning and Suicide Research

Jacobucci, R., Littlefield, A. K., Millner, A. J., Kleiman, E. M., & Steinley, D. (2021) · Clinical Psychological Science · doi:10.1177/2167702620954216 · Cited by 56 · see citing works

Supervised machine learning methods in psychology: A practical introduction with annotated R code

Rosenbusch, H., Soldner, F., Evans, A. M., & Zeelenberg, M. (2021) · Social and Personality Psychology Compass · doi:10.1111/spc3.12579 · Cited by 87 · see citing works

Beyond the hype: A simulation study evaluating the predictive performance of machine learning models in psychology.

Speck, K.-L., Jankowsky, K., Scharf, F., & Schroeders, U. (2026) · Psychological Methods · doi:10.1037/met0000832 · Cited by 2 · see citing works

Quantifying Optimization Bias in Model Evaluation when using Cross-Validation in Psychological Science: A Monte Carlo Simulation StudyPreprint

Moshontz, H., Fronk, G. E., Sant'Ana, S. J. K., & Curtin, J. J. (2020) · doi:10.31234/osf.io/ns9mj · Cited by 2 · see citing works

The prediction of performance in clinical psychology.

Kelly, E. L., & Fiske, D. W. (1951) · doi:10.1037/14626-000 · Cited by 167 · see citing works