The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Supervised Machine Learning: A Brief Primer
Open accessAbstract
Machine learning is increasingly used in mental health research and has the potential to advance our understanding of how to characterize, predict, and treat mental disorders and associated adverse health outcomes (e.g., suicidal behavior). Machine learning offers new tools to overcome challenges for which traditional statistical methods are not well-suited. This manuscript provides an overview of machine learning with a specific focus on supervised learning (i.e., methods that are designed to predict or classify an outcome of interest). Several common supervised learning methods are described, along with applied examples from the published literature. We also provide an overview of supervised learning model building, validation, and performance evaluation. Finally, challenges in creating robust and generalizable machine learning algorithms are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Confirmatory factor analysis for applied research
(2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,614
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013) · Wiley Series in Probability and Statistics · doi:10.1002/9781118548387 · Cited by 13,861
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Friedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,459
Supervised Machine Learning: A Review of Classification Techniques
Kotsiantis, S. B. (2007) · Informatica · Cited by 4,185
Unbiased Recursive Partitioning: A Conditional Inference Framework
Hothorn, T., Hornik, K., & Zeileis, A. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x133933 · Cited by 4,021
Bias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,688
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., Brown, T. A., Carpenter, W. T., Caspi, A., Clark, L. A., et al. (2017) · Journal of Abnormal Psychology · doi:10.1037/abn0000258 · Cited by 3,335
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Toward the future of psychiatric diagnosis: the seven pillars of RDoC
Cuthbert, B. N., & Insel, T. R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-126 · Cited by 3,053
Strobl, C., Malley, J., & Tutz, G. (2009) · Psychological Methods · doi:10.1037/a0016973 · Cited by 2,583
Hawkins, D. M. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci0342472 · Cited by 2,522
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · cited 1×
“Further, there is the potential for negligible incremental performance compared to traditional approaches and thus machine learning may not be preferable or efficient for certain data structures (Christodoulou et al., 2019).”
BackgroundClassification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777
van der Laan, M. J., Polley, E. C., & Hubbard, A. E. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,719
Variable selection – A review and recommendations for the practicing statistician
Heinze, G., Wallisch, C., & Dunkler, D. (2018) · Biometrical Journal · doi:10.1002/bimj.201700067 · Cited by 1,639
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631
BART: Bayesian additive regression trees
Chipman, H. A., George, E. I., & McCulloch, R. E. (2010) · The Annals of Applied Statistics · doi:10.1214/09-aoas285 · Cited by 1,542
Reduction Techniques for Instance-Based Learning Algorithms
Wilson, D. R., & Martinez, T. R. (2000) · Machine Learning · doi:10.1023/a:1007626913721 · Cited by 1,182
Hernán, M. A., Sauer, B. C., Hernández-Díaz, S., Platt, R., & Shrier, I. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.014 · Cited by 1,097
Machine learning in mental health: a scoping review of methods and applications
Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019) · Psychological Medicine · doi:10.1017/s0033291719000151 · Cited by 1,047
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
Improving propensity score weighting using machine learning
Lee, B. K., Lessler, J., & Stuart, E. A. (2010) · Statistics in Medicine · doi:10.1002/sim.3782 · Cited by 896
BART: Bayesian Additive Regression Trees
Chipman, H., George, E. I., & McCulloch, R. E. (2006) · ScholarlyCommons (University of Pennsylvania) · Cited by 889
Targeted Maximum Likelihood Learning
van der Laan, M. J., & Rubin, D. (2006) · The International Journal of Biostatistics · doi:10.2202/1557-4679.1043 · Cited by 823
Reporting of artificial intelligence prediction models
Collins, G. S., & Moons, K. G. M. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30037-6 · Cited by 701
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 1×
“Finally, one underreported metric of model performance is calibration, or goodness of fit (Bouwmeester et al., 2012).”
BackgroundPredicting Risk of Suicide Attempts Over Time Through Machine Learning
Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017) · Clinical Psychological Science · doi:10.1177/2167702617691560 · Cited by 567
Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.0174 · Cited by 540
Predicting sample size required for classification performance
Figueroa, R. L., Zeng-Treitler, Q., Kandula, S., & Ngo, L. H. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-8 · Cited by 520
A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks
Hernán, M. A., Hsu, J., & Healy, B. (2019) · CHANCE · doi:10.1080/09332480.2019.1579578 · Cited by 495
Kessler, R. C., Warner, C. H., Ivany, C., Petukhova, M. V., Rose, S., Bromet, E. J., Brown, M., Cai, T., Colpe, L. J., Cox, K. L., et al. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.1754 · Cited by 447 · cited 2×
“…characteristics (e.g., male, late age of enlistment), criminal offenses (e.g., weapons possession, verbal violence), prior suicidality, aspects of prior psychiatric inpatient and outpatient treatment, and specific disorders diagnosed during the hospitalizations (Kessler et al., 2015).”
BackgroundSelf-citationA study of the effect of different types of noise on the precision of supervised learning techniques
Nettleton, D. F., Orriols-Puig, A., & Fornells, A. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-010-9156-z · Cited by 446
Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies
Schuler, M. S., & Rose, S. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kww165 · Cited by 399
Chekroud, A. M., Gueorguieva, R., Krumholz, H. M., Trivedi, M. H., Krystal, J. H., & McCarthy, G. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0025 · Cited by 276
Multiple-imputation for measurement-error correction
Cole, S. R., Chu, H., & Greenland, S. (2006) · International Journal of Epidemiology · doi:10.1093/ije/dyl097 · Cited by 256
Kessler, R. C., Rose, S., Koenen, K. C., Karam, E. G., Stang, P. E., Stein, D. J., Heeringa, S. G., Hill, E. D., Liberzon, I., McLaughlin, K. A., et al. (2014) · World Psychiatry · doi:10.1002/wps.20150 · Cited by 255
Self-citationTackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251 · cited 1×
“This challenge has considerable overlap with current issues surrounding replication and reproducibility in psychological science (Tackett et al., 2017).”
BackgroundTargeted Maximum Likelihood Based Causal Inference: Part II
van der Laan, M. J. (2010) · The International Journal of Biostatistics · doi:10.2202/1557-4679.1241 · Cited by 239
Quantitative forecasting of PTSD from early trauma responses: A Machine Learning application
Galatzer-Levy, I. R., Karstoft, K.-I., Statnikov, A., & Shalev, A. Y. (2014) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2014.08.017 · Cited by 195
Predicting the Risk of Suicide by Analyzing the Text of Clinical Notes
Poulin, C., Shiner, B., Thompson, P., Vepstas, L., Young-Xu, Y., Goertzel, B., Watts, B., Flashman, L., & McAllister, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0085733 · Cited by 168
Kessler, R. C., Stein, M. B., Petukhova, M. V., Bliese, P., Bossarte, R. M., Bromet, E. J., Fullerton, C. S., Gilman, S. E., Ivany, C., Lewandowski-Romps, L., et al. (2017) · Molecular Psychiatry · doi:10.1038/mp.2016.110 · Cited by 163
Self-citationMånsson, K. N. T., Frick, A., Boraxbekk, C.-J., Marquand, A. F., Williams, S. C. R., Carlbring, P., Andersson, G., & Furmark, T. (2015) · Translational Psychiatry · doi:10.1038/tp.2015.22 · Cited by 154
Gradus, J. L., Rosellini, A. J., Horváth–Puhó, E., Street, A. E., Galatzer‐Levy, I. R., Jiang, T., Lash, T. L., & Sørensen, H. T. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.2905 · Cited by 149
Self-citationLuijken, K., Groenwold, R. H. H., CALSTER, B. V., Steyerberg, E. W., & van Smeden, M. (2019) · Statistics in Medicine · doi:10.1002/sim.8183 · Cited by 142
Wu, M., Mwangi, B., Bauer, I. E., Passos, I. C., Sanches, M., Zunta‐Soares, G. B., Meyer, T. D., Hasan, K. M., & Soares, J. C. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.02.016 · Cited by 119
Webb, C. A., Cohen, Z. D., Beard, C., Forgeard, M. J. C., Peckham, A., & Björgvinsson, Þ. (2019) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000451 · Cited by 103
Edwards, J. K., Cole, S. R., Troester, M. A., & Richardson, D. B. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kws340 · Cited by 85
Machine learning methods for developing precision treatment rules with observational data
Kessler, R. C., Bossarte, R. M., Luedtke, A. R., Zaslavsky, A. M., & Zubizarreta, J. R. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103412 · Cited by 85
Predicting posttraumatic stress disorder following a natural disaster
Rosellini, A. J., Francisca Dussaillant, L., Zubizarreta, J. R., Kessler, R. C., & Rose, S. (2017) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2017.09.010 · Cited by 84
Self-citationDiscriminant Analysis When the Initial Samples Are Misclassified
Lachenbruch, P. A. (1966) · Technometrics · doi:10.2307/1266637 · Cited by 80
Prediction of remission in obsessive compulsive disorder using a novel machine learning strategy
Askland, K. D., Garnaat, S., Sibrava, N. J., Boisseau, C. L., Strong, D., Mancebo, M., Greenberg, B., Rasmussen, S., & Eisen, J. (2015) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1463 · Cited by 75
Is the area under an ROC curve a valid measure of the performance of a screening or diagnostic test?
Wald, N., & Bestwick, J. (2014) · Journal of Medical Screening · doi:10.1177/0969141313517497 · Cited by 67
Empirically derived patterns of psychiatric symptoms in youth: A latent profile analysis
Kircanski, K., Zhang, S., Stringaris, A. K., Wiggins, J. L., Towbin, K. E., Pine, D. S., Leibenluft, E., & Brotman, M. A. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.09.016 · Cited by 59
Galatzer‐Levy, I. R., Ruggles, K. V., & Chen, Z. S. (2018) · Chronic Stress · doi:10.1177/2470547017747553 · Cited by 57
Ilgen, M. A., Downing, K., Zivin, K., Hoggatt, K. J., Kim, H. M., Ganoczy, D. A., Austin, K., McCarthy, J. F., Patel, J. M., & Valenstein, M. M. (2009) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.08m04795 · Cited by 44
Bayesian model selection for logistic regression with misclassified outcomes
Gerlach, R., & Stamey, J. D. (2007) · Statistical Modelling · doi:10.1177/1471082x0700700303 · Cited by 40
Estimating mutual information for feature selection in the presence of label noise
Frénay, B., Doquire, G., & Verleysen, M. (2013) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2013.05.001 · Cited by 32
Developing algorithms to predict adult onset internalizing disorders: An ensemble learning approach
Rosellini, A. J., Liu, S., Anderson, G. N., Sbi, S., Tung, E. S., & Knyazhanskaya, E. E. (2019) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2019.12.006 · Cited by 31
Self-citationRosellini, A. J., & Brown, T. A. (2014) · Psychological Assessment · doi:10.1037/pas0000023 · Cited by 26
Self-citationA Bayesian latent class approach for EHR‐based phenotyping
Hubbard, R. A., Huang, J. Y., Harton, J. G., Oganisian, A., Choi, G. H., Utidjian, L., Eneli, I., Bailey, L. C., & Chen, Y. (2018) · Statistics in Medicine · doi:10.1002/sim.7953 · Cited by 25
Hsu, K. J., McNamara, M. E., Shumake, J., Stewart, R. A., Labrada, J., Alario, A. A., Gonzalez, G. D. S., Schnyer, D. M., & Beevers, C. G. (2020) · Depression and Anxiety · doi:10.1002/da.23042 · Cited by 24
Fast PCA for processing calcium-imaging data from the brain of Drosophila melanogaster
Strauch, M., & Galizia, C. G. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-s1-s2 · Cited by 17
Hasanpour, H., Meibodi, R. G., Navi, K., & Asadi, S. (2018) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s173388 · Cited by 15
Robustness of Threshold-Based Feature Rankers with Data Sampling on Noisy and Imbalanced Data
Shanab, A. A., Khoshgoftaar, T. M., & Wald, R. (2012) · The Florida AI Research Society · Cited by 13
Galatzer‐Levy, I. R. (2015) · European Journal of Psychotraumatology · doi:10.3402/ejpt.v6.27515 · Cited by 12
Cited by
Using machine learning approaches for multi-omics data analysis: A review
Reel, P. S., Reel, S., Pearson, E., Trucco, E., & Jefferson, E. (2021) · Biotechnology Advances · doi:10.1016/j.biotechadv.2021.107739 · Cited by 903
A Review of Machine Learning and Deep Learning Approaches on Mental Health Diagnosis
Iyortsuun, N. K., Kim, S.-H., Jhon, M., Yang, H.-J., & Pant, S. (2023) · Healthcare · doi:10.3390/healthcare11030285 · Cited by 276
Using machine learning for healthcare challenges and opportunities
Alanazi, A. (2022) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2022.100924 · Cited by 256
Artificial intelligence in pancreatic cancer
Huang, B., Huang, H., Zhang, S., Zhang, D., Shi, Q., Liu, J., & Guo, J. (2022) · Theranostics · doi:10.7150/thno.77949 · Cited by 142
“By fitting a model that relates responses to predictors, supervised learning can accurately predict future observed responses or better understand the relationship between responses and predictors [49,50].”
BackgroundTheodorakis, N., Feretzakis, G., Tzelves, L., Paxinou, E., Hitas, C., Vamvakou, G., Verykios, V. S., & Nikolaou, M. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14090931 · Cited by 51
Metadata
- Authors
- Jiang, T., Gradus, J. L., & Rosellini, A. J.
- Venue
- Behavior Therapy
- Publisher
- Elsevier BV
- Published
- 2020-09
- Volume
- 51
- Issue
- 5
- Pages
- 675-687
- ISSN
- 0005-7894 (ISSN-L)
- OpenAlex
- W3025161810
- Cited by
- 847
- References
- 75
- FWCI
- 48.17
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.