Recommendations and future directions for supervised machine learning in psychiatry
Open accessThe DOI matches its Crossref record; 61 of the 62 references it cites were found in a registry or an index.
Abstract
Machine learning methods hold promise for personalized care in psychiatry, demonstrating the potential to tailor treatment decisions and stratify patients into clinically meaningful taxonomies. Subsequently, publication counts applying machine learning methods have risen, with different data modalities, mathematically distinct models, and samples of varying size being used to train and test models with the promise of clinical translation. Consequently, and in part due to the preliminary nature of such works, many studies have reported largely varying degrees of accuracy, raising concerns over systematic overestimation and methodological inconsistencies. Furthermore, a lack of procedural evaluation guidelines for non-expert medical professionals and funding bodies leaves many in the field with no means to systematically evaluate the claims, maturity, and clinical readiness of a project. Given the potential of machine learning methods to transform patient care, albeit, contingent on the rigor of employed methods and their dissemination, we deem it necessary to provide a review of current methods, recommendations, and future directions for applied machine learning in psychiatry. In this review we will cover issues of best practice for model training and evaluation, sources of systematic error and overestimation, model explainability vs. trust, the clinical implementation of AI systems, and finally, future directions for our field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“for which it is explicitly recommended that selection should work independently of model training (unlike in methods as forward and backward selection)”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [27]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One commonly referred strategy is feature selection13,15,27,47,76, for which it is explicitly recommended that selection should work independently of model training (unlike in methods as forward and backward selection) and is best informed - a priori - by medical expert knowledge or existing literature13,17,76.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is therefore considered important to compose a diverse development team17,22–25, create awareness and involve stakeholders in design choices22,24,26,27,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“although this may not eliminate the bias altogether.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Code for the complete model building pipeline should be published in well-documented scripts with computer environment requirements when possible7,8,13,14,16,19,20,24,25,27,33,50,62,64,65, including statements about any restrictions to access or re-use.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [27]
Experimental — accuracy not yet measured
11 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., Hauenstein, S., Lahoz‐Monfort, J. J., Schröder, B., Thuiller, W., et al. (2017) · Ecography · doi:10.1111/ecog.02881 · Cited by 3,689 · cited 1×
“Here, the solution is leave-group out cross-validation, also known as Monte Carlo or leave site out cross-validation 34,35 .”
MethodGenetic relationship between five psychiatric disorders estimated from genome-wide SNPs
Cross-Disorder Group of the Psychiatric Genomics Consortium (2013) · Nature Genetics · doi:10.1038/ng.2711 · Cited by 2,263
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning
Yarkoni, T., & Westfall, J. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617693393 · Cited by 1,916
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Statistics versus machine learning
Bzdok, D., Altman, N., & Krzywinski, M. (2018) · Nature Methods · doi:10.1038/nmeth.4642 · Cited by 1,508
(2015) · neural information processing systems · Cited by 1,305
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047
Xu, Q.-S., & Liang, Y.-Z. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(00)00122-2 · Cited by 1,043
No Free Lunch Theorems for SearchPreprint
Wolpert, D. H., & Macready, W. G. (1995) · RePEc: Research Papers in Economics · Cited by 1,036
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · cited 1×
“Given these circumstances, concerns have been raised that ML may face the same reproducibility crisis as that experienced by group level analyses in recent years 22 – 24 .”
BackgroundSingle subject prediction of brain disorders in neuroimaging: Promises and pitfalls
Arbabshirani, M. R., Plis, S., Sui, J., & Calhoun, V. D. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.02.079 · Cited by 912
Auto-Keras: An Efficient Neural Architecture Search System
Jin, H., Song, Q., & Hu, X. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330648 · Cited by 891
Machine Learning for Precision Psychiatry: Opportunities and Challenges
Bzdok, D., & Meyer-Lindenberg, A. (2018) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2017.11.007 · Cited by 796
Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.038 · Cited by 792
Leakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791
Feature selection for classification: A review
Tang, J., Alelyani, S., & Liu, H. (2014) · doi:10.1201/b17320-7 · Cited by 757
iml: An R package for Interpretable Machine Learning
Molnar, C. (2018) · Journal of Open Source Software · doi:10.21105/joss.00786 · Cited by 718
Understanding heterogeneity in clinical cohorts using normative models: Beyond case-control studies
Marquand, A. F., Rezek, I., Buitelaar, J., & Beckmann, C. F. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.12.023 · Cited by 683 · cited 1×
“Moving forward, only by parsing this heterogeneity in psychiatric phenotypes will we be able to lay the groundwork for more targeted models and improved patient care 65,66 .”
BackgroundCross-trial prediction of treatment outcome in depression: a machine learning approach
Chekroud, A. M., Zotti, R. J., Shehzad, Z., Gueorguieva, R., Johnson, M. K., Trivedi, M. H., Cannon, T. D., Krystal, J. H., & Corlett, P. R. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00471-x · Cited by 676 · cited 4×
“However, studies using machine learning methodologies (ML) have begun to identify subtypes of psychiatric disorders with differing symptomology 13 , illness trajectories 14,15 , and drug response pro fi les 16 .”
BackgroundMethodInfluentialDonahue, J., Krähenbühl, P., & Darrell, T. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09782 · Cited by 616
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 564
Prediction as a Humanitarian and Pragmatic Contribution from Human Cognitive Neuroscience
Gabrieli, J. D. E., Ghosh, S. S., & Whitfield-Gabrieli, S. (2015) · Neuron · doi:10.1016/j.neuron.2014.10.047 · Cited by 535
The Clinical Application of the Biopsychosocial Model
Engel, G. L. (1981) · Journal of Medicine and Philosophy · doi:10.1093/jmp/6.2.101 · Cited by 531
Listening to Prozac but hearing placebo: A meta-analysis of antidepressant medication.
Kirsch, I., & Sapirstein, G. (1998) · Prevention & Treatment · doi:10.1037/1522-3736.1.1.12a · Cited by 502
Truth or consequences: the validity of self‐report data in health services research on addictions
Del Boca, F. K., & Noll, J. A. (2000) · Addiction · doi:10.1046/j.1360-0443.95.11s3.5.x · Cited by 467
Nguyen, T.-M.-U., Caze, A. L., & Cottrell, N. (2014) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.12194 · Cited by 417
Biomarkers for depression: recent insights, current challenges and future prospects
Strawbridge, R., Young, A. H., & Cleare, A. J. (2017) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s114542 · Cited by 415 · cited 1×
“…degree of comorbidity between psychiatric conditions 4,5 , the lack of diagnostic biomarkers to delineate between disorders and illness trajectories 6,7 , the shared genetic origins of clinically disparate traits 8 , and the imprecision of symptom measures 9,10 has likely contributed to the…”
BackgroundKoutsouleris, N., Kambeitz-Ilankovic, L., Ruhrmann, S., Rosen, M., Ruef, A., Dwyer, D. B., Paolini, M., Chisholm, K., Kambeitz, J., Haidl, T., et al. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2165 · Cited by 341
Permutation Tests for Studying Classifier Performance
Ojala, M., & Garriga, G. C. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859913 · Cited by 322
Chekroud, A. M., Gueorguieva, R., Krumholz, H. M., Trivedi, M. H., Krystal, J. H., & McCarthy, G. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0025 · Cited by 276
Redlich, R., Opel, N., Grotegerd, D., Dohm, K., Zaremba, D., Bürger, C., Münker, S., Mühlmann, L., Wahl, P., Heindel, W., et al. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2016.0316 · Cited by 267
Marquand, A. F., Wolfers, T., Mennes, M., Buitelaar, J., & Beckmann, C. F. (2016) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2016.04.002 · Cited by 263
Individual differences and their measurement: A review of 100 years of research.
Sackett, P. R., Lievens, F., Van Iddekinge, C. H., & Kuncel, N. R. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000151 · Cited by 235
Detecting Neuroimaging Biomarkers for Psychiatric Disorders: Sample Size Matters
Schnack, H. G., & Kahn, R. S. (2016) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2016.00050 · Cited by 229
Koutsouleris, N., Kahn, R. S., Chekroud, A. M., Leucht, S., Falkai, P., Wobrock, T., Derks, E. M., Fleischhacker, W. W., & Hasan, A. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(16)30171-7 · Cited by 217
On estimating model accuracy with repeated cross-validation
Vanwinckelen, G., & Blockeel, H. (2012) · Lirias · Cited by 156
Integrating Neurobiological Markers of Depression
Hahn, T., Marquand, A. F., Ehlis, A.-C., Dresler, T., Kittel-Schneider, S., Jarczok, T. A., Lesch, K.-P., Jakob, P. M., Mourao-Miranda, J., Brammer, M. J., et al. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.178 · Cited by 142
Self-citationIenco, D., Pensa, R. G., & Meo, R. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2133360.2133361 · Cited by 135
Hahn, T., Kircher, T., Straube, B., Wittchen, H.-U., Konrad, C., Ströhle, A., Wittmann, A., Pfleiderer, B., Reif, A., Arolt, V., et al. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.1741 · Cited by 129
Self-citationComputational neuroscience approach to biomarkers and treatments for mental disorders
Yahata, N., Kasai, K., & Kawato, M. (2017) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.12502 · Cited by 111
Multimodal manifold-regularized transfer learning for MCI conversion prediction
Initiative, A. D. N., Cheng, B., Liu, M., Suk, H.-I., Shen, D., & Zhang, D. (2015) · Brain Imaging and Behavior · doi:10.1007/s11682-015-9356-x · Cited by 102
SCoRS—A Method Based on Stability for Feature Selection and Mapping in Neuroimaging
Rondina, J. M., Hahn, T., de Oliveira, L., Marquand, A. F., Dresler, T., Leitner, T., Fallgatter, A. J., Shawe-Taylor, J., & Mourao-Miranda, J. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2013.2281398 · Cited by 64
Self-citationAnker, E., Bendiksen, B., & Heir, T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019700 · Cited by 61
Efficient Neural Architecture Search with Network MorphismPreprint
Jin, H., Song, Q., & Hu, X. (2018) · arXiv (Cornell University) · Cited by 61
AL-Asadi, A. M., Klein, B., & Meyer, D. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4143 · Cited by 50
Hahn, T., Marquand, A. F., Plichta, M. M., Ehlis, A., Schecklmann, M. W., Dresler, T., Jarczok, T. A., Eirich, E., Leonhard, C., Reif, A., et al. (2013) · Human Brain Mapping · doi:10.1002/hbm.21497 · Cited by 27
Self-citationComputational Psychiatry: Embracing Uncertainty and Focusing on Individuals, Not Averages
Chekroud, A. M., Lane, C. E., & Ross, D. A. (2017) · Biological Psychiatry · doi:10.1016/j.biopsych.2017.07.011 · Cited by 19
Neuhaus, A. H., & Popescu, F. C. (2018) · Biological Psychiatry · doi:10.1016/j.biopsych.2017.09.032 · Cited by 12
Hahn, T., Ebner‐Priemer, U., & Meyer‐Lindenberg, A. (2018) · SSRN Electronic Journal · doi:10.31219/osf.io/uzehj · Cited by 10
Self-citationSpeelman, C. P., & McGann, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01234 · Cited by 10
Rondina, J. M., Hahn, T., Oliveira, L., et al. (2014) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2307811 · Cited by 5
Self-citationKambeitz, J. P., Cabral, C., Sacchet, M. D., Gotlib, I. H., Zahn, R., Serpa, M. H., Walter, M., Falkai, P., & Koutsouleris, N. (2018) · Biological Psychiatry · doi:10.1016/j.biopsych.2018.01.023 · Cited by 3
Research works printed but not linked: 2
- Goodfellow, I, Bengio, Y, Courville, A. Deep Learning (MIT press, 2016).
- Inoue, H. Data augmentation by pairing samples for images classification. Preprint at arXiv:180102929 (2018).
Cited by
de Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499
Deep learning for small and big data in psychiatry
Koppe, G., Meyer-Lindenberg, A., & Durstewitz, D. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0767-z · Cited by 189
“None-theless, leaking data used for model selection into the model assessment step is perhaps the most common mistake made in the literature (see also [77]).”
BackgroundMethodBickman, L. (2020) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-020-01065-8 · Cited by 184
“More suggestions for improving the quality of research on supervised machine learning can be found in the paper by Cearns et al. (2019).”
BackgroundA Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179
Translating promise into practice: a review of machine learning in suicide research and prevention
Kirtley, O. J., van Mens, K., Hoogendoorn, M., Kapur, N., & de Beurs, D. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00254-6 · Cited by 100
Metadata
- Authors
- Cearns, M., Hahn, T., & Baune, B. T.
- Affiliations
- The University of Adelaide (AU)
- University of Münster (DE)
- The University of Melbourne (AU)
- Corresponding author
- Baune, Bernhard T.
- Venue
- Translational Psychiatry
- Published
- 2019-10-22
- Volume
- 9
- Issue
- 1
- Pages
- 271-271
- ISSN
- 2158-3188 (ISSN-L)
- PMC
- PMC6805872
- OpenAlex
- W2982007782
- Cited by
- 127
- References
- 62
- Citations by year
- 2026: 12 · 2025: 19 · 2024: 19 · 2023: 17 · 2022: 27 · 2021: 22 · 2020: 11
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Translational Psychiatry (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Australian Government
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- supervised learning, personalized treatment, clinical translation, model evaluation, methodological quality, model explainability, clinical implementation of AI, clinical preparedness, best practices
- MeSH terms
- Supervised Machine Learning, Mental Disorders, Humans, Psychiatry, Practice Guidelines as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.