I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data
Open accessAbstract
Machine learning has enhanced the abilities of neuroscientists to interpret information collected through EEG, fMRI, and MEG data. With these powerful techniques comes the danger of overfitting of hyperparameters which can render results invalid. We refer to this problem as 'overhyping' and show that it is pernicious despite commonly used precautions. Overhyping occurs when analysis decisions are made after observing analysis outcomes and can produce results that are partially or even completely spurious. It is commonly assumed that cross-validation is an effective protection against overfitting or overhyping, but this is not actually true. In this article, we show that spurious results can be obtained on random data by modifying hyperparameters in seemingly innocuous ways, despite the use of cross-validation. We recommend a number of techniques for limiting overhyping, such as lock boxes, blind analyses, pre-registrations, and nested cross-validation. These techniques, are common in other fields that use machine learning, including computer science and physics. Adopting similar safeguards is critical for ensuring the robustness of machine-learning techniques in the neurosciences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Potential wrongful inflation of accuracy estimates (e.g. related to small sample sizes, absence of testing model performance in an external sample, and problematic validation methods) precludes reliable immediate implementation in clinical practice.
“potential wrongful inflation of accuracy estimates (e.g. related to small sample sizes, absence of testing model performance in an external sample, and problematic validation methods), precluding reliable immediate implementation in clinical practice (Hosseini et al., 2020; Jacobucci, Littlefield, Millner, Kleiman, & Steinley, 2020; Poldrack, Huckins, & Varoquaux, 2020).”
From Irremediability in psychiatric euthanasia: examining the objective standard (2023) · cites this paper as Hosseini et al., 2020
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredEvaluating models in the test dataset to select the best model hyperparameters is a form of leakage.
“While evaluating models in the test dataset to select the best model hyperparameters is a form of leakage, it has been previously studied in neuroimaging49 and therefore was not included in this study.”
From Data leakage inflates prediction performance in connectome-based machine learning models (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 40 · Cited by 165
Research works printed but not linked: 4
- Eklund, A., Nichols, T., Andersson, M., & Knutsson, H. (2015, April). Empirically investigating the statistical validity of SPM, FSL and AFNI for single subject fMRI analysis. In Biomedical Imaging (ISBI), 2015 IEEE 12th International Symposium on (pp. 1376-1380). IEEE.
- Flint, C., Cearns, M., Opel, N., Redlich, R., Mehler, D., Emden, D., ... & Krug, A. (2019). Systematic Overestimation of Machine Learning Performance in Neuroimaging Studies of Depression. arXiv preprint arXiv:1912.06686. · arXiv
- King, Jean-Rémi, Alexandre Gramfort, Aaron Schurger, Lionel Naccache, and Stanislas Dehaene. "Two distinct dynamic modes subtend the detection of unexpected sounds." PloS one 9, no. 1 (2014): e85791.
- Pernet, C. R., Latinus, M., Nichols, T. E., & Rousselet, G. A. (2015). Cluster-based computational methods for mass univariate analyses of event-related brain potentials/fields: A simulation study. Journal of neuroscience methods, 250, 85-93.
- Authors
- Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B.
- Affiliations
- University of Kent (GB)
- Pennsylvania State University (US)
- Corresponding author
- Wyble, Brad
- Venue
- Neuroscience & Biobehavioral Reviews
- Publisher
- Elsevier BV
- Published
- 2020-12
- Volume
- 119
- Pages
- 456-467
- ISSN
- 0149-7634 (ISSN-L)
- OpenAlex
- W3092041776
- Cited by
- 165
- References
- 40
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 31 · 2023: 31 · 2022: 31 · 2021: 19 · 2020: 2 · 2019: 2 · 2018: 3 · 2017: 5
- FWCI
- 0.72
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Neuroscience & Biobehavioral Reviews (publishedVersion)
- PubMed (publishedVersion)
- Kent Academic Repository (University of Kent) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Directorate for Social, Behavioral and Economic Sciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Keywords
- overfitting, hyperparameter optimization, cross-validation, spurious results, brain classification, EEG, functional MRI, MEG, nested cross-validation, preregistration, blind analysis, reproducibility
- MeSH terms
- Machine Learning, Brain, Humans, Magnetic Resonance Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
