The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
When Optimism Hurts: Inflated Predictions in Psychiatric Neuroimaging
Abstract
The ability to predict outcomes from neuroimaging data has the potential to answer important clinical questions such as which depressed patients will respond to treatment, which abstinent drug users will relapse, or which patients will convert to dementia. However, many prediction analyses require methods and techniques, not typically required in neuroimaging, to accurately assess a model's predictive ability. Regression models will tend to fit to the idiosyncratic characteristics of a particular sample and consequently will perform worse on unseen data. Failure to account for this inherent optimism is especially pernicious when the number of possible predictors is high relative to the number of participants, a common scenario in psychiatric neuroimaging. We show via simulated data that models can appear predictive even when data and outcomes are random, and we note examples of optimistic prediction in the literature. We provide some recommendations for assessment of model performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Whelan and Garavan64 previously demonstrated inflated performance via non-nested feature selection using random data, though we think it is also useful to demonstrate this with neuroimaging data.”
From Data leakage inflates prediction performance in connectome-based machine learning models (2024) · cites this paper as [64]
Citations
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Ridge regression: Biased estimation for nonorthogonal problems
Hoerl, A. E., & Kennard, R. W. (1970) · Technometrics · doi:10.1080/00401706.1970.10488634 · Cited by 9,648
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. W. (2000) · Technometrics · doi:10.2307/1271436 · Cited by 7,001
Relaxing the Rule of Ten Events per Variable in Logistic and Cox Regression
Vittinghoff, E., & McCulloch, C. E. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwk052 · Cited by 4,359
Calculation of signal detection theory measures
Stanislaw, H., & Todorov, N. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03207704 · Cited by 2,673
Research works printed but not linked: 1
- Efron, 1993, Vol. 57
Cited by
Sorted by citations
Building better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
Using connectome-based predictive modeling to predict individual behavior from brain connectivity
Shen, X.-L., Finn, E., Scheinost, D., Rosenberg, M., Chun, M. M., Papademetris, X., & Constable, R. T. (2017) · Nature Protocols · doi:10.1038/nprot.2016.178 · Cited by 1,088
Machine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
Metadata
- Authors
- Whelan, R., & Garavan, H.
- Affiliations
- University of Vermont (US)
- Corresponding author
- Garavan, Hugh
- Venue
- Biological Psychiatry
- Publisher
- Elsevier BV
- Published
- 2014-05
- Volume
- 75
- Issue
- 9
- Pages
- 746-748
- ISSN
- 0006-3223 (ISSN-L)
- OpenAlex
- W1979450493
- Cited by
- 209
- References
- 40
- Citations by year
- 2026: 13 · 2025: 12 · 2024: 11 · 2023: 17 · 2022: 24 · 2021: 22 · 2020: 17 · 2019: 22 · 2018: 17 · 2017: 15 · 2016: 20 · 2015: 12 · 2014: 6
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Biological Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- optimism bias, overfitting, model evaluation, psychiatric neuroimaging, outcome prediction, high-dimensional predictors, regression models, depression treatment response, recurrence prediction, conversion to dementia
- MeSH terms
- Brain, Computer Simulation, Humans, Regression Analysis, ROC Curve, Signal Processing, Computer-Assisted, Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
