Mind the gap: Performance metric evaluation in brain‐age prediction
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Estimating age based on neuroimaging‐derived data has become a popular approach to developing markers for brain integrity and health. While a variety of machine‐learning algorithms can provide accurate predictions of age based on brain characteristics, there is significant variation in model accuracy reported across studies. We predicted age in two population‐based datasets, and assessed the effects of age range, sample size and age‐bias correction on the model performance metrics Pearson's correlation coefficient ( r ), the coefficient of determination ( R 2 ), Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE). The results showed that these metrics vary considerably depending on cohort age range; r and R 2 values are lower when measured in samples with a narrower age range. RMSE and MAE are also lower in samples with a narrower age range due to smaller errors/brain age delta values when predictions are closer to the mean age of the group. Across subsets with different age ranges, performance metrics improve with increasing sample size. Performance metrics further vary depending on prediction variance as well as mean age difference between training and test sets, and age‐bias corrected metrics indicate high accuracy—also for models showing poor initial performance. In conclusion, performance metrics used for evaluating age prediction models depend on cohort and study‐specific data characteristics, and cannot be directly compared across different studies. Since age‐bias corrected metrics generally indicate high accuracy, even for poorly performing models, inspection of uncorrected model results provides important information about underlying model attributes such as prediction variance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson, B. C., Blacker, D., Buckner, R. L., Dale, A. M., Maguire, R. P., Hyman, B. T., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.021 · Cited by 13,324
Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., et al. (2002) · Neuron · doi:10.1016/s0896-6273(02)00569-x · Cited by 8,610
Multimodal population brain imaging in the UK Biobank prospective epidemiological study
Miller, K. L., Alfaro-Almagro, F., Bangerter, N. K., Thomas, D. L., Yacoub, E., Xu, J., Bartsch, A. J., Jbabdi, S., Sotiropoulos, S. N., Andersson, J. L. R., et al. (2016) · Nature Neuroscience · doi:10.1038/nn.4393 · Cited by 2,020
The Alzheimer's Disease Neuroimaging Initiative
Mueller, S. G., Weiner, M. W., Thal, L. J., Petersen, R. C., Jack, C., Jagust, W., Trojanowski, J. Q., Toga, A. W., & Beckett, L. (2005) · Neuroimaging Clinics of North America · doi:10.1016/j.nic.2005.09.008 · Cited by 1,525
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477 · cited 1×
“…assessment can be used to identify the MRI measures that are most prominently used in the model (Salih et al., 2021; Amoroso et al., 2019; Samek, Montavon, Lapuschkin, Anders, & Müller, 2021; Vercio et al., 2020), and partial dependence plots (Friedman, 2001; Zhao & Hastie, 2021) can…”
Cole, J. H., Poudel, R. P. K., Tsagkrasoulis, D., Caan, M. W. A., Steves, C., Spector, T. D., & Montana, G. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.07.059 · Cited by 872 · cited 2×
“…2021; Cole, 2020; Cole, Franke, & Cherbuin, 2019; de Lange et al., 2019; Dunås, Wåhlin, Nyberg, & Boraxbekk, 2021; Franke et al., 2020; Smith et al., 2020), and the overall evidence supports the use of brain-predicted age as a surrogate marker for brain integrity and health (Cole et al., 2017).”
Self-citationStorsve, A. B., Fjell, A. M., Tamnes, C. K., Westlye, L. T., Overbye, K., Aasland, H. W., & Walhovd, K. B. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.0391-14.2014 · Cited by 542
Self-citationConceptualizing mental disorders as deviations from normative functioning
Marquand, A. F., Kia, S. M., Zabihi, M., Wolfers, T., Buitelaar, J. K., & Beckmann, C. F. (2019) · Molecular Psychiatry · doi:10.1038/s41380-019-0441-1 · Cited by 433 · cited 2×
“From a methodological point of view, prediction models can benefit from advancements such as incorporating uncertainties into the predictions (Hahn et al., 2021; Marquand et al., 2019; Peng, Gong, Beckmann, Vedaldi, & Smith, 2021).”
Confound modelling in UK Biobank brain imaging
Alfaro-Almagro, F., McCarthy, P., Afyouni, S., Andersson, J. L. R., Bastiani, M., Miller, K. L., Nichols, T. E., & Smith, S. M. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117002 · Cited by 249 · cited 2×
“For UKB, the MRI data were residualised with respect to scanning site (Alfaro-Almagro et al., 2021; Solanes et al., 2021) using linear models.”
Advanced BrainAGE in older adults with type 2 diabetes mellitus
Franke, K., Gaser, C., Manor, B., & Novak, V. (2013) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2013.00090 · Cited by 221 · cited 3×
“…et al., 2021; Cole, 2020; Cole et al., 2018, 2020; de Lange et al., 2019, 2020; Franke, Gaser, & Alzheimer's Disease Neuroimaging Initiative, 2012; Franke, Gaser, Manor, & Novak, 2013; Franke, Ristow, & Gaser, 2014; Gaser et al., 2013; Høgestøl et al., 2019; Kaufmann et al., 2019; Pardoe et al.,…”
Self-citationBeck, D., de Lange, A.-M. G., Maximov, I. I., Richard, G., Andreassen, O. A., Nordvik, J. E., & Westlye, L. T. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117441 · Cited by 209
Self-citation
Metadata
- Authors
- de Lange, A. G., Anatürk, M., Rokicki, J., Han, L. K. M., Franke, K., Alnæs, D., Ebmeier, K. P., Draganski, B., Kaufmann, T., Westlye, L. T., Hahn, T., & Cole, J. H.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2022-03-21
- Volume
- 43
- Issue
- 10
- Pages
- 3113-3129
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- OpenAlex
- W4220800641
- Cited by
- 160
- References
- 13
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.