Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index; 29 of the 36 sources its claims cite were checked for support.

Data leakage inflates prediction performance in connectome-based machine learning models

Open access

Published version of a preprint

Abstract

Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Overview

CoChat Summary

TL;DR

This paper evaluates five forms of data leakage in connectome-based machine learning across four datasets and three phenotypes, showing that feature selection and repeated-subject leakage inflate prediction performance while other forms have minor or deflationary effects. Leakage effects are larger in small samples and can also alter model interpretation.

Key findings

  • Feature leakage inflated prediction performance in HCPD: Δr = 0.03 for age, Δr = 0.17 for matrix reasoning, and Δr = 0.47 for attention problems; attention problems went from chance-level (r = 0.01, q2 = −0.13) to moderate (r = 0.48, q2 = 0.22).
  • Subject leakage with 20% repeated subjects inflated performance in HCPD: Δr = 0.28 for attention problems, Δr = 0.04 for age, and Δr = 0.14 for matrix reasoning; across all datasets and phenotypes, subject leakage (20%) gave Δr = 0.06–0.29 and Δq2 = 0.03–0.24.
  • Leaky covariate regression was the only form that consistently deflated performance: in HCPD, Δr = −0.06 for attention problems, −0.02 for age, and −0.09 for matrix reasoning; across datasets, Δr = −0.09–0.00 and Δq2 = −0.17–0.00. Leaky site correction and family leakage had little to no effect.
  • Small samples increased leakage variability: for ABCD matrix reasoning, family leakage Δr ranged from −0.34 to 0.25 at N = 100 versus −0.12 to 0.13 at N = 400; taking medians over multiple k-fold iterations reduced but did not eliminate leakage effects.

Methods

  • Four datasets were used: ABCD (N = 7822–7969), HBN (N = 1024–1201), HCPD (N = 424–605), and PNC (N = 1119–1126), predicting age, attention problems, and matrix reasoning from functional connectomes; structural connectomes were also analyzed in 635 HCPD participants.
  • The main model was ridge regression with 5-fold cross-validation, 5% feature selection, and a grid search for the L2 regularization parameter; the gold standard included cross-validated covariate regression, site correction with ComBat, and family-aware splits where applicable. Over 400 pipelines tested feature leakage, covariate-related leakage, and subject-level leakage, with sensitivity analyses using SVR and CPM.
  • Performance was evaluated with Pearson’s r and cross-validation R2 (q2) over 100 random seeds; sample-size analyses resampled N = 100, 200, 300, and 400 with 10 resamples and 10 iterations of 5-fold cross-validation, and model coefficients were compared between leaky and non-leaky pipelines.

Limitations

  • The paper states it cannot cover all leakage forms, including hyperparameter selection leakage, temporal leakage, unsupervised dimensionality reduction, phenotype standardization, and illegitimate features; it also studied child, adolescent, and young adult cohorts in well-harmonized datasets, so other populations or dataset qualities could alter leakage effects.
  • The paper notes leakage is not always well-defined and can be application-dependent, and that effects were tested in only three phenotypes, which do not comprehensively cover all effect sizes; complex models such as neural networks may be more susceptible to leakage.
  • Evident from the text: many leakage types had little to no effect in these large, harmonized datasets, so results may not generalize to other datasets or pipelines; family leakage analyses were limited by the small proportion of multi-member families in the full datasets, and subject leakage was operationalized as exact scan duplication rather than longitudinal or repeated-measurement designs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-024-46150-w

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Summary of the prediction pipelines used in this study. The various forms of leakage that may occur are shown in orange. Feature leakage, leaky site correction, leaky covariate regression, and subject leakage may occur prior to splitting the data into training and test sets. Family leakage may occur during the splitting of data.
Fig. 1 Summary of the prediction pipelines used in this study. The various forms of leakage that may occur are shown in orange. Feature leakage, leaky site correction, leaky covariate regression, and subject leakage may occur prior to splitting the data into training and test sets. Family leakage may occur during the splitting of data.CC BY · PubMed Central
Fig. 2 Prediction performance in HCPD of the non-leaky pipelines, including the gold standard, omitting covariate regression, omitting site correction, and omitting both covariate regression and site correction. Rows represent different non-leaky analysis choices, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the exact value of the bar is shown as the median r, rmed. The histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.
Fig. 2 Prediction performance in HCPD of the non-leaky pipelines, including the gold standard, omitting covariate regression, omitting site correction, and omitting both covariate regression and site correction. Rows represent different non-leaky analysis choices, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the exact value of the bar is shown as the median r, rmed. The histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.CC BY · PubMed Central
Fig. 3 Prediction performance in HCPD of leaky feature selection, compared to the gold standard. Rows represent different leakage types, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.
Fig. 3 Prediction performance in HCPD of leaky feature selection, compared to the gold standard. Rows represent different leakage types, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.CC BY · PubMed Central
Fig. 4 Prediction performance in HCPD of covariate-related forms of leakage, including leaky site correction, and leaky covariate regression. Rows represent different leakage types, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.
Fig. 4 Prediction performance in HCPD of covariate-related forms of leakage, including leaky site correction, and leaky covariate regression. Rows represent different leakage types, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.CC BY · PubMed Central
Fig. 5 Prediction performance in HCPD of subject-level forms of leakage. These included family leakage and subject leakage at three different levels. Rows represent different leakage types, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.
Fig. 5 Prediction performance in HCPD of subject-level forms of leakage. These included family leakage and subject leakage at three different levels. Rows represent different leakage types, and columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 1. HCPD Human Connectome Project Development.CC BY · PubMed Central
Fig. 6 Prediction performance in ABCD comparing the gold standard to twin/family leakage. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 20-fold cross-validation. See also Supplementary Figs. 2–4. ABCD Adolescent Brain Cognitive Development.
Fig. 6 Prediction performance in ABCD comparing the gold standard to twin/family leakage. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 20-fold cross-validation. See also Supplementary Figs. 2–4. ABCD Adolescent Brain Cognitive Development.CC BY · PubMed Central
Fig. 7 Evaluation of performance differences between all pipelines and the gold standard pipeline across all datasets and phenotypes for Pearson’s r and q2. The plots are ranked from the most inflated performance (top) to the most deflated performance (bottom) by two different performance metrics, a) Pearson’s r and b) q2. Boxplot elements were defined as follows: the center line is the median across all datasets, phenotypes, and iterations (100 per dataset/phenotype combination); box limits are the upper and lower quartiles; whiskers are 1.5x the interquartile range; points are outliers. See also Supplementary Figs. 5 and 6.
Fig. 7 Evaluation of performance differences between all pipelines and the gold standard pipeline across all datasets and phenotypes for Pearson’s r and q2. The plots are ranked from the most inflated performance (top) to the most deflated performance (bottom) by two different performance metrics, a) Pearson’s r and b) q2. Boxplot elements were defined as follows: the center line is the median across all datasets, phenotypes, and iterations (100 per dataset/phenotype combination); box limits are the upper and lower quartiles; whiskers are 1.5x the interquartile range; points are outliers. See also Supplementary Figs. 5 and 6.CC BY · PubMed Central
Fig. 8 Similarity of coefficients between the gold standard and various forms of leakage. The boxes are colored by the leakage family: orange (non-leaky analysis choices), blue (feature leakage), green (covariate-related leakage), yellow (subject-level leakage). Boxplot elements were defined as follows: the center line is the median across 100 random iterations; box limits are the upper and lower quartiles; whiskers are 1.5x the interquartile range; points are outliers. Certain values, such as leaky site correction in PNC, are omitted because the relevant fields (e.g., site) do not exist. See also Supplementary Figs. 7 and 8. ABCD Adolescent Brain Cognitive Development, HBN Healthy Brain Network, HCPD Human Connectome Project Development, PNC Philadelphia Neurodevelopmental Cohort.
Fig. 8 Similarity of coefficients between the gold standard and various forms of leakage. The boxes are colored by the leakage family: orange (non-leaky analysis choices), blue (feature leakage), green (covariate-related leakage), yellow (subject-level leakage). Boxplot elements were defined as follows: the center line is the median across 100 random iterations; box limits are the upper and lower quartiles; whiskers are 1.5x the interquartile range; points are outliers. Certain values, such as leaky site correction in PNC, are omitted because the relevant fields (e.g., site) do not exist. See also Supplementary Figs. 7 and 8. ABCD Adolescent Brain Cognitive Development, HBN Healthy Brain Network, HCPD Human Connectome Project Development, PNC Philadelphia Neurodevelopmental Cohort.CC BY · PubMed Central
Fig. 9 Difference between leaky and gold standard performance for various types of leakage and four sample sizes (N = 100, 200, 300, 400). Rows represent the dataset, and columns show the phenotype. For each leakage type (x-axis), there are four results (N = 100, 200, 300, 400). For each sample size, we repeated 10 random seeds of resampling for 10 iterations of 5-fold cross-validation. Boxplot elements were defined as follows: the center line is the median across all subsampling seeds and cross-validation iterations; box limits are the upper and lower quartiles; whiskers are 1.5x the interquartile range; points are outliers. See also Supplementary Fig. 9. ABCD Adolescent Brain Cognitive Development, HBN Healthy Brain Network, HCPD Human Connectome Project Development, PNC Philadelphia Neurodevelopmental Cohort.
Fig. 9 Difference between leaky and gold standard performance for various types of leakage and four sample sizes (N = 100, 200, 300, 400). Rows represent the dataset, and columns show the phenotype. For each leakage type (x-axis), there are four results (N = 100, 200, 300, 400). For each sample size, we repeated 10 random seeds of resampling for 10 iterations of 5-fold cross-validation. Boxplot elements were defined as follows: the center line is the median across all subsampling seeds and cross-validation iterations; box limits are the upper and lower quartiles; whiskers are 1.5x the interquartile range; points are outliers. See also Supplementary Fig. 9. ABCD Adolescent Brain Cognitive Development, HBN Healthy Brain Network, HCPD Human Connectome Project Development, PNC Philadelphia Neurodevelopmental Cohort.CC BY · PubMed Central
Fig. 10 Evaluation of leakage types for matrix reasoning, attention problems, and age prediction in structural connectomes using r. The rows show different leakage types, and the columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 14. HCPD Human Connectome Project Development, DTI diffusion tensor imaging.
Fig. 10 Evaluation of leakage types for matrix reasoning, attention problems, and age prediction in structural connectomes using r. The rows show different leakage types, and the columns show different phenotypes. The black bar represents the median performance of the gold standard models across random iterations, and the histograms show prediction performance across 100 iterations of 5-fold cross-validation. See also Supplementary Fig. 14. HCPD Human Connectome Project Development, DTI diffusion tensor imaging.CC BY · PubMed Central

Verification

Claims the paper makes

19 Valid · 1 Not valid · 3 Not checked · 99 Own results and interpretations

72 more claims · 1 with their cited sources checked

Create a free account to see all 122 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 32
  • It may state two separate claims as one. 17
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 78 references

Research works printed but not linked: 2

  1. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J. & Song, D. The secret sharer: evaluating and testing unintended memorization in neural networks. in USENIX Security Symposium 267 (2019).
  2. Wechsler, D. WISC-V: Technical And Interpretive Manual (NCS Pearson, Incorporated, 2014).

Cited by

See all 236

Metadata

Authors
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D.
Affiliations
  • Yale University (US)
  • Northeastern University (US)
Corresponding author
Rosenblatt, Matthew
Venue
Nature Communications
Published
2024-02-28
Volume
15
Issue
1
Pages
1829-1829
ISSN
2041-1723 (ISSN-L)
OpenAlex
W4392232009
Cited by
236
References
78
Citations by year
2026: 150 · 2025: 61 · 2024: 24
FWCI
57.17
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • National Science Foundation (DGE2139841)
  • Gruber Science Fellowship
  • U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (R00MH130894, R01MH121095)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced Neuroimaging Techniques and Applications
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
data leakage, functional connectome, structural connectome, brain-behavior relationship, feature selection, small sample size, model generalization, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Data leakage inflates prediction performance in connectome-based machine learning models. https://app.cochat.ai/papers/10.1038/s41467-024-46150-w
The paper
Rosenblatt, M., Tejavibulya, L., Jiang, R., Noble, S., & Scheinost, D. (2024). Data leakage inflates prediction performance in connectome-based machine learning models. Nature Communications. https://doi.org/10.1038/s41467-024-46150-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.