Towards Reproducible Brain-Wide Association Studies
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Magnetic resonance imaging (MRI) continues to drive many important neuroscientific advances. However, progress in uncovering reproducible associations between individual differences in brain structure/function and behavioral phenotypes (e.g., cognition, mental health) may have been undermined by typical neuroimaging sample sizes (median N=25) 1,2 . Leveraging the Adolescent Brain Cognitive Development (ABCD) Study 3 (N=11,878), we estimated the effect sizes and reproducibility of these brain-wide associations studies (BWAS) as a function of sample size. The very largest, replicable brain-wide associations for univariate and multivariate methods were r=0.14 and r=0.34, respectively. In smaller samples, typical for brain-wide association studies (BWAS), irreproducible, inflated effect sizes were ubiquitous, no matter the method (univariate, multivariate). Until sample sizes started to approach consortium-levels, BWAS were underpowered and statistical errors assured. Multiple factors contribute to replication failures 4–6 ; here, we show that the pairing of small brain-behavioral phenotype effect sizes with sampling variability is a key element in wide-spread BWAS replication failure. Brain-behavioral phenotype associations stabilize and become more reproducible with sample sizes of N⪆2,000. While investigator-initiated brain-behavior research continues to generate hypotheses and propel innovation, large consortia are needed to usher in a new era of reproducible human brain-wide association studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in functional and structural MR image analysis and implementation as FSL
Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E. J., Johansen-Berg, H., Bannister, P. R., De Luca, M., Drobnjak, I., Flitney, D. E., et al. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.07.051 · Cited by 13,630
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
23andMe Research Team, Lee, J. J., COGENT (Cognitive Genomics Consortium), Social Science Genetic Association Consortium, Wedow, R., Okbay, A., Kong, E., Maghzian, O., Zacher, M., Nguyen-Viet, T. A., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0147-3 · Cited by 2,470
Resting-state connectivity biomarkers define neurophysiological subtypes of depression
Drysdale, A. T., Grosenick, L., Downar, J., Dunlop, K., Mansouri, F., Meng, Y., Fetcho, R. N., Zebley, B., Oathes, D. J., Etkin, A., et al. (2017) · Nature Medicine · doi:10.1038/nm.4246 · Cited by 2,170
Smith, S. M., Nichols, T. E., Vidaurre, D., Winkler, A. M., Behrens, T. E. J., Glasser, M. F., Ugurbil, K., Barch, D. M., Van Essen, D. C., & Miller, K. L. (2015) · Nature Neuroscience · doi:10.1038/nn.4125 · Cited by 859
Self-citationBarch, D. M., Albaugh, M. D., Avenevoli, S., Chang, L., Clark, D. B., Glantz, M. D., Hudziak, J. J., Jernigan, T. L., Tapert, S. F., Yurgelun-Todd, D., et al. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.10.010 · Cited by 814
Self-citationBorder, R., Johnson, E. C., Evans, L. M., Smolen, A., Berley, N., Sullivan, P. F., & Keller, M. C. (2019) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.18070881 · Cited by 629
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471
RICOPILI: Rapid Imputation for COnsortias PIpeLIne
Lam, M., Awasthi, S., Watson, H. J., Goldstein, J., Panagiotaropoulou, G., Trubetskoy, V., Karlsson, R., Frei, O., Fan, C.-C., De Witte, W., et al. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz633 · Cited by 315
Cited by
The Human Connectome Project: A retrospective
Elam, J. S., Glasser, M. F., Harms, M. P., Sotiropoulos, S. N., Andersson, J. L. R., Burgess, G. C., Curtiss, S. W., Oostenveld, R., Larson-Prior, L. J., Schoffelen, J.-M., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118543 · Cited by 307
Self-citationIntegrative frontal-parietal dynamics supporting cognitive control
Nee, D. E. (2021) · eLife · doi:10.7554/elife.57244 · Cited by 112
Psychedelic therapy's transdiagnostic effects: A research domain criteria (RDoC) perspective
Kelly, J. R., Gillan, C. M., Prenderville, J., Kelly, C., Harkin, A., Clarke, G., & O'Keane, V. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.800072 · Cited by 69
Lichenstein, S. D., Manco, N., Cope, L. M., Egbo, L., Garrison, K. A., Hardee, J., Hillmer, A. T., Reeder, K., Stern, E. F., Worhunsky, P., et al. (2022) · Neuropsychopharmacology · doi:10.1038/s41386-021-01226-9 · Cited by 63
Mo, K., Sadoway, T., Bonato, S., Ameis, S. H., Anagnostou, E., Lerch, J. P., Taylor, M. J., & Lai, M.-C. (2021) · NeuroImage: Clinical · doi:10.1016/j.nicl.2021.102811 · Cited by 53
“…brain regions/white matter tracts of interest; however, some reported effect sizes were smaller than the estimated MDE, which suggests an increased likelihood that some of these tests were not sufficiently powered and the observed effects may potentially be false positives (Marek et al., 2020).”
Metadata
- Authors
- Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Feczko, E., Miranda-Dominguez, O., Graham, A. M., Earl, E. A., Perrone, A. J., Cordova, M., Doyle, O., Moore, L. A., Conan, G., Uriarte, J., Snider, K., Tam, A., Chen, J., Newbold, D. J., Zheng, A., Seider, N. A., Van, A. N., Laumann, T. O., Thompson, W. K., Greene, D. J., Petersen, S. E., Nichols, T. E., Yeo, B. T. T., Barch, D. M., Garavan, H., Luna, B., Fair, D. A., & Dosenbach, N. U. F.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-08-22
- OpenAlex
- W3081365410
- Cited by
- 181
- References
- 11
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards Reproducible Brain-Wide Association Studies. https://app.cochat.ai/papers/10.1101/2020.08.21.257758
- The paper
- Marek, S., Tervo-Clemmens, B., Calabro, F. J., Montez, D. F., Kay, B. P., Hatoum, A. S., Donohue, M. R., Foran, W., Miller, R. L., Feczko, E., Miranda-Dominguez, O., Graham, A. M., Earl, E. A., Perrone, A. J., Cordova, M., Doyle, O., Moore, L. A., Conan, G., Uriarte, J., . . . Dosenbach, N. U. F. (2020). Towards Reproducible Brain-Wide Association Studies. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2020.08.21.257758
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.