Power-up: A Reanalysis of 'Power Failure' in Neuroscience Using Mixture Modeling
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Recently, evidence for endemically low statistical power has cast neuroscience findings into doubt. If low statistical power plagues neuroscience, then this reduces confidence in the reported effects. However, if statistical power is not uniformly low, then such blanket mistrust might not be warranted. Here, we provide a different perspective on this issue, analyzing data from an influential study reporting a median power of 21% across 49 meta-analyses (Button et al., 2013). We demonstrate, using Gaussian mixture modeling, that the sample of 730 studies included in that analysis comprises several subcomponents so the use of a single summary statistic is insufficient to characterize the nature of the distribution. We find that statistical power is extremely low for studies included in meta-analyses that reported a null result and that it varies substantially across subfields of neuroscience, with particularly low power in candidate gene association studies. Therefore, whereas power in neuroscience remains a critical issue, the notion that studies are systematically underpowered is not the full story: low power is far from a universal problem. SIGNIFICANCE STATEMENT Recently, researchers across the biomedical and psychological sciences have become concerned with the reliability of results. One marker for reliability is statistical power: the probability of finding a statistically significant result given that the effect exists. Previous evidence suggests that statistical power is low across the field of neuroscience. Our results present a more comprehensive picture of statistical power in neuroscience: on average, studies are indeed underpowered—some very seriously so—but many studies show acceptable or even exemplary statistical power. We show that this heterogeneity in statistical power is common across most subfields in neuroscience. This new, more nuanced picture of statistical power in neuroscience could affect not only scientific understanding, but potentially policy and funding decisions for neuroscience research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
(2007) · doi:10.5555/1283383.1283494 · Cited by 7,364
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Anthropometric Reference Data for Children and Adults: United States, 2003-2006: (623932009-001)
McDowell, M. A., Fryar, C. D., Ogden, C. L., & Flegal, K. M. (2008) · PsycEXTRA Dataset · doi:10.1037/e623932009-001 · Cited by 1,378
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Duncan, L. E., & Keller, M. C. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.11020191 · Cited by 1,032 · cited 2×
“Because numerous studies report that candidate gene association studies are severely underpowered (Klerk et al., 2002; Colhoun et al., 2003; Duncan and Keller, 2011), and given that candidate gene association studies comprised more than oneFigure 3.”
BackgroundResultDelayed reward discounting and addictive behavior: a meta-analysis
MacKillop, J., Amlung, M. T., Few, L. R., Ray, L. A., Sweet, L. H., & Munafò, M. R. (2011) · Psychopharmacology · doi:10.1007/s00213-011-2229-0 · Cited by 1,021
Frontocingulate Dysfunction in Depression: Toward Biomarkers of Treatment Response
Pizzagalli, D. A. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2010.166 · Cited by 945
Hannestad, J., DellaGioia, N., & Bloch, M. (2011) · Neuropsychopharmacology · doi:10.1038/npp.2011.132 · Cited by 914
Veehof, M. M., Oskam, M.-J., Schreurs, K. M. G., & Bohlmeijer, E. T. (2011) · PAIN · doi:10.1016/j.pain.2010.11.002 · Cited by 792
Publication bias in reports of animal stroke studies leads to major overstatement of efficacy
Sena, E. S., van der Worp, H. B., Bath, P. M. W., Howells, D. W., & Macleod, M. R. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000344 · Cited by 600 · cited 1×
“However, publication bias has long been reported in psychology (Francis, 2012) and neuroscience (Sena et al., 2010), so it is reasonable to assume that it has inflated estimates of statistical power in these analyses.”
BackgroundBrain-derived neurotrophic factor levels in schizophrenia: a systematic review with meta-analysis
Green, M. J., Matheson, S. L., Shepherd, A., Weickert, C. S., & Carr, V. J. (2011) · Molecular Psychiatry · doi:10.1038/mp.2010.88 · Cited by 453
A meta-analysis of diffusion tensor imaging in mild cognitive impairment and Alzheimer's disease
Sexton, C. E., Kalu, U. G., Filippini, N., Mackay, C. E., & Ebmeier, K. P. (2011) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2010.05.019 · Cited by 410
Meta-analysis of facial affect recognition difficulties after traumatic brain injury.
Babbage, D. R., Yim, J., Zupan, B., Neumann, D., Tomita, M. R., & Willer, B. (2011) · Neuropsychology · doi:10.1037/a0021908 · Cited by 198
Chamberlain, S. R., Robbins, T. W., Winder-Rhodes, S., Müller, U., Sahakian, B. J., Blackwell, A. D., & Barnett, J. H. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.08.019 · Cited by 178
Confidence and precision increase with high statistical power
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475-c4 · Cited by 121 · cited 13×
“Such variability across neuroscience subfields is consistent with the original report by Button et al. (2013), which reported the median power of animal studies (18% and 31% for two meta-analyses) and case-control structural brain imaging studies (8% across 41 meta-analyses).”
BackgroundMethodResultInfluentialWitteman, J., van IJzendoorn, M. H., van de Velde, D., van Heuven, V. J. J. P., & Schiller, N. O. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2011.09.028 · Cited by 112
Allergy and risk of glioma: a meta-analysis
Chen, C., Xu, T., Chen, J., Zhou, J., Yan, Y., Lu, Y., & Wu, S. (2011) · European Journal of Neurology · doi:10.1111/j.1468-1331.2010.03187.x · Cited by 100
Meta-analysis of the relationship between homocysteine, vitamin B12, folate, and multiple sclerosis
Zhu, Y., He, Z.-Y., & Liu, H.-N. (2011) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2010.12.022 · Cited by 75
Björkhem-Bergman, L., Asplund, A. B., & Lindh, J. D. (2011) · Journal of Psychopharmacology · doi:10.1177/0269881109353461 · Cited by 72
Barch, D. M., & Yarkoni, T. (2013) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-013-0201-7 · Cited by 67
Maneeton, N., Maneeton, B., Srisurapanont, M., & Martin, S. D. (2011) · Psychiatry and Clinical Neurosciences · doi:10.1111/j.1440-1819.2011.02264.x · Cited by 60
Cited by
Sex Differences in the Adult Human Brain: Evidence from 5216 UK Biobank Participants
Ritchie, S. J., Cox, S. R., Shen, X., Lombardo, M. V., Reus, L. M., Alloza, C., Harris, M. A., Alderson, H. L., Hunter, S., Neilson, E., et al. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhy109 · Cited by 855
“Given the surfeit of small-n studies in neuroscience (Button et al. 2013; Nord et al. 2017), it is of great importance to test hypotheses in large, well-powered samples, especially given that many neural sex differences are modest in size (Joel et al.”
BackgroundHippocampal volume across age: Nomograms derived from over 19,700 people in UK Biobank
Nobis, L., Manohar, S. G., Smith, S. M., Alfaro-Almagro, F., Jenkinson, M., Mackay, C. E., & Husain, M. (2019) · NeuroImage: Clinical · doi:10.1016/j.nicl.2019.101904 · Cited by 236
Disrupted Dorsal Mid-Insula Activation During Interoception Across Psychiatric Disorders
Nord, C. L., Lawson, R. P., & Dalgleish, T. (2021) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2020.20091340 · Cited by 171
Self-citation10 years of Bayesian theories of autism: A comprehensive review
Angeletos Chrysaitis, N., & Seriès, P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.105022 · Cited by 92
Sex/gender differences in cognitive abilities
Kheloui, S., Jacmin-Park, S., Larocque, O., Kerr, P., Rossi, M., Cartier, L., & Juster, R.-P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105333 · Cited by 78
Metadata
- Authors
- Nord, C. L., Valton, V., Wood, J., & Roiser, J. P.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2017-07-13
- Volume
- 37
- Issue
- 34
- Pages
- 8051-8061
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2735379969
- Cited by
- 86
- References
- 23
- FWCI
- 10.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.