CoChat
Sign Up
Cited by 7,723Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods in Epidemiology

Power failure: why small sample size undermines the reliability of neuroscience

Open access
Button, Katherine S., Ioannidis, John P. A., Mokrysz, Claire, Nosek, Brian A., Flint, Jonathan, Robinson, Emma S. J., Munafò, Marcus R.·2013·Nature Reviews Neuroscience
DOI 10.1038/nrn3475
Open this paper in CoChat — free accountOpen PDF

Abstract

A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.

AI snapshot

Written from the paper’s full text.

TL;DR

This review argues that low statistical power is widespread in neuroscience and that it makes published findings less likely to be true, more likely to overstate effects, and harder to replicate. Across meta-analyses from several subfields, median power was often below 31%, with some estimates as low as 8%.

Key findings

  • In a sample of 48 articles, 49 meta-analyses and 730 primary studies, the median statistical power in neuroscience was 21%; excluding seven unusually high-powered neurological meta-analyses, it fell to 18%.
  • The observed number of nominally significant studies was 349, compared with 254 expected under the meta-analytic summary effects, indicating excess significance (p < 0.0001).
  • For brain volume studies, median power was 8% across 461 individual studies from 41 meta-analyses in eight articles; for animal water-maze and radial-maze studies, median power was 18% and 31%, respectively, with average sample sizes of 22 and 24 animals.
  • With pre-study odds of 0.25 and α = 0.05, 20% power yielded a positive predictive value of 0.50, whereas 80% power yielded 0.80; simulations suggested initial effect estimates from studies powered between about 8% and 31% are inflated by 25% to 50%.

Methods

  • The authors searched Web of Science on 2 February 2012 for 2011 articles using the terms “neuroscience” and “meta-analysis,” screened records independently, and included 48 articles containing 49 meta-analyses and 730 primary studies.
  • They extracted summary effect sizes and study-level sample sizes or event counts, then calculated each primary study’s achieved power to detect its corresponding meta-analytic summary effect at α = 5% using G*Power, reporting mean and median power and testing for excess significance.
  • Because the main sample underrepresented neuroimaging and animal-model research, they added representative meta-analyses from those areas outside the 2011 sampling frame.

Limitations

  • The paper states that true effect sizes are unknown, so meta-analytic summary effects were used as proxies; because those summaries may themselves be inflated by publication and other biases, the power estimates are likely optimistic.
  • The authors note that the main analysis underrepresented neuroimaging and animal-model subfields, and that the animal-model findings rest on only two meta-analyses, so extrapolation should be cautious.
  • Evident from the text: the search was limited to 2011 Web of Science records and to meta-analyses with extractable data, and the excess-significance test depends on the assumption that the summary effects approximate the true effects.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/nrn3475

Cites

110 of 110 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 39 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Researchers have strong incentives to engage in research practices that make findings publishable quickly, even if those practices reduce the likelihood that findings reflect a true effect.

    “As a consequence, researchers have strong incentives to engage in research practices that make their findings publishable quickly, even if those practices reduce the likelihood that the findings reflect a true (that is, non-null) effect 4 .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [4] Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012). source
  • Premisecitation

    If studies in a field have 20% power, then out of 100 genuine non-null effects, they are expected to discover only 20.

    “When studies in a given field are designed with a power of 20%, it means that if there are 100 genuine non-null effects to be discovered in that field, these studies are expected to discover only 20 of them 11 .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [11] Sterne, J. A. & Davey Smith, G. Sifting the evidence — what's wrong with significance tests? BMJ 322, 226–231 (2001). source
  • Premisecitation

    The lower the power of a study, the lower the probability that an observed effect passing nominal statistical significance actually reflects a true effect.

    “Second, the lower the power of a study, the lower the probability that an observed effect that passes the required threshold of claiming its discovery (that is, reaching nominal statistical significance, such as p < 0.05) actually reflects a true effect 1,12 .”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [1] Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power. source
    • cites [12] Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011). source
  • Premisecitation

    Previous analyses of studies using animal models have shown that small studies consistently give more favourable (positive) results than larger studies.

    “Previous analyses of studies using animal models have shown that small studies consistently give more favourable (that is, 'positive') results than larger studies 78”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [78] Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010). source

      The Results (Prevalence of Publication Bias; page not provided) report that all 16 datasets had an excess of imprecise studies with large effect sizes, supporting the favourable-results pattern in animal stroke studies. But the Discussion (page not provided) cautions that precision in these studies relates more to biological variability and measurement error than to study size. The paper therefore does not establish the claimed comparison of small studies with larger studies. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    Study quality is inversely related to effect size in animal model studies.

    “study quality is inversely related to effect size [79][80][81][82]”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [79] Ioannidis, J. P. Extrapolating from animals to humans. Sci. Transl. Med. 4, 151ps15 (2012). source
    • cites [80] Jonasson, Z. Meta-analysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data. Neurosci. Biobehav. Rev. 28, 811–825 (2005). source
    • cites [81] Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008). source

      The Results report that estimated efficacy was highest in lower-quality studies of NXY-059 in animal models of stroke. This supports the inverse relationship in the studies analyzed, but does not establish it across animal model studies generally. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 19

  • May combine two claims.

    Premisecitation

    Such practices include using flexible study designs, flexible statistical analyses, and running small studies with low statistical power.

    “Such practices include using flexible study designs and flexible statistical analyses and running small studies with low statistical power 1,5 .”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [1] Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power. source
    • cites [5] Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false. source

  • Classified as probably not a substantial claim.

    Premisecitation

    Many (and possibly most) conclusions drawn from biomedical research are probably false.

    “It has been claimed and demonstrated that many (and possibly most) of the conclusions drawn from biomedical research are probably false 1 .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [1] Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power. source

  • May combine two claims.

    Premisecitation

    Publication bias, selective data analysis, and selective reporting of outcomes and analyses are more likely to affect smaller, underpowered studies.

    “Second, publication bias, selective data analysis and selective reporting of outcomes are more likely to affect low-powered studies.”
    • Also stated in: § Low power in the presence of other biases · hedged
      “Publication bias and selective reporting of outcomes and analyses are also more likely to affect smaller, underpowered studies 17 .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [17] Dwan, K. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081 (2008). source

      The Discussion (page unavailable) reports that positive or statistically significant studies are more likely to be published and significant outcomes more likely to be fully reported. The paper does not establish that publication bias or selective reporting is more likely in smaller, low-powered studies; the Results mention power calculations but do not make that comparison. Its Introduction describes selective reporting of analyses but does not assess whether it varies by study power. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Previously almost all of the proposed candidate gene associations from small studies were false, with some exceptions, but collaborative consortia have substantially improved power, and the replicated results can be considered highly reliable.

    “Although previously almost all of the proposed candidate gene associations from small studies were false 99 (with some exceptions 100 ), collaborative consortia have substantially improved power, and the replicated results can be considered highly reliable.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [99] Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011). source
    • cites [100] Siontis, K. C., Patsopoulos, N. A. & Ioannidis, J. P. Replication of past candidate loci for common diseases and phenotypes in 100 genome-wide association studies. Eur. J. Hum. Genet. 18, 832–837 (2010). source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    These policies are uncommonly adhered to, and thus the ability for independent experts to repeat published analysis remains low.

    “However, these policies are uncommonly adhered to 95 , and thus the ability for independent experts to repeat published analysis remains low 96 .”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [95] Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H. & Ioannidis, J. P. Public availability of published research data in high-impact journals. PLoS ONE 6, e24357 (2011). source
    • cites [96] Ioannidis, J. P. et al. Repeatability of published microarray gene expression analyses. Nature Genet. 41, 149–155 (2009). source

  • May combine two claims.

    Findingown data

    The median statistical power for the water maze studies and the radial maze studies to detect these medium to large effects was 18% and 31%, respectively.

    “Our results indicate that the median statistical power for the water maze studies and the radial maze studies to detect these medium to large effects was 18% and 31%, respectively (TABLE 2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The average sample size in these studies was 22 animals for the water maze and 24 for the radial maze experiments.

    “The average sample size in these studies was 22 animals for the water maze and 24 for the radial maze experiments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    To achieve 80% power to detect the most probable true effects as indicated by the meta-analysis, a sample size of 134 animals would be required for the water maze experiment (assuming d = 0.49) and 68 animals for the radial maze experiment (assuming d = 0.69); to achieve 95% power, these sample sizes would need to increase to 220 and 112, respectively.

    “In order to achieve 80% power to detect, in a single study, the most probable true effects as indicated by the meta-analysis, a sample size of 134 animals would be required for the water maze experiment (assuming an effect size of d = 0.49) and 68 animals for the radial maze experiment (assuming an effect size of d = 0.69); to achieve 95% power, these sample sizes would need to increase to 220 and”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The total numbers of animals actually used in the studies contributing to the meta-analyses were 420 for the water maze experiments and 514 for the radial maze experiments.

    “the total numbers of animals actually used in the studies contributing to the meta-analyses were even larger: 420 for the water maze experiments and 514 for the radial maze experiments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The meta-analyses (n = 7) that comprised studies with high average power in excess of 90% had their broadly neurological subject matter in common.

    “The meta-analyses (n = 7) that comprised studies with high average power in excess of 90% had their broadly neurological subject matter in common.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Low power means the chance of discovering genuinely true effects is low.

    “First, low power, by definition, means that the chance of discovering effects that are genuinely true is low.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    False positives may heavily contaminate the neuroscience literature, and this problem may affect prominent journals at least as much or more.

    “Given that these publishing biases are pervasive across scientific practice, it is possible that false positives heavily contaminate the neuroscience literature as well, and this problem may affect at least as much, if not even more so, the most prominent journals 9,10 .”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [9] Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855–3859 (2011). source
    • cites [10] Munafo, M. R., Stothart, G. & Flint, J. Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119–120 (2009). source

  • May combine two claims.

    Interpretationreasoning

    Pre-registration and full reporting of analysis plans clarifies the distinction between confirmatory and exploratory analysis, encourages well-powered studies, and reduces the file-drawer effect.

    “Pre-registration -and, ultimately, full reporting of analysis plans -clarifies the distinction between confirmatory and exploratory analysis, encourages well-powered studies (at least in the case of confirmatory analyses) and reduces the file-drawer effect.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The principles of the 'three Rs' in animal research (reduce, refine and replace) require appropriate experimental design and statistics; both too many and too few animals present an issue as they reduce the value of research outputs.

    “The principles of the 'three Rs' in animal research (reduce, refine and replace) 83 require appropriate experimental design and statistics -both too many and too few animals present an issue as they reduce the value of research outputs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [83] Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Methuen, 1958).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All articles identified via the electronic search were screened independently for suitability by two authors (K.S.B. and M.R.M.).

    “All of the articles that were identified via this electronic search were screened independently for suitability by two authors (K.S.B. and M.R.M.).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Low power has an ethical dimension: unreliable research is inefficient and wasteful.

    “Low power therefore has an ethical dimensionunreliable research is inefficient and wasteful.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The results described in this section are based on only two meta-analyses, and we should be appropriately cautious in extrapolating from this limited evidence.

    “The results described in this section are based on only two meta-analyses, and we should be appropriately cautious in extrapolating from this limited evidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The results are consistent with those observed in other fields, such as the neuroimaging and neuroscience studies described above.

    “Nevertheless, it is notable that the results are so consistent with those observed in other fields, such as the neuroimaging and neuroscience studies that we have described above.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    False positives occur more frequently and go unnoticed when degrees of freedom in data analysis and reporting are undisclosed.

    “False positives occur more frequently and go unnoticed when degrees of freedom in data analysis and reporting are undisclosed 5 .”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false. source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Making research materials available will improve the quality of studies aimed at replicating and extending research findings.

    “Making research materials available will improve the quality of studies aimed at replicating and extending research findings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Data extraction was performed independently by K.S.B. and M.R.M. or C.M. and verified collaboratively.

    “Data extraction was performed independently by K.S.B. and M.R.M. or C.M. and verified collaboratively.”
    • Also stated in: § Figure 2 | Flow diagram of articles selected for inclusion. · plain
      “Data extraction was independently performed by K.S.B. and either M.R.M. or C.M. and verified collaboratively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Meta-analysis provides the best estimate of the true effect size, albeit with limitations, including that contributing studies are subject to the problems described above.

    “Meta-analysis provides the best estimate of the true effect size, albeit with limitations, including the limitation that the individual studies that contribute to a meta-analysis are themselves subject to the problems described above.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Several other biases are also likely to reduce the probability that a research finding reflects a true effect.

    “As we have already discussed, several other biases are also likely to reduce the probability that a research finding reflects a true effect.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Inflated effect estimates make it difficult to determine an adequate sample size for replication studies, increasing the probability of type II errors.

    “Inflated effect estimates make it difficult to determine an adequate sample size for replication studies, increasing the probability of type II errors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Null findings locked in file drawers bias the literature, whereas exploratory analyses are only useful and valid if you acknowledge the caveats and limitations.

    “Null findings locked in file drawers bias the literature, whereas exploratory analyses are only useful and valid if you acknowledge the caveats and limitations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Assuming that the initially observed effect being replicated reflects the true effect, the potential distribution of replication effect estimates would be similar to the distribution of the original study.

    “Assuming, as one might in a power calculation, that the initially observed effect we are trying to replicate reflects the true effect, the potential distribution of these replication effect estimates would be similar to the distribution of the original study (dashed green curve).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The initially observed effect that a replication study attempts to replicate reflects the true effect.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Some fields are confronting the problem of the poor reliability of research findings that arises from low-powered studies.

    “Some fields are confronting the problem of the poor reliability of research findings that arises from low-powered studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Increasing the sample size of a replication study reduces the standard error of the mean and thereby increases the study's power.

    “We can increase the power of the replication study (coloured area under the orange curve) by increasing the sample size so as to reduce the sem.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    In genetic epidemiology, sample sizes increased dramatically with the widespread understanding that the effects being sought are likely to be extremely small.

    “For example, in genetic epidemiology sample sizes increased dramatically with the widespread understanding that the effects being sought are likely to be extremely small.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    If a study finds evidence for an effect at p = 0.05, then the difference between the mean of the null distribution and the mean of the observed distribution is 1.96 times the standard error of the mean.

    “If a study finds evidence for an effect at p = 0.05, then the difference between the mean of the null distribution (indicated by the solid blue curve) and the mean of the observed distribution (dashed blue curve) is 1.96 × sem.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Making raw data available will improve data aggregation methods and confidence in reported results.

    “Making raw data available will improve data aggregation methods and confidence in reported results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    Combining data increases the total sample size and therefore statistical power, while minimizing the labour and resource impact on any one contributor.

    “Combining data increases the total sample size (and therefore power) while minimizing the labour and resource impact on any one contributor.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Large, high-powered studies can readily detect both small and large effects and so are less biased, as both over- and underestimations of the true effect size will pass the threshold for discovery.

    “By contrast, large, high-powered studies can readily detect both small and large effects and so are less biased, as both over-and underestimations of the true effect size will pass the threshold for 'discovery'.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    The winner's curse is the phenomenon that studies that find evidence of an effect often provide inflated estimates of the size of that effect.

    “The winner's curse refers to the phenomenon that studies that find evidence of an effect often provide inflated estimates of the size of that effect.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites REF. 103

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

34 of 34

  • The authors state this as an assumption.

    The summary effect size reported in each study is close to the true effect size.

    “Importantly, these calculations assume that the summary effect size reported in each study is close to the true effect size, but it is likely that they are inflated owing to publication and other biases described above.”

  • The authors state this as an assumption.

    The initially observed effect we are trying to replicate reflects the true effect.

    “Assuming, as one might in a power calculation, that the initially observed effect we are trying to replicate reflects the true effect, the potential distribution of these replication effect estimates would be similar to the distribution of the original study (dashed green curve).”

  • The paper relies on this without support.

    The 49 meta-analyses included are representative of neuroscience research.

    “Our results indicate that the median statistical power in neuroscience is 21%.”

  • The paper relies on this without support.

    Most structural and volumetric MRI studies are very small and have minimal power to detect differences between compared groups.

    “Most structural and volumetric MRI studies are very small and have minimal power to detect differences between compared groups (for example, healthy people versus those with mental health diseases).”

  • The paper relies on this without support.

    The effects now sought in neuroscience experiments are often smaller and more subtle than before.

    “A consequence of the remarkable growth in neuroscience over the past 50 years has been that the effects we now seek in our experiments are often smaller and more subtle than before as opposed to when mostly easily discernible 'low-hanging fruit' were targeted.”

  • The paper relies on this without support.

    Computational analysis of very large datasets is now relatively straightforward, so that an enormous number of tests can be run in a short time on the same dataset.

    “At the same time, computational analysis of very large datasets is now relatively straightforward, so that an enormous number of tests can be run in a short time on the same dataset.”

  • The paper relies on this without support.

    These dramatic advances in the flexibility of research design and analysis have occurred without accompanying changes to other aspects of research design, particularly power.

    “These dramatic advances in the flexibility of research design and analysis have occurred without accompanying changes to other aspects of research design, particularly power.”

  • The paper relies on this without support.

    Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings.

    “Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings.”

  • The authors state this as an assumption.

    The initially observed effect that a replication study attempts to replicate reflects the true effect.

    “Assuming, as one might in a power calculation, that the initially observed effect we are trying to replicate reflects the true effect”

  • The paper relies on this without support.

    In an exploratory research field such as much of neuroscience, the pre-study odds are often low.

    “Notably, in an exploratory research field such as much of neuroscience, the pre-study odds are often low.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Due to differences in age and sample size between protocols and phases, effect sizes could not be directly compared.

    “Due to differences in age and sample size between protocols and phases, effect sizes could not be directly compared (Button et al., 2013), but the directions of associations with age were largely consistent for ADNI2 and ADNI3 phases separately, and each ADNI3 protocol (Figures 1, 2).”

    From Diffusion MRI Indices and Their Relation to Cognitive Impairment in Brain Aging: The Updated Multi-protocol Approach in ADNI3 (2019) · cites this paper as Button et al., 2013

    The “Effect size” section (page unavailable) says standardized effect sizes allow estimates from different studies to be compared directly, contrary to the cited rationale that differing sample sizes prevent direct comparison. The paper does not address the particular age differences, protocols, or phases in the citing sentence. (LLM, conf=0.78)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The average power of studies in neuroscience in general is 21%.

    “but not very different from studies in neuroscience in general (average power = 21%) (14).”

    From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [14]

    The Results section reports 21% as the median statistical power across the studies analyzed, not the arithmetic average. The paper discusses this estimate in relation to neuroscience generally, but the cited figure is mislabeled as an average. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The small sample size precludes the detection of small or medium-sized effects.

    “First, the sample size is small, which precludes the detection of small or medium-sized effects.43”

    From Effects of Insufficient Sleep on Pituitary–Adrenocortical Response to CRH Stimulation in Healthy Men (2017) · cites this paper as [43]

    The Animal model studies section reports that small studies had low power to detect medium-to-large effects and needed very large effects for 80% power. But the paper does not say a small sample precludes detection: its discussion of the winner’s curse explains that an underpowered study can detect a medium-sized effect by chance. The cited paper also does not assess the citing study’s particular sample size. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The lower response rate at week 26 may also result in inflated effect sizes and lower reproducibility of results.

    “The lower response rate at week 26 may also result in inflated effects sizes and lower reproducibility of results [63].”

    From Benchmark findings from a veteran electronic patient-reported outcomes evaluation from a chronic pain management telehealth program (2024) · cites this paper as [63]

    The “Winner’s curse” section explains that small studies can yield inflated effect estimates and make replication less likely; “Low power in the presence of other biases” also links low power to less reliable evidence. The paper does not examine response rates or a week-26 follow-up, so it supports the general concern about a smaller sample, not the specific claim about the lower response rate. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Neuroimaging studies are costly and tend to have small sample sizes.

    “This may benefit studies in particular that focus on neuroimaging, which are costly and tend to have small sample sizes (Button et al. 2013), those that employ electrophysiological techniques that might be extremely time-consuming, or those that include genetics.”

    From New directions in hypnosis research: strategies for advancing the cognitive and clinical neuroscience of hypnosis (2017) · cites this paper as Button et al. 2013

    The “Neuroimaging studies” section says most structural and volumetric MRI studies are very small, supporting the sample-size part of the claim. Our copy does not establish that neuroimaging studies are costly; the discussion of funding in “Low power in the presence of other biases” concerns large studies generally, not neuroimaging specifically. Page numbers are unavailable in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Detectable effect sizes are reversed for the two animal-study sample sizes

    Minor

    The animal-model section assigns d = 1.20 to n = 22 and d = 1.26 to n = 24 at 80% power. The ethical-implications section assigns the same effect sizes to the opposite sample sizes, changing which effect each typical study could detect.

1 of 1

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

  • Cites a retracted work: 10.1128/iai.05661-11

    Info

    The cited work was retracted; the date of the notice is not on record.

    Cited work 10.1128/iai.05661-11 · retraction date not known · this paper published 2013-04-10

1 of 1

Cites

1 retracted

Flagged first, then by citations

Research works printed but not linked: 2

  1. David, S. P. et al. Potential reporting bias in small fMRI studies of the brain. PLoS Biol. (in the press).
  2. Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011). · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R.
Affiliations
  • University of Bristol (GB)
  • Stanford Medicine (US)
  • University of Virginia (US)
ORCID
Venue
Nature Reviews Neuroscience
Publisher
Springer Science and Business Media LLC
Published
2013-04-10
Volume
14
Issue
5
Pages
365-376
ISSN
  • 1471-003X (ISSN-L)
  • 1471-0048
DOI
10.1038/nrn3475
OpenAlex
W2105824687
Semantic Scholar
c2bc880156c10fc8d1752cd3d528e4ef96cbaac4
Cited by
7,641
References
106
Citations by year
2026: 584 · 2025: 611 · 2024: 606 · 2023: 553 · 2022: 658 · 2021: 692 · 2020: 752 · 2019: 683 · 2018: 679 · 2017: 597 · 2016: 486 · 2015: 352 · 2014: 268 · 2013: 109
FWCI
82.23
Type
journal article
Language
en
Update notices
erratum (2013-04-15)
Open access
Licence
Locations
  • Nature reviews. Neuroscience (publishedVersion)
  • PubMed (publishedVersion)
  • Explore Bristol Research (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Funders
  • Wellcome Trust (090532)
  • United Kingdom Clinical Research Collaboration
  • Cancer Research UK
  • National Institute for Health and Care Research
  • British Heart Foundation
  • Medical Research Council
  • Economic and Social Research Council
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
statistical power, small sample size, effect size overestimation, reproducibility, research ethics, research waste, research practices, brain research
MeSH terms
Humans, Neurosciences, Probability, Reproducibility of Results, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review