The DOI matches its Crossref record; 110 of the 110 references it cites were found in a registry or an index; 11 of the 39 sources its claims cite were checked for support.
Power failure: why small sample size undermines the reliability of neuroscience
Open accessAbstract
A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This review argues that low statistical power is widespread in neuroscience and that it makes published findings less likely to be true, more likely to overstate effects, and harder to replicate. Across meta-analyses from several subfields, median power was often below 31%, with some estimates as low as 8%.
Key findings
- In a sample of 48 articles, 49 meta-analyses and 730 primary studies, the median statistical power in neuroscience was 21%; excluding seven unusually high-powered neurological meta-analyses, it fell to 18%.
- The observed number of nominally significant studies was 349, compared with 254 expected under the meta-analytic summary effects, indicating excess significance (p < 0.0001).
- For brain volume studies, median power was 8% across 461 individual studies from 41 meta-analyses in eight articles; for animal water-maze and radial-maze studies, median power was 18% and 31%, respectively, with average sample sizes of 22 and 24 animals.
- With pre-study odds of 0.25 and α = 0.05, 20% power yielded a positive predictive value of 0.50, whereas 80% power yielded 0.80; simulations suggested initial effect estimates from studies powered between about 8% and 31% are inflated by 25% to 50%.
Methods
- The authors searched Web of Science on 2 February 2012 for 2011 articles using the terms “neuroscience” and “meta-analysis,” screened records independently, and included 48 articles containing 49 meta-analyses and 730 primary studies.
- They extracted summary effect sizes and study-level sample sizes or event counts, then calculated each primary study’s achieved power to detect its corresponding meta-analytic summary effect at α = 5% using G*Power, reporting mean and median power and testing for excess significance.
- Because the main sample underrepresented neuroimaging and animal-model research, they added representative meta-analyses from those areas outside the 2011 sampling frame.
Limitations
- The paper states that true effect sizes are unknown, so meta-analytic summary effects were used as proxies; because those summaries may themselves be inflated by publication and other biases, the power estimates are likely optimistic.
- The authors note that the main analysis underrepresented neuroimaging and animal-model subfields, and that the animal-model findings rest on only two meta-analyses, so extrapolation should be cautious.
- Evident from the text: the search was limited to 2011 Web of Science records and to meta-analyses with extractable data, and the excess-significance test depends on the assumption that the summary effects approximate the true effects.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
45 Own interpretation · 15 No basis given · 12 Own result · 9 No full text of the sources · 5 Partly supported · 4 Sources not found · 3 Supported · 3 Supported so far · 3 Own method · 1 Not checked yet · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“As a consequence, researchers have strong incentives to engage in research practices that make their findings publishable quickly, even if those practices reduce the likelihood that the findings reflect a true (that is, non-null) effect 4 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
What the sources say
- Supportedcites [4] Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012). source
The Abstract says publication incentives encourage research decisions that produce positive results and inflate the rate of false effects. The section “Strategies That Will Accelerate the Accumulation of Knowledge” contrasts the immediate payoff of publishing with the longer-term goal of accuracy, supporting the claim’s emphasis on publishing quickly. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“When studies in a given field are designed with a power of 20%, it means that if there are 100 genuine non-null effects to be discovered in that field, these studies are expected to discover only 20 of them 11 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
What the sources say
- Supportedcites [11] Sterne, J. A. & Davey Smith, G. Sifting the evidence — what's wrong with significance tests? BMJ 322, 226–231 (2001). source
Table 3 defines 20% power as rejecting the null hypothesis 20% of the time when it is false. Thus, among 100 genuine non-null effects, the expected number detected is 20. The supplied copy does not specify a page number. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Second, the lower the power of a study, the lower the probability that an observed effect that passes the required threshold of claiming its discovery (that is, reaching nominal statistical significance, such as p < 0.05) actually reflects a true effect 1,12 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
What the sources say
- Supportedcites [1] Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power. source
Corollary 1 states that, other factors being equal, the probability that a research finding is true decreases as study power decreases. Table 4 identifies this probability as the positive predictive value of a finding under a nominal significance threshold of α = 0.05. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites [12] Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous analyses of studies using animal models have shown that small studies consistently give more favourable (that is, 'positive') results than larger studies 78”
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
What the sources say
- Partly supportedcites [78] Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010). sourceExperimental — accuracy not yet measured
The Results (Prevalence of Publication Bias; page not provided) report that all 16 datasets had an excess of imprecise studies with large effect sizes, supporting the favourable-results pattern in animal stroke studies. But the Discussion (page not provided) cautions that precision in these studies relates more to biological variability and measurement error than to study size. The paper therefore does not establish the claimed comparison of small studies with larger studies. (LLM, conf=0.86)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“study quality is inversely related to effect size [79][80][81][82]”
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
What the sources say
- Not checked — no full text of the sourcecites [79] Ioannidis, J. P. Extrapolating from animals to humans. Sci. Transl. Med. 4, 151ps15 (2012). source
- Not checked — no full text of the sourcecites [80] Jonasson, Z. Meta-analysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data. Neurosci. Biobehav. Rev. 28, 811–825 (2005). source
- Partly supportedcites [81] Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008). sourceExperimental — accuracy not yet measured
The Results report that estimated efficacy was highest in lower-quality studies of NXY-059 in animal models of stroke. This supports the inverse relationship in the studies analyzed, but does not establish it across animal model studies generally. (LLM, conf=0.84)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A requirement for sample size and power calculation is included in the Animal Research: Reporting In Vivo Experiments (ARRIVE) guidelines 84”
§ Power failure: why small sample size undermines the reliability of neuroscience > Ethical implications.
What the sources say
- Partly supportedcites [84] Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010). sourceExperimental — accuracy not yet measured
Table 2, item 10 (Sample size), asks authors to report animal numbers, explain how they were decided, and provide details of any sample-size calculation used. It does not require a calculation or mention a power calculation; the accompanying text also says the guidelines are not intended to be mandatory. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that the median statistical power in neuroscience is 21%.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Results.
hedgedn = 730- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 49 meta-analyses included are representative of neuroscience research.
In the paper
“Our results indicated that the median statistical power of these studies was 8% across 461 individual studies contributing to 41 separate meta-analyses, which were drawn from eight articles that were published between 2006 and 2009.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Neuroimaging studies.
What the sources say
- Not checked — no full text of the sourcecites [73] Ioannidis, J. P. Excess significance bias in the literature on brain volume abnormalities. Arch. Gen. Psychiatry 68, 773–780 (2011). source
hedgedn = 461- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, seven meta-analyses comprised studies with high (>90%) average power 24,26,31,57,63,68,71 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Results.
What the sources say
- Not checked — no full text of the sourcecites [24] Babbage, D. R. et al. Meta-analysis of facial affect recognition difficulties after traumatic brain injury. Neuropsychology 25, 277–285 (2011). source
- Not checked — no full text of the sourcecites [26] Bjorkhem-Bergman, L., Asplund, A. B. & Lindh, J. D. Metformin for weight reduction in non-diabetic patients on antipsychotic drugs: a systematic review and meta-analysis. J. Psychopharmacol. 25, 299–305 (2011). source
- Not checked — no full text of the sourcecites [31] Chen, C. et al. Allergy and risk of glioma: a meta-analysis. Eur. J. Neurol. 18, 387–395 (2011). source
- Not checked — no full text of the sourcecites [57] Tian, Y., Kang, L. G., Wang, H. Y. & Liu, Z. Y. Meta-analysis of transcranial magnetic stimulation to treat post-stroke dysfunction. Neural Regen. Res. 6, 1736–1741 (2011).
- Not checked — no full text of the sourcecites [63] Witteman, J., van Ijzendoorn, M. H., van de Velde, D., van Heuven, V. J. & Schiller, N. O. The nature of hemispheric specialization for linguistic and emotional prosodic perception: a meta-analysis of the lesion literature. Neuropsychologia 49, 3722–3738 (2011). source
- Not matchedcites [68] Yuan, H. et al. Meta-analysis of tau genetic polymorphism and sporadic progressive supranuclear palsy susceptibility. Neural Regen. Res. 6, 353–359 (2011).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [71] Zhu, Y., He, Z. Y. & Liu, H. N. Meta-analysis of the relationship between homocysteine, vitamin B(12), folate, and multiple sclerosis. J. Clin. Neurosci. 18, 933–938 (2011). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If we exclude these 'outlying' meta-analyses, the median statistical power falls to 18%.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Results.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 19
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Such practices include using flexible study designs and flexible statistical analyses and running small studies with low statistical power 1,5 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
What the sources say
- Supportedcites [1] Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power. source
Corollary 4 says that flexibility in study designs and analytical methods makes findings less likely to be true. Corollary 1 says that smaller studies have lower statistical power and that the probability a finding is true falls as power decreases. Together, these support the practices attributed to the paper. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites [5] Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“It has been claimed and demonstrated that many (and possibly most) of the conclusions drawn from biomedical research are probably false 1 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
What the sources say
- Supportedcites [1] Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power. source
The Abstract says there is increasing concern that most published research findings are false and that simulations show a research claim is more likely to be false than true for most study designs and settings. The paper’s discussion of biomedical research makes the citing sentence a fair paraphrase of its argument. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, publication bias, selective data analysis and selective reporting of outcomes are more likely to affect low-powered studies.”
- Also stated in: § Low power in the presence of other biases · hedged
“Publication bias and selective reporting of outcomes and analyses are also more likely to affect smaller, underpowered studies 17 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Low power in the presence of other biases
What the sources say
- Partly supportedcites [17] Dwan, K. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081 (2008). sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) reports that positive or statistically significant studies are more likely to be published and significant outcomes more likely to be fully reported. The paper does not establish that publication bias or selective reporting is more likely in smaller, low-powered studies; the Results mention power calculations but do not make that comparison. Its Introduction describes selective reporting of analyses but does not assess whether it varies by study power. (LLM, conf=0.83)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Low power in the presence of other biases · hedged
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although previously almost all of the proposed candidate gene associations from small studies were false 99 (with some exceptions 100 ), collaborative consortia have substantially improved power, and the replicated results can be considered highly reliable.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Incentivizing replication.
What the sources say
- Not checked — no full text of the sourcecites [99] Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011). source
- Partly supportedcites [100] Siontis, K. C., Patsopoulos, N. A. & Ioannidis, J. P. Replication of past candidate loci for common diseases and phenotypes in 100 genome-wide association studies. Eur. J. Hum. Genet. 18, 832–837 (2010). sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, these policies are uncommonly adhered to 95 , and thus the ability for independent experts to repeat published analysis remains low 96 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Improving availability of materials and data.
What the sources say
- Supportedcites [95] Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H. & Ioannidis, J. P. Public availability of published research data in high-impact journals. PLoS ONE 6, e24357 (2011). source
The Discussion states that more than half of publications subject to journal data-availability instructions did not adhere to them. This supports the policy-adherence clause attributed to this source; the separate claim about repeating analyses carries citation 96. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [96] Ioannidis, J. P. et al. Repeatability of published microarray gene expression analyses. Nature Genet. 41, 149–155 (2009). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that the median statistical power for the water maze studies and the radial maze studies to detect these medium to large effects was 18% and 31%, respectively (TABLE 2).”
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average sample size in these studies was 22 animals for the water maze and 24 for the radial maze experiments.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
plainn = 22- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In order to achieve 80% power to detect, in a single study, the most probable true effects as indicated by the meta-analysis, a sample size of 134 animals would be required for the water maze experiment (assuming an effect size of d = 0.49) and 68 animals for the radial maze experiment (assuming an effect size of d = 0.69); to achieve 95% power, these sample sizes would need to increase to 220 and”
§ Power failure: why small sample size undermines the reliability of neuroscience > Ethical implications.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the total numbers of animals actually used in the studies contributing to the meta-analyses were even larger: 420 for the water maze experiments and 514 for the radial maze experiments.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Ethical implications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The meta-analyses (n = 7) that comprised studies with high average power in excess of 90% had their broadly neurological subject matter in common.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
plaincorrelationaln = 7- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, low power, by definition, means that the chance of discovering effects that are genuinely true is low.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Given that these publishing biases are pervasive across scientific practice, it is possible that false positives heavily contaminate the neuroscience literature as well, and this problem may affect at least as much, if not even more so, the most prominent journals 9,10 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Abstract
What the sources say
- Not checked — no full text of the sourcecites [9] Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855–3859 (2011). source
- Not checked — no full text of the sourcecites [10] Munafo, M. R., Stothart, G. & Flint, J. Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119–120 (2009). source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pre-registration -and, ultimately, full reporting of analysis plans -clarifies the distinction between confirmatory and exploratory analysis, encourages well-powered studies (at least in the case of confirmatory analyses) and reduces the file-drawer effect.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Registration of confirmatory analysis plan.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The principles of the 'three Rs' in animal research (reduce, refine and replace) 83 require appropriate experimental design and statistics -both too many and too few animals present an issue as they reduce the value of research outputs.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Ethical implications.
What the sources say
- Not matchedcites [83] Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Methuen, 1958).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All of the articles that were identified via this electronic search were screened independently for suitability by two authors (K.S.B. and M.R.M.).”
§ Power failure: why small sample size undermines the reliability of neuroscience > Methods.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Low power therefore has an ethical dimensionunreliable research is inefficient and wasteful.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Ethical implications.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The results described in this section are based on only two meta-analyses, and we should be appropriately cautious in extrapolating from this limited evidence.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, it is notable that the results are so consistent with those observed in other fields, such as the neuroimaging and neuroscience studies that we have described above.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“False positives occur more frequently and go unnoticed when degrees of freedom in data analysis and reporting are undisclosed 5 .”
§ Power failure: why small sample size undermines the reliability of neuroscience > Conclusions and future directions
What the sources say
- Not checked — no full text of the sourcecites [5] Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Making research materials available will improve the quality of studies aimed at replicating and extending research findings.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Improving availability of materials and data.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Data extraction was performed independently by K.S.B. and M.R.M. or C.M. and verified collaboratively.”
- Also stated in: § Figure 2 | Flow diagram of articles selected for inclusion. · plain
“Data extraction was independently performed by K.S.B. and either M.R.M. or C.M. and verified collaboratively.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Figure 2 | Flow diagram of articles selected for inclusion.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Figure 2 | Flow diagram of articles selected for inclusion. · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Meta-analysis provides the best estimate of the true effect size, albeit with limitations, including the limitation that the individual studies that contribute to a meta-analysis are themselves subject to the problems described above.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Empirical evidence from neuroscience
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we have already discussed, several other biases are also likely to reduce the probability that a research finding reflects a true effect.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Implications for the likelihood that a research finding reflects a true effect.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Inflated effect estimates make it difficult to determine an adequate sample size for replication studies, increasing the probability of type II errors.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Null findings locked in file drawers bias the literature, whereas exploratory analyses are only useful and valid if you acknowledge the caveats and limitations.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Disclose methods and findings transparently
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assuming, as one might in a power calculation, that the initially observed effect we are trying to replicate reflects the true effect, the potential distribution of these replication effect estimates would be similar to the distribution of the original study (dashed green curve).”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: The initially observed effect that a replication study attempts to replicate reflects the true effect.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some fields are confronting the problem of the poor reliability of research findings that arises from low-powered studies.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Concluding remarks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can increase the power of the replication study (coloured area under the orange curve) by increasing the sample size so as to reduce the sem.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, in genetic epidemiology sample sizes increased dramatically with the widespread understanding that the effects being sought are likely to be extremely small.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Concluding remarks.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If a study finds evidence for an effect at p = 0.05, then the difference between the mean of the null distribution (indicated by the solid blue curve) and the mean of the observed distribution (dashed blue curve) is 1.96 × sem.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Making raw data available will improve data aggregation methods and confidence in reported results.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Improving availability of materials and data.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Combining data increases the total sample size (and therefore power) while minimizing the labour and resource impact on any one contributor.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast, large, high-powered studies can readily detect both small and large effects and so are less biased, as both over-and underestimations of the true effect size will pass the threshold for 'discovery'.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The winner's curse refers to the phenomenon that studies that find evidence of an effect often provide inflated estimates of the size of that effect.”
§ Power failure: why small sample size undermines the reliability of neuroscience > Work collaboratively to increase power and replicate findings
What the sources say
- Not matchedcites REF. 103
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The authors state this as an assumption.
The summary effect size reported in each study is close to the true effect size.
“Importantly, these calculations assume that the summary effect size reported in each study is close to the true effect size, but it is likely that they are inflated owing to publication and other biases described above.”
The authors state this as an assumption.
The initially observed effect we are trying to replicate reflects the true effect.
“Assuming, as one might in a power calculation, that the initially observed effect we are trying to replicate reflects the true effect, the potential distribution of these replication effect estimates would be similar to the distribution of the original study (dashed green curve).”
The paper relies on this without support.
The 49 meta-analyses included are representative of neuroscience research.
“Our results indicate that the median statistical power in neuroscience is 21%.”
The paper relies on this without support.
Most structural and volumetric MRI studies are very small and have minimal power to detect differences between compared groups.
“Most structural and volumetric MRI studies are very small and have minimal power to detect differences between compared groups (for example, healthy people versus those with mental health diseases).”
The paper relies on this without support.
The effects now sought in neuroscience experiments are often smaller and more subtle than before.
“A consequence of the remarkable growth in neuroscience over the past 50 years has been that the effects we now seek in our experiments are often smaller and more subtle than before as opposed to when mostly easily discernible 'low-hanging fruit' were targeted.”
The paper relies on this without support.
Computational analysis of very large datasets is now relatively straightforward, so that an enormous number of tests can be run in a short time on the same dataset.
“At the same time, computational analysis of very large datasets is now relatively straightforward, so that an enormous number of tests can be run in a short time on the same dataset.”
The paper relies on this without support.
These dramatic advances in the flexibility of research design and analysis have occurred without accompanying changes to other aspects of research design, particularly power.
“These dramatic advances in the flexibility of research design and analysis have occurred without accompanying changes to other aspects of research design, particularly power.”
The paper relies on this without support.
Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings.
“Low power is a problem in practice because of the normative publishing standards for producing novel, significant, clean results and the ubiquity of null hypothesis significance testing as the means of evaluating the truth of research findings.”
The authors state this as an assumption.
The initially observed effect that a replication study attempts to replicate reflects the true effect.
“Assuming, as one might in a power calculation, that the initially observed effect we are trying to replicate reflects the true effect”
The paper relies on this without support.
In an exploratory research field such as much of neuroscience, the pre-study odds are often low.
“Notably, in an exploratory research field such as much of neuroscience, the pre-study odds are often low.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Due to differences in age and sample size between protocols and phases, effect sizes could not be directly compared (Button et al., 2013), but the directions of associations with age were largely consistent for ADNI2 and ADNI3 phases separately, and each ADNI3 protocol (Figures 1, 2).”
From Diffusion MRI Indices and Their Relation to Cognitive Impairment in Brain Aging: The Updated Multi-protocol Approach in ADNI3 (2019) · cites this paper as Button et al., 2013
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“but not very different from studies in neuroscience in general (average power = 21%) (14).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, the sample size is small, which precludes the detection of small or medium-sized effects.43”
From Effects of Insufficient Sleep on Pituitary–Adrenocortical Response to CRH Stimulation in Healthy Men (2017) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The lower response rate at week 26 may also result in inflated effects sizes and lower reproducibility of results [63].”
From Benchmark findings from a veteran electronic patient-reported outcomes evaluation from a chronic pain management telehealth program (2024) · cites this paper as [63]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may benefit studies in particular that focus on neuroimaging, which are costly and tend to have small sample sizes (Button et al. 2013), those that employ electrophysiological techniques that might be extremely time-consuming, or those that include genetics.”
From New directions in hypnosis research: strategies for advancing the cognitive and clinical neuroscience of hypnosis (2017) · cites this paper as Button et al. 2013
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The current study is limited in that the sample sizes and sample demographics available for each protocol vary, complicating direct comparison of the protocols (Button et al., 2013).”
From Diffusion MRI Indices and Their Relation to Cognitive Impairment in Brain Aging: The Updated Multi-protocol Approach in ADNI3 (2019) · cites this paper as Button et al., 2013
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This might indicate that results from previous studies do not always reflect a true effect (Button et al. 2013)”
From Terror Management Theory in the Consumer Domain: A Systematic Review and Meta‐Analysis on Mortality Salience Driven Consumer Responses (2025) · cites this paper as Button et al. 2013
The cited source says what the claim attributes to it.
In the citing paper
“Another limitation was the small sample size, due to the duration of the project (just over a year) and the availability of families to take their children for evaluation within this period. This could have reduced the statistical power of the study, potentially lowering the probability of detecting a true effect if one exists and reducing the likelihood that a statistically significant result”
From Enhancing Ecological Validity: Virtual Reality Assessment of Executive Functioning in Children and Adolescents with ADHD (2024) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Button et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“Given the tendency for small sample sizes to produce inflated effect size estimates (Button et al., 2013), future studies should include larger samples in order to produce more precise effect size estimates.”
From Working Memory Training Does Not Improve Performance on Measures of Intelligence or Other Measures of “Far Transfer”: Evidence From a Meta-Analytic Review (2016) · cites this paper as Button et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“Fourth, due to the typical practices in neuroscience research of obtaining and relying on small and underpowered samples (Button et al., 2013).”
From Is the Putative Mirror Neuron System Associated with Empathy? A Systematic Review and Meta-Analysis (2021) · cites this paper as Button et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“Sample sizes of included studies were, on average, relatively small, which is consistent with the recent findings of Button et al. (2013) that suggest such sample sizes are typical in neuroscience research.”
From Is the Putative Mirror Neuron System Associated with Empathy? A Systematic Review and Meta-Analysis (2021) · cites this paper as Button et al. (2013)
The cited source says what the claim attributes to it.
In the citing paper
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“An important qualification about this result is the possibility that the original studies have inflated effect sizes due to publication, selection, reporting, or other biases ( 9, 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“There is good evidence that low power of research designs combined with publication bias favoring positive results over negative results produces a literature with upwardly biased effect sizes ( 14, 16, 30, 31 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [14]
The cited source says what the claim attributes to it.
In the citing paper
“and has contributed, in part, to the replication crisis in other fields (Button et al., 2013; Szucs and Ioannidis, 2017; Ioannidis et al., 2017).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Button et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“when significant effects are found, they are likely to be noisier and have lower positive predictive value (Button et al., 2013).”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Button et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“Low power has three effects: first, within the experiment, real effects are more likely to be missed; second, when an effect is detected, this will often be an overestimation of the true effect size [24]; and finally, when low power is combined with publication bias, there is an increase in the false positive rate in the published literature [36].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [36]
The cited source says what the claim attributes to it.
In the citing paper
“The reproducibility of scientific findings has been called into question.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as [6]
In the citing paper
“Furthermore, sample sizes are often underpowered (4-6), at times leading to conclusions that do not hold up with larger-sized replications or metaanalyses (7).”
From Generic language in scientific communication (2019) · cites this paper as [5]
In the citing paper
“Testing more rather than fewer participants will make inferences more robust and more widely applicable as this can reduce random variation, increasing the probability that a statistically significant result reflects a true effect also likely to occur in retesting with similar groups [5].”
From Hasty generalizations and generics in medical research: A systematic review (2024) · cites this paper as [5]
In the citing paper
“Less clear, perhaps, is that low power can also increase the false discovery rate and the likelihood that reported effect sizes are inflated, due to their reduced ability to mute stochastic noise [13,63,64].”
From The natural selection of bad science (2016) · cites this paper as [63]
In the citing paper
“For most published research findings to be false in psychology and neuroscience, for example, one must assume that the hypotheses tested in these disciplines are correct much less than 50% of the time (14, 18).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [18]
In the citing paper
“Indeed, power analyses show that resting-state functional connectivity for individual links in people is not robust with small samples (Button et al., 2013; Poldrack, 2019), especially in populations with psychiatric disorders, as we also have found.”
From The orbitofrontal cortex: reward, emotion and depression (2020) · cites this paper as Button et al., 2013
In the citing paper
“For example, in neuroscience, even with lenient assumptions, the average power of an experiment is about 20%13,14.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [13]
In the citing paper
“Overall the sample size was relatively small, which may limit reproducibility of results (Button et al., 2013), but was comparable to other relevant fMRI studies (Christakou et al., 2013, Richey et al., 2012, Rubia et al., 2009b).”
From Differentiating neural reward responsiveness in autism versus ADHD (2014) · cites this paper as Button et al., 2013
In the citing paper
“even if a lack of statistical power is a major cause of concern in neuroscience and may be imputed for several studies (Button et al., 2013)”
From What visual illusions teach us about schizophrenia (2014) · cites this paper as Button et al., 2013
In the citing paper
“Many studies included relatively small sample sizes potentially limiting the ability to observe true effects (Button et al., 2013).”
From Neural networks involved in adolescent reward processing: an activation likelihood estimation meta-analysis of functional neuroimaging studies (2015) · cites this paper as Button et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Power failure: why small sample size undermines the reliability of neuroscience > Animal model studies.
“Studies of this size can only detect very large effects (d = 1.20 for n = 22, and d = 1.26 for n = 24) with 80% power”
§ Power failure: why small sample size undermines the reliability of neuroscience > Ethical implications.
“the average sample size of 22 animals for the water maze experiments was only sufficient to detect an effect size of d = 1.26 with Table 1 (cont.)| Characteristics of included meta-analyses Study k N Summary effect size Power Refs Median (range) Cohen's d OR Random or fixed effects Median (range) Yang 3 51 (18-205) 0.67 NA 0.65 (0.27-1.00) 67 Yuan 14 116.5 (19-1178) 4.98 Fixed 0.92”
Cited works that were retracted
Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.
A note, not an issue: for example, a cited work that was retracted only after this paper was published, which its authors could not have known.
In the paper
“Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855–3859 (2011).”
Cited work 10.1128/iai.05661-11 · retraction date not known · this paper published 2013-04-10
Citations
Cites
1 retracted
Retracted Science and the Retraction Index
Fang, F. C., & Casadevall, A. (2011) · Infection and Immunity · doi:10.1128/iai.05661-11 · Cited by 405
RetractedFaul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationImproving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
The Principles of Humane Experimental Technique
(1960) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1960.tb73127.x · Cited by 5,045
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationBelief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Research works printed but not linked: 2
- David, S. P. et al. Potential reporting bias in small fMRI studies of the brain. PLoS Biol. (in the press).
- Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011). · DOI · Paper page
Cited by
Sorted by citations
Items 1,401–1,500 of 7,720
Hudson, N. W., Lucas, R. E., & Donnellan, M. B. (2019) · Personality and Social Psychology Bulletin · doi:10.1177/0146167219867960 · Cited by 48
“…given the small sample size of divorced individuals, the probability that this statistically significant effect reflects a true population effect (i.e., its positive predictive value) is low, and our study has likely dramatically overestimated the true parameter’ size (see Button et al., 2013).”
BackgroundWyatt, T. D. (2020) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2019.0262 · Cited by 52
“The second is small-scale experiments with low statistical power, with little chance of finding a nominally statistically significant finding that actually reflects a true effect and that when a true effect is discovered gives an exaggerated estimate [54].”
BackgroundFu, Q., Hoijtink, H., & Moerbeek, M. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01408-1 · Cited by 47
“with respect to insufficient power is steadily increasing (Maxwell, 2004; Button et al., 2013; Simonsohn et al., 2014). Numerous articles have criticized the Neyman–Pearson approach to hypothesis testing in the classical framework (e.g., Cohen (1994), Nickerson (2000), Sellke et al.”
BackgroundMethodInfluentialFeature—Is a Replicability Crisis on the Horizon for Environmental and Resource Economics?
Ferraro, P. J., & Shukla, P. (2020) · Review of Environmental Economics and Policy · doi:10.1093/reep/reaa011 · Cited by 52
Rocque, C. L. L., Mazurka, R., Stuckless, T. J. R., Pyke, K., & Harkness, K. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.10.067 · Cited by 51
Ni, H., Chen, Y., Chao, Y., Wu, C., Wu, Y., Liang, S. H., Chin, W., Chou, T., Gau, S. S., Huang, Y., et al. (2021) · Autism · doi:10.1177/1362361321990534 · Cited by 51
“…may be partially explained by the fact that this study with a larger sample size, compared to those low-powered studies (Barahona-Correa et al., 2018; Enticott et al., 2014), was less susceptible to exaggeration of effect size from inter-individual random variation (Button et al., 2013).”
ResultKjeldgaard, M. K., Hewlett, J. A., & Eubanks, M. D. (2021) · Ecological Monographs · doi:10.1002/ecm.1451 · Cited by 53
Dubreucq, J., Haesebaert, F., Plasse, J., Dubreucq, M., & Franck, N. (2022) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-021-05058-w · Cited by 49
Zekelman, L., Zhang, F., Makris, N., He, J., Chen, Y., Xue, T., Liera, D., Drane, D. L., Yogesh Rathi, A., Golby, A. J., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118739 · Cited by 51
“Small sample sizes are known to be particularly problematic for neuroimaging and psychology studies because they produce unreliable results (Button et al., 2013; Maxwell et al., 2015).”
Tan, J., Cheng, L. J., Chan, E., Lau, Y., & Lau, S. T. (2022) · Sleep Medicine · doi:10.1016/j.sleep.2022.01.013 · Cited by 52
The Potential for Outdoor Nature-Based Interventions in the Treatment and Prevention of Depression
Owens, M., & Bunce, H. L. I. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.740210 · Cited by 55
Biological markers and psychosocial factors predict chronic pain conditions
Fillingim, M., Tanguay-Sabourin, C., Parisien, M., Zare, A., Guglietti, G. V., Norman, J., Petre, B., Bortsov, A. V., Ware, M. A., Pérez, J., et al. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02156-y · Cited by 61
“Other limiting factors hindering the identification of reliable biomarkers for pain include small sample sizes 5 , lack of dataset generalizability 6 , restricted availability of biological features and the investigation of a limited range of pain phenotypes 7 .”
Neural markers of individual differences in decision-making
Kable, J. W., & Levy, I. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.08.004 · Cited by 46
Detante, O., Jaillard, A. S., Moisan, A., Barbieux, M., Favre, I., Garambois, K., Hommel, M., & Rémy, C. (2014) · Revue Neurologique · doi:10.1016/j.neurol.2014.10.005 · Cited by 53
“[91] Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al.”
BackgroundImproving functional magnetic resonance imaging reproducibility
Pernet, C. R., & Poline, J. (2015) · GigaScience · doi:10.1186/s13742-015-0055-8 · Cited by 47
“Recently, a number of authors have questioned the validity of many findings in epidemiology or in neuroscience [4,5].”
BackgroundKishi, T., Matsunaga, S., & Iwata, N. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbw043 · Cited by 54
Seven Pervasive Statistical Flaws in Cognitive Training Interventions
Moreau, D., Kirk, I. J., & Waldie, K. E. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00153 · Cited by 54
Magnetic Resonance Spectroscopy in Patients with Insomnia: A Repeated Measurement Study
Spiegelhalder, K., Regen, W., Nissen, C., Feige, B., Baglioni, C., Riemann, D. W., Hennig, J., & Lange, T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0156771 · Cited by 53
“This, however, results in a rather low power for the between-subject analyses, which, unfortunately, has been identified as a ubiquitous limitation in neuroscience [56].”
BackgroundTrautwein, F., Singer, T., & Kanske, P. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhw225 · Cited by 48
“Note that the only study which explicitly addressed overlap on a neural level (Fan et al. 2005) relied on a sample size of n = 16, and it is possible that overlap may be detected in larger samples with higher statistical power (Button et al. 2013; Friston 2013; Ingre 2013; Lindquist et al. 2013).”
BackgroundMethodFunctional connectivity of hippocampal subregions in PTSD: relations with symptoms
Malivoire, B. L., Girard, T. A., Patel, R., & Monson, C. M. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1716-9 · Cited by 52
“To address reproducibility concerns, we propose future pre-registration of larger-scale, collaborative replications [52, 53].”
Tschentscher, N., Ruisinger, A., Blank, H., Díaz, B., & von Kriegstein, K. (2019) · Journal of Neuroscience · doi:10.1523/jneurosci.1435-18.2018 · Cited by 50
“However, whether the effects reported here observed in two sensory domains are inflated due to a modest sample size (Button et al., 2013) can only be answered by future (large-sample) DTI studies on developmental disorders.”
BackgroundHandedness and sex effects on lateral biases in human cradling: Three meta-analyses
Packheiser, J., Schmitz, J., Berretz, G., et al. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.06.035 · Cited by 49
Practical Considerations for Navigating Registered Reports
Kiyonaga, A., & Scimeca, J. M. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.07.003 · Cited by 46
Beraldo, F., Palmer, D., Memar, S., Wasserman, D., Lee, W.-J. V., Liang, S., Creighton, S., Kolisnyk, B., Cowan, M. F., Mels, J., et al. (2019) · eLife · doi:10.7554/elife.49630 · Cited by 52
McDonough, I. M., Festini, S. B., & Wood, M. M. (2020) · Ageing Research Reviews · doi:10.1016/j.arr.2020.101133 · Cited by 52
Moseley, P., Aleman, A., Allen, P., Bell, V., Bless, J., Bortolon, C., Cella, M., Garrison, J., Hugdahl, K., Kozáková, E., et al. (2021) · Psychological Science · doi:10.1177/0956797620985832 · Cited by 46
Relationship between the Implicit Association Test and intergroup behavior: A meta-analysisPreprint
Kurdi, B., Seitchik, A. E., Axt, J. R., Carroll, T. J., Karapetyan, A., Kaushik, N., Tomezsko, D., Greenwald, A. G., & Banaji, M. R. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/582gh · Cited by 53
Sundermann, B., Bode, J., Lueken, U., Westphal, D., Gerlach, A. L., Straube, B., Wittchen, H.-U., Ströhle, A., Wittmann, A., Konrad, C., et al. (2017) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2017.00099 · Cited by 46
“It is not possible to identify unequivocally the reasons why our diagnostic approach has failed, nonetheless, we believe that it is important to report this negative finding since the methodologically diverse field of diagnostic MVPA in mental disorders is particularly susceptible to publication bias (7, 78, 79).”
BackgroundChen, Y., Juhás, M., Greenshaw, A. J., Hu, Q., Meng, X., Cui, H., Ding, Y., Kang, L., Zhang, Y., Wang, Y., et al. (2016) · Neuroscience Letters · doi:10.1016/j.neulet.2016.04.030 · Cited by 47
Challenging the challenge hypothesis on testosterone in fathers: Limited meta-analytic support
Meijer, W. M., van IJzendoorn, M. H., & Bakermans - Kranenburg, M. J. (2019) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2019.104435 · Cited by 49
“Neuroscientific studies have been characterized as underpowered, and have been suggested to suffer from power failure (Button et al., 2013) with high risk of false positive results due to too small sample sizes.”
The Study of Walking, Walkability and Wellbeing in Immersive Virtual Environments
Birenboim, A., Bloom, P. B.-N., Levit, H., & Omer, I. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18020364 · Cited by 46
“This problem may worsen with increasing measurement errors and sample variation [72].”
BackgroundUnderstanding Vulnerability and Adaptation in Early Brain Development using Network Neuroscience
Graham, A. M., Marr, M. C., Buß, C., Sullivan, E. L., & Fair, D. A. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.01.008 · Cited by 50
Rapid online assessment of reading ability
Yeatman, J. D., Tang, K. A., Donnelly, P. M., Yablonski, M., Ramamurthy, M., Karipidis, I. I., Caffarra, S., Takada, M. E., Kanopka, K., Ben‐Shachar, M., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-85907-x · Cited by 54
Hendricks, B. E., Landsman, W. R., & Peña-Romera, F. D. (2021) · The Accounting Review · doi:10.2308/tar-2019-0649 · Cited by 48
Rintala, A., Kossi, O., Bonnechère, B., Evers, L. M., Printemps, E., & Feys, P. (2022) · Disability and Rehabilitation · doi:10.1080/09638288.2022.2140844 · Cited by 49
“This may partly explain also the null findings of several included studies, as a low sample size study has the risk to miss a significant effect [48].”
ResultThe tip of the iceberg: A call to embrace anti-localizationism in human neuroscience research
Noble, S., Curtiss, J. E., Pessoa, L., & Scheinost, D. (2024) · Imaging Neuroscience · doi:10.1162/imag_a_00138 · Cited by 60
“A recent flurry of reports has demonstrated that many typical research findings reflect only modest associations with psychological processes, low power, and poor reliability (e.g., Button et al., 2013; Marek et al., 2022; Noble et al., 2022).”
Botvinik‐Nezer, R., Petre, B., Čeko, M., Lindquist, M. A., Friedman, N. P., & Wager, T. D. (2024) · Nature Communications · doi:10.1038/s41467-024-50103-8 · Cited by 50
Zheng, Q., Wu, H., Yu, Q., Kim, D. H., Lipton, J. H., Angelini, S., Soverini, S., Vivona, D., Takahashi, N., & Cao, J. (2014) · The Pharmacogenomics Journal · doi:10.1038/tpj.2014.54 · Cited by 47
Evidence on the deleterious impact of sustained use of polynomial regression on causal inference
Gelman, A., & Zelizer, A. (2015) · Research & Politics · doi:10.1177/2053168015569830 · Cited by 46
“That is a discontinuity right there, and researchers in various fields (for example, Button et al., 2013) have found evidence that it introduces endogeneity in the forcing variable.”
BackgroundWillingness to pay lip service? Applying a neuroscience-based method to WTP for green electricity
Herbes, C., Friege, C., Baldo, D., & Mueller, K.-M. (2015) · Energy Policy · doi:10.1016/j.enpol.2015.10.001 · Cited by 48
A Tutorial on Hunting Statistical Significance by Chasing N
Szűcs, D. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01444 · Cited by 47
“Also note that larger Ns also boost power which is often very low in psychological and neuroscience research (Sedlmeier and Gigerenzer, 1989; Rossi, 1990; Button et al., 2013).”
BackgroundLongitudinal changes of cortical microstructure in Parkinson's disease assessed with T1 relaxometry
Nürnberger, L., Gracien, R., Hok, P., Hof, S., Rüb, U., Steinmetz, H., Hilker, R., Klein, J. C., Deichmann, R., & Baudrexel, S. (2016) · NeuroImage Clinical · doi:10.1016/j.nicl.2016.12.025 · Cited by 48
“Sample bias and an increased false discovery rate are other problems inherent to small sample studies that potentially limit the generalizability of the results (Button et al., 2013; Friston, 2012).”
Joye, N., Broc, L., Olive, T., & Dockrell, J. E. (2018) · Scientific Studies of Reading · doi:10.1080/10888438.2018.1491584 · Cited by 56
“Meta-analyses provide the possibility of synthesizing findings across studies (Hedges & Cooper, 1994), giving a more accurate indication of effect size (Button et al., 2013).”
MethodOpioidergic Regulation of Emotional Arousal: A Combined PET–fMRI Study
Karjalainen, T., Seppälä, K., Glerean, E., Karlsson, H. K., Lahnakoski, J. M., Nuutila, P., Jääskeläinen, I. P., Hari, R., Sams, M. E., & Nummenmaa, L. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhy281 · Cited by 46
“Finally, although large for a PET–fMRI fusion study, the sample size of 35 is slightly lower than recent recommendations arising from worries about false positive findings in fMRI studies (Button et al. 2013; Poldrack et al. 2017).”
BackgroundHengartner, M. P. (2019) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2019-111161 · Cited by 48
Consequences of measurement error in qPCR telomere data: A simulation study
Nettle, D., Seeker, L. A., Nussey, D. H., Froy, H., & Melissa Bateson, B. (2019) · PLoS ONE · doi:10.1371/journal.pone.0216118 · Cited by 51
“Perhaps less obviously, ‘significant’ results that are found are more likely to represent false positives when measurement error is greater [31].”
BackgroundTechnology Acceptance and Quality of Life among Older People Using a TUI Application
Bong, W. K., Bergland, A., & Chen, W. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16234706 · Cited by 49
“The small sample size limits the precision of estimation, and thus reduces the chance of detecting the true effect of Tangible Cup as statistically significant [52].”
BackgroundAndereggen, L., Frey, J., Andres, R. H., Luedi, M. M., El‐Koussy, M., Widmer, H. R., Beck, J., Mariani, L., Seiler, R. W., & Christ, E. (2021) · Journal of Endocrinological Investigation · doi:10.1007/s40618-021-01569-6 · Cited by 48
“It is conceivable that the smaller sample size of patients with macroadenomas conceals a true effect [57].”
BackgroundKaplan, G., Fine, L., Lukyanov, V., et al. (2021) · Land · doi:10.3390/land10070680 · Cited by 48
“According to the goals set in this study and due to a finite amount of available SAR imagery and ground truth data, all the available data were used to calibrate the empirical models to achieve the models’ maximum reliability and estimation accuracy [104].”
MethodDisclosure versus Recognition: Inferences from Subsequent EventsPreprint
Michels, J. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2324394 · Cited by 48
Ko, W. K. D., & Bézard, E. (2017) · Experimental Neurology · doi:10.1016/j.expneurol.2017.07.020 · Cited by 48
“…in PD can benefit by strictly adhering to randomisation, treatment/assessment blinding and sample size calculated designs (Bebarta et al., 2003; Button et al., 2013; Perel et al., 2007; Van Der Worp et al., 2005), to avoid poor data reproduction and ineffective translational progress (Hackam,…”
BackgroundMethodHow, M.-L., Chan, Y. J., & Cheah, S.-M. (2020) · Sustainability · doi:10.3390/su12156272 · Cited by 51
Individual preparation for collaborative learning: Systematic review and synthesis
Mende, S., Proske, A., & Narciss, S. (2021) · Educational Psychologist · doi:10.1080/00461520.2020.1828086 · Cited by 47
Noble, D. W. A., Pottier, P., Lagisz, M., Burke, S., Drobniak, S. M., O’Dea, R. E., & Nakagawa, S. (2022) · Journal of Experimental Biology · doi:10.1242/jeb.243225 · Cited by 49
Structural insight into the individual variability architecture of the functional brain connectome
Sun, L., Liang, X., Duan, D., Liu, J., Chen, Y., Wang, X., Liao, X., Xia, M., Zhao, T., & Yong He, A. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119387 · Cited by 48
Ten simple rules for good research practice
Schwab, S., Janiaud, P., Dayan, M., Amrhein, V., Panczak, R., Palagi, P. M., Hemkens, L. G., Ramon, M., Rothen, N., Senn, S. J., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010139 · Cited by 53
“The reason is that with low power, studies that due to sampling variation find larger (overestimated) effects are much more likely to be statistically significant than those that happen to find smaller (more realistic) effects [30,32,33].”
BackgroundBrain asymmetries from mid- to late life and hemispheric brain age
Korbmacher, M., van der Meer, D., Beck, D., de Lange, A. G., Eikefjord, E., Lundervold, A., Andreassen, O. A., Westlye, L. T., & Maximov, I. I. (2024) · Nature Communications · doi:10.1038/s41467-024-45282-3 · Cited by 44
Chen, X., Reich, P. B., Taylor, A. R., An, Z., & Chang, S. X. (2024) · Nature Communications · doi:10.1038/s41467-024-53004-y · Cited by 56
Janus kinase-1 and Janus kinase-2 inhibitors for treating myelofibrosis
Martí-Carvajal, A. J., Anand, V., & Solà, I. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010298.pub2 · Cited by 46
Display of individuality in avoidance behavior and risk assessment of inbred mice
Hager, T., Jansen, R. F., Pieneman, A. W., Manivannan, S. N., Golani, I., van der Sluis, S., Smit, A. B., Verhage, M., & Stiedl, O. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00314 · Cited by 51
“In classical behavior tests individual variation, the quantitative measure for observed differences of a particular phenotype, is considered a disadvantage since it negatively affects statistical power and replicability (Button et al., 2013).”
BackgroundResultInfluentialMansell, T., Vuillermin, P., Ponsonby, A., Collier, F., Saffery, R., Barwon Infant Study Investigator Team, & Ryan, J. L. (2016) · Development and Psychopathology · doi:10.1017/s0954579416000183 · Cited by 51
“These are less likely to represent the true nature of exposure–outcome relationships and often overestimate the effect size (Button et al., 2013).”
BackgroundCai, W., Chen, T., Ide, J. S., Li, C. R., & Menon, V. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhw219 · Cited by 47
Network classification with applications to brain connectomics
Relión, J. D. A., Kessler, D. A., Levina, E., & Taylor, S. F. (2019) · The Annals of Applied Statistics · doi:10.1214/19-aoas1252 · Cited by 66
“…lll lllll l l lll llllllllllllllll lllllll lll l l ll ll lllllllllllll ll l ll ll l l l l ll ll llllllllllllll lll ll l l l l l l llllllllllllllllll COBRE UMich C V A cc u r a cy particularly challenging to identify significant variables because of small sample sizes (Button et al., 2013).”
BackgroundGlobally weaker and topologically different: resting-state connectivity in youth with autism
Yerys, B. E., Herrington, J. D., Satterthwaite, T. D., Guy, L., Schultz, R. T., & Bassett, D. S. (2017) · Molecular Autism · doi:10.1186/s13229-017-0156-6 · Cited by 48
“Thus, we avoided the pitfalls of small sample studies [62], while maintaining diagnostic fidelity across”
BackgroundThe effect of sleep restriction on empathy for pain: An fMRI study in younger and older adults
Tamm, S., Nilsonne, G., Schwarz, J., Lamm, C., Kecklund, G., Petrović, P., Fischer, H., Åkerstedt, T., & Lekander, M. (2017) · Scientific Reports · doi:10.1038/s41598-017-12098-9 · Cited by 46
Cognitive and Neural Correlates of Mathematical Giftedness in Adults and Children: A Review
Myers, T., Carey, E., & Szűcs, D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01646 · Cited by 49
“…negatives), exaggerating measured effect sizes associated with statistically significant effects, and having a high false report probability (i.e., the probability that a statistically significant finding is false; Pollard and Richardson, 1987; Button et al., 2013; Szucs and Ioannidis, 2017).”
BackgroundWhen and Why to Replicate: As Easy as 1, 2, 3?
Field, S. M., Hoekstra, R., Bringmann, L. F., & van Ravenzwaaij, D. (2019) · Collabra Psychology · doi:10.1525/collabra.218 · Cited by 45
Contemplative interventions and employee distress: A meta‐analysis
Slemp, G. R., Jach, H. K., Chia, A., Loton, D. J., & Kern, M. L. (2019) · Stress and Health · doi:10.1002/smi.2857 · Cited by 53
Campbell, A. K., Grace, F., Ritchie, L., Beaumont, A. J., & Sculthorpe, N. F. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.00031 · Cited by 47
“Firstly, the majority of both observational and cohort studies recruited only a small number of participants (<20 participants per group) which increases the chances of type 2 error, while also increasing the risk of finding a disproportionately large effect size (Button et al., 2013).”
BackgroundHarty, S., & Kadosh, R. C. (2019) · Psychological Science · doi:10.1177/0956797619856658 · Cited by 43
“Assuming that these effect sizes could be inflated (Button et al., 2013), we used G*Power software (Faul, Erdfelder, Lang, & Buchner, 2007) to determine what sample size we would need to have 80% power to detect a small effect (Cohen’s d = 0.”
MethodHermiller, M. S., Karp, E. L., Nilakantan, A. S., & Voss, J. L. (2019) · Brain and Behavior · doi:10.1002/brb3.1393 · Cited by 46
“However, replication using the same memory outcome is important for a variety of reasons, including testing whether anyone finding is robust to the variety of factors that can differ across experiments and robust given sample size (Button et al., 2013).”
BackgroundBadhwar, A., Collin-Verreault, Y., Orban, P., Urchs, S. G. W., Chouinard, I., Vogel, J. W., Potvin, O., Duchesne, S., & Bellec, P. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.116210 · Cited by 47
How does interpreting experience enhance working memory and short-term memory: A meta-analysis
Wen, H., & Dong, Y. (2019) · Journal of Cognitive Psychology · doi:10.1080/20445911.2019.1674857 · Cited by 57
“…(Cooper et al., 2009), the present meta-analysis was in a better position to overcome weaknesses in primary studies (e.g. insufficient sample size, see Button et al., 2013) and yield a more accurate estimate of the population effect for the issue of an interpreter advantage in WM and STM spans.”
BackgroundResultPower to the People: Power, Negative Results and Sample Size
Gaskill, B. N., & Garner, J. P. (2019) · Journal of the American Association for Laboratory Animal Science · doi:10.30802/aalas-jaalas-19-000042 · Cited by 51
Klink, K., Peter, J., Wyss, P., & Klöppel, S. (2020) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2020.00066 · Cited by 49
“Meta-analyses provide better estimates of true effect size, but these estimates might still be inflated due to different biases (e.g., publication bias; Button et al., 2013).”
BackgroundUsing multiple short epochs optimises the stability of infant EEG connectivity parameters
Haartsen, R., van der Velde, B., Jones, E. J. H., Johnson, M. H., & Kemner, C. (2020) · Scientific Reports · doi:10.1038/s41598-020-68981-5 · Cited by 52
The case for preregistering all region of interest (ROI) analyses in neuroimaging research
Gentili, C., Cecchetti, L., Handjaras, G., Lettieri, G., & Cristea, I. A. (2020) · European Journal of Neuroscience · doi:10.1111/ejn.14954 · Cited by 45
“These have been attributed to small sample sizes and resulting small statistical power to detect effects that are not large (Button et al., 2013), combined with selective reporting biases (Szucs & Ioannidis, 2017) and inadequate analytic choices (Eklund, Nichols, & Knutsson, 2016).”
BackgroundClinical Psychological Science
Wright, A. G. C., & Hallquist, M. N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781316995808.001 · Cited by 47
Palmer, D. D., Dumont, J. R., Dexter, T. D., Prado, M. A. M., Finger, E. C., Bussey, T. J., & Saksida, L. M. (2021) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2021.107443 · Cited by 54
Zhang, W., Kaldewaij, R., Hashemi, M. M., Koch, S. B. J., Smit, A. S., van Ast, V. A., Beckmann, C. F., Klumpers, F., & Roelofs, K. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-01798-0 · Cited by 49
“, N between 15 and 50) [22–24] were likely statistically underpowered according to recent guidelines [25, 26] and therefore require replications with large sample sizes.”
BackgroundMigueis, D. P., Lopes, M. C., Casella, E. B., Soares, P. V., Soster, L. A., & Spruyt, K. (2023) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2023.101770 · Cited by 51
Moreno-López, L., Perales, J. C., Son, D. V., Albein‐Urios, N., Soriano‐Mas, C., Martínez-González, J. M., Wiers, R. W. H. J., & Verdejo‐García, A. (2014) · Addiction Biology · doi:10.1111/adb.12143 · Cited by 46
Spalek, K., Fastenrath, M., Ackermann, S., Auschra, B., Coynel, D., Frey, J. N., Gschwind, L., Hartmann, F., van der Maarel, N., Papassotiropoulos, A., et al. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.2384-14.2015 · Cited by 49
“Small sample size has been identified as an issue undermining the reliability of findings in neuroscience (Ioannidis, 2008; Button et al., 2013).”
BackgroundAlfarra, R., Fins, A. I., Chayo, I., & Tartar, J. L. (2014) · Biological Psychology · doi:10.1016/j.biopsycho.2014.11.001 · Cited by 47
Querné, L., Fall, S., Moing, A. G. L., Bourel‐Ponchel, E., Delignières, A., Simonnot, A., de Broca, A., Gondry‐Jouet, C., Boucart, M., & Berquin, P. C. (2014) · Journal of Attention Disorders · doi:10.1177/1087054713517542 · Cited by 46
“Studies with low statistical power have a reduced chance of detecting a true effect (false negative bias), but also reduces the likelihood that a statistically significant result reflects a true effect (false positive bias; for meta-analysis, see Button et al., 2013; David et al., 2013).”
BackgroundHupé, J. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00018 · Cited by 44
“Most studies in neuroimaging (Yarkoni et al., 2010) and even neurosciences (Button et al., 2013) are underpowered.”
BackgroundKnecht, S., Roßmüller, J., Unrath, M., Stephan, K., Berger, K., & Studer, B. (2015) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp-2015-310344 · Cited by 44
Intact Ventral Striatal Prediction Error Signaling in Medicated Schizophrenia Patients
Culbreth, A. J., Westbrook, A., Xu, Z., Barch, D. M., & Waltz, J. A. (2016) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2016.07.007 · Cited by 44
“These small samples are problematic because positive findings with small samples represent estimates of effect sizes with high uncertainty (27).”
BackgroundLee, H., & Park, S. (2016) · Journal of Health Communication · doi:10.1080/10810730.2016.1245801 · Cited by 46
“A follow-up study with a larger sample may help determine whether the relatively low statistical power of this study negatively affects the likelihood that a statistically significant finding actually reflects a true effect (Button et al., 2013).”
BackgroundSullivan, S. A., Thompson, A., Kounali, D., Lewis, G., & Zammit, S. (2017) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-017-1359-z · Cited by 47
“The sample size was also smaller than ours (n = 89), and it is increasingly recognised that very small sample sizes can be associated with effect inflation [57].”
BackgroundLeeners, B., Kruger, T. H., Geraedts, K., Tronci, E., Mancini, T., Ille, F., Egli, M., Röblitz, S., Saleh, L., Spanaus, K., et al. (2017) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2017.00120 · Cited by 54
“As comprehensively reviewed by Button et al. (2013), underpowered small samples substantially undermine the reliability of research findings by producing severely inflated effect sizes and both false-positive and false-negative results.”
BackgroundTakeuchi, H., Taki, Y., Nouchi, R., Yokoyama, R., Kotozaki, Y., Nakagawa, S., Sekiguchi, A., Iizuka, K., Yamamoto, Y., Hanawa, S., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-10104-8 · Cited by 46
Plimmer, G., Bryson, J., & Teo, S. (2017) · Personnel Review · doi:10.1108/pr-10-2016-0275 · Cited by 47
“The small sample size means that these exploratory results need to be treated with considerable caution (Button et al., 2013), but we believe they provide useful insights for future research.”
BackgroundThe Prior Odds of Testing a True Effect in Cognitive and Social Psychology
Wilson, B. M., & Wixted, J. T. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918767122 · Cited by 47
“As it turns out, if researchers in a field typically run low-powered experiments, the significant effects reported in the field’s scientific journals will often be false positives, not real effects with large effect sizes (Button et al., 2013). The probability that a significant effect is real is known as the positive…”
BackgroundInfluentialA Big Data Analytics Approach for Construction Firms Failure Prediction Models
Alaka, H., Oyedele, L. O., Owolabi, H. A., Akinadé, O. O., Bilal, M., & Ajayi, S. O. (2018) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2018.2856376 · Cited by 46
“Since larger sample size increases reliability [44], option one was nullified.”
BackgroundMultilevel Surgery for Children With Cerebral Palsy: A Meta-analysis
Amirmudin, N. A., Lavelle, G., Theologis, T. N., Thompson, N., & Ryan, J. M. (2019) · PEDIATRICS · doi:10.1542/peds.2018-3390 · Cited by 47
“Sample sizes may have impacted results by either inflating effect sizes or resulting in statistically nonsignificant findings.(101,102)”
BackgroundPérez-Báez, J., Risco, C. A., Chebel, R. C., et al. (2019) · Journal of Dairy Science · doi:10.3168/jds.2018-15879 · Cited by 46
Puoliväli, T., Palva, S., & Palva, J. M. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108654 · Cited by 47
Smith, E., Hokstad, S., & Næss, K.-A. B. (2020) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2020.105992 · Cited by 46
“It is valuable to aggregate studies for a greater level of power to investigate overall effects when individual studies are small scale (Button et al., 2013), as is commonly the case in the Down syndrome research field.”
Limiting data loss in infant EEG: putting hunches to the test
van der Velde, B., & Junge, C. (2020) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2020.100809 · Cited by 46
Metadata
- Authors
- Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R.
- Affiliations
- University of Bristol (GB)
- Stanford Medicine (US)
- University of Virginia (US)
- Venue
- Nature Reviews Neuroscience
- Published
- 2013-04-10
- Volume
- 14
- Issue
- 5
- Pages
- 365-376
- ISSN
- 1471-003X (ISSN-L)
- 1471-0048
- OpenAlex
- W2105824687
- Semantic Scholar
- c2bc880156c10fc8d1752cd3d528e4ef96cbaac4
- Cited by
- 7,641
- References
- 106
- Citations by year
- 2026: 584 · 2025: 611 · 2024: 606 · 2023: 553 · 2022: 658 · 2021: 692 · 2020: 752 · 2019: 683 · 2018: 679 · 2017: 597 · 2016: 486 · 2015: 352 · 2014: 268 · 2013: 109
- FWCI
- 82.23
- Type
- journal article
- Language
- en
- Update notices
- erratum (2013-04-15)
- Open access
- bronze
- doi.org
- Locations
- Nature reviews. Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Explore Bristol Research (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Wellcome Trust (090532)
- United Kingdom Clinical Research Collaboration
- Cancer Research UK
- National Institute for Health and Care Research
- British Heart Foundation
- Medical Research Council
- Economic and Social Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- statistical power, small sample size, effect size overestimation, reproducibility, research ethics, research waste, research practices, brain research
- MeSH terms
- Humans, Neurosciences, Probability, Reproducibility of Results, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
