The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Most meta-analyses include data from one or more small studies that, individually, do not have power to detect an intervention effect. The relative influence of adequately powered and underpowered studies in published meta-analyses has not previously been explored. We examine the distribution of power available in studies within meta-analyses published in Cochrane reviews, and investigate the impact of underpowered studies on meta-analysis results. METHODS AND FINDINGS: For 14,886 meta-analyses of binary outcomes from 1,991 Cochrane reviews, we calculated power per study within each meta-analysis. We defined adequate power as ≥50% power to detect a 30% relative risk reduction. In a subset of 1,107 meta-analyses including 5 or more studies with at least two adequately powered and at least one underpowered, results were compared with and without underpowered studies. In 10,492 (70%) of 14,886 meta-analyses, all included studies were underpowered; only 2,588 (17%) included at least two adequately powered studies. 34% of the meta-analyses themselves were adequately powered. The median of summary relative risks was 0.75 across all meta-analyses (inter-quartile range 0.55 to 0.89). In the subset examined, odds ratios in underpowered studies were 15% lower (95% CI 11% to 18%, P<0.0001) than in adequately powered studies, in meta-analyses of controlled pharmacological trials; and 12% lower (95% CI 7% to 17%, P<0.0001) in meta-analyses of controlled non-pharmacological trials. The standard error of the intervention effect increased by a median of 11% (inter-quartile range -1% to 35%) when underpowered studies were omitted; and between-study heterogeneity tended to decrease. CONCLUSIONS: When at least two adequately powered studies are available in meta-analyses reported by Cochrane reviews, underpowered studies often contribute little information, and could be left out if a rapid review of the evidence is required. However, underpowered studies made up the entirety of the evidence in most Cochrane reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we pooled results excluding studies with fewer than 50 participants in the experimental group, as a planned sensitivity analysis”
From Effectiveness of psychological interventions in prison to reduce recidivism: a systematic review and meta-analysis of randomised controlled trials (2021) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“there is a risk that untested approaches may introduce new forms of bias 18 .”
From Software support for environmental evidence synthesis (2018) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“On collating the evidence, some studies may appear underpowered for the outcome(s) of interest to us (Turner 2013), yet were adequately powered for the studies' primary measure.”
From Physical activity and exercise for chronic pain in adults: an overview of Cochrane Reviews (2017) · cites this paper as Turner 2013
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Although our number of effect sizes (n=37) is relatively large in terms of outcomes analyzed, it is small in terms of statistical analyses that combine empirical studies [100].”
From Generative AI Mental Health Chatbots as Therapeutic Tools: Systematic Review and Meta-Analysis of Their Role in Reducing Mental Health Issues (2025) · cites this paper as [100]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 2×
“In each meta-analysis j, we fitted a random-effects model, using a method-of-moments estimate for the between study variance [17], and calculated the variance Vj of the combined intervention effect (on the log relative risk scale).”
MethodSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · cited 1×
“[24], who published new guidelines.”
Self-citationKjaergard, L. L., Villumsen, J., & Gluud, C. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,283 · cited 1×
“Alternatively, small-study effects may arise from biases caused by methodological flaws arising more frequently in small studies [3], or may be due to true differences in the underlying effects between smaller and larger studies.”
BackgroundPublication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
Wetterslev, J., Thorlund, K., Brok, J., & Gluud, C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.013 · Cited by 1,881
Brok, J., Thorlund, K., Gluud, C., & Wetterslev, J. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.007 · Cited by 1,113 · cited 1×
“[20], who had, however, excluded both reviews with fewer than three trials and those in which all trials had a high risk of bias, in which metaanalytic size is likely to have been smaller.”
ResultTotal Plasma Homocysteine and Cardiovascular Risk Profile: The Hordaland Homocysteine Study
Nygård, O., Vollset, S. E., Refsum, H., Stensvold, I., Tverdal, A., Nordrehaug, J. E., Ueland, P. M., & Kvåle, G. (1995) · JAMA · doi:10.1001/jama.1995.03530190040032 · Cited by 1,042
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897 · cited 1×
“Researchers who choose to undertake a study that is capable of detecting only an unrealistically large effect may lack understanding of both scientific methods and ethics [10].”
BackgroundThe power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716 · cited 1×
“where W is the cumulative standard normal distribution function and Ca=2~W {1 1{a=2 ð Þ[18].”
BackgroundMeta‐analysis of drug‐related deaths soon after release from prison
Merrall, E. L. C., Kariminia, A., Binswanger, I. A., Hobbs, M. S., Farrell, M., Marsden, J., Hutchinson, S. J., & Bird, S. M. (2010) · Addiction · doi:10.1111/j.1360-0443.2010.02990.x · Cited by 674 · cited 1×
“In teaching, Bird has long advocated that trials should not be started unless they could deliver at least 50% power in respect of a priori plausible, worthwhile effect sizes [6].”
BackgroundSelf-citation
Cited by
Physical activity and exercise for chronic pain in adults: an overview of Cochrane Reviews
Geneen, L. J., Moore, R. A., Clarke, C., Martin, D., Colvin, L. A., & Smith, B. H. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011279.pub3 · Cited by 1,222
Trial Sequential Analysis in systematic reviews with meta-analysis
Wetterslev, J., Jakobsen, J. C., & Gluud, C. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0315-7 · Cited by 1,189
“Only 22% of the metaanalyses in The Cochrane Library have 80% power to conclude whether there is an intervention effect of 30% or not when the usual maximal risks of type I error (α) of 5% and type II error (β) of 20% are applied [1].”
BackgroundInfluentialYe, Q., Zou, B., Yeo, Y. H., Li, J., Huang, D. Q., Wu, Y., Yang, H., Liu, C., Kam, L. Y., Tan, X. X. E., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(20)30077-7 · Cited by 925
Physical activity and exercise for chronic pain in adults: an overview of Cochrane Reviews
Geneen, L. J., Moore, R. A., Clarke, C., Martin, D., Colvin, L. A., & Smith, B. H. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011279.pub2 · Cited by 719
Jakobsen, J. C., Wetterslev, J., Winkel, P., Lange, T., & Gluud, C. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-120 · Cited by 704
“Most review authors do not assess if an accrued information size is sufficient or not to detect or reject a given intervention effect, and this is problematic as nearly all meta-analyses in systematic reviews are underpowered (the meta-analyses do not have enough information to reject or accept the null hypothesis whe…”
Background
Metadata
- Authors
- Turner, R. M., Bird, S. M., & Higgins, J. P. T.
- Affiliations
- MRC Biostatistics Unit (GB)
- University of Bristol (GB)
- Corresponding author
- Turner, Rebecca M.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-03-27
- Volume
- 8
- Issue
- 3
- Pages
- e59202
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3609745
- OpenAlex
- W2015824469
- Cited by
- 672
- References
- 25
- Citations by year
- 2026: 46 · 2025: 51 · 2024: 51 · 2023: 59 · 2022: 70 · 2021: 70 · 2020: 59 · 2019: 49 · 2018: 56 · 2017: 60 · 2016: 45 · 2015: 31 · 2014: 20 · 2013: 4
- FWCI
- 36.78
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- UCL Discovery (University College London) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion)
- Funders
- Medical Research Council (U105260794)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- underpowered studies, study size, statistical power, meta-analysis, binary outcomes, relative risk, standard errors, study heterogeneity, rapid review
- MeSH terms
- Humans, Review Literature as Topic, Meta-Analysis as Topic, Odds Ratio, Sample Size, Risk Reduction Behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews. https://app.cochat.ai/papers/10.1371/journal.pone.0059202
- The paper
- Turner, R. M., Bird, S. M., & Higgins, J. P. T. (2013). The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews. PLoS ONE. https://doi.org/10.1371/journal.pone.0059202
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
