A Guide for Calculating Study-Level Statistical Power for Meta-Analyses
Open accessPublished version of a preprint
Abstract
Meta-analysis is a popular approach in the psychological sciences for synthesizing data across studies. However, the credibility of meta-analysis outcomes depends on the evidential value of studies included in the body of evidence used for data synthesis. One important consideration for determining a study’s evidential value is the statistical power of the study’s design/statistical test combination for detecting hypothetical effect sizes of interest. Studies with a design/test combination that cannot reliably detect a wide range of effect sizes are more susceptible to questionable research practices and exaggerated effect sizes. Therefore, determining the statistical power for design/test combinations for studies included in meta-analyses can help researchers make decisions regarding confidence in the body of evidence. Because the one true population effect size is unknown when hypothesis testing, an alternative approach is to determine statistical power for a range of hypothetical effect sizes. This tutorial introduces the metameta R package and web app, which facilitates the straightforward calculation and visualization of study-level statistical power in meta-analyses for a range of hypothetical effect sizes. Readers will be shown how to reanalyze data using information typically presented in meta-analysis forest plots or tables and how to integrate the metameta package when reporting novel meta-analyses. A step-by-step companion screencast video tutorial is also provided…
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Meta-analyses are considered the gold standard for synthesizing empirical research findings.
“Meta‐analyses are considered the goal standard for synthesizing empirical research findings (Quintana, 2023)”
From Review: Social media use and adolescent mental health - an umbrella review and power analysis. (2026) · cites this paper as Quintana, 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283 · see citing works
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1× · see citing works
“In light of these factors, the contribution of low statistical power to the reproducibility crisis in the psychological sciences has become increasingly recognized (Button et al., 2013; Munafò et al., 2017; Walum et al., 2016).”
BackgroundAssessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256 · see citing works
On the practice of dichotomization of quantitative variables.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.19 · Cited by 3,538 · cited 1× · see citing works
“…metrics is the comparison of studies in which researchers dichotomize one continuous variable into two groups— although this practice in most circumstances has been the subject of critique (MacCallum et al., 2002)—with studies that evaluate the relationship between two continuous variables.”
BackgroundEffect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860 · see citing works
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962 · see citing works
Doing Meta-Analysis with R: A Hands-On Guide
Harrer, M., Cuijpers, P., Furukawa, T. A., & Ebert, D. D. (2021) · doi:10.1201/9781003107347 · Cited by 1,850 · see citing works
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819 · see citing works
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · cited 1× · see citing works
“The use of prior effect sizes reported in the literature is one suggestion among others for determining an SESOI (Keefe et al., 2013; Lakens et al., 2018), which will be used an example for how to interpret information provided by the metameta package.”
MethodHow Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294 · see citing works
Interpreting effect sizes of education interventions
Kraft, M. A. (2020) · Educational Researcher · doi:10.3102/0013189x20912798 · Cited by 1,281 · see citing works
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969 · see citing works
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774 · see citing works
<i>p</i> -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717 · see citing works
The prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · see citing works
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461 · cited 1× · see citing works
“Various tools have more recently been developed for detecting and/or correcting for publication bias, such as robust Bayesian meta-analysis (Bartoš et al., 2022), selection models (Maier et al., 2022; Vevea & Woods, 2005), p-curve (Simonsohn et al., 2014), and z-curve (Brunner & Schimmack, 2020).”
MethodBakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2013) · Translational Psychiatry · doi:10.1038/tp.2013.34 · Cited by 383 · see citing works
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 345 · cited 2× · see citing works
“, 2017), which has partly been attributed to study designs with low statistical power (Quintana, 2020; Walum et al., 2016).”
BackgroundIs meta-analysis the platinum standard of evidence?
Stegenga, J. (2011) · Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2011.07.003 · Cited by 193 · see citing works
Ooi, Y., Weng, S.-J., Kossowsky, J., Gerger, H., & Sung, M. (2016) · Pharmacopsychiatry · doi:10.1055/s-0042-109400 · Cited by 131 · see citing works
Brown, N. J. L., & Heathers, J. A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550616673876 · Cited by 127 · see citing works
Alvares, G. A., Quintana, D. S., & Whitehouse, A. J. O. (2017) · Autism Research · doi:10.1002/aur.1692 · Cited by 91 · see citing works
Self-citationEstimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance
Brunner, J., & Schimmack, U. (2020) · Meta-Psychology · doi:10.15626/mp.2018.874 · Cited by 76 · see citing works
Heathers, J. A., Anaya, J., van der Zee, T., & Brown, N. J. (2018) · PeerJ Preprints · doi:10.7287/peerj.preprints.26968v1 · Cited by 44 · cited 2× · see citing works
“Recently developed tools can evaluate the presence of reporting errors, such as GRIM (Brown & Heathers, 2017), SPRITE (Heathers et al., 2018), and statcheck (Nuijten & Polanin, 2020).”
Boen, R., Quintana, D. S., Ladouceur, C. D., & Tamnes, C. K. (2022) · Psychophysiology · doi:10.1111/psyp.14003 · Cited by 40 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 40
- Alexander, R. A., Scozzaro, M. J., & Borodkin, L. J. (1989). Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis. Psychological Bulletin , 106 , 329-331. https://doi.org/10.1037/0033-2909.106.2.329 · DOI
- Barto š , F., Maier, M., Quintana, D. S., & Wagenmakers, E.-J. (2022). Adjusting for Publication Bias in JASP and R: Selection Models, PET-PEESE, and Robust Bayesian Meta-Analysis. Advances in Methods and Practices in Psychological Science , 5 (3), 25152459221109260. https://doi.org/10.1177/25152459221109259 · DOI
- Barto š , F., Maier, M., Quintana, D., & Wagenmakers, E.-J. (2020). Adjusting for Publication Bias in JASP & R - Selection Models, PET-PEESE, and Robust Bayesian Meta-Analysis . PsyArXiv. https://doi.org/10.31234/osf.io/75bqn · DOI
- Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to Meta-Analysis . John Wiley & Sons.
- Borenstien, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2009). Publication Bias. In Introduction to Meta-Analysis (pp. 277-292). John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470743386.ch30 · DOI
- Bryan, C. J., Tipton, E., & Yeager, D. S. (2021). Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature Human Behaviour , 5 (8), 980989. https://doi.org/10.1038/s41562-021-01143-3 · DOI
- Cherubini, J. M., & MacDonald, M. J. (2021). Statistical Inferences Using Effect Sizes in Human Endothelial Function Research. Artery Research , 27 (4), Article 4. https://doi.org/10.1007/s44200-021-00006-6 · DOI
- Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Elm, E. V., Gamble, C., Ghersi, D., Ioannidis, J. P., Simes, J., & Williamson, P. R. (2008). Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias. PLOS ONE , 3 (8), e3081. https://doi.org/10.1371/journal.pone.0003081 · DOI · Paper page
- Gallyer, A. J., Dougherty, S. P., Burani, K., Albanese, B. J., Joiner, T. E., & Hajcak, G. (2021). Suicidal thoughts, behaviors, and event-related potentials: A systematic review and meta-analysis. Psychophysiology , 58 (12), e13939. https://doi.org/10.1111/psyp.13939 · DOI
- Ghai, S. (2021). It's time to reimagine sample diversity and retire the WEIRD dichotomy. Nature Human Behaviour , 5 (8), Article 8. https://doi.org/10.1038/s41562-021-011759 · DOI
- Halpern, S. D., Karlawish, J. H. T., & Berlin, J. A. (2002). The Continuing Unethical Conduct of Underpowered Clinical Trials. JAMA , 288 (3), 358-362. https://doi.org/10.1001/jama.288.3.358 · DOI
- Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in metaanalysis. Psychological Methods , 9 (4), 426-445. https://doi.org/10.1037/1082989X.9.4.426 · DOI
- Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine , 21 (11), 1539-1558. https://doi.org/10.1002/sim.1186 · DOI · Paper page
- Ioannidis, J. P. (2005). Why Most Published Research Findings Are False. PLOS Medicine , 2 (8), e124. https://doi.org/10.1371/journal.pmed.0020124 · DOI · Paper page
- Ioannidis, J. P. (2008). Why most discovered true associations are inflated. Epidemiology , 19 , 640-648. https://doi.org/10.1097/EDE.0b013e31818131e7 · DOI · Paper page
- Jacobs, P., & Viechtbauer, W. (2017). Estimation of the biserial correlation and its sampling variance for use in meta-analysis. Research Synthesis Methods , 8 (2), 161-180. https://doi.org/10.1002/jrsm.1218 · DOI
- Jurek, B., & Neumann, I. D. (2018). The oxytocin receptor: From intracellular signaling to behavior. Physiological Reviews , 98 (3), 1805-1908. https://doi.org/10.1152/physrev.00031.2017 · DOI · Paper page
- Kadlec, D., Sainani, K. L., & Nimphius, S. (2022). With Great Power Comes Great Responsibility: Common Errors in Meta-Analyses and Meta-Regressions in Strength & Conditioning Research. Sports Medicine . https://doi.org/10.1007/s40279-02201766-0 · DOI
- Keech, B., Crowe, S., & Hocking, D. R. (2018). Intranasal oxytocin, social cognition and neurodevelopmental disorders: A meta-analysis. Psychoneuroendocrinology , 87 , 9-19. https://doi.org/10.1016/j.psyneuen.2017.09.022 · DOI · Paper page
- Kossmeier, M., Tran, U. S., & Voracek, M. (2020). Power-enhanced funnel plots for metaanalysis: The sunset funnel plot. Zeitschrift Für Psychologie , 228 (1), 43-49. https://doi.org/10.1027/2151-2604/a000392 · DOI
- Kvarven, A., Strømland, E., & Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour , 4 (4), 423-434. https://doi.org/10.1038/s41562-019-0787-z · DOI · Paper page
- Leng, G., & Leng, R. I. (2021). Oxytocin: A citation network analysis of 10 000 papers. Journal of Neuroendocrinology , e13014. https://doi.org/10.1111/jne.13014 · DOI
- Linden, A. H., & Hönekopp, J. (2021). Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science. Perspectives on Psychological Science , 16 (2), 358-376. https://doi.org/10.1177/1745691620964193 · DOI
- Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis (pp. ix, 247). Sage Publications, Inc.
- Maier, M., & Lakens, D. (2022). Justify Your Alpha: A Primer on Two Practical Approaches. Advances in Methods and Practices in Psychological Science , 5 (2), 25152459221080396. https://doi.org/10.1177/25152459221080396 · DOI
- Maier, M., VanderWeele, T. J., & Mathur, M. B. (2022). Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use. Campbell Systematic Reviews , 18 (3), e1256. https://doi.org/10.1002/cl2.1256 · DOI
- Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour , 1 (1), 0021. https://doi.org/10.1038/s41562-016-0021 · DOI · Paper page
- Nordahl-Hansen, A., Cogo-Moreira, H., Panjeh, S., & Quintana, D. (2022). Redefining Effect Size Interpretations for Psychotherapy RCTs in Depression . OSF Preprints. https://doi.org/10.31219/osf.io/erhmw · DOI
- Nuijten, M. B., & Polanin, J. R. (2020). 'statcheck': Automatically detect statistical reporting inconsistencies to increase reproducibility of meta-analyses. Research Synthesis Methods , 11 (5), 574-579. https://doi.org/10.1002/jrsm.1408 · DOI · Paper page
- Quintana, D. S. (2015). From pre-registration to publication: A nontechnical primer for conducting a meta-analysis to synthesize correlational data. Frontiers in Psychology , 6 , 1549. https://doi.org/10.3389/fpsyg.2015.01549 · DOI
- Quintana, D. S. (2016). Statistical considerations for reporting and planning heart rate variability case-control studies. Psychophysiology , 54 (3), 344-349. https://doi.org/10.1111/psyp.12798 · DOI · Paper page
- Quintana, D. S. (2021). Replication studies for undergraduate theses to improve science and education. Nature Human Behaviour , 1-2. https://doi.org/10.1038/s41562-021-011928 · DOI
- Quintana, D. S., & Guastella, A. J. (2020). An allostatic theory of oxytocin. Trends in Cognitive Sciences , 24 (7), 515-528. https://doi.org/10.1016/j.tics.2020.03.008 · DOI
- Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences , 115 (45), 11401-11405. https://doi.org/10.1073/pnas.1721165115 · DOI · Paper page
- Rochefort-Maranda, G. (2021). Inflated effect sizes and underpowered tests: How the severity measure of evidence is affected by the winner's curse. Philosophical Studies , 178 (1), 133-145. https://doi.org/10.1007/s11098-020-01424-z · DOI · Paper page
- Simes, R. J. & The PPP and CTT Investigators. (1995). Prospective meta-analysis of cholesterol-lowering studies: The prospective pravastatin pooling (PPP) project and the cholesterol treatment trialists (CTT) collaboration. The American Journal of
- Szucs, D., & Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLOS Biology , 15 (3), e2000797. https://doi.org/10.1371/journal.pbio.2000797 · DOI · Paper page
- van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019). Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis. PLoS ONE , 14 (4), e0215052. https://doi.org/10.1371/journal.pone.0215052 · DOI
- Viechtbauer, W. (2010). Conducting Meta-Analyses in R with the metafor Package. Journal of Statistical Software , 36 , 1-48. https://doi.org/10.18637/jss.v036.i03 · DOI · Paper page
- Wagge, J. R., Brandt, M. J., Lazarevic, L. B., Legate, N., Christopherson, C., Wiggins, B., & Grahe, J. E. (2019). Publishing Research With Undergraduate Students via Replication Work: The Collaborative Replications and Education Project. Frontiers in Psychology , 10 . https://doi.org/10.3389/fpsyg.2019.00247 · DOI
Cited by
Sorted by citations
Review: Social media use and adolescent mental health - an umbrella review and power analysis.
Tølbøll, K. B. (2026) · Child and Adolescent Mental Health · doi:10.1111/camh.70071 · Cited by 8 · see citing works
- Authors
- Quintana, D. S.
- Affiliations
- Oslo University Hospital (NO)
- Corresponding author
- Quintana, Daniel S.
- ORCID
- Quintana, Daniel S.
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2023-01
- Volume
- 6
- Issue
- 1
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W4361802251
- Cited by
- 95
- References
- 26
- Citations by year
- 2026: 30 · 2025: 28 · 2024: 16 · 2023: 19 · 2022: 2
- FWCI
- 17.31
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Advances in Methods and Practices in Psychological Science (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- norges forskningsråd (301767; 324783)
- kavlifondet
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- statistical power, meta-analysis, questionable research practices, forest plots, reanalysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
