Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

Anderson, Samantha F.; Kelley, Ken; Maxwell, Scott E.

doi:10.1177/0956797617723724

Cited by 416Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods in Epidemiology

Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

Anderson, Samantha F., Kelley, Ken, Maxwell, Scott E.·2017·Psychological Science

DOI 10.1177/0956797617723724

Open this paper in CoChat — free account

Abstract

The sample size necessary to obtain a desired level of statistical power depends in part on the population value of the effect size, which is, by definition, unknown. A common approach to sample-size planning uses the sample effect size from a prior study as an estimate of the population value of the effect to be detected in the future study. Although this strategy is intuitively appealing, effect-size estimates, taken at face value, are typically not accurate estimates of the population effect size because of publication bias and uncertainty. We show that the use of this approach often results in underpowered studies, sometimes to an alarming degree. We present an alternative approach that adjusts sample effect sizes for bias and uncertainty, and we demonstrate its effectiveness for several experimental designs. Furthermore, we discuss an open-source R package, BUCSS, and user-friendly Web applications that we have made available to researchers so that they can easily implement our suggested methods.

Integrity signals

Registry

Crossref · 10.1177/0956797617723724

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Psychology studies often have insufficient statistical power.
“Furthermore, sample sizes are often underpowered (4-6), at times leading to conclusions that do not hold up with larger-sized replications or metaanalyses (7).”
From Generic language in scientific communication (2019) · cites this paper as [4]

Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971 · see citing works
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,385 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1× · see citing works
“However, even when researchers conduct a well-intentioned power analysis to determine the sample size of a replication study, the power of the original study limits their ability to determine an accurate sample size, sometimes severely (Anderson & Maxwell, 2016, 2017; Button et al., 2013).”
Background
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
How many subjects does it take to do a regression analysis
Green, S. B. (1991) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr2603_7 · Cited by 3,916 · cited 1× · see citing works
“Strategies based on prior sample size alone (“rules of thumb”) can result in selecting sample sizes that are either too large or too small, depending on the power of the prior studies involved (Green, 1991; see Anderson & Maxwell, 2017, for a simulation).”
Background
Designing Experiments and Analyzing Data: A Model Comparison Perspective
Maxwell, S., & Delaney, H. D. (1990) · Journal of the American Statistical Association · doi:10.2307/2289744 · Cited by 2,593 · see citing works
Self-citation
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
The role and interpretation of pilot studies in clinical research
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.10.008 · Cited by 1,917 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343 · see citing works
Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Designing Experiments and Analysing Data: A Model Comparison Perspective.
Freeman, G. H., Maxwell, S. E., & Delaney, H. D. (1991) · Biometrics · doi:10.2307/2532173 · Cited by 1,229 · see citing works
Self-citation
Design Sensitivity: Statistical Power for Experimental Research.
Eliason, S. R., & Lipsey, M. W. (1991) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2072167 · Cited by 1,051 · see citing works
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,018 · see citing works
Self-citation
On effect size.
Kelley, K., & Preacher, K. J. (2012) · Psychological Methods · doi:10.1037/a0028086 · Cited by 1,002 · see citing works
Self-citation
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 <i>t</i> Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945 · see citing works
How Many Subjects?: Statistical Power Analysis in Research
KRAEMER, H. C., & Blasey, C. M. (2016) · SAGE Publications Ltd eBooks · doi:10.4135/9781483398761 · Cited by 918 · see citing works
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860 · cited 3× · see citing works
“Using Taylor and Muller’s (1996) 50th-percentile method to adjust the sample estimate of 0.”
BackgroundMethodInfluential
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
Sample Size Calculations in Clinical Research
Chow, S., Wang, H., & Shao, J. (2003) · CRC Press eBooks · doi:10.1201/9780203911341 · Cited by 813 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742 · see citing works
Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations
Olejnik, S. F., & Algina, J. J. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.2000.1040 · Cited by 703 · see citing works
The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655 · see citing works
Self-citation
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E., Zehetleitner, M., & Perugini, M. (2015) · Psychological Methods · doi:10.1037/met0000061 · Cited by 562 · see citing works
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542 · see citing works
Self-citation
Performing high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414 · see citing works
The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409 · see citing works
Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant.
Kelley, K., & Maxwell, S. E. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.305 · Cited by 310 · see citing works
Self-citation
Designing Experiments and Analyzing Data: Model Comparison Perspective
Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018) · Thammasat University Digital Collections · Cited by 298 · see citing works
Self-citation
Estimation of effect size under nonrandom sampling: the effects of censoring studies yielding statistically insignificant mean differences
Hedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241 · see citing works
Applied analysis of variance in behavioral science
Edwards, L. K. (1993) · M. Dekker eBooks · Cited by 235 · see citing works
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203 · see citing works
On the Post Hoc Power in Testing Mean Differences
Yuan, K.-H., & Maxwell, S. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030002141 · Cited by 168 · see citing works
Self-citation
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167 · see citing works
Self-citation
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167 · see citing works
Estimating effect size: Bias resulting from the significance criterion in editorial decisions
Lane, D. M., & Dunlap, W. P. (1978) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1978.tb00578.x · Cited by 161 · see citing works
Researchers’ Intuitions About Power in Psychological Research
Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016) · Psychological Science · doi:10.1177/0956797616647519 · Cited by 147 · see citing works
Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power
Anderson, S. F., & Maxwell, S. E. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 133 · see citing works
Self-citation
Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals.
Kelley, K., & Rausch, J. R. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.363 · Cited by 116 · see citing works
Self-citation
Handbook of Ethics in Quantitative Methodology
Panter, A. T. (2011) · Routledge eBooks · doi:10.4324/9780203840023 · Cited by 109 · see citing works
Regression and ANOVA: An Integrated Approach Using SAS Software
Gunst, R. F. (2003) · Technometrics · doi:10.1198/tech.2003.s159 · Cited by 109 · see citing works
Introduction to Statistical Methods for Clinical Trials
Cook, T. D., & DeMets, D. L. (2007) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420009965 · Cited by 96 · see citing works
Bias and precision of some classical ANOVA effect sizes when assumptions are violated
Skidmore, S. T., & Thompson, B. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0257-2 · Cited by 76 · see citing works
Comment: What p -hacking really looks like: A comment on Masicampo and LaLande (2012)
Lakens, D. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.982664 · Cited by 56 · see citing works
Accuracy of Effect Size Estimates from Published Psychological Research
Brand, A., Bradley, M. T., Best, L. A., & Stoica, G. (2008) · Perceptual and Motor Skills · doi:10.2466/pms.106.2.645-649 · Cited by 52 · see citing works
Bias in linear model power and sample size calculation due to estimating noncentrality
Taylor, D. J., & Muller, K. E. (1996) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929608831787 · Cited by 49 · see citing works
On the challenges of drawing conclusions from p -values just below 0.05
Lakens, D. (2015) · PeerJ · doi:10.7717/peerj.1142 · Cited by 44 · see citing works
Associative Activation and Its Relation to Exploration and Exploitation in the Brain
Baror, S., & Bar, M. (2016) · Psychological Science · doi:10.1177/0956797616634487 · Cited by 37 · see citing works
The perils with the misuse of predictive power
Dallow, N. S., & Fina, P. (2010) · Pharmaceutical Statistics · doi:10.1002/pst.467 · Cited by 34 · see citing works
Attention’s Accelerator
Reinhart, R. M. G., McClenahan, L. J., & Woodman, G. F. (2016) · Psychological Science · doi:10.1177/0956797616636416 · Cited by 32 · see citing works
Accuracy in parameter estimation for a general class of effect sizes: A sequential approach.
Kelley, K., Darku, F. B., & Chattopadhyay, B. (2017) · Psychological Methods · doi:10.1037/met0000127 · Cited by 31 · see citing works
Self-citation
Adaptive Trial Designs
Lai, T. L., Lavori, P. W., & Shih, M. (2011) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010611-134504 · Cited by 25 · see citing works
Going All In
Ackerman, J. M., Maner, J. K., & Carpenter, S. M. (2016) · Psychological Science · doi:10.1177/0956797616636631 · Cited by 25 · see citing works
Estimating the standardized mean difference with minimum risk: Maximizing accuracy and minimizing cost with sequential estimation.
Chattopadhyay, B., & Kelley, K. (2016) · Psychological Methods · doi:10.1037/met0000089 · Cited by 14 · see citing works
Self-citation
Clinical trial methodology
(2010) · Taylor & Francis eBooks · doi:10.4324/9780203448304_chapter_10 · Cited by 4 · see citing works

Flagged first, then by citations

1 of the 63 references couldn’t be matched to a known paper.

Sorted by citations

Sample Size Justification
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962 · see citing works
How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
A Practical Primer To Power Analysis for Simple Experimental Designs
Perugini, M., Gallucci, M., & Costantini, G. (2018) · International Review of Social Psychology · doi:10.5334/irsp.181 · Cited by 426 · see citing works
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 345 · see citing works
Power to Detect What? Considerations for Planning and Evaluating Sample Size
Giner‐Sorolla, R., Montoya, A. K., Reifman, A. S., Carpenter, T., Lewis, N. A., Aberson, C. L., Bostyn, D. H., Conrique, B. G., Ng, B., Schoemann, A. M., et al. (2024) · Personality and Social Psychology Review · doi:10.1177/10888683241228328 · Cited by 292 · see citing works
A tutorial on Bayes Factor Design Analysis using an informed prior
Stefan, A. M., Gronau, Q. F., Schönbrodt, F. D., & Wagenmakers, E.-J. (2019) · Behavior Research Methods · doi:10.3758/s13428-018-01189-8 · Cited by 284 · see citing works
Gamification and serious games: A literature meta-analysis and integrative model
Baptista, G., & Oliveira, T. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.11.030 · Cited by 263 · see citing works
The interpretation of statistical power after the data have been gathered
Dziak, J. J., Dierker, L., & Abar, B. W. (2018) · Current Psychology · doi:10.1007/s12144-018-0018-1 · Cited by 208 · see citing works
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 184 · see citing works
Resting-State Functional Brain Connectivity Best Predicts the Personality Dimension of Openness to Experience
Dubois, J., Galdi, P., Han, Y., Paul, L. K., & Adolphs, R. (2018) · Personality Neuroscience · doi:10.1017/pen.2018.8 · Cited by 168 · see citing works
Avoid Cohen’s ‘Small’, ‘Medium’, and ‘Large’ for Power Analysis
Correll, J., Mellinger, C., McClelland, G. H., & Judd, C. M. (2020) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.12.009 · Cited by 167 · see citing works
Summary-statistics-based power analysis: A new and practical method to determine sample size for mixed-effects modeling.
Murayama, K., Usami, S., & Sakaki, M. (2022) · Psychological Methods · doi:10.1037/met0000330 · Cited by 165 · see citing works
Generic language in scientific communication
DeJesus, J. M., Callanan, M. A., Solis, G., & Gelman, S. A. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1817706116 · Cited by 157 · see citing works
Beyond Experiments
Diener, E., Northcott, R., Zyphur, M. J., & West, S. G. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211037670 · Cited by 157 · see citing works
A Primer for Conducting Experiments in Human–Robot Interaction
Hoffman, G., & Zhao, X. (2020) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3412374 · Cited by 138 · see citing works
Sample Size and Shapiro-Wilk Test: An Analysis for Soybean Grain Yield
de Souza, R. R., Toebe, M. J., Mello, A. C., & Bittencourt, K. C. (2022) · European Journal of Agronomy · doi:10.1016/j.eja.2022.126666 · Cited by 138 · see citing works
Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 98 · see citing works
Systematic review of the status of pfhrp2 and pfhrp3 gene deletion, approaches and methods used for its estimation and reporting in Plasmodium falciparum populations in Africa: review of published studies 2010–2019
Agaba, B. B., Yeka, A., Nsobya, S. L., Arinaitwe, E., Nankabirwa, J. I., Opigo, J., Mbaka, P., Lim, C. S., Kalyango, J. N., Karamagi, C., et al. (2019) · Malaria Journal · doi:10.1186/s12936-019-2987-4 · Cited by 95 · see citing works
Power Analysis and Exploratory Research
Haile, Z. T. (2023) · Journal of Human Lactation · doi:10.1177/08903344231195625 · Cited by 91 · see citing works
Spontaneous Vicarious Perception of the Content of Another’s Visual Perspective
Ward, E., Ganis, G., & Bach, P. (2019) · Current Biology · doi:10.1016/j.cub.2019.01.046 · Cited by 89 · see citing works
Using multilevel models for the analysis of event-related potentials
Volpert‐Esmond, H. I., Page‐Gould, E., & Bartholow, B. D. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.006 · Cited by 89 · see citing works
The profile of the skilled reader: An investigation into the role of reading enjoyment and student characteristics
Rogiers, A., Keer, H. V., & Merchie, E. (2019) · International Journal of Educational Research · doi:10.1016/j.ijer.2019.101512 · Cited by 85 · see citing works
Counterfeit diversity: How strategically misrepresenting gender diversity dampens organizations’ perceived sincerity and elevates women’s identity threat concerns.
Kroeper, K. M., Williams, H. E., & Murphy, M. C. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000348 · Cited by 86 · see citing works
The impact of sample size on the reproducibility of voxel-based lesion-deficit mappings
Lorca‐Puls, D. L., Gajardo‐Vidal, A., White, J., Seghier, M. L., Leff, A. P., Green, D. W., Crinion, J., Ludersdorfer, P., Hope, T. M. H., Bowman, H., et al. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.03.014 · Cited by 82 · see citing works
Estimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance
Brunner, J., & Schimmack, U. (2020) · Meta-Psychology · doi:10.15626/mp.2018.874 · Cited by 76 · see citing works
Calibrate your confidence in research findings: A tutorial on improving research methods and practices
Frost, A. D. S., & Ledgerwood, A. (2020) · Journal of Pacific Rim Psychology · doi:10.1017/prp.2020.7 · Cited by 75 · see citing works
Non-operative treatment for lumbar spinal stenosis with neurogenic claudication: an updated systematic review
Ammendolia, C., Hofkirchner, C., Plener, J., Bussières, A., Schneider, M., Young, J. J., Furlan, A. D., Stuber, K. J., Ahmed, A., Cancelliere, C., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-057724 · Cited by 73 · see citing works
Human studies of mitochondrial biology demonstrate an overall lack of binary sex differences: A multivariate meta‐analysis
Junker, A., Wang, J., Gouspillou, G., Ehinger, J. K., Elmér, E., Sjövall, F., Fisher‐Wellman, K. H., Neufer, P. D., Molina, A. J. A., Ferrucci, L., et al. (2022) · The FASEB Journal · doi:10.1096/fj.202101628r · Cited by 72 · see citing works
Examining investors' sentiments, behavioral biases and investment decisions during COVID-19 in the emerging stock market: a case of Pakistan stock market
Parveen, S., Satti, Z. W., Subhan, Q. A., Riaz, N., Babar, S. F., & Bashir, T. (2021) · Journal of economic and administrative sciences. · doi:10.1108/jeas-08-2020-0153 · Cited by 71 · see citing works
Intrusive memories and voluntary memory of a trauma film: Differential effects of a cognitive interference task after encoding.
Lau‐Zhu, A., Henson, R. N., & Holmes, E. A. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000598 · Cited by 70 · see citing works
A Meta-Analysis of Expressive Writing on Posttraumatic Stress, Posttraumatic Growth, and Quality of Life
Pavlacic, J. M., Buchanan, E. M., Maxwell, N. P., Hopke, T. G., & Schulenberg, S. E. (2019) · Review of General Psychology · doi:10.1177/1089268019831645 · Cited by 69 · see citing works
Statistical power analysis and sample size planning for moderated mediation models
Xu, Z., Gao, F., Fa, A., Qu, W., & Zhang, Z. J. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02342-2 · Cited by 68 · see citing works
The prevalence and factors associated with alcohol use disorder among people living with HIV/AIDS in Africa: a systematic review and meta-analysis
Necho, M., Belete, A., & Getachew, Y. (2020) · Substance Abuse Treatment Prevention and Policy · doi:10.1186/s13011-020-00301-6 · Cited by 67 · see citing works
A Historically Based Review of Empirical Work on Color and Psychological Functioning: Content, Methods, and Recommendations for Future Research
Elliot, A. J. (2018) · Review of General Psychology · doi:10.1037/gpr0000170 · Cited by 66 · see citing works
The Sample Size Matters: To What Extent the Participant Reduction Affects the Outcomes of a Neuroscientific Research. A Case-Study in Neuromarketing Field
Vozzi, A., Ronca, V., Aricò, P., Borghini, G., Sciaraffa, N., Cherubino, P., Trettel, A., Babiloni, F., & Flumeri, G. D. (2021) · Sensors · doi:10.3390/s21186088 · Cited by 65 · see citing works
A meta-analysis of factors affecting acceptance of information technology in auditing
afsay, A., Tahriri, A., & Rezaee, Z. (2023) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2022.100608 · Cited by 63 · see citing works
Understanding the factors affecting AI services adoption in hospitality: The role of behavioral reasons and emotional intelligence
Rasheed, H. M. W., Chen, Y., Khizar, H. M. U., & Safeer, A. A. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e16968 · Cited by 53 · see citing works
Finger pinching and imagination classification: A fusion of CNN architectures for IoMT-enabled BCI applications
Varone, G., Boulila, W., Driss, M., Kumari, S., Khan, M. K., Gadekallu, T. R., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102006 · Cited by 51 · see citing works
Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2269357 · Cited by 51 · see citing works
Embodiment of supernumerary robotic limbs in virtual reality
Arai, K. I., Saito, H., Fukuoka, M., Ueda, S., Sugimoto, M., Kitazaki, M., & İnami, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-13981-w · Cited by 50 · see citing works
Obtaining Evidence for No Effect
Dienes, Z. P. (2021) · Collabra Psychology · doi:10.1525/collabra.28202 · Cited by 50 · see citing works
Playing a video game is more than mere procrastination
Nordby, K., Løkken, R. A., & Pfuhl, G. (2019) · BMC Psychology · doi:10.1186/s40359-019-0309-9 · Cited by 49 · see citing works
Suicidal ideation, attempt, and its associated factors among HIV/AIDS patients in Africa: a systematic review and meta-analysis study
Necho, M., Tsehay, M., & Zenebe, Y. (2021) · International Journal of Mental Health Systems · doi:10.1186/s13033-021-00437-3 · Cited by 49 · see citing works
Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research
Pietschnig, J., Siegel, M., Eder, J. S. N., & Gittler, G. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02874 · Cited by 48 · see citing works
Estimating the Replicability of Sports and Exercise Science Research
Murphy, J., Caldwell, A. R., Mesquida, C., Ladell, A. J. M., Martínez, A. E., Tual, A., Denys, A. T., Cameron, B., Hooren, B. V., Parr, B., et al. (2025) · Sports Medicine · doi:10.1007/s40279-025-02201-w · Cited by 48 · see citing works
Big Five Personality Traits and Life Satisfaction: The Mediating Role of Religiosity
Szczęśniak, M., Sopińska, B., & Kroplewski, Z. (2019) · Religions · doi:10.3390/rel10070437 · Cited by 48 · see citing works
Sample-size determination for the Bayesian t test and Welch’s test using the approximate adjusted fractional Bayes factor
Fu, Q., Hoijtink, H., & Moerbeek, M. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01408-1 · Cited by 47 · see citing works
From Outcome to Process Focus: Fostering a More Robust Psychological Science Through Registered Reports and Results-Blind Reviewing
Grand, J. A., Rogelberg, S. G., Banks, G. C., Landis, R. S., & Tonidandel, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691618767883 · Cited by 45 · see citing works
Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis
Nuijten, M. B., van Assen, M. A. L. M., Augusteijn, H. E. M., Crompvoets, E. A. V., & Wicherts, J. M. (2020) · Journal of Intelligence · doi:10.3390/jintelligence8040036 · Cited by 42 · see citing works
A spotlight on cross-sectional research: Addressing the issues of confounding and adjustment
Asiamah, N., Mends-Brew, E., & Boison, B. T. K. (2019) · International Journal of Healthcare Management · doi:10.1080/20479700.2019.1621022 · Cited by 42 · see citing works
Misinterpreting p: The discrepancy between p values and the probability the null hypothesis is true, the influence of multiple testing, and implications for the replication crisis.
Anderson, S. F. (2019) · Psychological Methods · doi:10.1037/met0000248 · Cited by 41 · see citing works
Self-citation
Effects of transcranial magnetic stimulation (TMS) current direction and pulse waveform on cortico‐cortical connectivity: A registered report TMS‐EEG study
Guidali, G., Zazio, A., Lucarelli, D., Marcantoni, E., Stango, A., Barchiesi, G., & Bortoletto, M. (2023) · European Journal of Neuroscience · doi:10.1111/ejn.16127 · Cited by 40 · see citing works
Online Advertising Suppresses Visual Competition during Planned Purchases
van der Lans, R., Pieters, R. G. M., & Wedel, M. (2021) · Journal of Consumer Research · doi:10.1093/jcr/ucab017 · Cited by 39 · see citing works
Sample Size Planning for Detecting Mediation Effects: A Power Analysis Procedure Considering Uncertainty in Effect Size Estimates
Liu, X., & Wang, L. (2019) · Multivariate Behavioral Research · doi:10.1080/00273171.2019.1593814 · Cited by 37 · see citing works
Estimating the Reproducibility of Experimental PhilosophyPreprint
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · doi:10.31234/osf.io/sxdah · Cited by 37 · see citing works
Basic statistical considerations for physiology: The journal Temperature toolbox
Caldwell, A. R., & Cheuvront, S. N. (2019) · Temperature · doi:10.1080/23328940.2019.1624131 · Cited by 35 · see citing works
A strengths-based, culturally responsive family intervention improves Latino kindergarteners’ vocabulary and approaches to learning
Leyva, D., Weiland, C., Shapiro, A., Yeomans‐Maldonado, G., & Febles, A. (2021) · Child Development · doi:10.1111/cdev.13698 · Cited by 35 · see citing works
When masters of abstraction run into a concrete wall: Experts failing arithmetic word problems
Gros, H., Sander, E., & Thibaut, J. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01628-3 · Cited by 35 · see citing works
Best (but oft forgotten) practices: sample size planning for powerful studies
Anderson, S. F. (2019) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqz058 · Cited by 34 · see citing works
Self-citation
Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size
Bakker, M., Veldkamp, C. L. S., van den Akker, O. R., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., & Wicherts, J. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236079 · Cited by 34 · see citing works
Open Minded
Newell, B. R., & Shanks, D. R. (2023) · The MIT Press eBooks · doi:10.7551/mitpress/14922.001.0001 · Cited by 33 · see citing works
Does Temporal Distance Influence Abstraction? A Large Pre-Registered Experiment
Sánchez, A. M., Coleman, C., & Ledgerwood, A. (2021) · Social Cognition · doi:10.1521/soco.2021.39.3.352 · Cited by 33 · see citing works
Power of averaging: Noise reduction by ensemble coding of multiple faces.
Sun, J., & Chong, S. C. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000667 · Cited by 31 · see citing works
Lexical Acquisition Through Category Matching: 12-Month-Old Infants Associate Words to Visual Categories
Pomiechowska, B., & Gliga, T. (2018) · Psychological Science · doi:10.1177/0956797618817506 · Cited by 30 · see citing works
Attitudes Towards Seeking Psychological Help: An Integrative Model Based on Contact, Essentialist Beliefs About Mental Illness, and Stigma
Hantzi, A., Anagnostopoulos, F., & Alexiou, E. (2018) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-018-9573-8 · Cited by 30 · see citing works
Effect size: a statistical basis for clinical practice
Valladares‐Neto, J. (2018) · Revista Odonto Ciência · doi:10.15448/1980-6523.2018.1.29437 · Cited by 29 · see citing works
Zur Methodologie von Replikationsstudien
Erdfelder, E., & Ulrich, R. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000387 · Cited by 29 · see citing works
Examining the impact of social media de-influencing on audiences
Elhajjar, S., & Itani, O. S. (2025) · Internet Research · doi:10.1108/intr-04-2024-0574 · Cited by 28 · see citing works
Eye movements reveal the contributions of early and late processes of enhancement and suppression to the guidance of visual search
Hamblin-Frohman, Z., Chang, S., Egeth, H. E., & Becker, S. I. (2022) · Attention Perception & Psychophysics · doi:10.3758/s13414-022-02536-w · Cited by 28 · see citing works
Psychological Wellbeing during the COVID-19 Pandemic: the influence of personality traits in the Italian population
Rossi, C., Bonanomi, A., & Oasi, O. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18115862 · Cited by 27 · see citing works
Statistical power in clinical trials of interventions for mood, anxiety, and psychotic disorders
de Vries, Y. A., Schoevers, R. A., Higgins, J. P. T., Munafo, M. R., & Bastiaansen, J. A. (2022) · Psychological Medicine · doi:10.1017/s0033291722001362 · Cited by 27 · see citing works
Adaptation to mean and variance: Interrelationships between mean and variance representations in orientation perception
Jeong, J., & Chong, S. C. (2020) · Vision Research · doi:10.1016/j.visres.2020.01.002 · Cited by 27 · see citing works
Preregistration in infant research—A primer
Havron, N., Bergmann, C., & Tsuji, S. (2020) · Infancy · doi:10.1111/infa.12353 · Cited by 27 · see citing works
Assessing the Evidential Value of Mental Fatigue and Exercise Research
Holgado, D., Mesquida, C., & Román-Caballero, R. (2023) · Sports Medicine · doi:10.1007/s40279-023-01926-w · Cited by 27 · see citing works
Visual motion prediction and verbal false memory performance in autistic children
Tewolde, F. G., Bishop, D. V. M., & Manning, C. (2018) · Autism Research · doi:10.1002/aur.1915 · Cited by 25 · see citing works
Power and sample size analysis for longitudinal mixed models of health in populations exposed to environmental contaminants: a tutorial
Harrall, K. K., Muller, K. E., Starling, A. P., Dabelea, D., Barton, K. E., Adgate, J. L., & Glueck, D. H. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01819-y · Cited by 25 · see citing works
How state and trait versions of self-esteem and depressive symptoms affect their interplay: A longitudinal experimental investigation.
Braun, L., Göllner, R., Rieger, S., Trautwein, U., & Spengler, M. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000295 · Cited by 25 · see citing works
The influence of agents’ negligence in shaping younger and older adults’ moral judgment
Margoni, F., Geipel, J., Hadjichristidis, C., & Surian, L. (2019) · Cognitive Development · doi:10.1016/j.cogdev.2018.12.002 · Cited by 25 · see citing works
Meaningful change definitions: sample size planning for experimental intervention research
Gruijters, S. L. K., & Peters, G.-J. (2020) · Psychology and Health · doi:10.1080/08870446.2020.1841762 · Cited by 24 · see citing works
Physical correlates of human-like softness elicit high tactile pleasantness
Kitada, R., Ng, M., Tan, Z. Y., Lee, X. E., & Kochiyama, T. (2021) · Scientific Reports · doi:10.1038/s41598-021-96044-w · Cited by 24 · see citing works
Religious/Spiritual Struggles and Life Satisfaction among Young Roman Catholics: The Mediating Role of Gratitude
Szczęśniak, M., Bielecka, G., Bajkowska, I., Czaprowska, A., & Madej, D. (2019) · Religions · doi:10.3390/rel10060395 · Cited by 24 · see citing works
What is top-down about seeing enemies? Social anxiety and attention to threat
Delchau, H. L., Christensen, B. K., O’Kearney, R., & Goodhew, S. C. (2019) · Attention Perception & Psychophysics · doi:10.3758/s13414-019-01920-3 · Cited by 24 · see citing works
Emotion, Sophistication and Political Behavior: Evidence From a Laboratory Experiment
Lamprianou, I., & Ellinas, A. A. (2018) · Political Psychology · doi:10.1111/pops.12536 · Cited by 23 · see citing works
Nurturing the reading brain: home literacy practices are associated with children’s neural response to printed words through vocabulary skills
Girard, C., Bastelica, T., Léone, J., Epinat‐Duclos, J., Longo, L., & Prado, J. (2021) · npj Science of Learning · doi:10.1038/s41539-021-00112-9 · Cited by 23 · see citing works
Support from a TMS/MEP study for a direct link between positive/negative stimuli and approach/avoidance tendencies
Fini, C., Fischer, M., Bardi, L., Braß, M., & Moors, A. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2020.107496 · Cited by 23 · see citing works
Seven steps toward transparency and replicability in psychological science.
Lindsay, D. S. (2020) · Canadian Psychology/Psychologie canadienne · doi:10.1037/cap0000222 · Cited by 23 · see citing works
The Role of Cultural Values in National-Level Innovation: Evidence from 106 Countries
Bonetto, É., Pichot, N., & Adam‐Troian, J. (2022) · Cross-Cultural Research · doi:10.1177/10693971221078087 · Cited by 22 · see citing works
Peer-Review Guidelines Promoting Replicability and Transparency in Psychological Science
Davis, W. E., Giner‐Sorolla, R., Lindsay, D. S., Lougheed, J. P., Makel, M. C., Meier, M. E., Sun, J., Vaughn, L. A., & Zelenski, J. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918806489 · Cited by 22 · see citing works
Affordance matching predictively shapes the perceptual representation of others’ ongoing actions.
McDonough, K. L., Costantini, M., Hudson, M., Ward, E., & Bach, P. (2020) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0000745 · Cited by 22 · see citing works
Have I Got the Power? Analysing and Reporting Statistical Power in HRI
Bartlett, M. E., Edmunds, C. E. R., Belpaeme, T., & Thill, S. (2022) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3495246 · Cited by 21 · see citing works
Replication is fundamental, but is it common? A call for scientific self-reflection and contemporary research practices in gambling-related research
LaPlante, D. A. (2019) · International Gambling Studies · doi:10.1080/14459795.2019.1672768 · Cited by 21 · see citing works
Analytical Strategies and Tactics in Salivary Bioscience
Riis, J. L., Chen, F. R., Dent, A. L., Laurent, H. K., & Bryce, C. I. (2020) · Springer eBooks · doi:10.1007/978-3-030-35784-9_4 · Cited by 21 · see citing works
Open Science Practices in Gambling Research Publications (2016–2019): A Scoping Review
Louderback, E. R., Gainsbury, S., Heirene, R. M., Amichia, K., Grossman, A., Bernhard, B. J., & LaPlante, D. A. (2022) · Journal of Gambling Studies · doi:10.1007/s10899-022-10120-y · Cited by 20 · see citing works
Optimizing practice scheduling requires quantitative tracking of individual item performance
Eglington, L. G., & Pavlik, P. I. (2020) · npj Science of Learning · doi:10.1038/s41539-020-00074-4 · Cited by 20 · see citing works
Mechanisms underlying memory enhancement for disgust over fear
Moeck, E. K., Matson, L. A., & Takarangi, M. K. T. (2021) · Cognition & Emotion · doi:10.1080/02699931.2021.1936460 · Cited by 20 · see citing works
“Emoji, I can feel your pain” – Neural responses to facial and emoji expressions of pain
Liao, W., Zhang, Y., Huang, X., Xu, X., & Peng, X. (2021) · Biological Psychology · doi:10.1016/j.biopsycho.2021.108134 · Cited by 20 · see citing works
The Collaborative Roots of Corruption? A Replication of Weisel & Shalvi (2015)
Wouda, J., Bijlstra, G., Frankenhuis, W. E., & Wigboldus, D. H. J. (2017) · Collabra Psychology · doi:10.1525/collabra.97 · Cited by 20 · see citing works
Impact of basic psychological support on stigma and mental well-being of people with disabilities due to leprosy and lymphatic filariasis: a proof-of-concept study
Agarwal, A., Nayak, P., van Brakel, W. H., van Wijk, R., Jain, A., Broekkamp, H., Mol, M. M., & Mishra, C. P. (2023) · International Health · doi:10.1093/inthealth/ihad096 · Cited by 19 · see citing works
Effect Size Estimation From t-Statistics in the Presence of Publication Bias: A Brief Review of Existing Approaches With Some Extensions
Ulrich, R., Miller, J., & Erdfelder, E. (2018) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000319 · Cited by 15 · see citing works

Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach.
McShane, B. B., & Böckenholt, U. (2015) · Psychological Methods · doi:10.1037/met0000036 · Cited by 48 · see citing works
Using prior information to plan appropriately powered regression studies: A tutorial using BUCSS.
Anderson, S. F. (2020) · Psychological Methods · doi:10.1037/met0000366 · Cited by 11 · see citing works
Self-citation
Sample size and replication: A census using the Reproducibility Project PsychologyPreprint
Anderson, S. F., Kelley, K., Liu, X., & Guarino, K. J. (2024) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/dn3fk
Self-citation
Approaches to effect size selection for power analysisPreprint
Thorn, F. S. (2019) · OSF Preprints (OSF Preprints)
Uncertainty limits the use of power analysis.
Pek, J. C. X., Pitt, M. A., & Wegener, D. T. (2024) · Journal of Experimental Psychology General · doi:10.1037/xge0001273 · Cited by 18 · see citing works

Authors: Anderson, S. F., Kelley, K., & Maxwell, S. E.
Affiliations: University of Notre Dame (US)
Corresponding author: Anderson, Samantha F.
ORCID: Anderson, Samantha F.
Kelley, Ken
Maxwell, Scott E.
Venue: Psychological Science
Publisher: SAGE Publications
Published: 2017-09-13
Volume: 28
Issue: 11
Pages: 1547-1562
ISSN: 0956-7976 (ISSN-L)
1467-9280
DOI: 10.1177/0956797617723724
OpenAlex: W2756450828
Cited by: 406
References: 63
Citations by year: 2026: 31 · 2025: 39 · 2024: 46 · 2023: 55 · 2022: 52 · 2021: 53 · 2020: 44 · 2019: 47 · 2018: 34 · 2017: 3
FWCI: 31.51
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Psychological Science (publishedVersion)
PubMed (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords: sample size determination, statistical power, effect size estimation, publication bias, underpowered studies
MeSH terms: Data Interpretation, Statistical, Humans, Publication Bias, Sample Size, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks