Cited by 416Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods in Epidemiology

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.

Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

Anderson, Samantha F., Kelley, Ken, Maxwell, Scott E.·2017·Psychological Science

DOI 10.1177/0956797617723724

Open this paper in CoChat — free account

Abstract

The sample size necessary to obtain a desired level of statistical power depends in part on the population value of the effect size, which is, by definition, unknown. A common approach to sample-size planning uses the sample effect size from a prior study as an estimate of the population value of the effect to be detected in the future study. Although this strategy is intuitively appealing, effect-size estimates, taken at face value, are typically not accurate estimates of the population effect size because of publication bias and uncertainty. We show that the use of this approach often results in underpowered studies, sometimes to an alarming degree. We present an alternative approach that adjusts sample effect sizes for bias and uncertainty, and we demonstrate its effectiveness for several experimental designs. Furthermore, we discuss an open-source R package, BUCSS, and user-friendly Web applications that we have made available to researchers so that they can easily implement our suggested methods.

Integrity signals

Registry

Crossref · 10.1177/0956797617723724

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,421
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“However, even when researchers conduct a well-intentioned power analysis to determine the sample size of a replication study, the power of the original study limits their ability to determine an accurate sample size, sometimes severely (Anderson & Maxwell, 2016, 2017; Button et al., 2013).”
Background
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Ioannidis (2005) showed that, all other things being equal, there is a higher probability that a published research claim is false in a literature with lower-powered studies.”
Background
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
How many subjects does it take to do a regression analysis
Green, S. B. (1991) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr2603_7 · Cited by 3,916 · cited 1×
“Strategies based on prior sample size alone (“rules of thumb”) can result in selecting sample sizes that are either too large or too small, depending on the power of the prior studies involved (Green, 1991; see Anderson & Maxwell, 2017, for a simulation).”
Background
Designing Experiments and Analyzing Data: A Model Comparison Perspective
Maxwell, S., & Delaney, H. D. (1990) · Journal of the American Statistical Association · doi:10.2307/2289744 · Cited by 2,593
Self-citation
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
The role and interpretation of pilot studies in clinical research
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.10.008 · Cited by 1,917
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343
Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Designing Experiments and Analysing Data: A Model Comparison Perspective.
Freeman, G. H., Maxwell, S. E., & Delaney, H. D. (1991) · Biometrics · doi:10.2307/2532173 · Cited by 1,229
Self-citation
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,099
Self-citation
On effect size.
Kelley, K., & Preacher, K. J. (2012) · Psychological Methods · doi:10.1037/a0028086 · Cited by 1,084
Self-citation
Design Sensitivity: Statistical Power for Experimental Research.
Eliason, S. R., & Lipsey, M. W. (1991) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2072167 · Cited by 1,038
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
How Many Subjects?: Statistical Power Analysis in Research
KRAEMER, H. C., & Blasey, C. M. (2016) · SAGE Publications Ltd eBooks · doi:10.4135/9781483398761 · Cited by 918
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 876
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860 · cited 3×
“Using Taylor and Muller’s (1996) 50th-percentile method to adjust the sample estimate of 0.”
BackgroundMethodInfluential
Sample Size Calculations in Clinical Research
Chow, S., Wang, H., & Shao, J. (2003) · CRC Press eBooks · doi:10.1201/9780203911341 · Cited by 813
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Do studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations
Olejnik, S. F., & Algina, J. J. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.2000.1040 · Cited by 703
The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655
Self-citation
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E., Zehetleitner, M., & Perugini, M. (2015) · Psychological Methods · doi:10.1037/met0000061 · Cited by 599
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542
Self-citation
Performing high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414
The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409
Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant.
Kelley, K., & Maxwell, S. E. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.305 · Cited by 310
Self-citation
Designing Experiments and Analyzing Data: Model Comparison Perspective
Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018) · Thammasat University Digital Collections · Cited by 298
Self-citation
Estimation of effect size under nonrandom sampling: the effects of censoring studies yielding statistically insignificant mean differences
Hedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241
Applied analysis of variance in behavioral science
Edwards, L. K. (1993) · M. Dekker eBooks · Cited by 233
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 203
On the Post Hoc Power in Testing Mean Differences
Yuan, K.-H., & Maxwell, S. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030002141 · Cited by 168
Self-citation
There’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167
Self-citation
Sample size in psychological research over the past 30 years
Marszalek, J. M., Barber, C., Kohlhart, J., & Cooper, B. H. (2011) · Perceptual and Motor Skills · doi:10.2466/03.11.pms.112.2.331-348 · Cited by 167
Estimating effect size: Bias resulting from the significance criterion in editorial decisions
Lane, D. M., & Dunlap, W. P. (1978) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1978.tb00578.x · Cited by 161
Researchers’ Intuitions About Power in Psychological Research
Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016) · Psychological Science · doi:10.1177/0956797616647519 · Cited by 158
Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals.
Kelley, K., & Rausch, J. R. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.363 · Cited by 116
Self-citation
Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power
Anderson, S. F., & Maxwell, S. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 116
Self-citation
Handbook of Ethics in Quantitative Methodology
Panter, A. T. (2011) · Routledge eBooks · doi:10.4324/9780203840023 · Cited by 109
Regression and ANOVA: An Integrated Approach Using SAS Software
Gunst, R. F. (2003) · Technometrics · doi:10.1198/tech.2003.s159 · Cited by 109
Introduction to Statistical Methods for Clinical Trials
Cook, T. D., & DeMets, D. L. (2007) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420009965 · Cited by 96
Bias and precision of some classical ANOVA effect sizes when assumptions are violated
Skidmore, S. T., & Thompson, B. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0257-2 · Cited by 76
Comment: What p -hacking really looks like: A comment on Masicampo and LaLande (2012)
Lakens, D. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.982664 · Cited by 56
Accuracy of Effect Size Estimates from Published Psychological Research
Brand, A., Bradley, M. T., Best, L. A., & Stoica, G. (2008) · Perceptual and Motor Skills · doi:10.2466/pms.106.2.645-649 · Cited by 52
Bias in linear model power and sample size calculation due to estimating noncentrality
Taylor, D. J., & Muller, K. E. (1996) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929608831787 · Cited by 49
On the challenges of drawing conclusions from p -values just below 0.05
Lakens, D. (2015) · PeerJ · doi:10.7717/peerj.1142 · Cited by 45
Associative Activation and Its Relation to Exploration and Exploitation in the Brain
Baror, S., & Bar, M. (2016) · Psychological Science · doi:10.1177/0956797616634487 · Cited by 40
The perils with the misuse of predictive power
Dallow, N. S., & Fina, P. (2010) · Pharmaceutical Statistics · doi:10.1002/pst.467 · Cited by 34
Attention’s Accelerator
Reinhart, R. M. G., McClenahan, L. J., & Woodman, G. F. (2016) · Psychological Science · doi:10.1177/0956797616636416 · Cited by 32
Accuracy in parameter estimation for a general class of effect sizes: A sequential approach.
Kelley, K., Darku, F. B., & Chattopadhyay, B. (2017) · Psychological Methods · doi:10.1037/met0000127 · Cited by 31
Self-citation
Going All In
Ackerman, J. M., Maner, J. K., & Carpenter, S. M. (2016) · Psychological Science · doi:10.1177/0956797616636631 · Cited by 26
Adaptive Trial Designs
Lai, T. L., Lavori, P. W., & Shih, M. (2011) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010611-134504 · Cited by 25
Estimating the standardized mean difference with minimum risk: Maximizing accuracy and minimizing cost with sequential estimation.
Chattopadhyay, B., & Kelley, K. (2016) · Psychological Methods · doi:10.1037/met0000089 · Cited by 14
Self-citation
Clinical trial methodology
(2010) · Taylor & Francis eBooks · doi:10.4324/9780203448304_chapter_10 · Cited by 4

Research works printed but not linked: 3

10.1002/per1919 · DOI
Lakens D. (in press). Equivalence tests: A practical primer for t-tests, correlations, and meta-analyses. Social Psychological & Personality Science.
10.2466/03.11.pms112.2.331-348 · DOI

Sorted by citations

Sample Size Justification
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables
Brysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
A Practical Primer To Power Analysis for Simple Experimental Designs
Perugini, M., Gallucci, M., & Costantini, G. (2018) · International Review of Social Psychology · doi:10.5334/irsp.181 · Cited by 482
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 365

See all 416

Authors: Anderson, S. F., Kelley, K., & Maxwell, S. E.
Affiliations: University of Notre Dame (US)
Corresponding author: Anderson, Samantha F.
ORCID: Anderson, Samantha F.
Kelley, Ken
Maxwell, Scott E.
Venue: Psychological Science
Publisher: SAGE Publications
Published: 2017-09-13
Volume: 28
Issue: 11
Pages: 1547-1562
ISSN: 0956-7976 (ISSN-L)
1467-9280
DOI: 10.1177/0956797617723724
OpenAlex: W2756450828
Cited by: 406
References: 63
Citations by year: 2026: 31 · 2025: 39 · 2024: 46 · 2023: 55 · 2022: 52 · 2021: 53 · 2020: 44 · 2019: 47 · 2018: 34 · 2017: 3
FWCI: 31.51
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Psychological Science (publishedVersion)
PubMed (publishedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords: sample size determination, statistical power, effect size estimation, publication bias, underpowered studies
MeSH terms: Data Interpretation, Statistical, Humans, Publication Bias, Sample Size, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks