The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias
Open accessAbstract
When designing a study, the planned sample size is often based on power analyses. One way to choose an effect size for power analyses is by relying on pilot data. A-priori power analyses are only accurate when the effect size estimate is accurate. In this paper we highlight two sources of bias when performing a-priori power analyses for between-subject designs based on pilot data. First, we examine how the choice of the effect size index (η2, ω2 and ε2) affects the sample size and power of the main study. Based on our observations, we recommend against the use of η2 in a-priori power analyses. Second, we examine how the maximum sample size researchers are willing to collect in a main study (e.g. due to time or financial constraints) leads to overestimated effect size estimates in the studies that are performed. Determining the required sample size exclusively based on the effect size estimates from pilot data, and following up on pilot studies only when the sample size estimate for the main study is considered feasible, creates what we term follow-up bias. We explain how follow-up bias leads to underpowered main studies. Our simulations show that designing main studies based on effect sizes estimated from small pilot studies does not yield desired levels of power due to accuracy bias and follow-up bias, even when publication bias is not an issue. We urge researchers to consider alternative approaches to determining the sample size of their studies, and discuss several options.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303
Self-citationEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 3×
“…are exactly the situations least representative of current practices in psychology, where e ff ect sizes are often not large (Richard, Bond, & Stokes-Zoota, 2003), studies have low sample sizes (Fraley & Vazire, 2014), and η 2 is used more often than ε 2 or ω 2 (Open Science Collaboration, 2015).”
BackgroundMethodInfluentialNursing Research: Principles and Methods
McFarland, G. K., Polit, D. F., & Hungler, B. P. (1987) · The American Journal of Nursing · doi:10.2307/3425916 · Cited by 6,942
Effect size estimates: Current use, calculations, and interpretation.
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024338 · Cited by 5,199
Jaeschke, R., Singer, J., & Guyatt, G. H. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90005-6 · Cited by 4,982
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 2,161 · cited 1×
“For example, if researchers decide their SESOI is a medium e ff ect size of η 2 = .0588 a study with 87 participants in each of two groups will in the long run have a power of 0.9 to detect the SESOI, or reject it in an equivalence test (Lakens, 2017).”
BackgroundInfluentialSelf-citationA Primer of Multivariate Statistics
Strauss, D. J., & Harris, R. J. (1975) · Biometrics · doi:10.2307/2529445 · Cited by 1,975
The role and interpretation of pilot studies in clinical research
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.10.008 · Cited by 1,917 · cited 1×
“Pilot studies can be performed because they have other uses, such as determining the feasibility of performing the designed study in practice, which is especially useful when trying out new methods or procedures (Leon et al., 2011).”
BackgroundBoneau, C. A., & Hays, W. L. (1966) · The American Journal of Psychology · doi:10.2307/1420739 · Cited by 1,881
Sur les fonctions convexes et les inégalités entre les valeurs moyennes
Jensen, J. L. W. V. (1906) · Acta Mathematica · doi:10.1007/bf02418571 · Cited by 1,811
Sequential Tests of Statistical Hypotheses
Wald, A. (1945) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731118 · Cited by 1,671
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Sequential Tests of Statistical Hypotheses
Wald, A. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_18 · Cited by 1,345
Statistics for Psychologists: An Intermediate Course
Everitt, B. S. (2001) · Psychology Press eBooks · doi:10.4324/9781410605542 · Cited by 1,045
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“…more likely to follow-up on initial studies that yielded higher e ff ect size estimates than initial studies that yielded lower e ff ect size estimates (cf. Greenwald, 1975), simply because these studies require less resources to observe a statistically signi fi cant result in the expected direction.”
ResultEta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998
Sequential Analysis, Tests and Confidence Intervals
Schmee, J. (1987) · Technometrics · doi:10.1080/00401706.1987.10488200 · Cited by 719
. Caution regarding the use of pilot studies to guide power calculations for study proposals.
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.484 · Cited by 702
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
pwr: Basic Functions for Power Analysis
Champely, S. (2006) · CRAN: Contributed Packages · doi:10.32614/cran.package.pwr · Cited by 550
A Primer of Multivariate Statistics
Harris, R. J. (2001) · Psychology Press eBooks · doi:10.4324/9781410600455 · Cited by 506
Performing high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414
Self-citationFraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409
When effect sizes disagree: The case of r and d.
McGrath, R. E., & Meyer, G. J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.386 · Cited by 399
How to report statistics in medicine: annotated guidelines for authors, editors, and reviewers
(1997) · Choice Reviews Online · doi:10.5860/choice.35-1556 · Cited by 308
Statistical Power and Effect Size in Marketing Research
Sawyer, A. G., & Ball, A. D. (1981) · Journal of Marketing Research · doi:10.1177/002224378101800302 · Cited by 269
Lakens, D., & Evers, E. R. K. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528520 · Cited by 266
Self-citationNursing Research— Principles and Methods
Lancaster, D. R. (1992) · The Journal of Cardiovascular Nursing · doi:10.1097/00005082-199207000-00011 · Cited by 228
A Method of Sampling Inspection
Dodge, H. F., & Romig, H. G. (1929) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1929.tb01240.x · Cited by 215
Safeguard Power as a Protection Against Imprecise Power Estimates
Perugini, M., Gallucci, M., & Costantini, G. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528519 · Cited by 198
L., J. D. (1928) · Agronomy Journal · doi:10.2134/agronj1928.00021962002000080010x · Cited by 188
An Unbiased Correlation Ratio Measure
Kelley, T. L. (1935) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.21.9.554 · Cited by 153
Statistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1928) · Journal of the American Statistical Association · doi:10.2307/2276756 · Cited by 145
Anderson, S. F., & Maxwell, S. E. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1289361 · Cited by 143
Is Omega Squared Less Biased? a Comparison of Three Major Effect Size Indices in One-Way Anova
Okada, K. (2013) · Behaviormetrika · doi:10.2333/bhmk.40.129 · Cited by 128
Sampling Characteristics of Kelley's ε and Hays' ω
Carroll, R. M., & Nordholm, L. A. (1975) · Educational and Psychological Measurement · doi:10.1177/001316447503500304 · Cited by 85
Sakaluk, J. K. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.013 · Cited by 76
Persuasiveness of one- and two-sided communications as a function of awareness there are two sides
Jones, R., & Brehm, J. W. (1970) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(70)90075-2 · Cited by 58
Bias in linear model power and sample size calculation due to estimating noncentrality
Taylor, D. J., & Muller, K. E. (1996) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929608831787 · Cited by 49
On a Correction to be Made to the Correlation Ratio η
Pearson, K. (1911) · Biometrika · doi:10.2307/2331454 · Cited by 24
Cited by
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
Self-citationBrysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R
Kumle, L., Võ, M. L.-H., & Draschkow, D. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01546-0 · Cited by 552
A Practical Primer To Power Analysis for Simple Experimental Designs
Perugini, M., Gallucci, M., & Costantini, G. (2018) · International Review of Social Psychology · doi:10.5334/irsp.181 · Cited by 408
Ros, T., Enriquez-Geppert, S., Zotev, V., Young, K. D., Wood, G., Whitfield-Gabrieli, S., Wan, F., Vuilleumier, P., Vialatte, F., Van De Ville, D., et al. (2020) · Brain · doi:10.1093/brain/awaa009 · Cited by 399
Metadata
- Authors
- Albers, C., & Lakens, D.
- Venue
- Journal of Experimental Social Psychology
- Publisher
- Elsevier BV
- Published
- 2018-01
- Volume
- 74
- Pages
- 187-195
- ISSN
- 0022-1031 (ISSN-L)
- OpenAlex
- W2766522838
- Cited by
- 357
- References
- 45
- FWCI
- 20.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.