Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli.
Abstract
Researchers designing experiments in which a sample of participants responds to a sample of stimuli are faced with difficult questions about optimal study design. The conventional procedures of statistical power analysis fail to provide appropriate answers to these questions because they are based on statistical models in which stimuli are not assumed to be a source of random variation in the data, models that are inappropriate for experiments involving crossed random factors of participants and stimuli. In this article, we present new methods of power analysis for designs with crossed random factors, and we give detailed, practical guidance to psychology researchers planning experiments in which a sample of participants responds to a sample of stimuli. We extensively examine 5 commonly used experimental designs, describe how to estimate statistical power in each, and provide power analysis results based on a reasonable set of default parameter values. We then develop general conclusions and formulate rules of thumb concerning the optimal design of experiments in which a sample of participants responds to a sample of stimuli. We show that in crossed designs, statistical power typically does not approach unity as the number of participants goes to infinity but instead approaches a maximum attainable power value that is possibly small, depending on the stimulus sample. We also consider the statistical merits of designs involving multiple stimulus blocks. Finally, we provide a s…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most NLP tasks do not take the form of standard experiments in other sciences, so it is nontrivial to run power analyses for many tasks of interest.
“Because most NLP tasks do not take the form of standard experiments in other sciences (Kraemer and Blasey, 2015; Westfall et al., 2014), it is nontrivial to run power analyses for many tasks of interest.”
From With Little Power Comes Great Responsibility (2020) · cites this paper as Westfall et al., 2014
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
Mixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540 · see citing works
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 2× · see citing works
“…salient in light of the growing awareness in psychology and other experimental sciences of the problems created and maintained by a proliferation of chronically underpowered studies (Asendorpf et al., 2013; Bakker, van Dijk, & Wicherts, 2012; Button et al., 2013; Ioannidis, 2008; Schimmack, 2012).”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Partitioning Variation in Multilevel Models
Goldstein, H., Browne, W., & Rasbash, J. (2002) · Understanding Statistics · doi:10.1207/s15328031us0104_02 · Cited by 874 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Standard Errors and Sample Sizes for Two-Level Research
Snijders, T. A. B., & Bosker, R. J. (1993) · Journal of Educational Statistics · doi:10.3102/10769986018003237 · Cited by 487 · cited 2× · see citing works
“…there is an extensive literature in education and applied statistics on nested designs, commonly referred to as hierarchically nested designs, and this literature has considered issues of power and optimal design (Raudenbush, 1997; Raudenbush & Liu, 2000; Snijders, 2001; Snijders & Bosker, 1993).”
BackgroundThe ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
Average Values of Mean Squares in Factorials
Cornfield, J., & Tukey, J. W. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728067 · Cited by 292 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210 · see citing works
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962 · see citing works
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420 · see citing works
“First, Westfall et al. (2014) showed how you can calculate the effect size (measured as d) for a design with random participants and random items.”
BackgroundMethodInfluentialExperiments with More Than One Random Factor: Designs, Analytic Models, and Statistical Power
Judd, C. M., Westfall, J., & Kenny, D. A. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033702 · Cited by 466 · see citing works
Self-citationPatil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243 · see citing works
Understanding Mixed-Effects Models Through Data Simulation
DeBruine, L. M., & Barr, D. J. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920965119 · Cited by 148 · see citing works
Mehta, A., Niles, A. N., Vargas, J. H., Marafon, T., Couto, D. D., & Gross, J. J. (2021) · Journal of Medical Internet Research · doi:10.2196/26771 · Cited by 143 · see citing works
Ketamine can reduce harmful drinking by pharmacologically rewriting drinking memories
Das, R. K., Gale, G., Walsh, K., Hennessy, V. E., Iskandar, G., Mordecai, L. A., Brandner, B., Kindt, M., Curran, H. V., & Kamboj, S. K. (2019) · Nature Communications · doi:10.1038/s41467-019-13162-w · Cited by 122 · see citing works
Goodale, B. M., Shilaih, M., Falco, L., Dammeier, F., Hamvas, G., & Leeners, B. (2019) · Journal of Medical Internet Research · doi:10.2196/13404 · Cited by 123 · see citing works
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Chen, X., Affourtit, J., Ryskin, R., Regev, T. I., Norman-Haignere, S., Jouravlev, O., Malik-Moraleda, S., Kean, H., Varley, R., & Fedorenko, E. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhad087 · Cited by 57 · see citing works
Kraus, B., & Kitayama, S. (2019) · Biological Psychology · doi:10.1016/j.biopsycho.2019.107733 · Cited by 46 · see citing works
Articulatory suppression delays processing of abstract words: the role of inner speech
Fini, C., Zannino, G. D., Orsoni, M., Carlesimo, G. A., Benassi, M., & Borghi, A. M. (2022) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218211053623 · Cited by 41 · see citing works
Soncini, A., Matteucci, M. C., & Butera, F. (2021) · European Journal of Psychology of Education · doi:10.1007/s10212-020-00494-1 · Cited by 39 · see citing works
Mazurek Melnyk, B., Hoying, J., & Tan, A. (2022) · Journal of American College Health · doi:10.1080/07448481.2020.1782922 · Cited by 36 · see citing works
Arch, J. J., Mitchell, J. L., Genung, S. R., Fisher, R., Andorsky, D. J., & Stanton, A. L. (2019) · BMC Cancer · doi:10.1186/s12885-019-5289-x · Cited by 19 · see citing works
Thompson, E. M., Albertella, L., Viskovich, S., Pakenham, K. I., & Fontenelle, L. F. (2024) · Behaviour Research and Therapy · doi:10.1016/j.brat.2024.104595 · Cited by 8 · see citing works
Yoga is effective for treating chronic pain in veterans with Gulf War Illness at long-term follow-up
Allende, S., Mathersul, D. C., Schulz-Heik, J. R., Avery, T. J., Mahoney, L., & Bayley, P. J. (2023) · BMC Complementary Medicine and Therapies · doi:10.1186/s12906-023-04145-y · Cited by 5 · see citing works
“To derive a Cohen’s d effect size at each follow-up time period, we divided the difference in treatment means at each time point by the square root of the sum of variances of the random effects (i.e., intercept, slope, and residual variance components) [33].”
Methodde Haas, S. M. H., Blaesing, L., Oosterhoff, R., Rozental, A., & Scheres, A. P. J. (2026) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2025.2543893 · Cited by 3 · see citing works
“Additionally, we reported standardized effect sizes (i.e. labelled “d”) for the confirmatory mixed models, following the equation outlined by Westfall et al. (2014), which are similar in interpretation to the more traditional Cohen’s d in Anova-analyses.”
Rubel, J., Quest, J., Pruessner, L., Timm, C., Hartmann, S., Barnow, S., Rittmeyer, L., Rosenbaum, D., & Lalk, C. (2023) · JMIR Research Protocols · doi:10.2196/41440 · Cited by 2 · see citing works
“Hedges [50] and Westfall et al [51] propose an effect size based on the Cohen d, which is also used for power analysis [47]: To assess the magnitude of the treatment effects, the fixed interaction effect of time and group assignment is divided by the root of the summed variances of the random effects.”
Method
- Authors
- Westfall, J., Kenny, D. A., & Judd, C. M.
- Affiliations
- University of Connecticut (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014
- Volume
- 143
- Issue
- 5
- Pages
- 2020-2045
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W2329873939
- Cited by
- 928
- References
- 9
- Citations by year
- 2026: 61 · 2025: 201 · 2024: 100 · 2023: 126 · 2022: 92 · 2021: 104 · 2020: 90 · 2019: 49 · 2018: 35 · 2017: 28 · 2016: 27 · 2015: 10 · 2014: 2
- FWCI
- 28.21
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- statistical power analysis, optimal experimental design, stimulus sampling, default parameter values
- MeSH terms
- Humans, Research Design, Statistics as Topic, Models, Statistical, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
