Researchers’ Intuitions About Power in Psychological Research
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers’ experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…centrality of power in the debate regarding reproducibility and replicability of research in psychology and beyond (e.g., Asendorpf et al., 2013; Button et al., 2013; Gilbert, King, Pettigrew, & Wilson, 2016; Open Science Collaboration, 2015), we surveyed psychology researchers on their…”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 2×
“In a classic study, Tversky and Kahneman (1971) showed that even quantitatively oriented psychologists underestimated the randomness in small samples.”
BackgroundResultMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“…ES was 0.39, which is somewhat lower than the estimated mean ES obtained in large-scale meta-analyses of psychological research (d = 0.5 on average; Anderson, Lindsay, & Bushman, 1999; Lipsey & Wilson, 1993; Meyer et al., 2001; Richard, Bond, & StokesZoota, 2003; Tett, Meyer, & Roese, 1994).”
InfluentialThings I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Cited by
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences
Hedge, C., Powell, G., & Sumner, P. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0935-1 · Cited by 1,457
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
“Also, typical study designs are woefully under-powered for studying moderation (Frazier, Tix, & Barron, 2004; McClelland, 1997), perhaps because researchers intuitively overestimate the power of various research designs (Bakker, Hartgerink, Wicherts, & van der Maas, 2016).”
BackgroundBrysbaert, M. (2019) · Journal of Cognition · doi:10.5334/joc.72 · Cited by 996
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Self-citationAnderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Metadata
- Authors
- Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J.
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2016-07-20
- Volume
- 27
- Issue
- 8
- Pages
- 1069-1077
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2475350511
- Cited by
- 146
- References
- 22
- FWCI
- 22.31
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Researchers’ Intuitions About Power in Psychological Research. https://app.cochat.ai/papers/10.1177/0956797616647519
- The paper
- Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016). Researchers’ Intuitions About Power in Psychological Research. Psychological Science. https://doi.org/10.1177/0956797616647519
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.