The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication.
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Repeated investigations of the same phenomenon typically yield effect sizes that vary more than one would expect from sampling error alone. Such variation is even found in exact replication studies, suggesting that it is not only because of identifiable moderators but also to subtler random variation across studies. Such heterogeneity of effect sizes is typically ignored, with unfortunate consequences. We consider its implications for power analyses, the precision of estimated effects, and the planning of original and replication research. With heterogeneity and an interest in generalizing to a population of studies, the usual power calculations and confidence intervals are likely misleading, and the preference for single definitive large-N studies is misguided. Researchers and methodologists need to recognize that effects are often heterogeneous and plan accordingly. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · cited 4×
“Meta-analysts typically test whether this quantity differs from zero using a 2 test of homogeneity, symbolized as Q (Cochran, 1954). (Several studies, e.g., Chung, Rabe-Hesketh, and Choi [2013], have shown that this test has relatively low power to detect heterogeneity.) Not always reported is the estimated true varia…”
MethodInfluentialBelief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 1×
“In their classic paper on “The Law of Small Numbers,” Tversky and Kahneman (1971) described an experimenter who does the same study twice and in the first study, he or she obtains a significant effect, whereas in the second study the effect is no longer significant.”
BackgroundFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Pigott, T. D. (2012) · doi:10.1007/978-1-4614-2278-5 · Cited by 453 · cited 1×
“Following the original suggestion by Pigott (2012); McShane and Böckenholt (2014) suggest using 0.10 for small heterogeneity, 0.20 for medium, and 0.35 for large.”
MethodInfluentialMcShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398
Cited by
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
What can be learned from growth mindset controversies?
Yeager, D. S., & Dweck, C. S. (2020) · American Psychologist · doi:10.1037/amp0000794 · Cited by 868
“…made it difficult to distinguish between studies that call into question the foundations of a field and studies that are a part of the normal accumulation of knowledge about heterogeneity (about where effects are likely to be stronger or weaker; see Kenny & Judd, 2019; Tipton et al., 2019).”
BackgroundBehavioural science is unlikely to change the world without a heterogeneity revolution
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01143-3 · Cited by 537
Bayesian Meta-Analysis with Weakly Informative Prior Distributions
Williams, D. R., Rast, P., & Bürkner, P. C. (2018) · doi:10.31234/osf.io/7tbrm · Cited by 124
“This topic has received little consideration in the psychological literature, but has serious implications for cumulative science (Kenny & Judd, 2017).”
Background
Metadata
- Authors
- Kenny, D. A., & Judd, C. M.
- Venue
- Psychological Methods
- Published
- 2019-10
- Volume
- 24
- Issue
- 5
- Pages
- 578-589
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2797364217
- Cited by
- 156
- References
- 13
- FWCI
- 13.85
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication. https://app.cochat.ai/papers/10.1037/met0000209
- The paper
- Kenny, D. A., & Judd, C. M. (2019). The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication. Psychological Methods. https://doi.org/10.1037/met0000209
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.