The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Limitations of PET-PEESE and Other Meta-Analysis Methods
Open accessAbstract
A novel meta-regression method, precision-effect test and precision-effect estimate with standard errors (PET-PEESE), predicts and explains recent high-profile failures to replicate in psychology. The central purpose of this article is to identify the limitations of PET-PEESE for application to social/personality psychology. Using typical conditions found in social/personality research, our simulations identify three areas of concern. PET-PEESE performs poorly in research areas where there are only a few studies, all studies use small samples, and where there is very high heterogeneity of results from study to study. Nonetheless, the statistical properties of conventional meta-analysis approaches are much worse than PET-PEESE under these same conditions. Our simulations suggest alterations to conventional research practice and ways to moderate PET-PEESE weaknesses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall, PET-PEESE has a low false positive rate, but its estimation of the effect size tends to be imprecise with a sample size of small, heterogeneous studies, resulting in large standard errors, such that 'if all studies are small, PET-PEESE has little power to identify a genuine empirical effect' (p. 590, Stanley, 2017)5.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley, 2017
In the citing paper
“However, this counterintuitive finding may be explained by the fact that PET is prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking (Stanley & Doucouliagos, 2014; Stanley 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley 2017
In the citing paper
“However, when effects are highly heterogeneous and when sample sizes are small, the PET-PEESE analysis is known to yield imprecise estimates (Stanley & Doucouliagos, 2014; Stanley, 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley, 2017
In the citing paper
“Furthermore, its estimate should be interpreted with caution in small meta-analyses (with 20 studies or less; Stanley, 2017).”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Stanley, 2017
In the citing paper
“Regarding the adjustment of the effect, the original PET-PEESE (which assumes independence) performed worse with smaller true effects and higher heterogeneity, consistently underestimating the true effect.”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Stanley, 2017
In the citing paper
“There are two main threats to the validity of meta-analytic results: the systematic omission of studies (e.g., publication bias suppressing studies with null results36,37), and the flexible selection of analysis approaches within each study inflating the rate of false positives (e.g., p-hacking38).”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [37]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,377
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4×
“For decades, publication bias has been widely acknowledged to be a serious, yet intractable, problem for empirical science (Begg & Berlin, 1988; Glass et al., 1981; Hedges & Oklin, 1985; Rosenthal, 1979; Schmidt & Hunter, 2014; Sterling, 1959).”
InfluentialEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,629
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Research works printed but not linked: 2
- Greene W. E., 1990, Econometric analysis
- Johnson N., 1970, Distributions in statistics: Continuous univariate distribution
Cited by
Sorted by citations
A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808
Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 399
Physical activity and subjective well-being in healthy individuals: a meta-analytic review
Buecker, S., Simacek, T., Ingwersen, B., Terwiel, S., & Simonsmeier, B. A. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1760728 · Cited by 374
Metadata
- Authors
- Stanley, T. D.
- Affiliations
- Hendrix College (US)
- Corresponding author
- Stanley, T. D.
- ORCID
- Stanley, T. D.
- Venue
- Social Psychological and Personality Science
- Publisher
- SAGE Publications
- Published
- 2017-02-28
- Volume
- 8
- Issue
- 5
- Pages
- 581-591
- ISSN
- 1948-5506 (ISSN-L)
- 1948-5514
- OpenAlex
- W2591641564
- Semantic Scholar
- 62d9746bdeace29af787162583753de042a82d8a
- Cited by
- 229
- References
- 37
- Citations by year
- 2026: 23 · 2025: 29 · 2024: 21 · 2023: 33 · 2022: 25 · 2021: 27 · 2020: 21 · 2019: 27 · 2018: 18 · 2017: 5
- FWCI
- 17.48
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- Social Psychological and Personality Science (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- meta-regression, replication failure, small sample bias, effect size heterogeneity, meta-analysis limitations, social-personality psychology, simulation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
