The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Limitations of PET-PEESE and Other Meta-Analysis Methods
Open accessAbstract
A novel meta-regression method, precision-effect test and precision-effect estimate with standard errors (PET-PEESE), predicts and explains recent high-profile failures to replicate in psychology. The central purpose of this article is to identify the limitations of PET-PEESE for application to social/personality psychology. Using typical conditions found in social/personality research, our simulations identify three areas of concern. PET-PEESE performs poorly in research areas where there are only a few studies, all studies use small samples, and where there is very high heterogeneity of results from study to study. Nonetheless, the statistical properties of conventional meta-analysis approaches are much worse than PET-PEESE under these same conditions. Our simulations suggest alterations to conventional research practice and ways to moderate PET-PEESE weaknesses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall, PET-PEESE has a low false positive rate, but its estimation of the effect size tends to be imprecise with a sample size of small, heterogeneous studies, resulting in large standard errors, such that 'if all studies are small, PET-PEESE has little power to identify a genuine empirical effect' (p. 590, Stanley, 2017)5.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley, 2017
In the citing paper
“However, this counterintuitive finding may be explained by the fact that PET is prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking (Stanley & Doucouliagos, 2014; Stanley 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley 2017
In the citing paper
“However, when effects are highly heterogeneous and when sample sizes are small, the PET-PEESE analysis is known to yield imprecise estimates (Stanley & Doucouliagos, 2014; Stanley, 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley, 2017
In the citing paper
“Furthermore, its estimate should be interpreted with caution in small meta-analyses (with 20 studies or less; Stanley, 2017).”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Stanley, 2017
In the citing paper
“Regarding the adjustment of the effect, the original PET-PEESE (which assumes independence) performed worse with smaller true effects and higher heterogeneity, consistently underestimating the true effect.”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Stanley, 2017
In the citing paper
“There are two main threats to the validity of meta-analytic results: the systematic omission of studies (e.g., publication bias suppressing studies with null results36,37), and the flexible selection of analysis approaches within each study inflating the rate of false positives (e.g., p-hacking38).”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [37]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,377
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4×
“For decades, publication bias has been widely acknowledged to be a serious, yet intractable, problem for empirical science (Begg & Berlin, 1988; Glass et al., 1981; Hedges & Oklin, 1985; Rosenthal, 1979; Schmidt & Hunter, 2014; Sterling, 1959).”
InfluentialEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,629
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Research works printed but not linked: 2
- Greene W. E., 1990, Econometric analysis
- Johnson N., 1970, Distributions in statistics: Continuous univariate distribution
Cited by
Sorted by citations
Items 201–232 of 232
Fifty Years of Anchoring Effects: A Theoretical Reintegration and Meta-Analysis
Schley, D. R., & Weingarten, E. (2026) · Management Science · doi:10.1287/mnsc.2023.03238 · Cited by 2
Rotella, A. (2020) · The Atrium (University of Guelph) · Cited by 1
Supplemental Material for Risk Factors for Child Sexual Abuse Victimization: A Meta-Analytic Review
(2019) · Psychological Bulletin · doi:10.1037/bul0000188.supp · Cited by 2
Beyond Statistical Significance: Five Principles for the New Era of Data Analysis and ReportingPreprint
Wedel, M., & Gal, D. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6kvnf · Cited by 1
Does digital intergroup contact reduce prejudice? A meta-analysisPreprint
da Costa, L. P., Bierwiaczonek, K., & Bianchi, M. (2023) · doi:10.31234/osf.io/j937s · Cited by 1
Effectiveness of Multimedia Condition on Learning Second-Language Vocabulary: A Meta-analysisPreprint
Zhang, C., Sala, G., & Gobet, F. R. (2023) · doi:10.31234/osf.io/yge9n · Cited by 1
Bartos̆, F., Maier, M., Stanley, T. D., & Wagenmakers, E. (2023) · doi:10.31234/osf.io/98xb5 · Cited by 1
Self-citationThe Unconscious: What we know, should know, and what we (should not) have forgotten.Preprint
Tsikandilakis, M., Bali, P., Karlis, A.-K., Morfi, P., Mével, P.-A., Madan, C. R., Derrfuß, J., & Milbank, A. G. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/j46ye · Cited by 1
Revisiting the Unconscious: What we really know and what we should not have forgottenPreprint
Tsikandilakis, M., Bali, P., Mével, P.-A., Madan, C. R., & Milbank, A. G. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/45ygf · Cited by 2
Pan and Rickard (2018) Transfer of test-enhanced learning meta-analysis (preprint)Preprint
Pan, S. C. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4qpyn · Cited by 1
Sex differences in trust and trustworthinessPreprint
van den Akker, O. R. (2018) · OSF Preprints (OSF Preprints)
Augusteijn, H. E. M., van Aert, R. C. M., & van Assen, M. A. L. M. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/avkgj
A systematic review and meta-analysis of vineyard techniques used to delay ripening.Preprint
Previtali, P., Giorgini, F., Mullen, R. S., Dokoozlian, N., Wilkinson, K. L., & Ford, C. M. (2021) · doi:10.31220/agrirxiv.2021.00112
vanAert_Niemeyer21Preprint
van Aert, R. C. M., & Niemeyer, H. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/3rdys
Bruns, A. K., Henriksson, I., Nordström, T., & Carlsson, R. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1389
СПЕЦИФІКА ЗВʼЯЗКУ ОСОБИСТІСНИХ ХАРАКТЕРИСТИК ТА ПЕРЕЖИВАННЯ САМОТНОСТІ ЧОЛОВІКАМИ ЗРІЛОГО ВІКУ
Стеценко, І. І., & Kubrichenko, T. V. (2024) · Cai fu bi ji · doi:10.15421/102510
Construct validity of the Child Abuse Potential Inventory: A meta-analytic review
Milner, J. S., Ammar, J., Gibson, C., Menon, S. V., & Crouch, J. L. (2025) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2025.107586 · Cited by 1
Rethinking Type S and M ErrorsPreprint
Lakens, D., Mesquida, C., Xavier-Quintais, G., Rasti, S., Toffalini, E., & Altoè, G. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2phzb_v1
Park, S.-H., & Kim, E. Y. (2026) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2026.105269
The Behavioral Impact of Moral Appeals: An Integrated Framework and Meta-Analysis
Xu, S. Y., Mulder, L. B., Rink, F. A., Bijmolt, T. H. A., & Leliveld, M. C. (2026) · Personality and Social Psychology Bulletin · doi:10.1177/01461672261437934 · Cited by 1
Effect of episodic specificity induction on divergent thinking: A meta-analytic study
Sánchez, G., Bajo, M. T., & Marful, A. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-026-02859-x
Unemployment, inactivity, and hiring chances: A systematic review and meta-analysis
D’hert, L., Baert, S., & Lippens, L. (2026) · Socio-Economic Review · doi:10.1093/ser/mwag023
Rethinking Type S and Type M Errors
Lakens, D., Mesquida, C., Xavier-Quintais, G., Rasti, S., Toffalini, E., & Altoè, G. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261432530
Donolato, E., Caviola, S., Masi, A., Lervåg, A., & Melby‐Lervåg, M. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100815 · Cited by 1
Bodynek, J., Gaigulo, D., Mayer, A., & Jonas, K. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100831 · Cited by 1
The Returns to Education: A Comment
Crawfurd, L. (2026) · Kyklos · doi:10.1111/kykl.70081 · Cited by 1
Cong, Q., van Geel, M., Zhao, X., & Vedder, P. H. (2026) · Trauma Violence & Abuse · doi:10.1177/15248380261476689
Guidelines for Meta‐Analytic Structural Equation Modeling: Steps, Resources, and Illustrations
Waheed, H., & Lim, W. M. (2026) · Psychology and Marketing · doi:10.1002/mar.70266
Workplace Interventions to Promote Employees' Pro‐Environmental Behaviour: A Meta‐Analysis
Brack, F., Petrowsky, H. M., Hörisch, J., & Loschelder, D. D. (2026) · Human Resource Management Journal · doi:10.1111/1748-8583.70066
Chen, M. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03206-7
Sala, G., & Gobet, F. (2019) · Collabra: Psychology · doi:10.2478/collabra.2019.5.1.18
Metadata
- Authors
- Stanley, T. D.
- Affiliations
- Hendrix College (US)
- Corresponding author
- Stanley, T. D.
- ORCID
- Stanley, T. D.
- Venue
- Social Psychological and Personality Science
- Publisher
- SAGE Publications
- Published
- 2017-02-28
- Volume
- 8
- Issue
- 5
- Pages
- 581-591
- ISSN
- 1948-5506 (ISSN-L)
- 1948-5514
- OpenAlex
- W2591641564
- Semantic Scholar
- 62d9746bdeace29af787162583753de042a82d8a
- Cited by
- 229
- References
- 37
- Citations by year
- 2026: 23 · 2025: 29 · 2024: 21 · 2023: 33 · 2022: 25 · 2021: 27 · 2020: 21 · 2019: 27 · 2018: 18 · 2017: 5
- FWCI
- 17.48
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- Social Psychological and Personality Science (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- meta-regression, replication failure, small sample bias, effect size heterogeneity, meta-analysis limitations, social-personality psychology, simulation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
