Limitations of PET-PEESE and Other Meta-Analysis Methods

Stanley, T. D.

doi:10.1177/1948550617693062

Cited by 232Meta-analysis and systematic reviewsPsychology Research and BibliometricsStatistical Methods and Bayesian Inference

Limitations of PET-PEESE and Other Meta-Analysis Methods

Open access

Stanley, T. D.·2017·Social Psychological and Personality Science

DOI 10.1177/1948550617693062

Open this paper in CoChat — free account

Abstract

A novel meta-regression method, precision-effect test and precision-effect estimate with standard errors (PET-PEESE), predicts and explains recent high-profile failures to replicate in psychology. The central purpose of this article is to identify the limitations of PET-PEESE for application to social/personality psychology. Using typical conditions found in social/personality research, our simulations identify three areas of concern. PET-PEESE performs poorly in research areas where there are only a few studies, all studies use small samples, and where there is very high heterogeneity of results from study to study. Nonetheless, the statistical properties of conventional meta-analysis approaches are much worse than PET-PEESE under these same conditions. Our simulations suggest alterations to conventional research practice and ways to moderate PET-PEESE weaknesses.

Integrity signals

Registry

Crossref · 10.1177/1948550617693062

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

PET-PEESE has a low false positive rate, but its estimation of the effect size tends to be imprecise with a sample size of small, heterogeneous studies, resulting in large standard errors, such that if all studies are small, PET-PEESE has little power to identify a genuine empirical effect.
“Overall, PET-PEESE has a low false positive rate, but its estimation of the effect size tends to be imprecise with a sample size of small, heterogeneous studies, resulting in large standard errors, such that 'if all studies are small, PET-PEESE has little power to identify a genuine empirical effect' (p. 590, Stanley, 2017)5.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley, 2017
The counterintuitive PET finding may be explained by PET being prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking.
“However, this counterintuitive finding may be explained by the fact that PET is prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking (Stanley & Doucouliagos, 2014; Stanley 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley 2017
When effects are highly heterogeneous and sample sizes are small, PET-PEESE analysis is known to yield imprecise estimates.
“However, when effects are highly heterogeneous and when sample sizes are small, the PET-PEESE analysis is known to yield imprecise estimates (Stanley & Doucouliagos, 2014; Stanley, 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Stanley, 2017
PET-PEESE's estimate should be interpreted with caution in small meta-analyses (with 20 studies or less).
“Furthermore, its estimate should be interpreted with caution in small meta-analyses (with 20 studies or less; Stanley, 2017).”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Stanley, 2017
The original PET-PEESE (which assumes independence) performed worse with smaller true effects and higher heterogeneity, consistently underestimating the true effect.
“Regarding the adjustment of the effect, the original PET-PEESE (which assumes independence) performed worse with smaller true effects and higher heterogeneity, consistently underestimating the true effect.”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Stanley, 2017

Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Book Reviews: Statistical Methods for Meta-Analysis Larry V. Hedges and Ingram Olkin New York: Academic Press, 1985. xxii + 369 pp
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 4× · see citing works
“For decades, publication bias has been widely acknowledged to be a serious, yet intractable, problem for empirical science (Begg & Berlin, 1988; Glass et al., 1981; Hedges & Oklin, 1985; Rosenthal, 1979; Schmidt & Hunter, 2014; Sterling, 1959).”
Influential
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765 · see citing works
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,628 · see citing works
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,896 · see citing works
Nicotine replacement therapy for smoking cessation
Stead, L. F., Perera, R., Bullen, C., Mant, D., Hartmann‐Boyce, J., Cahill, K., & Lancaster, T. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000146.pub4 · Cited by 2,514 · see citing works
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
A Multilab Preregistered Replication of the Ego-Depletion Effect
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H. J. E. M., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brand, M., Brewer, G., Bruyneel, S. D., et al. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616652873 · Cited by 1,097 · see citing works
Estimation of a Single Effect Size: Parametric and Nonparametric Methods
Hedges, L. V., & Olkin, I. (1985) · Statistical Methods for Meta-Analysis · doi:10.1016/b978-0-08-057065-5.50010-5 · Cited by 983 · see citing works
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 3× · see citing works
“For decades, publication bias has been widely acknowledged to be a serious, yet intractable, problem for empirical science (Begg & Berlin, 1988; Glass et al., 1981; Hedges & Oklin, 1985; Rosenthal, 1979; Schmidt & Hunter, 2014; Sterling, 1959).”
Meta-regression approximations to reduce publication selection bias: T. D. STANLEY AND H. DOUCOULIAGOS
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
Self-citation
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
Meta-Regression Analysis in Economics and Business
Stanley, T. D., & Doucouliagos, H. (2012) · Routledge eBooks · doi:10.4324/9780203111710 · Cited by 788 · see citing works
Self-citation
Improving fluid intelligence with training on working memory: a meta-analysis
Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0699-x · Cited by 671 · cited 1× · see citing works
“Nonetheless, these sample sizes are found in at least one psychological meta-analysis on the transfer of working memory to fluid intelligence (Au et al., 2015; Boggs & Lasecki, 2015).”
Background
The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.
Maxwell, S. E. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.2.147 · Cited by 655 · see citing works
Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection*
Stanley, T. D. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2007.00487.x · Cited by 605 · see citing works
Self-citation
A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource.
Carter, E. C., Kofler, L., Forster, D. E., & McCullough, M. E. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000083 · Cited by 539 · see citing works
Meta-analysis, funnel plots and sensitivity analysis
Copas, J. (2000) · Biostatistics · doi:10.1093/biostatistics/1.3.247 · Cited by 503 · see citing works
The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power
Fraley, R. C., & Vazire, S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109019 · Cited by 409 · see citing works
Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398 · see citing works
Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study
Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373 · see citing works
Self-citation
Neither fixed nor random: weighted least squares meta‐analysis
Stanley, T. D., & Doucouliagos, H. (2015) · Statistics in Medicine · doi:10.1002/sim.6481 · Cited by 249 · see citing works
Self-citation
Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p -Uniform and p -Curve
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219 · see citing works
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210 · see citing works
Neither fixed nor random: weighted least squares meta‐regression
Stanley, T. D., & Doucouliagos, H. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1211 · Cited by 205 · see citing works
Self-citation
Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications
Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., & Ades, A. E. (2009) · BMJ · doi:10.1136/bmj.b2981 · Cited by 184 · see citing works
Could It Be Better to Discard 90% of the Data? A Statistical Paradox
Stanley, T. D., Jarrell, S. B., & Doucouliagos, H. (2010) · The American Statistician · doi:10.1198/tast.2009.08205 · Cited by 170 · see citing works
Self-citation
Finding the power to reduce publication bias
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7228 · Cited by 131 · see citing works
Self-citation
A generalized weighting regression‐derived meta‐analysis estimator robust to small‐study effects and heterogeneity
Moreno, S. G., Sutton, A. J., Thompson, J. R., Ades, A. E., Abrams, K. R., & Cooper, N. J. (2012) · Statistics in Medicine · doi:10.1002/sim.4488 · Cited by 87 · see citing works
Reliable gains? Evidence for substantially underpowered designs in studies of working memory training transfer to fluid intelligence
Bogg, T., & Lasecki, L. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01589 · Cited by 41 · see citing works

Flagged first, then by citations

Sorted by citations

A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808 · see citing works
Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807 · see citing works
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588 · see citing works
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O’Dea, R. E. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 455 · see citing works
Physical activity and subjective well-being in healthy individuals: a meta-analytic review
Buecker, S., Simacek, T., Ingwersen, B., Terwiel, S., & Simonsmeier, B. A. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1760728 · Cited by 374 · see citing works
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 345 · see citing works
Risk factors for child sexual abuse victimization: A meta-analytic review.
Assink, M., van der Put, C. E., Meeuwsen, M. W. C. M., de Jong, N. M., Oort, F. J., Stams, G. J. J. M., & Hoeve, M. (2019) · Psychological bulletin · doi:10.1037/bul0000188 · Cited by 322 · see citing works
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307 · see citing works
Self-citation
Cognitive Training Does Not Enhance General Cognition
Sala, G., & Gobet, F. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.10.004 · Cited by 303 · see citing works
The impact of shared book reading on children's language skills: A meta-analysis
Noble, C. H., Sala, G., Peter, M., Lingwood, J., Rowland, C. F., Gobet, F. R., & Pine, J. M. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.100290 · Cited by 326 · see citing works
Transfer of test- enhanced learning: meta-analytic review and synthesis
Pan, S. C., & Rickard, T. C. (2018) · Psychological Bulletin · doi:10.1037/bul0000151 · Cited by 286 · see citing works
Is bilingualism related to a cognitive advantage in children? A systematic review and meta-analysis.
Gunnerud, H. L., ten Braak, D., Reikerås, E. K. L., Donolato, E., & Melby‐Lervåg, M. (2020) · Psychological Bulletin · doi:10.1037/bul0000301 · Cited by 303 · see citing works
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271 · see citing works
Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation
Pennycook, G., & Rand, D. G. (2022) · Nature Communications · doi:10.1038/s41467-022-30073-5 · Cited by 270 · see citing works
Cognitive and academic benefits of music training with children: A multilevel meta-analysis
Sala, G., & Gobet, F. (2020) · Memory & Cognition · doi:10.3758/s13421-020-01060-2 · Cited by 235 · see citing works
A meta-analysis of the facial feedback literature: Effects of facial feedback on emotional experience are small and variable.
Coles, N. A., Larsen, J. T., & Lench, H. C. (2019) · Psychological Bulletin · doi:10.1037/bul0000194 · Cited by 231 · see citing works
Trait mindfulness and health behaviours: a meta-analysis
Sala, M., Rochefort, C., Lui, P. P., & Baldwin, A. S. (2020) · Health Psychology Review · doi:10.1080/17437199.2019.1650290 · Cited by 199 · see citing works
The Perils of Misinterpreting and Misusing “Publication Bias” in Meta-analyses: An Education Review on Funnel Plot-Based Methods
Afonso, J. R., Ramírez‐Campillo, R., Clemente, F. M., Büttner, F., & Andrade, R. (2023) · Sports Medicine · doi:10.1007/s40279-023-01927-9 · Cited by 213 · see citing works
A meta-analytical evaluation of the dual-hormone hypothesis: Does cortisol moderate the relationship between testosterone and status, dominance, risk taking, aggression, and psychopathy?
Dekkers, T. J., van Rentergem, J. A. A., Meijer, B., Popma, A., Wagemaker, E., & Huizenga, H. M. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.12.004 · Cited by 147 · see citing works
A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration
Coles, N. A., March, D. S., Marmolejo‐Ramos, F., Larsen, J. T., Arinze, N. C., Ndukaihe, I. L. G., Willis, M. L., Foroni, F., Reggev, N., Mokady, A., et al. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01458-9 · Cited by 152 · see citing works
Robust Bayesian meta‐analysis: Model‐averaging across complementary publication bias adjustment methods
Bartos̆, F., Maier, M., Wagenmakers, E., Doucouliagos, H., & Stanley, T. D. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1594 · Cited by 138 · see citing works
Self-citation
The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews
Steel, P. D. G., Beugelsdijk, S., & Aguinis, H. (2021) · Journal of International Business Studies · doi:10.1057/s41267-020-00385-z · Cited by 5 · see citing works
Revisiting the Integration Hypothesis: Correlational and Longitudinal Meta-Analyses Demonstrate the Limited Role of Acculturation for Cross-Cultural Adaptation
Bierwiaczonek, K., & Kunst, J. R. (2021) · Psychological Science · doi:10.1177/09567976211006432 · Cited by 133 · see citing works
Do ‘watching eyes’ influence antisocial behavior? A systematic review & meta-analysis
Dear, K., Dutton, K., & Fox, E. (2019) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2019.01.006 · Cited by 127 · see citing works
Ironic Effects of Thought Suppression: A Meta-Analysis
Wang, D., Hagger, M. S., & Chatzisarantis, N. L. D. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619898795 · Cited by 128 · see citing works
The cognitive and academic benefits of Cogmed: A meta-analysis
Aksayli, N. D., Sala, G., & Gobet, F. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.04.003 · Cited by 123 · see citing works
“If PET suggests the presence of a real effect (i.e., intercept different from zero; p < .100, one-tailed), the PEESE estimator must be preferred over the PET estimator (Stanley, 2017; Stanley & Doucouliagos, 2014).”
BackgroundMethod
Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence
Román-Caballero, R., Vadillo, M. A., Trainor, L. J., & Lupiáñez, J. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100436 · Cited by 118 · see citing works
Does democracy cause growth? A meta-analysis (of 2000 regressions)
Colagrossi, M., Rossignoli, D., & Maggioni, M. A. (2019) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2019.101824 · Cited by 114 · see citing works
Adjusting for Publication Bias in JASP and R: Selection Models, PET-PEESE, and Robust Bayesian Meta-Analysis
Bartos̆, F., Maier, M., Quintana, D. S., & Wagenmakers, E. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221109259 · Cited by 115 · see citing works
The relationship between sleep quality and quality of life in aging: a systematic review and meta-analysis
Sella, E., Miola, L., Toffalini, E., & Borella, E. (2021) · Health Psychology Review · doi:10.1080/17437199.2021.1974309 · Cited by 96 · see citing works
Meta-Analysis of Present-Bias Estimation using Convex Time Budgets
Imai, T., Rutter, T. A., & Camerer, C. F. (2020) · The Economic Journal · doi:10.1093/ej/ueaa115 · Cited by 94 · see citing works
Music Training and Nonmusical Abilities
Schellenberg, E. G., & Lima, C. F. (2023) · Annual Review of Psychology · doi:10.1146/annurev-psych-032323-051354 · Cited by 94 · see citing works
Sex differences in trust and trustworthiness: A meta-analysis of the trust game and the gift-exchange game
van den Akker, O. R., van Assen, M. A. L. M., van Vugt, M., & Wicherts, J. M. (2020) · Journal of Economic Psychology · doi:10.1016/j.joep.2020.102329 · Cited by 85 · see citing works
Educational gender gaps and economic growth: A systematic review and meta-regression analysis
Minasyan, A., Zenker, J., Klasen, S., & Vollmer, S. (2019) · World Development · doi:10.1016/j.worlddev.2019.05.006 · Cited by 87 · see citing works
When is narcissism associated with low empathy? A meta-analytic review
Urbonaviciute, G., & Hepper, E. G. (2020) · Journal of Research in Personality · doi:10.1016/j.jrp.2020.104036 · Cited by 91 · see citing works
Meta-analysis of the effects of transcranial direct current stimulation on inhibitory control
Schroeder, P. A., Schwippel, T., Wolz, I., & Svaldi, J. (2020) · Brain stimulation · doi:10.1016/j.brs.2020.05.006 · Cited by 83 · see citing works
Detecting publication selection bias through excess statistical significance
Stanley, T. D., Doucouliagos, H., Ioannidis, J. P. A., & Carter, E. C. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1512 · Cited by 79 · see citing works
Self-citation
Effects of action video game play on cognitive skills: A meta-analysis.
Bediou, B., Rodgers, M. A., Tipton, E., Mayer, R. E., Green, C. S., & Bavelier, D. (2023) · Technology Mind and Behavior · doi:10.1037/tmb0000102 · Cited by 90 · see citing works
Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology
Yang, Y., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Koricheva, J., Jennions, M. D., Parker, T. H., Lagisz, M., & Nakagawa, S. (2023) · BMC Biology · doi:10.1186/s12915-022-01485-y · Cited by 81 · see citing works
Meta‐analyzing partial correlation coefficients using Fisher's z transformation
van Aert, R. C. M. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1654 · Cited by 78 · see citing works
The Effects of Transcranial Direct Current Stimulation on Chronic Ankle Instability
Leicht, A. S. B., Howard, J. S., van Werkhoven, H., McBride, J. M., & Needle, A. R. (2019) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000002129 · Cited by 77 · see citing works
Heterogeneity in direct replications in psychology and its association with effect size.
Olsson-Collentine, A., Wicherts, J. M., & van Assen, M. A. L. M. (2020) · Psychological Bulletin · doi:10.1037/bul0000294 · Cited by 78 · see citing works
Does music training enhance auditory and linguistic processing? A systematic review and meta-analysis of behavioral and brain evidence
Neves, L., Correia, A. I., Castro, S. L., Martins, D., & Lima, C. F. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104777 · Cited by 76 · see citing works
Physical activity and cortisol regulation: A meta-analysis
Moyers, S. A., & Hagger, M. S. (2023) · Biological Psychology · doi:10.1016/j.biopsycho.2023.108548 · Cited by 83 · see citing works
Sexual (in)equality? A meta‐analysis of sex differences in thermal acclimation capacity across ectotherms
Pottier, P., Burke, S., Drobniak, S. M., Lagisz, M., & Nakagawa, S. (2021) · Functional Ecology · doi:10.1111/1365-2435.13899 · Cited by 67 · see citing works
Are we facing a “causality crisis” in acculturation research? The need for a methodological (r)evolution
Kunst, J. R. (2021) · International Journal of Intercultural Relations · doi:10.1016/j.ijintrel.2021.08.003 · Cited by 66 · see citing works
Child physical abuse risk factors: A systematic review and a meta-analysis
Milner, J. S., Crouch, J. L., McCarthy, R. J., Ammar, J., Dominguez-Martinez, R., Thomas, C. L., & Jensen, A. P. (2022) · Aggression and Violent Behavior · doi:10.1016/j.avb.2022.101778 · Cited by 66 · see citing works
Working memory training does not enhance older adults' cognitive skills: A comprehensive meta-analysis
Sala, G., Aksayli, N. D., Tatlidil, K. S., Gondo, Y., & Gobet, F. (2019) · Intelligence · doi:10.1016/j.intell.2019.101386 · Cited by 63 · see citing works
A meta-analysis of the emotional victim effect for female adult rape complainants: Does complainant distress influence credibility?
Nitschke, F., McKimmie, B. M., & Vanman, E. J. (2019) · Psychological Bulletin · doi:10.1037/bul0000206 · Cited by 61 · see citing works
How to Detect Publication Bias in Psychological Research
Renkewitz, F., & Keiner, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000386 · Cited by 56 · see citing works
Paths to the light and dark sides of human nature: A meta-analytic review of the prosocial benefits of autonomy and the antisocial costs of control.
Donald, J. N., Bradshaw, E. L., Conigrave, J. H., Parker, P. D., Byatt, L. L., Noetel, M., & Ryan, R. M. (2021) · Psychological Bulletin · doi:10.1037/bul0000338 · Cited by 57 · see citing works
Game-thinking; utilizing serious games and gamification in nursing education – a systematic review and meta-analysis
Nylén‐Eriksen, M., Stojiljkovic, M., Lillekroken, D., Lindeflaten, K., Hessevaagbakke, E., Flølo, T. N., Hovland, O. J., Solberg, A. S., Hansen, S., Bjørnnes, A. K., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-024-06531-7 · Cited by 55 · see citing works
Cultivation and social media: A meta-analysis
Hermann, E., Morgan, M., & Shanahan, J. E. (2023) · New Media & Society · doi:10.1177/14614448231180257 · Cited by 66 · see citing works
Digital mental health interventions for the treatment of depression: A multiverse meta-analysis
Plessen, C. Y., Panagiotopoulou, O. M., Tong, L., Cuijpers, P., & Karyotaki, E. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.10.018 · Cited by 53 · see citing works
Times are changing, bias isn’t: A meta-meta-analysis on publication bias detection practices, prevalence rates, and predictors in industrial/organizational psychology.
Siegel, M., Eder, J. S. N., Wicherts, J. M., & Pietschnig, J. (2021) · Journal of Applied Psychology · doi:10.1037/apl0000991 · Cited by 55 · see citing works
Sleep spindles and general cognitive ability – A meta-analysis
Ujma, P. P. (2018) · Sleep Spindles & Cortical Up States · doi:10.1556/2053.2.2018.01 · Cited by 49 · see citing works
Emotion regulation and mental health across cultures: a systematic review and meta-analysis
Chen, S., Cai, Q., Omari, D. I., Sanghvi, D. E., Lyu, S., & Bonanno, G. A. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02168-8 · Cited by 61 · see citing works
Suicidality among individuals with gambling problems: A meta-analytic literature review.
Kristensen, J. H., Pallesen, S., Bauer, J., Leino, T. M., Griffiths, M. D., & Erevik, E. K. (2023) · Psychological Bulletin · doi:10.1037/bul0000411 · Cited by 55 · see citing works
Television, Continuity, and Change: A Meta-Analysis of Five Decades of Cultivation Research
Hermann, E., Morgan, M., & Shanahan, J. E. (2021) · Journal of Communication · doi:10.1093/joc/jqab014 · Cited by 51 · see citing works
Manipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Braß, M., & Schooler, J. W. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683221087527 · Cited by 60 · see citing works
Quantitative Political Science Research Is Greatly Underpowered
Arel‐Bundock, V., Briggs, R. C., Doucouliagos, H., Aviña, M. M., & Stanley, T. D. (2024) · The Journal of Politics · doi:10.1086/734279 · Cited by 49 · see citing works
Self-citation
Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research
Pietschnig, J., Siegel, M., Eder, J. S. N., & Gittler, G. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02874 · Cited by 48 · see citing works
How identity fusion predicts extreme pro-group orientations: A meta-analysis
Varmann, A. H., Kruse, L., Bierwiaczonek, K., Gómez, Á., Vázquez, A., & Kunst, J. (2022) · European Review of Social Psychology · doi:10.1080/10463283.2023.2190267 · Cited by 40 · see citing works
Using Monte Carlo experiments to select meta‐analytic estimators
Hong, S., & Reed, W. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1467 · Cited by 38 · see citing works
Does adult-child co-use during digital media use improve children's learning aged 0–6 years? A systematic review with meta-analysis
Taylor, G., Sala, G., Kołak, J., Gerhardstein, P. C., & Lingwood, J. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100614 · Cited by 70 · see citing works
Olfactory perception and blindness: a systematic review and meta-analysis
Sorokowska, A., Sorokowski, P., Karwowski, M., Larsson, M. U., & Hummel, T. R. (2018) · Psychological Research · doi:10.1007/s00426-018-1035-2 · Cited by 40 · see citing works
Resting Frontal Eeg Asymmetry and Personality Traits: A Meta–Analysis
Kuper, N., Käckenmester, W., & Wacker, J. (2019) · European Journal of Personality · doi:10.1002/per.2197 · Cited by 40 · see citing works
Has the evidence for moral licensing been inflated by publication bias?
Kuper, N., & Bott, A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.878 · Cited by 39 · see citing works
Meta-analysis of the reduced relative feedback frequency effect on motor learning and performance
McKay, B., Hussien, J., Vinh, M.-A., Mir-Orefice, A., Brooks, H., & Ste‐Marie, D. M. (2022) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2022.102165 · Cited by 37 · see citing works
The Wages of Mothers' Labor: A Meta‐Regression Analysis
Leonard, M. D. L., & Stanley, T. D. (2020) · Journal of Marriage and the Family · doi:10.1111/jomf.12693 · Cited by 40 · see citing works
Self-citation
Early Psychological Interventions for Somatic Symptom Disorder and Functional Somatic Syndromes: A Systematic Review and Meta-Analysis
Berezowski, L., Ludwig, L., Martin, A., Löwe, B., & Mora, M. C. S. (2021) · Psychosomatic Medicine · doi:10.1097/psy.0000000000001011 · Cited by 40 · see citing works
Examining the Reproducibility of Meta-Analyses in Psychology: A Preliminary ReportPreprint
Lakens, D., Page‐Gould, E., van Assen, M. A. L. M., Spellman, B., Schönbrodt, F. D., Hasselman, F., Corker, K. S., Grange, J., Sharples, A. E., Cavender, C., et al. (2017) · doi:10.31222/osf.io/xfbjf · Cited by 35 · see citing works
How robust is our cumulative knowledge on turnover?
Field, J. G., Bosco, F. A., & Kepes, S. (2020) · Journal of Business and Psychology · doi:10.1007/s10869-020-09687-3 · Cited by 34 · see citing works
Dyslexia treatment studies: A systematic review and suggestions on testing treatment efficacy with small effects and small samples
Toffalini, E., Giofrè, D., Pastore, M., Carretti, B., Fraccadori, F., & Szűcs, D. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01549-x · Cited by 34 · see citing works
Beyond Random Effects: When Small-Study Findings Are More Heterogeneous
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221120427 · Cited by 34 · see citing works
Self-citation
Does Mindfulness Improve Intergroup Bias, Internalized Bias, and Anti-Bias Outcomes?: A Meta-Analysis of the Evidence and Agenda for Future Research
Chang, D. F., Donald, J. N., Whitney, J. M., Miao, I. Y., & Sahdra, B. K. (2023) · Personality and Social Psychology Bulletin · doi:10.1177/01461672231178518 · Cited by 33 · see citing works
Oral language interventions can improve language outcomes in children with neurodevelopmental disorders: A systematic review and meta-analysis
Donolato, E., Toffalini, E., Rogde, K., Nordahl‐Hansen, A., Lervåg, A., Norbury, C., & Melby‐Lervåg, M. (2023) · Campbell Systematic Reviews · doi:10.1002/cl2.1368 · Cited by 62 · see citing works
Reinforcement Learning in Education: A Systematic Literature Review
Riedmann, A., Schäper, P., & Lugrin, B. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00494-6 · Cited by 42 · see citing works
Implications of the open science era for educational psychology research syntheses
Patall, E. A. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1897009 · Cited by 34 · see citing works
The relationship of divergent thinking with broad retrieval ability and processing speed: A meta-analysis
Miroshnik, K. G., Forthmann, B., Karwowski, M., & Benedek, M. (2023) · Intelligence · doi:10.1016/j.intell.2023.101739 · Cited by 31 · see citing works
Meta-Analysis of Present-Bias Estimation Using Convex Time Budgets
Imai, T., Rutter, T., & Camerer, C. F. (2019) · Open access LMU (Ludwid Maxmilian's Universitat Munchen) · doi:10.5282/ubm/epub.69188 · Cited by 30 · see citing works
A meta‐regression analysis on the willingness‐to‐pay for country‐of‐origin labelling
Yeh, C., & Hirsch, S. (2023) · Journal of Agricultural Economics · doi:10.1111/1477-9552.12528 · Cited by 28 · see citing works
Efficacy of memory training interventions targeting metacognition for older adults: a systematic review and meta-analysis
Sella, E., Carbone, E., Vincenzi, M., Toffalini, E., & Borella, E. (2022) · Aging & Mental Health · doi:10.1080/13607863.2022.2122931 · Cited by 27 · see citing works
Retrospective median power, false positive meta‐analysis and large‐scale replication
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1529 · Cited by 28 · see citing works
Self-citation
Studying abroad and earnings: A meta‐analysis
Pietro, G. D., European Commission, & IZA (2021) · Journal of Economic Surveys · doi:10.1111/joes.12472 · Cited by 27 · see citing works
Inflation targeting: Genuine effects or publication selection bias?
Balima, H. W., Kilama, E. G., & Tapsoba, R. (2020) · European Economic Review · doi:10.1016/j.euroecorev.2020.103520 · Cited by 26 · see citing works
Self-efficacy in gifted and non-gifted students: A multilevel meta-analysis
Alabbasi, A. M. A., Sultan, Z. M., Karwowski, M., Cross, T. L., & Ayoub, A. (2023) · Personality and Individual Differences · doi:10.1016/j.paid.2023.112244 · Cited by 26 · see citing works
Adjusting for Publication Bias Reveals That Evidence for and Size of Construal Level Theory Effects is Substantially OverestimatedPreprint
Maier, M., Bartos̆, F., Oh, M., Wagenmakers, E., Shanks, D. R., & Harris, A. J. L. (2022) · doi:10.31234/osf.io/r8nyu · Cited by 25 · see citing works
Does Digital Intergroup Contact Reduce Prejudice? A Meta-Analysis
da Costa, L. P., Bierwiaczonek, K., & Bianchi, M. (2024) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2023.0591 · Cited by 26 · see citing works
THE EURO'S TRADE EFFECT: A META‐ANALYSIS
Polák, P. (2018) · Journal of Economic Surveys · doi:10.1111/joes.12264 · Cited by 24 · see citing works
Evaluating the effects of maternal positions in childbirth: An overview of Cochrane Systematic Reviews
Kibuka, M., Price, A. I., Onakpoya, I., Tierney, S., & Clarke, M. J. (2021) · European Journal of Midwifery · doi:10.18332/ejm/142781 · Cited by 24 · see citing works
A meta‐analysis on the effects of just‐below versus round prices
Troll, E. S., Frankenbach, J., Friese, M., & Loschelder, D. D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1353 · Cited by 25 · see citing works
Assessing the association between parents’ math talk and children’s math performance: A preregistered meta-analysis
Silver, A. M., Alvarez-Vargas, D., Bailey, D. H., & Libertus, M. E. (2024) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2024.105920 · Cited by 24 · see citing works
Exploring the potent enhancement effects of plyometric training on vertical jumping and sprinting ability in sports individuals
Xie, L., Chen, J., Dai, J., Zhang, W., Chen, L., Jian, S., Gao, X., Song, J., & Shen, H. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1435011 · Cited by 23 · see citing works
An evaluation of binge eating characteristics in individuals with eating disorders: A systematic review and meta-analysis
Mourilhe, C., de Moraes, C. E. F., da Veiga, G. V., da Luz, F. Q., Trindade, A. P., Nazar, B. P., Coutinho, E. S. F., Hay, P., & Appolinário, J. C. (2021) · Appetite · doi:10.1016/j.appet.2021.105176 · Cited by 23 · see citing works
P ‐hacking in meta‐analyses: A formalization and new meta‐analytic methods
Mathur, M. B. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1701 · Cited by 24 · see citing works
The Role of Need for Cognition in Well-Being – Review and Meta-Analyses of Associations and Potentially Underlying Mechanisms
Zerna, J., Strobel, A., & Strobel, A. (2024) · Collabra Psychology · doi:10.1525/collabra.92885 · Cited by 22 · see citing works
A systematic review and meta-analysis of vineyard techniques used to delay ripening
Previtali, P., Giorgini, F., Mullen, R. S., Dokoozlian, N., Wilkinson, K. L., & Ford, C. M. (2022) · Horticulture Research · doi:10.1093/hr/uhac118 · Cited by 22 · see citing works
Skating on Thin Evidence: Implications for Public Policy
Doucouliagos, H., Paldam, M., & Stanley, T. D. (2018) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2018.03.004 · Cited by 22 · see citing works
Self-citation
Anatomic and non‐anatomic reconstruction improves post‐operative outcomes in chronic acromio‐clavicular instability: a systematic review
Xará-Leite, F., Andrade, R., Moreira, P. S., Coutinho, L., Ayeni, O. R., Sevivas, N., & Espregueira‐Mendes, J. (2019) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-019-05408-3 · Cited by 21 · see citing works

Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? A replication study of Alinaghi & Reed (Research Synthesis Methods, 2018)
Hong, S. (2019) · RePEc: Research Papers in Economics · doi:10.18718/81781.13 · Cited by 1 · see citing works
Correcting for bias in psychology: A comparison of meta-analytic methodsPreprint
Schönbrodt, F., Gervais, W. M., Hilgard, J., & Carter, E. (2019) · OSF Preprints (OSF Preprints)
Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? A reply to Hong (International Journal for Re-Views in Empirical Economics, 2019)
Reed, W. R. (2019) · Econstor (Econstor) · doi:10.18718/81781.14
A Comparative Evaluation of Psychometric Meta‐Analysis Methods in Management and Applied Psychology: Toward a Nuanced Understanding of Their Accuracy
Oh, I., Le, H., & Schmidt, F. L. (2025) · Personnel Psychology · doi:10.1111/peps.70001
Pre-Analysis Plan: Is Psychology Research Consistent with Random Effects?Preprint
Stanley, T. D. (2022) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/wdg5y
Self-citation

Authors: Stanley, T. D.
Affiliations: Hendrix College (US)
Corresponding author: Stanley, T. D.
ORCID: Stanley, T. D.
Venue: Social Psychological and Personality Science
Publisher: SAGE Publications
Published: 2017-02-28
Volume: 8
Issue: 5
Pages: 581-591
ISSN: 1948-5506 (ISSN-L)
1948-5514
DOI: 10.1177/1948550617693062
OpenAlex: W2591641564
Semantic Scholar: 62d9746bdeace29af787162583753de042a82d8a
Cited by: 229
References: 37
Citations by year: 2026: 23 · 2025: 29 · 2024: 21 · 2023: 33 · 2022: 25 · 2021: 27 · 2020: 21 · 2019: 27 · 2018: 18 · 2017: 5
FWCI: 17.48
Type: journal article
Language: en
Open access: green
figshare.com
Licence: cc-by
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Social Psychological and Personality Science (publishedVersion)
Figshare (submittedVersion, cc-by)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords: meta-regression, replication failure, small sample bias, effect size heterogeneity, meta-analysis limitations, social-personality psychology, simulation study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Limitations of PET-PEESE and Other Meta-Analysis Methods

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks