The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Open accessPublished version of a preprint
Abstract
Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, it is not clear which methods work best for data typically seen in psychology. Here, we present a comprehensive simulation study in which we examined how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We simulated several levels of questionable research practices, publication bias, and heterogeneity, and used study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all the others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change depending on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts to improve the primary literature and conduct large-scale, preregistered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/ .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The absence of publication bias is highly unlikely unless a set of preregistered reports is examined (Carter et al. 2019).”
From The gamification of learning: a meta-analysis (2020) · cites this paper as Carter et al. 2019
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Bias adjustments can improve performance relative to a meta-analysis that does not adjust for publication bias (Carter et al., 2019; McShane et al., 2016; Moreno et al., 2009; Simonsohn et al., 2014).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“negative and statistically significant SPET estimates have been observed in both simulation studies (Carter et al., 2019) and meta-analysis (Hilgard et al., 2019).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“On the one hand, this may be an underestimate of average study power—in simulation studies, p-hacking adds large p values to the p-curve, canceling out the right skew and biasing the p-curve estimate downward (Carter et al., 2019).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“SPET and SPEESE have a downward bias in the presence of p-hacking (Carter et al., 2019)”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“There may also be downward bias inflicted by p-hacking (Carter et al., 2019).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Further, PET-PEESE can underestimate the effect size when there is publication bias and when p-hacking is present (Carter et al., 2019), which are two conditions likely affecting the literature.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“However, this counterintuitive finding may be explained by the fact that PET is prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking (Stanley & Doucouliagos, 2014; Stanley 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Simulation studies by Carter et al. (2019) showed that WAAP-WLS can provide a reasonably unbiased estimate of the effect size with a high power to detect the effect given that a true underlying effect exists.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al. (2019)
In the citing paper
“This could lead selection model to underestimate effects in the presence of publication bias and p-hacking, as shown in simulation studies (Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Lastly, PET-PEESE can underestimate the size of small effects in the presence of p-hacking and publication bias (Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“However, when effects are highly heterogeneous and when sample sizes are small, the PET-PEESE analysis is known to yield imprecise estimates (Stanley & Doucouliagos, 2014; Stanley, 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“there is a danger that uncritical application can lead to inappropriate conclusions being drawn 68,73,74”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [74]
In the citing paper
“the available evidence suggests that no single method outperforms others in all scenarios”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [74]
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Ego depletion and the strength model of self-control: a meta-analysis
Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010) · Psychological Bulletin · doi:10.1037/a0019486 · Cited by 2,265
Research works printed but not linked: 1
- Augusteijn H. E. M., 2019, Q, 24, 116
Cited by
Sorted by citations
Items 501–578 of 578
Multiverse Meta-Analysis: Proposing an Exploratory FrameworkPreprint
Rasti, S. (2023) · doi:10.31237/osf.io/racx9 · Cited by 1
Learning from comics versus non-comics material in education: Systematic review and meta-analysisPreprint
Pagkratidou, M., Cohn, N., Phylactou, P., Παπαδάτου-Παστού, Μ., & Duffy, G. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/ceda3 · Cited by 1
Commentary: A Science Teacher-Friendly Primer on Implicit Bias
Miller, K. K. (2022) · The Science Teacher · doi:10.1080/00368555.2022.12293732 · Cited by 2
Rotella, A., Jung, J., Chinn, C., & Barclay, P. (2025) · doi:10.31234/osf.io/tmhe9_v2 · Cited by 1
Rotella, A., Jung, J., Chinn, C., & Barclay, P. (2023) · doi:10.31234/osf.io/tmhe9_v1 · Cited by 1
Schimmack, U., & Soto, M. (2026) · Communication Methods and Measures · doi:10.1080/19312458.2026.2658505 · Cited by 2
Bakshi, K., & Gait, P. (2026) · Research Square · doi:10.21203/rs.3.rs-9952939/v1 · Cited by 1
Lobermeier, M., & Peters, S. D. (2020) · Psi Chi Journal of Psychological Research · doi:10.24839/2325-7342.jn25.1.65
Tybur, J. M., & Lieberman, D. (2021) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20000576
Reimer, N. K., Hughes, J., Blaylock, D., Donnelly, C., Wölfer, R., & Hewstone, M. (2021) · doi:10.31234/osf.io/5mkwh
Doing Science: Methods and PhilosophyPreprint
van Dongen, N. N. N. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/yegvs
Do your eyes protect your memory?
Houben, S. T. L. (2021) · doi:10.26481/dis.20210527sh
Augusteijn, H. E. M., van Aert, R. C. M., & van Assen, M. A. L. M. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/avkgj
(2021) · Wiley eBooks · doi:10.1002/9781119558378.ref
Tressoldi, P. E., & Storm, L. (2020) · F1000Research · doi:10.12688/f1000research.24868.2
Publication Bias and Psychotherapy
Haushofer, J. (2017) · AEA Randomized Controlled Trials · doi:10.1257/rct.2574
vanAert_Niemeyer21Preprint
van Aert, R. C. M., & Niemeyer, H. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/3rdys
Almgren, H., Parsons, N., Bossche, S. V. D., & Marinazzo, D. (2020) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/9hg3j
Assessment of Evidential Value Requires More Than a Single Data PointPreprint
Imhoff, R. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/cpjr7
A comprehensive Meta-Analysis on Spontaneous Trait Inferences from Behavior - preprintPreprint
Bott, A., Brockmann, L., Henken, E., Denneberg, I., Kuper, N., Kruse, F., & Degner, J. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xfw84
The Signaling Effects of Blood Glucose and Appetitive Stimuli on Delay Discounting: A Meta-AnalysisPreprint
Lu, J. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/zkgwt
Interventionsdesign, Evaluationsverfahren und Wirksamkeit von Interventionen
Reis, D., & Meier, L. L. (2022) · Springer eBooks · doi:10.1007/978-3-658-28651-4_24
ORMA: A strategy to reduce Psychology's replication problems
Howard, G. S., & Maxwell, S. E. (2022) · New Ideas in Psychology · doi:10.1016/j.newideapsych.2022.100991
Gieseler, K., Loschelder, D. D., & Friese, M. (2023) · Springer eBooks · doi:10.1007/978-3-031-17529-9_1
Znanstvena reforma i replikacijska kriza u psihologiji
Flis, I. (2023) · Psihologijske teme · doi:10.31820/pt.32.2.1 · Cited by 1
Terror Management Theory and the Existential Psychological Function of Heroes
Schindler, S. (2023) · Springer eBooks · doi:10.1007/978-3-031-17125-3_142-1
Bruns, A. K., Henriksson, I., Nordström, T., & Carlsson, R. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1389
Garofalo, S. (2024) · Peer Community In Registered Reports · doi:10.24072/pci.rr.100743
Tressoldi, P. E., & Storm, L. (2024) · F1000Research · doi:10.12688/f1000research.51746.4 · Cited by 1
Chernikova, O., Heitzmann, N., Holzberger, D., Neuhaus, B. J., Kron, S., Ufer, S., & Fischer, F. (2024) · Proceedings. · doi:10.22318/icls2024.767690
Parenting Attitudes and Attributional Style in the First Secondary School Students of kazeron City
(2024) · Journal of Psychology and Neuroscience · doi:10.47485/2693-2490.1092
Vicarious Dissonance: Pre-Registered Meta-Analysis
Jaubert, S., Fillon, A. A., Souchet, L., & Girandola, F. (2024) · Personality and Social Psychology Bulletin · doi:10.1177/01461672241266653 · Cited by 1
Child maltreatment and interoception: A meta-analytic reviewPreprint
Ditzer, J., Woll-Weber, C. F. J., Burger, C., Ernst, A., Boehm, I., Garthus‐Niegel, S., & Zietlow, A. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/n8tpc
Terror Management Theory and the Existential Psychological Function of Heroes
Schindler, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-48129-1_142
Meta-analysis with Jeffreys priors: Empirical frequentist properties
Mathur, M. B. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2024.2
Hussey, I., & Elson, M. (2025) · doi:10.71240/lcyc.052239
Hussey, I., & Elson, M. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/84e7t_v2
León, S. P., Garrido, M. G., Martínez, I. G., & Lipnevich, A. A. (2025) · doi:10.35542/osf.io/c5qsr_v2
Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., & Wicherts, J. M. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2uynm_v1
Fillon, A. A., Souchet, L., Pascual, A., & Girandola, F. (2020) · doi:10.31234/osf.io/3ds26_v1
Ingebrigtsen, M., & Lukic, M. (2025) · doi:10.31234/osf.io/wgu56_v1
Anikin, A. (2025) · Psychonomic Bulletin & Review · doi:10.3758/s13423-025-02740-3
Neurostimulation: Exploring Perceptual & Cognitive Enhancement
(2025) · Frontiers research topics · doi:10.3389/978-2-8325-6506-3
Replikationen in der Sportwissenschaft
Schweizer, G. (2025) · Journal of Applied Sport and Exercise Psychology · doi:10.1026/2941-7597/a000053
Espinoza, J. A., Allen, N. J., Su, S., & Thompson, B. (2025) · Team Performance Management · doi:10.1108/tpm-02-2024-0014
PublicationBiasBenchmark: Benchmark for Publication Bias Correction Methods
Bartos̆, F., Pawel, S., Siepe, B. S., & Čala, P. (2025) · doi:10.32614/cran.package.publicationbiasbenchmark
Falahati, G., Biglarian, A., Behboudi‐Gandevani, S., & Yarandi, R. B. (2025) · Health Science Reports · doi:10.1002/hsr2.71626
Addressing Outcome Reporting Bias in Meta‐Analysis: A Selection Model Perspective
Saracini, A. G., & Held, L. (2025) · Statistics in Medicine · doi:10.1002/sim.70238
Outcomes associated with believing in free will: Meta-analysis Registered ReportPreprint
Nanakdewa, K., Bulchand, D., Chia, R.-J., Lim, V., Ong, C. W., Savani, K., & Feldman, G. (2020) · doi:10.17605/osf.io/57sau
Additional file 1 of Evaluation of a decided sample size in machine learning applications
Rajput, D., Wang, W.-J., & Chen, C. (2024) · Figshare · doi:10.6084/m9.figshare.26568583
Plate clearing and body mass index: a meta-analysisPreprint
Meule, A., Dietlmeier, L., & Kolar, D. (2024) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/d4zh3
Meta-analytical evidence of a self–other discrepancy in climate change-related risk perceptions
Sandlund, I., Bjälkebring, P., & Bergquist, M. (2026) · Nature Sustainability · doi:10.1038/s41893-025-01717-3
Kansızoğlu, H. B., Kansızoğlu, N., & Memiş, M. (2026) · Journal of Computer and Education Research · doi:10.18009/jcer.1701299
Mortality salience: A critical synthesis of its effects on psychological symptoms
Letzner, R. D., Nedelcea, C., & Boroi, D. (2026) · Death Studies · doi:10.1080/07481187.2026.2630951
Khatun, K., Oshan, Y. A., & Hossain, A. (2025) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca66035.2025.11430763
Høgetveit, A. M., Canrinus, E. T., Donolato, E., & Melby‐Lervåg, M. (2026) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.70138 · Cited by 2
Park, S.-H., & Kim, E. Y. (2026) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2026.105269
A meta-scientific approach to understanding stress regulation
Sparacio, A. (2023) · theses.fr (ABES) · doi:10.70675/757862f2z87a4z4ec7zabc9z5a275e01261b
Alcaraz-Ibáñez, M., Paterna, A., Diaz-Carrasco, A. F., Sicilia, Á., & Griffiths, M. D. (2026) · Scandinavian Journal of Psychology · doi:10.1111/sjop.70097
The Behavioral Impact of Moral Appeals: An Integrated Framework and Meta-Analysis
Xu, S. Y., Mulder, L. B., Rink, F. A., Bijmolt, T. H. A., & Leliveld, M. C. (2026) · Personality and Social Psychology Bulletin · doi:10.1177/01461672261437934 · Cited by 1
Effect of episodic specificity induction on divergent thinking: A meta-analytic study
Sánchez, G., Bajo, M. T., & Marful, A. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-026-02859-x
Alsalti, T., Hussey, I., Elson, M., Krause, R. W., & Pohl, S. (2026) · European Journal of Neuroscience · doi:10.1111/ejn.70533
Moving in the presence of others – a systematic review and meta-analysis on social facilitationPreprint
Strauß, B., Greve, J., & van Meurs, E. (2021) · OSF Preprints (OSF Preprints)
Employing Meta-Analysis to Build Validity Evidence in Personnel Selection
Wee, S. (2026) · Oxford University Press eBooks · doi:10.1093/9780197809013.003.0009
Ferraz, J. S., & Ribeiro, J. R. I. (2026) · Revista Brasileira de Botânica · doi:10.1007/s40415-026-01155-1
A meta-analysis of variables associated with financial satisfaction
Bayramoğlu, G. (2026) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2026.102614
León, S. P., Lipnevich, A. A., & Garrido, M. G. (2026) · Review of Educational Research · doi:10.3102/00346543261458396
A Conceptual Framework for Computational Reproductions: Formal Definitions and Epistemic Functions
Kohrt, F., Melinščak, F., McElreath, R. L., & Schönbrodt, F. D. (2026) · Journal of Trial and Error · doi:10.36850/9ec0-4fab
Self-citationBeyond bias: building a more inclusive and rigorous research culture in animal behaviour
Wascher, C. A. F., Koyama, N. F., & Nityananda, V. (2026) · Animal Behaviour · doi:10.1016/j.anbehav.2026.123671
Problematic videogame use and academic achievement: A systematic review and meta-analysis
Paterna, A., Alcaraz‐Ibáñez, M., Diaz‐Carrasco, A. F., & Griffiths, M. D. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100835
The Returns to Education: A Reply
Clark, G., & Nielsen, C. A. A. (2026) · Kyklos · doi:10.1111/kykl.70080
Workplace Interventions to Promote Employees' Pro‐Environmental Behaviour: A Meta‐Analysis
Brack, F., Petrowsky, H. M., Hörisch, J., & Loschelder, D. D. (2026) · Human Resource Management Journal · doi:10.1111/1748-8583.70066
A meta-scientific approach to understanding stress regulation
Sparacio, A. (2023) · Cronfa (Swansea University) · doi:10.23889/suthesis.63454
Open Science and Meta-Analyses in Clinical Psychology: Lessons Learned
Cuijpers, P. J., Miguel, C., Harrer, M., & Karyotaki, E. (2026) · Clinical Psychological Science · doi:10.1177/21677026251401874
Creative tendency of Chinese students: a meta-analysis, 1999–2025
Wang, Y., & Fan, H. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1957163
Chen, M. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03206-7
Sala, G., & Gobet, F. (2019) · Collabra: Psychology · doi:10.2478/collabra.2019.5.1.18
Metadata
- Authors
- Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J.
- Affiliations
- DEVCOM Army Research Laboratory (US)
- Ludwig-Maximilians-Universität München (DE)
- University of Kentucky (US)
- Illinois State University (US)
- Corresponding author
- Carter, Evan C.
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2019-06
- Volume
- 2
- Issue
- 2
- Pages
- 115-144
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W2781957993
- Cited by
- 588
- References
- 76
- Citations by year
- 2026: 43 · 2025: 60 · 2024: 69 · 2023: 86 · 2022: 83 · 2021: 87 · 2020: 79 · 2019: 54
- FWCI
- 50.69
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Advances in Methods and Practices in Psychological Science (publishedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- publication bias, questionable research practices, effect size overestimation, meta-analytic methods, sensitivity analysis, heterogeneity, simulation study, pre-registered replication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
