The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Open accessPublished version of a preprint
Abstract
Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, it is not clear which methods work best for data typically seen in psychology. Here, we present a comprehensive simulation study in which we examined how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We simulated several levels of questionable research practices, publication bias, and heterogeneity, and used study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all the others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change depending on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts to improve the primary literature and conduct large-scale, preregistered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/ .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The absence of publication bias is highly unlikely unless a set of preregistered reports is examined (Carter et al. 2019).”
From The gamification of learning: a meta-analysis (2020) · cites this paper as Carter et al. 2019
In the citing paper
“The trim-and-fill method (Duval & Tweedie, 2000) tends to adjust too little when bias is strong (Carter et al., 2019), and Fail-Safe N (Rosenthal, 1979) neither tests nor adjusts for the presence of bias.”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Bias adjustments can improve performance relative to a meta-analysis that does not adjust for publication bias (Carter et al., 2019; McShane et al., 2016; Moreno et al., 2009; Simonsohn et al., 2014).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“negative and statistically significant SPET estimates have been observed in both simulation studies (Carter et al., 2019) and meta-analysis (Hilgard et al., 2019).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“On the one hand, this may be an underestimate of average study power—in simulation studies, p-hacking adds large p values to the p-curve, canceling out the right skew and biasing the p-curve estimate downward (Carter et al., 2019).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“SPET and SPEESE have a downward bias in the presence of p-hacking (Carter et al., 2019)”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“There may also be downward bias inflicted by p-hacking (Carter et al., 2019).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Further, PET-PEESE can underestimate the effect size when there is publication bias and when p-hacking is present (Carter et al., 2019), which are two conditions likely affecting the literature.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“However, this counterintuitive finding may be explained by the fact that PET is prone to producing imprecise and biased estimates under high heterogeneity, small sample sizes, and a large proportion of p-hacking (Stanley & Doucouliagos, 2014; Stanley 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Simulation studies by Carter et al. (2019) showed that WAAP-WLS can provide a reasonably unbiased estimate of the effect size with a high power to detect the effect given that a true underlying effect exists.”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al. (2019)
In the citing paper
“This could lead selection model to underestimate effects in the presence of publication bias and p-hacking, as shown in simulation studies (Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“Lastly, PET-PEESE can underestimate the size of small effects in the presence of p-hacking and publication bias (Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“However, when effects are highly heterogeneous and when sample sizes are small, the PET-PEESE analysis is known to yield imprecise estimates (Stanley & Doucouliagos, 2014; Stanley, 2017; Carter et al., 2019).”
From Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature (2023) · cites this paper as Carter et al., 2019
In the citing paper
“there is a danger that uncritical application can lead to inappropriate conclusions being drawn 68,73,74”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [74]
In the citing paper
“the available evidence suggests that no single method outperforms others in all scenarios”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [74]
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Ego depletion and the strength model of self-control: a meta-analysis
Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010) · Psychological Bulletin · doi:10.1037/a0019486 · Cited by 2,265
Research works printed but not linked: 1
- Augusteijn H. E. M., 2019, Q, 24, 116
Cited by
Sorted by citations
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
The gamification of learning: a meta-analysis
Sailer, M., & Homner, L. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09498-w · Cited by 2,594
Simulation-Based Learning in Higher Education: A Meta-Analysis
Chernikova, O., Heitzmann, N., Stadler, M., Holzberger, D., Seidel, T., & Fischer, F. (2020) · Review of Educational Research · doi:10.3102/0034654320933544 · Cited by 1,072
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Self-citationIs bilingualism associated with enhanced executive functioning in adults? A meta-analytic review.
Lehtonen, M., Soveri, A., Laine, A., Järvenpää, J., de Bruin, A., & Antfolk, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000142 · Cited by 711
Rodgers, M. A., & Pustejovsky, J. E. (2021) · Psychological Methods · doi:10.1037/met0000300 · Cited by 492
A meta-analysis of the health action process approach.
Zhang, C.-Q., Zhang, R., Schwarzer, R., & Hagger, M. S. (2019) · Health Psychology · doi:10.1037/hea0000728 · Cited by 485
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 399
Music therapy for stress reduction: a systematic review and meta-analysis
de Witte, M., Pinho, A. D. S., Stams, G.-J., Moonen, X., Bos, A. E. R., & van Hooren, S. (2022) · Health Psychology Review · doi:10.1080/17437199.2020.1846580 · Cited by 444
Sexual orientation concealment and mental health: A conceptual and meta-analytic review.
Pachankis, J. E., Mahon, C. P., Jackson, S. D., Fetzner, B. K., & Bränström, R. (2020) · Psychological Bulletin · doi:10.1037/bul0000271 · Cited by 433
Macnamara, B. N., & Burgoyne, A. P. (2023) · Psychological Bulletin · doi:10.1037/bul0000352 · Cited by 415
Evaluation of a decided sample size in machine learning applications
Rajput, D., Wang, W.-J., & Chen, C.-C. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05156-9 · Cited by 411
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2018) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 438
Loneliness and the Big Five Personality Traits: A Meta–Analysis
Buecker, S., Maes, M., Denissen, J. J. A., & Luhmann, M. (2020) · European Journal of Personality · doi:10.1002/per.2229 · Cited by 452
The association between weight stigma and mental health: A meta‐analysis
Emmer, C., Bošnjak, M., & Mata, J. (2019) · Obesity Reviews · doi:10.1111/obr.12935 · Cited by 416
Donald, J. N., Sahdra, B. K., Van Zanden, B., Duineveld, J. J., Atkins, P. W. B., Marshall, S. L., & Ciarrochi, J. (2019) · British Journal of Psychology · doi:10.1111/bjop.12338 · Cited by 371
Cuijpers, P., Miguel, C., Harrer, M., Plessen, C. Y., Ciharova, M., Ebert, D., & Karyotaki, E. (2023) · World Psychiatry · doi:10.1002/wps.21069 · Cited by 354
Testing for funnel plot asymmetry of standardized mean differences
Pustejovsky, J. E., & Rodgers, M. A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1332 · Cited by 354
Association of Drug Cues and Craving With Drug Use and Relapse
Vafaie, N., & Hedy Kober, A. (2022) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2022.1240 · Cited by 378
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0215052 · Cited by 365
Testing (quizzing) boosts classroom learning: a systematic and meta-analytic review
Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021) · Psychological Bulletin · doi:10.1037/bul0000309 · Cited by 333
Joel, S., Eastwick, P. W., Allison, C. J., Arriaga, X. B., Baker, Z. G., Bar-Kalifa, E., Bergeron, S., Birnbaum, G. E., Brock, R. L., Brumbaugh, C. C., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1917036117 · Cited by 315
Is bilingualism related to a cognitive advantage in children? A systematic review and meta-analysis.
Gunnerud, H. L., ten Braak, D., Reikerås, E. K. L., Donolato, E., & Melby‐Lervåg, M. (2020) · Psychological Bulletin · doi:10.1037/bul0000301 · Cited by 330
Agostini, M., & van Zomeren, M. (2021) · Psychological Bulletin · doi:10.1037/bul0000256 · Cited by 301
Utesch, T., Bardid, F., Büsch, D., & Strauß, B. (2019) · Sports Medicine · doi:10.1007/s40279-019-01068-y · Cited by 331
Page, M. J., Sterne, J. A. C., Higgins, J. P. T., & Egger, M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1468 · Cited by 294
A tutorial on Bayes Factor Design Analysis using an informed prior
Stefan, A. M., Gronau, Q. F., Schönbrodt, F. D., & Wagenmakers, E.-J. (2019) · Behavior Research Methods · doi:10.3758/s13428-018-01189-8 · Cited by 284
Self-citationComparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Promoting memory consolidation during sleep: A meta-analysis of targeted memory reactivation
Hu, X., Cheng, L. Y., Chiu, M. H., & Paller, K. A. (2020) · Psychological Bulletin · doi:10.1037/bul0000223 · Cited by 232
Social Value Orientation, Expectations, and Cooperation in Social Dilemmas: A Meta–Analysis
Pletzer, J. L., Balliet, D. P., Joireman, J. A., Kuhlman, D. M., Voelpel, S. C., & Lange, P. A. M. V. (2018) · European Journal of Personality · doi:10.1002/per.2139 · Cited by 247
A century of research on conscientiousness at work
Wilmot, M. P., & Ones, D. S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1908430116 · Cited by 215
Agarwal, P. K., Nunes, L. D., & Blunt, J. R. (2021) · Educational Psychology Review · doi:10.1007/s10648-021-09595-9 · Cited by 211
Trait mindfulness and health behaviours: a meta-analysis
Sala, M., Rochefort, C., Lui, P. P., & Baldwin, A. S. (2020) · Health Psychology Review · doi:10.1080/17437199.2019.1650290 · Cited by 199
A meta-analysis of factors related to recycling
Geiger, J. L., Steg, L., van der Werff, E., & Ünal, A. B. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.05.004 · Cited by 219
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019) · Psychophysiology · doi:10.1111/psyp.13437 · Cited by 205
The Past, Present, and Future of Ego Depletion
Inzlicht, M., & Friese, M. (2019) · Social Psychology · doi:10.1027/1864-9335/a000398 · Cited by 192
Association of Testosterone Treatment With Alleviation of Depressive Symptoms in Men
Walther, A., Breidenstein, J., & Miller, R. E. (2018) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2734 · Cited by 182
Galuschka, K., Görgen, R., Kalmar, J., Haberstroh, S., Schmalz, X., & Schulte‐Körne, G. (2020) · Educational Psychologist · doi:10.1080/00461520.2019.1659794 · Cited by 198
Morgenroth, T., Kirby, T. A., Ryan, M. K., & Sudkämper, A. (2020) · Psychological Bulletin · doi:10.1037/bul0000234 · Cited by 169
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
An extended theory of planned behavior for parent-for-child health behaviors: A meta-analysis.
Hamilton, K., van Dongen, A., & Hagger, M. S. (2020) · Health Psychology · doi:10.1037/hea0000940 · Cited by 181
Lessons From Pinocchio: Cues to Deception May Be Highly Exaggerated
Luke, T. J. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619838258 · Cited by 174
An Aberrant Abundance of Cronbach’s Alpha Values at .70
Hussey, I., Alsalti, T., Bosco, F. A., Elson, M., & Arslan, R. C. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241287123 · Cited by 197
An umbrella review of randomized control trials on the effects of physical exercise on cognition
Ciria, L. F., Román-Caballero, R., Vadillo, M. A., Holgado, D., Luque-Casado, A., Perakakis, P., & Sanabria, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01554-4 · Cited by 157
Internalization of body shape ideals and body dissatisfaction: A systematic review and meta‐analysis
Paterna, A., Alcaraz‐Ibáñez, M., Fuller‐Tyszkiewicz, M., & Sicilia, Á. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23568 · Cited by 178
A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration
Coles, N. A., March, D. S., Marmolejo‐Ramos, F., Larsen, J. T., Arinze, N. C., Ndukaihe, I. L. G., Willis, M. L., Foroni, F., Reggev, N., Mokady, A., et al. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01458-9 · Cited by 170
Täschner, J., Dicke, T., Reinhold, S., & Holzberger, D. (2024) · Review of Educational Research · doi:10.3102/00346543231221499 · Cited by 175
Wartenberg, G., Aldrup, K., Grund, S., & Klusmann, U. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09831-4 · Cited by 180
Big little lies: a compendium and simulation of p -hacking strategies
Stefan, A. M., & Schönbrodt, F. D. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 177
Self-citationRobust Bayesian meta-analysis: Addressing publication bias with model-averaging.
Maier, M., Bartos̆, F., & Wagenmakers, E. (2022) · Psychological Methods · doi:10.1037/met0000405 · Cited by 153
Longitudinal tests of the theory of planned behaviour: A meta-analysis
Hagger, M. S., & Hamilton, K. (2023) · European Review of Social Psychology · doi:10.1080/10463283.2023.2225897 · Cited by 165
Bartos̆, F., Maier, M., Wagenmakers, E., Doucouliagos, H., & Stanley, T. D. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1594 · Cited by 156
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
Long-term Persuasive Effects in Narrative Communication Research: A Meta-Analysis
Oschatz, C., & Marker, C. (2020) · Journal of Communication · doi:10.1093/joc/jqaa017 · Cited by 149
Agreeableness and Its Consequences: A Quantitative Review of Meta-Analytic Findings
Wilmot, M. P., & Ones, D. S. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683211073007 · Cited by 130
Ironic Effects of Thought Suppression: A Meta-Analysis
Wang, D., Hagger, M. S., & Chatzisarantis, N. L. D. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691619898795 · Cited by 136
Do ‘watching eyes’ influence antisocial behavior? A systematic review & meta-analysis
Dear, K., Dutton, K., & Fox, E. (2019) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2019.01.006 · Cited by 143
Almaatouq, A., Griffiths, T. L., Suchow, J. W., Whiting, M. E., Evans, J., & Watts, D. J. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002874 · Cited by 134
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Current practices in meta‐regression in psychology, education, and medicine
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1339 · Cited by 133
Replication is important for educational psychology: Recent developments and key issues
Plucker, J. A., & Makel, M. C. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1895796 · Cited by 138
Siegel, R., König, C. J., & Lazar, V. (2022) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2022.100227 · Cited by 121
The Relationships of Intergroup Ideologies to Ethnic Prejudice: A Meta-Analysis
Whitley, B. E., & Webster, G. D. (2018) · Personality and Social Psychology Review · doi:10.1177/1088868318761423 · Cited by 138
The use of commercial computerised cognitive games in older adults: a meta-analysis
Bonnechère, B., Langley, C., & Sahakian, B. J. (2020) · Scientific Reports · doi:10.1038/s41598-020-72281-3 · Cited by 124
Is personality associated with dementia risk? A meta-analytic investigation
Aschwanden, D., Strickhouser, J. E., Luchetti, M., Stéphan, Y., Sutin, A. R., & Terracciano, A. (2021) · Ageing Research Reviews · doi:10.1016/j.arr.2021.101269 · Cited by 129
Román-Caballero, R., Vadillo, M. A., Trainor, L. J., & Lupiáñez, J. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100436 · Cited by 118
Aging and recognition memory: A meta-analysis.
Fraundorf, S. H., Hourihan, K. L., Peters, R. A., & Benjamin, A. S. (2019) · Psychological Bulletin · doi:10.1037/bul0000185 · Cited by 118
Spaniol, M., & Danielsson, H. (2021) · Journal of Intellectual Disability Research · doi:10.1111/jir.12878 · Cited by 134
Intuitive Honesty Versus Dishonesty: Meta-Analytic Evidence
Köbis, N., Verschuère, B., Bereby‐Meyer, Y., Rand, D. G., & Shalvi, S. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619851778 · Cited by 127
Pair-bonding, fatherhood, and the role of testosterone: a meta-analytic review
Grebe, N. M., Sarafin, R. E., Strenth, C. R., & Zilioli, S. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.01.010 · Cited by 116
Oser, J. L., Grinson, A., Boulianne, S., & Halperin, E. (2022) · Political Communication · doi:10.1080/10584609.2022.2086329 · Cited by 127
Bartos̆, F., Maier, M., Quintana, D. S., & Wagenmakers, E. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221109259 · Cited by 122
Aulbach, M. B., Knittle, K. P., & Haukkala, A. (2019) · Health Psychology Review · doi:10.1080/17437199.2019.1571933 · Cited by 121
Rhodes, R. E., Gray, S. M., & Husband, C. (2018) · Health Psychology Review · doi:10.1080/17437199.2018.1530067 · Cited by 119
Lin, H., Saunders, B., Friese, M., Evans, N. J., & Inzlicht, M. (2020) · Psychological Science · doi:10.1177/0956797620904990 · Cited by 118
McBreen, M., & Savage, R. S. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09584-4 · Cited by 110
Correcting for publication bias in a meta-analysis with the p-uniform* method
van Aert, R. C. M., & van Assen, M. A. L. M. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-025-02812-4 · Cited by 118
Nudging Social Media toward Accuracy
Pennycook, G., & Rand, D. G. (2022) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/00027162221092342 · Cited by 106
Meta-Analysis of Cognition in Parkinson’s Disease Mild Cognitive Impairment and Dementia Progression
Wallace, E. R., Segerstrom, S. C., van Horne, C. G., Schmitt, F. A., & Koehl, L. (2021) · Neuropsychology Review · doi:10.1007/s11065-021-09502-7 · Cited by 98
Effects of habit and intention on behavior: Meta-analysis and test of key moderators.
Hagger, M. S., Hamilton, K., Phipps, D. J., Protogerou, C., Zhang, C., Girelli, L., Mallia, L., & Lucidi, F. (2023) · Motivation Science · doi:10.1037/mot0000294 · Cited by 102
Effectiveness of conceptual change strategies in science education: A meta‐analysis
Paçacı, Ç., Üstün, U., & Özdemir, Ö. F. (2023) · Journal of Research in Science Teaching · doi:10.1002/tea.21887 · Cited by 111
Meta-Analysis of Present-Bias Estimation using Convex Time Budgets
Imai, T., Rutter, T. A., & Camerer, C. F. (2020) · The Economic Journal · doi:10.1093/ej/ueaa115 · Cited by 103
Hagger, M. S., & Hamilton, K. (2020) · European Journal of Personality · doi:10.1177/0890207020962330 · Cited by 108
When is narcissism associated with low empathy? A meta-analytic review
Urbonaviciute, G., & Hepper, E. G. (2020) · Journal of Research in Personality · doi:10.1016/j.jrp.2020.104036 · Cited by 97
Corruption Information and Vote Share: A Meta-Analysis and Lessons for Experimental Design
Incerti, T. (2020) · American Political Science Review · doi:10.1017/s000305542000012x · Cited by 92
Handling effect size dependency in meta-analysis
Gucciardi, D. F., Lines, R. L. J., & Ntoumanis, N. (2021) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1946835 · Cited by 94
McAuliffe, W. H. B., Carter, E. C., Berhane, J., Snihur, A., & McCullough, M. E. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868319887599 · Cited by 92
Self-citationSchaerer, M., du Plessis, C., Nguyen, M. H. B., van Aert, R. C. M., Tiokhin, L., Lakens, D., Giulia Clemente, E., Pfeiffer, T., Dreber, A., Johannesson, M., et al. (2023) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2023.104280 · Cited by 87
How Strongly Is Personality Associated With Burnout Among Teachers? A Meta-analysis
Roloff, J., Kirstges, J., Grund, S., & Klusmann, U. (2022) · Educational Psychology Review · doi:10.1007/s10648-022-09672-7 · Cited by 105
Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project
Rohrer, J. M., Tierney, W., Uhlmann, E. L., DeBruine, L. M., Heyman, T., Jones, B. C., Schmukle, S. C., Silberzahn, R., Willén, R. M., Carlsson, R., et al. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620964106 · Cited by 87
Unemployment and health: A meta‐analysis
Picchio, M., & Ubaldi, M. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12588 · Cited by 99
Problematic smartphone use and academic achievement: A systematic review and meta-analysis
Paterna, A., Alcaraz-Ibáñez, M., Aguilar-Parra, J. M., Salavera, C., Demetrovics, Z., & Griffiths, M. D. (2024) · Journal of Behavioral Addictions · doi:10.1556/2006.2024.00014 · Cited by 83
A comprehensive meta-analysis of money priming.
Lodder, P., Ong, H. H., Grasman, R. P. P. P., & Wicherts, J. M. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000570 · Cited by 86
Physical activity and cortisol regulation: A meta-analysis
Moyers, S. A., & Hagger, M. S. (2023) · Biological Psychology · doi:10.1016/j.biopsycho.2023.108548 · Cited by 89
A systematic review and meta-analysis of the evidence for unaware fear conditioning
Mertens, G., & Engelhard, I. M. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.11.012 · Cited by 81
Gieseler, K., Loschelder, D. D., & Friese, M. (2019) · Springer eBooks · doi:10.1007/978-3-030-13788-5_1 · Cited by 80
Detecting publication selection bias through excess statistical significance
Stanley, T. D., Doucouliagos, H., Ioannidis, J. P. A., & Carter, E. C. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1512 · Cited by 83
Self-citationHeterogeneity in direct replications in psychology and its association with effect size.
Olsson-Collentine, A., Wicherts, J. M., & van Assen, M. A. L. M. (2020) · Psychological Bulletin · doi:10.1037/bul0000294 · Cited by 81
Metadata
- Authors
- Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J.
- Affiliations
- DEVCOM Army Research Laboratory (US)
- Ludwig-Maximilians-Universität München (DE)
- University of Kentucky (US)
- Illinois State University (US)
- Corresponding author
- Carter, Evan C.
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2019-06
- Volume
- 2
- Issue
- 2
- Pages
- 115-144
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W2781957993
- Cited by
- 588
- References
- 76
- Citations by year
- 2026: 43 · 2025: 60 · 2024: 69 · 2023: 86 · 2022: 83 · 2021: 87 · 2020: 79 · 2019: 54
- FWCI
- 50.69
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Advances in Methods and Practices in Psychological Science (publishedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- publication bias, questionable research practices, effect size overestimation, meta-analytic methods, sensitivity analysis, heterogeneity, simulation study, pre-registered replication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
