Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes.
Open accessPublished version of a preprint
Abstract
Selective reporting of results based on their statistical significance threatens the validity of meta-analytic findings. A variety of techniques for detecting selective reporting, publication bias, or small-study effects are available and are routinely used in research syntheses. Most such techniques are univariate, in that they assume that each study contributes a single, independent effect size estimate to the meta-analysis. In practice, however, studies often contribute multiple, statistically dependent effect size estimates, such as for multiple measures of a common outcome construct. Many methods are available for meta-analyzing dependent effect sizes, but methods for investigating selective reporting while also handling effect size dependencies require further investigation. Using Monte Carlo simulations, we evaluate three available univariate tests for small-study effects or selective reporting, including the trim and fill test, Egger's regression test, and a likelihood ratio test from a three-parameter selection model (3PSM), when dependence is ignored or handled using ad hoc techniques. We also examine two variants of Egger's regression test that incorporate robust variance estimation (RVE) or multilevel meta-analysis (MLMA) to handle dependence. Simulation results demonstrate that ignoring dependence inflates Type I error rates for all univariate tests. Variants of Egger's regression maintain Type I error rates when dependent effect sizes are sampled or handled using RVE or MLMA. The 3PSM likelihood ratio test does not fully control Type I error rates. With the exception of the 3PSM, all methods have limited power to detect selection bias except under strong selection for statistically significant effects. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Publication bias was assessed both visually, using funnel plots, and statistically, using a multilevel Egger-type regression test, in which the standard error of the effect size was included as a predictor in a mixed-effects model.
“Publication bias was assessed both visually, using funnel plots, and statistically, using a multilevel Egger-type regression test, in which the standard error of the effect size was included as a predictor in a mixed-effects model (39).”
From Carbohydrate ingestion during prolonged exercise and net skeletal muscle glycogen utilization: A meta-analysis. (2026) · cites this paper as [39]
The opening summary (p. 1) says the paper evaluates an Egger regression variant using multilevel meta-analysis to detect selective reporting. The introduction (approximately p. 2) describes Egger’s test as a test of funnel-plot asymmetry. The retrieved passages do not establish visual inspection of funnel plots or specify that the effect-size standard error is entered as a predictor in a mixed-effects model. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Parry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H., & Quintana, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01117-5 · Cited by 793 · see citing works
“Egger’s regression test (incorporating RVE per the Egger–sandwich test) 82 , indicated no evidence of small study bias in this sample ( β = 0.55, P = 0.136); see Fig.”
MethodInfluentialCipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636 · see citing works
“These practices of data treatment allowed us to build upon a comprehensive catalog of data for review, and reduced researcher bias while maintaining methodological transparency (Pigott & Polanin, 2020; Rodgers & Pustejovsky, 2021).”
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O’Dea, R. E. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 455 · see citing works
Visual attentional orienting by eye gaze: A meta-analytic review of the gaze-cueing effect.
McKay, K. T., Grainger, S. A., Coundouris, S. P., Skorich, D. P., Phillips, L. H., & Henry, J. D. (2021) · Psychological Bulletin · doi:10.1037/bul0000353 · Cited by 443 · see citing works
Baldwin, J. R., Wang, B., Karwatowska, L., Schoeler, T., Tsaligopoulou, A., Munafò, M. R., & Pingault, J.-B. (2023) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20220174 · Cited by 313 · see citing works
Palmer, C. A., Bower, J. L., Cho, K. W., Clementi, M. A., Lau, S., Oosterhoff, B. J., & Alfano, C. A. (2023) · Psychological Bulletin · doi:10.1037/bul0000410 · Cited by 237 · see citing works
Johnson, D., Policelli, J., Li, M., Dharamsi, A., Hu, Q., Sheridan, M. A., McLaughlin, K. A., & Wade, M. (2021) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2021.2511 · Cited by 179 · see citing works
Polanin, J. R., Espelage, D. L., Grotpeter, J. K., Ingram, K., Michaelson, L., Spinney, E., Valido, A., Sheikh, A. E., Torgal, C., & Robinson, L. (2022) · Prevention Science · doi:10.1007/s11121-021-01259-y · Cited by 171 · see citing works
Hall, C., Dahl‐Leonard, K., Cho, E., Solari, E. J., Capin, P., Conner, C. L., Henry, A. R., Cook, L., Hayes, L. L., Vargas, I., et al. (2022) · Reading Research Quarterly · doi:10.1002/rrq.477 · Cited by 169 · see citing works
Examining the relations between spatial skills and mathematical performance: A meta-analysis
Atit, K., Power, J., Pigott, T., Lee, J., Geer, E. A., Uttal, D. H., Ganley, C. M., & Sorby, S. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-021-02012-w · Cited by 160 · see citing works
“To examine possible publication bias in the data, we implemented a modified version of Egger ’ s test of funnel plot asymmetry (Egger et al., 1997), or Egger ’ s sandwich (Rodgers & Pustejovsky, 2020), that can be used when dependent effect sizes exist in the meta-analysis data.”
MethodSandbank, M. P., Bottema‐Beutel, K., Crowley, S., Feldman, J. I., Barrett, D. J., Caldwell, N., Dunham-Carr, K., Crank, J., Albarran, S., & Woynaroski, T. G. (2023) · BMJ · doi:10.1136/bmj-2023-076733 · Cited by 187 · see citing works
“Results from the Egger multilevel meta-analysis test 31 indicated that, across all outcomes analyzed in our models, there was evidence for funnel plot asymmetry (B=2.98, P 0.001).”
MethodInfluentialAn umbrella review of randomized control trials on the effects of physical exercise on cognition
Ciria, L. F., Román-Caballero, R., Vadillo, M. A., Holgado, D., Luque-Casado, A., Perakakis, P., & Sanabria, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01554-4 · Cited by 157 · see citing works
Influencing climate change attitudes in the United States: A systematic review and meta-analysis
Rode, J. B., Dent, A. L., Benedict, C. N., Brosnahan, D., Martinez, R. L., & Ditto, P. H. (2021) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2021.101623 · Cited by 144 · see citing works
Wang, L. A. L., Petrulla, V., Zampella, C., Waller, R. E., & Schultz, R. T. (2022) · Psychological Bulletin · doi:10.1037/bul0000358 · Cited by 141 · see citing works
Baldwin, J., Coleman, O., Francis, E. R., & Danese, A. (2024) · JAMA psychiatry · doi:10.1001/jamapsychiatry.2024.0818 · Cited by 124 · see citing works
Román-Caballero, R., Vadillo, M. A., Trainor, L. J., & Lupiáñez, J. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100436 · Cited by 118 · see citing works
“A way to account for dependence is to combine all the effect sizes coming from the same sample generating an average estimate for each study, and conduct the classic methods on these aggregated estimates (Rodgers & Pustejovsky, 2020).”
BackgroundMethodInfluentialYuan, T., Hu, Y., Zhou, X., et al. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101613 · Cited by 118 · see citing works
Wang, B., Baldwin, J. R., Schoeler, T., Cheesman, R., Barkhuizen, W., Dudbridge, F., Bann, D., Morris, T. T., & Pingault, J.-B. (2021) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2021.07.010 · Cited by 103 · see citing works
Oser, J. L., Grinson, A., Boulianne, S., & Halperin, E. (2022) · Political Communication · doi:10.1080/10584609.2022.2086329 · Cited by 116 · see citing works
“The results provide evidence of modest publication bias for OffPP but no evidence of such bias for OnPP, although there is not yet consensus regarding thresholds for publication bias at the effect-level in multilevel meta-analysis (Fernández-Castilla et al., 2021; Rodgers & Pustejovsky, 2021).”
BackgroundThe role of teaching staff in fostering perceived employability of university students
Petruzziello, G., Mariani, M. G., Guglielmi, D., HEIJDEN, B. I. J. M. V. D., de Jong, J. P., & Chiesa, R. (2022) · Studies in Higher Education · doi:10.1080/03075079.2022.2105830 · Cited by 94 · see citing works
Kittel, A. F. D., Olderbak, S. G., & Wilhelm, O. (2021) · Assessment · doi:10.1177/1073191121996469 · Cited by 86 · see citing works
“We could not estimate trim and fill analyses for theory of mind, cognitive empathy, or affective empathy because the nonindependent in the effect sizes is ignored by trim and fill analyses and results could lead to Type I errors (Rodgers & Pustejovsky, 2020).”
BackgroundJoyal‐Desmarais, K., Scharmer, A., Madzelan, M. K., See, J. V., Rothman, A. J., & Snyder, M. (2022) · Psychological Bulletin · doi:10.1037/bul0000377 · Cited by 88 · see citing works
Schoenherr, J., Strohmaier, A., & Schukajlow, S. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100639 · Cited by 83 · see citing works
Effects of action video game play on cognitive skills: A meta-analysis.
Bediou, B., Rodgers, M. A., Tipton, E., Mayer, R. E., Green, C. S., & Bavelier, D. (2023) · Technology Mind and Behavior · doi:10.1037/tmb0000102 · Cited by 90 · see citing works
Self-citationReinebo, G., Alfonsson, S., Jansson‐Fröjmark, M., Rozental, A., & Lundgren, T. (2023) · Sports Medicine · doi:10.1007/s40279-023-01931-z · Cited by 111 · see citing works
“An adapted version of Egger’s regression was also conducted to test for funnel plot asymmetry [45, 46].”
MethodThe sleep benefit in episodic memory: An integrative review and a meta-analysis.
Berres, S., & Erdfelder, E. (2021) · Psychological Bulletin · doi:10.1037/bul0000350 · Cited by 74 · see citing works
Depression and episodic memory across the adult lifespan: A meta-analytic review.
James, T., Weiss-Cowie, S., Hopton, Z. W., Verhaeghen, P., Dotson, V. M., & Duarte, A. (2021) · Psychological Bulletin · doi:10.1037/bul0000344 · Cited by 77 · see citing works
“Regression analysis on the funnel plot (Figure 2) was conducted using an Egger test appropriate for multilevel metaanalysis (Rodgers & Pustejovsky, 2020).”
MethodNickow, A., Oreopoulos, P., & Quan, V. (2024) · American Educational Research Journal · doi:10.3102/00028312231208687 · Cited by 82 · see citing works
“Regressing ES on W i using the same RVE models as for our main estimates, we then observe the significance of the W i term and especially changes in the regression constant, which represents pooled ES (Rodgers & Pustejovsky, 2021).”
MethodPredictors of Political Violence Outcomes among Young People: A Systematic Review and Meta‐Analysis
Jahnke, S., Borger, K. A., & Beelmann, A. (2021) · Political Psychology · doi:10.1111/pops.12743 · Cited by 68 · see citing works
“Rodgers and Pustejovsky (2019) have recently introduced a new version of the traditional Egger test that is based on RVE and can be used for dependent samples.”
BackgroundBackhaus, S., Leijten, P. H. O., Jochim, J., Meléndez‐Torres, G. J., & Gardner, F. E. M. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102003 · Cited by 69 · see citing works
Sex drive: Theoretical conceptualization and meta-analytic review of gender differences.
Frankenbach, J., Weber, M., Loschelder, D. D., Kilger, H., & Friese, M. (2022) · Psychological Bulletin · doi:10.1037/bul0000366 · Cited by 68 · see citing works
Wollburg, C., Steinert, J., Reeves, A., & Nye, E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281283 · Cited by 41 · see citing works
“It should be noted that funnel plot asymmetry may occur for reasons other than publication bias and traditional methods may not adequately account for effect size dependency [65,67].”
BackgroundWang, F., Cheung, A. C. K., Neitzel, A. J., & Chai, C. S. (2025) · Review of Educational Research · doi:10.3102/00346543241255621 · Cited by 64 · see citing works
“Other methods to test for and assess the degree of publication bias exist, but few have been utilized with robust variance estimation and may rely on larger sample sizes (Rodgers & Pustejovsky, 2021).”
McWeeny, S., Choi, S. J., Choe, J., LaTourrette, A. S., Roberts, M. Y., & Norton, E. S. (2022) · Reading Research Quarterly · doi:10.1002/rrq.467 · Cited by 74 · see citing works
Emmer, C., Dorn, J., & Mata, J. (2024) · Psychological Bulletin · doi:10.1037/bul0000419 · Cited by 78 · see citing works
Salmerón, L., Altamura, L., Delgado, P., Karagiorgi, A., & Vargas, C. (2023) · Journal of Educational Psychology · doi:10.1037/edu0000830 · Cited by 71 · see citing works
Self-regulated learning strategies in continuing education: A systematic review and meta-analysis
Hemmler, Y. M., & Ifenthaler, D. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100629 · Cited by 95 · see citing works
“However, reliable methods for detecting publication bias in three-level meta-analyses are still lacking, and research has shown that the three-level version of the Egger ’ s regression test may suffer from a lack of statistical power (Rodgers & Pustejovsky, 2021).”
InfluentialStucke, N. J., & Doebel, S. (2024) · Psychological Bulletin · doi:10.1037/bul0000445 · Cited by 69 · see citing works
Myers, J. A., Witzel, B., Powell, S. R., Li, H.-L., Pigott, T. D., Yan, P.-X., & Hughes, E. M. (2022) · Review of Educational Research · doi:10.3102/00346543211070049 · Cited by 70 · see citing works
de Roo, M., Veenstra, R., & Kretschmer, T. (2022) · Social Development · doi:10.1111/sode.12590 · Cited by 66 · see citing works
“To assess the likelihood of publication bias, we conducted Egger’s test for funnel plot asymmetry using sample size as amoderator to account for dependence among effect sizes (Rodgers & Pustejovsky, 2020).”
MethodWan, S., Lauermann, F., Bailey, D. H., & Eccles, J. S. (2021) · Psychological Bulletin · doi:10.1037/bul0000340 · Cited by 61 · see citing works
Roest, J. J., de Poll, M. J. W.-V., van der Helm, P. G., Stams, G. J. J. M., & Hoeve, M. (2022) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2022.2093210 · Cited by 60 · see citing works
“Moreover, for multilevel meta-analysis, specific methods to take into account dependency of effect sizes in the assessment of publication bias are still under development (Assink & Wibbelink, 2016; Fernández-Castilla et al., 2021; Rodgers & Pustejovsky, 2021).”
MethodSchmaußer, M., Hoffmann, S., Raab, M., & Laborde, S. (2022) · Journal of Neuroscience Research · doi:10.1002/jnr.25062 · Cited by 59 · see citing works
Bourke, M., Wang, H., McNaughton, S. A., Thomas, G., Firth, J., Trott, M., & Cairney, J. (2025) · Clinical Psychology Review · doi:10.1016/j.cpr.2025.102585 · Cited by 65 · see citing works
A meta-analysis of Libet-style experiments
Braun, M. N., Wessler, J., & Friese, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.018 · Cited by 59 · see citing works
Meta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 55 · see citing works
Schwabe, A., Lind, F., Kosch, L., & Boomgaarden, H. G. (2022) · Media Psychology · doi:10.1080/15213269.2022.2070216 · Cited by 75 · see citing works
“However, the detected funnel plot asymmetry could have another cause than a publication bias (Rodgers & Pustejovsky, 2021).”
BackgroundMethodReal-time fMRI neurofeedback as a new treatment for psychiatric disorders: A meta-analysis
Pindi, P., Houenou, J., Piguet, C., & Favre, P. (2022) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2022.110605 · Cited by 55 · see citing works
Dietrichson, J., Filges, T., Seerup, J. K., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Eiberg, M. (2021) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1152 · Cited by 54 · see citing works
Chacón-Candia, J. A., Román-Caballero, R., Aranda‐Martín, B., Casagrande, M., Lupiáñez, J., & Marotta, A. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104993 · Cited by 60 · see citing works
Winters, K. L., Jasso, J., Pustejovsky, J. E., & Byrd, C. T. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00017 · Cited by 54 · see citing works
Self-citationManipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Braß, M., & Schooler, J. W. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683221087527 · Cited by 60 · see citing works
Patall, E. A., Yates, N., Lee, J., Chen, M., Bhat, B. H., Lee, K., Beretvas, S. N., Lin, S., Hooper, S. Y., Jacobson, N. G., et al. (2023) · Educational Psychologist · doi:10.1080/00461520.2023.2274104 · Cited by 67 · see citing works
“Finally, we examined the possibility of publication bias and funnel plot asymmetry by conducting a modified version of Egger’s regression (Egger et al., 1997) that accounted for the dependent effect sizes (Rodgers & Pustejovsky, 2020).”
MethodA Meta-Analysis on Teachers’ Growth Mindset
Bardach, L., Bostwick, K. C. P., Fütterer, T., Kopatz, M., Hobbi, D. M., Klassen, R. M., & Pietschnig, J. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09925-7 · Cited by 69 · see citing works
“Therefore, we used three methods that have been shown to be suitable for multilevel meta-analyses in targeted investigations (Rodgers & Pustejovsky, 2021).”
MethodInfluentialImproving Low-Performing Schools: A Meta-Analysis of Impact Evaluation Studies
Schueler, B. E., Asher, C. A., Larned, K. E., Mehrotra, S., & Pollard, C. (2021) · American Educational Research Journal · doi:10.3102/00028312211060855 · Cited by 49 · see citing works
“We implement RVE in Stata using the Robumeta package (Tanner-Smith & Tipton, 2014) parameterized for hierarchical effects.”
MethodInfluentialNeglect of publication bias compromises meta-analyses of educational research
Ropovik, I., Adamkovic, M., & Greger, D. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252415 · Cited by 52 · see citing works
“The meta-analyst then needs to be aware that if dependencies are not properly modeled (see [61]) or handled otherwise (e.”
Yang, Y., Macleod, M. R., Pan, J., Lagisz, M., & Nakagawa, S. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.105016 · Cited by 49 · see citing works
Girdwood, M., Culvenor, A. G., Rio, E., Patterson, B., Haberfield, M. J., Couch, J. L., Mentiplay, B. F., Hedger, M., & Crossley, K. M. (2024) · British Journal of Sports Medicine · doi:10.1136/bjsports-2023-107977 · Cited by 48 · see citing works
Dudek, E. M., & Dodell‐Feder, D. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.020 · Cited by 44 · see citing works
A systematic review and meta-analysis of strategies to promote vaccination uptake
Liu, S., Durantini, M. R., Calabrese, C., Sánchez, F., & Albarracín, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01940-6 · Cited by 46 · see citing works
“…effect sizes and involves one-sided t -tests to detect the linear relation between effect size magnitude and the associated precision 170,171 , together with the visual inspection of the funnel plot, whose asymmetry of data point distribution suggests missing studies due to reporting…”
MethodIdentifying Key Antecedents of Quiet Quitting Among Nurses: A Cross‐Profession Meta‐Analytic Review
Geng, R., Geng, X., & Geng, S. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.16934 · Cited by 51 · see citing works
Cross-cultural variation in cooperation: A meta-analysis.
Spadaro, G., Graf, C., Jin, S., Arai, S., Inoue, Y., Lieberman, E., Rinderu, M. I., Yuan, M., Lissa, C. J. V., & Balliet, D. P. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000389 · Cited by 44 · see citing works
Denton, C. A., Hall, C., Cho, E., Cannon, G., Scammacca, N., & Wanzek, J. (2021) · Learning and Individual Differences · doi:10.1016/j.lindif.2021.102062 · Cited by 51 · see citing works
Speech‐in‐noise perception in musicians and non‐musicians: A multi‐level meta-analysis
Hennessy, S., Mack, W. J., & Habibi, A. (2022) · Hearing Research · doi:10.1016/j.heares.2022.108442 · Cited by 41 · see citing works
Roberts, B. R. T., MacLeod, C. M., & Fernandes, M. A. (2022) · Psychological Bulletin · doi:10.1037/bul0000360 · Cited by 47 · see citing works
“To assess publication bias while maintaining robust variance estimates, we used the “ Egger sandwich ” analysis method (Rodgers & Pustejovsky, 2020).”
MethodHow universal is preference for visual curvature? A systematic review and meta‐analysis
Chuquichambi, E. G., Vartanian, O. A., Skov, M., Corradi, G. B., Nadal, M., Silvia, P. J., & Munar, E. (2022) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.14919 · Cited by 51 · see citing works
Wallrich, L., Opara, V., Wesołowska, M., Barnoth, D., & Yousefi, S. (2024) · Journal of Business and Psychology · doi:10.1007/s10869-024-09977-0 · Cited by 59 · see citing works
A broadscale analysis of host‐symbiont cophylogeny reveals the drivers of phylogenetic congruence
Hayward, A., Poulin, R., & Nakagawa, S. (2021) · Ecology Letters · doi:10.1111/ele.13757 · Cited by 42 · see citing works
Vembye, M. H., Weiss, F., & Bhat, B. H. (2023) · Review of Educational Research · doi:10.3102/00346543231186588 · Cited by 62 · see citing works
“Moreover, selection models based on dependent effect sizes aggregated to the study level do not fully calibrate the nominal Type-I error rate (Rodgers & Pustejovsky, 2021), and the WAAP test used was based on just three adequately powered studies.”
MethodMcNally, X., Webb, T. L., Smith, C., Moss, A., & Miller, J. G. (2024) · Conservation Biology · doi:10.1111/cobi.14237 · Cited by 43 · see citing works
Zhou, L., Sommer, I. E., Yang, P., Sikirin, L., van Os, J., Bentall, R. P., Varese, F., & Begemann, M. (2025) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20240456 · Cited by 51 · see citing works
Effects of the 5E Instructional Model: A Systematic Review and Meta-Analysis
Polanin, J. R., Austin, M. J., Taylor, J. A., Steingut, R. R., Rodgers, M. A., & Williams, R. T. (2024) · AERA Open · doi:10.1177/23328584241269866 · Cited by 74 · see citing works
Self-citationJoshi, M., Pustejovsky, J. E., & Beretvas, S. N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1554 · Cited by 39 · see citing works
Self-citationEmotional intelligence and romantic relationship satisfaction: A systematic review and meta-analysis
Jardine, B. B., Vannier, S. A., & Voyer, D. (2022) · Personality and Individual Differences · doi:10.1016/j.paid.2022.111713 · Cited by 44 · see citing works
Vieira, A. P. A., Peng, P., Antoniuk, A., DeVries, J., Rothou, K. M., Parrila, R., & Georgiou, G. K. (2023) · Annals of Dyslexia · doi:10.1007/s11881-023-00294-4 · Cited by 48 · see citing works
Demangeon, A., Claudel-Valentin, S., Aubry, A., & Tazouti, Y. (2023) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2023.102182 · Cited by 55 · see citing works
Ye, L., Li, Y., Deng, Q., Zhao, X., Zhong, L., & Yang, L. (2024) · PLoS ONE · doi:10.1371/journal.pone.0301226 · Cited by 43 · see citing works
Kong, F., Meng, S., Deng, H., Wang, M., & Sun, X. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09746-0 · Cited by 45 · see citing works
Meta-analysis of second language research with complex research designs
Norouzian, R., & Bui, G. (2023) · Studies in Second Language Acquisition · doi:10.1017/s0272263123000311 · Cited by 38 · see citing works
January, S. A., & Klingbeil, D. A. (2020) · Journal of School Psychology · doi:10.1016/j.jsp.2020.08.007 · Cited by 38 · see citing works
Ivaniushina, V. A., & Викторовна, Т. В. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250169 · Cited by 39 · see citing works
“While no bias was detected, it should be noted that such tests do not have strong power, especially when the number of articles is small, as in case of our meta-analysis [34].”
BackgroundMethodThe Impasse on Gender Differences in Intelligence: a Meta-Analysis on WISC Batteries
Giofrè, D., Allen, K., Toffalini, E., & Caviola, S. (2022) · Educational Psychology Review · doi:10.1007/s10648-022-09705-1 · Cited by 45 · see citing works
“In addition, regression 1 3 tests (PET-PEESE is a modification of Egger’s test; Stanley & Doucouliagos, 2014) can be used within the multi-level meta-analytic framework (Rodgers & Pustejovsky, 2021).”
MethodDanielsen, P. A., Mentzoni, R. A., & Låg, T. (2023) · Addictive Behaviors · doi:10.1016/j.addbeh.2023.107887 · Cited by 35 · see citing works
Cui, X., Zhang, S., Yu, S., Ding, Q., & Li, X. (2024) · Behaviour Research and Therapy · doi:10.1016/j.brat.2024.104549 · Cited by 37 · see citing works
Effects of peer observation on risky decision-making in adolescence: A meta-analytic review.
Powers, K. E., Schaefer, L., Figner, B., & Somerville, L. H. (2022) · Psychological Bulletin · doi:10.1037/bul0000382 · Cited by 36 · see citing works
“In addition, we implemented Vevea and Hedges (1995) three-parameter model that provides greater power and can be interpreted as a more direct test of publication bias (Rodgers & Pustejovsky, 2020).”
MethodChang, D. F., Donald, J. N., Whitney, J. M., Miao, I. Y., & Sahdra, B. K. (2023) · Personality and Social Psychology Bulletin · doi:10.1177/01461672231178518 · Cited by 33 · see citing works
Anxiety and Depression in Gifted Individuals: A Systematic and Meta-Analytic Review
Duplenne, L., Bourdin, B., Fernandez, D., Blondelle, G., & Aubry, A. (2023) · Gifted Child Quarterly · doi:10.1177/00169862231208922 · Cited by 46 · see citing works
“This test could be adapted with effect size dependence (Fernández-Castilla et al., 2021; Rodgers & Pustejovsky, 2021).”
MethodSwaine, M., Bergna, A., Oyserman, B. O., et al. (2025) · FEMS Microbiology Ecology · doi:10.1093/femsec/fiaf052 · Cited by 34 · see citing works
“Publication bias was assessed using Egger’s test for multilevel models (Rodgers and Pustejovsky 2021 ).”
Patten, R. V., Mulhauser, K. R. W., Austin, T. A., Bellone, J. A., Cotton, E., Chan, L. C. N., Twamley, E. W., Sawyer, K., & LaFrance, W. C., Jr. (2025) · Clinical Psychology Review · doi:10.1016/j.cpr.2025.102648 · Cited by 38 · see citing works
Webber, H. E., de Dios, C., Kessler, D. A., Schmitz, J. M., Lane, S. D., & Suchting, R. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104835 · Cited by 34 · see citing works
Donolato, E., Toffalini, E., Rogde, K., Nordahl‐Hansen, A., Lervåg, A., Norbury, C., & Melby‐Lervåg, M. (2023) · Campbell Systematic Reviews · doi:10.1002/cl2.1368 · Cited by 62 · see citing works
“Unfortunately, the p ‐ curve method may present the problem of not accounting for the presence of dependency structures between the effect sizes, and it is known to perform poorly when there is between ‐ study heterogeneity (Rodgers & Pustejovsky, 2021).”
Dietrichson, J., Filges, T., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Jensen, U. H. (2020) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1081 · Cited by 33 · see citing works
Геннадьевич, М. К., Forthmann, B., Karwowski, M., & Benedek, M. (2023) · Intelligence · doi:10.1016/j.intell.2023.101739 · Cited by 31 · see citing works
The effect of notetaking method on academic performance: A systematic review and meta-analysis
Voyer, D., Ronis, S. T., & Byers, N. (2021) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2021.102025 · Cited by 38 · see citing works
Siepe, B. S., Bartos̆, F., Morris, T. P., Boulesteix, A., Heck, D. W., & Pawel, S. (2024) · Psychological Methods · doi:10.1037/met0000695 · Cited by 57 · see citing works
“Hypothesis testing, e.g., comparing different tests of publication bias (Rodgers & Pustejovsky, 2021).”
Hagger, M. S. (2022) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1966824 · Cited by 36 · see citing works
“For a more detailed explanation, see Rodgers and Pustejovsky (2021).”
BackgroundGolan, O. K., Totaram, R., Perry, E. W., Fortson, K., Rivera-Atilano, R., Entress, R. M., Golan, M., Andraka‐Christou, B., Whitaker, D. J., & Pigott, T. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2022.109445 · Cited by 34 · see citing works
Conservation interventions are required to improve bird breeding performance in artificial wetlands
Cheng, C., & Ma, Z. (2022) · Biological Conservation · doi:10.1016/j.biocon.2022.109872 · Cited by 31 · see citing works
Zhang, Y., Zhou, S., Wu, X., & Cheung, A. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100627 · Cited by 45 · see citing works
The effectiveness of refutation text in confronting scientific misconceptions: A meta-analysis
Danielson, R., Jacobson, N. G., Patall, E. A., Sinatra, G. M., Adesope, O., Kennedy, A. A. U., Bhat, B. H., Ramazan, O., Akinrotimi, B., Nketah, G., et al. (2024) · Educational Psychologist · doi:10.1080/00461520.2024.2365628 · Cited by 35 · see citing works
“…of publication bias and funnel plot asymmetry by creating funnel plots for IG and IGRM effects and each outcome separately and conducting a modified version of Egger’s regression (Egger et al., 1997) for each outcome that accounted for the dependent effect sizes (Rodgers & Pustejovsky, 2020).”
- Authors
- Rodgers, M. A., & Pustejovsky, J. E.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- Psychological Methods
- Published
- 2021-04
- Volume
- 26
- Issue
- 2
- Pages
- 141-160
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W3043790905
- Cited by
- 492
- References
- 0
- Citations by year
- 2026: 111 · 2025: 111 · 2024: 95 · 2023: 72 · 2022: 61 · 2021: 34 · 2020: 8
- FWCI
- 26.68
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- MetArXiv (OSF Preprints) (submittedVersion)
- MetArXiv (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- selective reporting, publication bias, small-study effects, dependent effect sizes, robust variance estimation, multilevel meta-analysis, type I error rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
