Robust point and variance estimation for ecological and evolutionary meta-analyses with selective reporting and dependent effect sizes

Abstract

Meta-analysis produces a quantitative synthesis of evidence-based knowledge, shaping not only research trends but also policy and practices in ecology and evolution. However, two statistical issues, selective reporting and statistical dependence, can severally distort meta-analytic evidence. Here, we propose a two-step procedure to tackle these challenges concurrently and re-analyse 448 ecological and evolutionary meta-analyses. First, we employ bias-robust weighting schemes under the generalized least square estimator to obtain less biased population mean effect sizes by mitigating selective reporting. Second, we use cluster-robust variance estimation to account for statistical dependence and reduce bias in estimating standard errors, ensuring valid statistical inference. Re-analyses of 448 meta-analyses show that ignoring the two issues tends to overestimate effect sizes by an average of 110% and underestimate standard errors by 120%. Our approach is effective at mitigating these biases in meta-analytic evidence. To facilitate the implementation, we have developed a website showing the step-by-step tutorial available on our website. Complementing the current meta-analytic practice with the proposed method can facilitate a transition to a more pluralistic approach in quantitative evidence synthesis in ecology and evolution.