CoChat
Sign Up
PreprintCited by 2

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

Robust point and variance estimation for ecological and evolutionary meta-analyses with selective reporting and dependent effect sizes

Yang, Yefeng, Lagisz, Malgorzata, Williams, Coralie, Pan, Jinming, Noble, Daniel, Nakagawa, Shinichi·2023
DOI 10.32942/x20g6q
Open this paper in CoChat — free account

Abstract

Meta-analysis produces a quantitative synthesis of evidence-based knowledge, shaping not only research trends but also policy and practices in ecology and evolution. However, two statistical issues, selective reporting and statistical dependence, can severally distort meta-analytic evidence. Here, we propose a two-step procedure to tackle these challenges concurrently and re-analyse 448 ecological and evolutionary meta-analyses. First, we employ bias-robust weighting schemes under the generalized least square estimator to obtain less biased population mean effect sizes by mitigating selective reporting. Second, we use cluster-robust variance estimation to account for statistical dependence and reduce bias in estimating standard errors, ensuring valid statistical inference. Re-analyses of 448 meta-analyses show that ignoring the two issues tends to overestimate effect sizes by an average of 110% and underestimate standard errors by 120%. Our approach is effective at mitigating these biases in meta-analytic evidence. To facilitate the implementation, we have developed a website showing the step-by-step tutorial available on our website. Complementing the current meta-analytic practice with the proposed method can facilitate a transition to a more pluralistic approach in quantitative evidence synthesis in ecology and evolution.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.32942/x20g6q

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Figure 2 Comparison of the population mean effect size estimates 𝛽 ̂ and their standard errors SE(𝛽 ̂ ) obtained from different meta-analytic models. The dashed diagonal line is 𝑦 = 𝑥 , indicating where 𝛽 ̂ and SE(𝛽 ̂ ) would fall if there were no discrepancies between the models. The benchmark method, MLMA (multilevel meta-analysis model), is considered a standard method for addressing statistical dependence. The UWLS (unrestricted weighted least squares) and FE + VCV (fixed-effect model with
Figure 2 Comparison of the population mean effect size estimates 𝛽 ̂ and their standard errors SE(𝛽 ̂ ) obtained from different meta-analytic models. The dashed diagonal line is 𝑦 = 𝑥 , indicating where 𝛽 ̂ and SE(𝛽 ̂ ) would fall if there were no discrepancies between the models. The benchmark method, MLMA (multilevel meta-analysis model), is considered a standard method for addressing statistical dependence. The UWLS (unrestricted weighted least squares) and FE + VCV (fixed-effect model withCC-BY · From the PDF
Figure 3 Paired comparison of the estimated population mean effect size estimates 𝛽 ̂ obtained from the benchmark method (MLMA) and the first-step procedure of the proposed approach (UWLS and FE +VCV). For other details, refer to the legend in Figure 2.
Figure 3 Paired comparison of the estimated population mean effect size estimates 𝛽 ̂ obtained from the benchmark method (MLMA) and the first-step procedure of the proposed approach (UWLS and FE +VCV). For other details, refer to the legend in Figure 2.CC-BY · From the PDF
Figure 4 The relationship between the difference in the population mean effect size estimates 𝛽 ̂ and the severity of publication bias.
Figure 4 The relationship between the difference in the population mean effect size estimates 𝛽 ̂ and the severity of publication bias.CC-BY · From the PDF
Figure 5 The relationship between the difference in the estimated standard error of the mean effect sizes SE(𝛽 ̂ ) and the indicator of degree of statistical dependence. Intraclass correlation coefficient (ICC) represents the degree of dependence among effect sizes in a meta-analysis. For other details, refer to the legends in Figures 2 and 4.
Figure 5 The relationship between the difference in the estimated standard error of the mean effect sizes SE(𝛽 ̂ ) and the indicator of degree of statistical dependence. Intraclass correlation coefficient (ICC) represents the degree of dependence among effect sizes in a meta-analysis. For other details, refer to the legends in Figures 2 and 4.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 6 The implementation of the proposed two-step meta-analytic modelling approach, along with a novel visualization, using open-source software. The metafor 56 package is utilized for building twostep models, while the orchaRd 54 package is employed for graphical representation. For a detailed step-by-step tutorial demonstrating two alternative first-step models, please refer to the webpage (https://yefeng0920.github.io/BiasRobustMA_tutorial/). The bottom graph provides a visual solution to
Figure 6 The implementation of the proposed two-step meta-analytic modelling approach, along with a novel visualization, using open-source software. The metafor 56 package is utilized for building twostep models, while the orchaRd 54 package is employed for graphical representation. For a detailed step-by-step tutorial demonstrating two alternative first-step models, please refer to the webpage (https://yefeng0920.github.io/BiasRobustMA_tutorial/). The bottom graph provides a visual solution toCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 22

  1. Nakagawa, S., Yang, Y., Macartney, E. L., Spake, R. & Lagisz, M. Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences. Environmental Evidence 12, 8, doi:10.1186/s13750- 023-00301-6 (2023). · DOI
  2. Pigott, T. D. & Polanin, J. R. Methodological guidance paper: High-quality meta- analysis in a systematic review. Review of Educational Research 90, 24-46 (2020).
  3. Cheung, M. W.-L. A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychology review 29, 387-396 (2019).
  4. Hansen, C., Steinmetz, H. & Block, J. 1-19 (Springer, 2021).
  5. Stanley, T. D. et al. Meta-analysis of economics research reporting guidelines. Journal of economic surveys 27, 390-394 (2013).
  6. Andrews, I. & Kasy, M. Identification of and correction for publication bias. American Economic Review 109, 2766-2794 (2019).
  7. Borenstein, M., Hedges, L. V., Higgins, J. P. & Rothstein, H. R. Introduction to meta- analysis. (John Wiley & Sons, 2021).
  8. Cheung, M. W.-L. Multivariate meta-analysis as structural equation models. Structural Equation Modeling: A Multidisciplinary Journal 20, 429-454 (2013).
  9. Pustejovsky, J. E. & Tipton, E. Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics 36, 672-683 (2018).
  10. Sidik, K. & Jonkman, J. N. A note on variance estimation in random effects meta- regression. Journal of biopharmaceutical statistics 15, 823-838 (2005).
  11. Rosenthal, R. The file drawer problem and tolerance for null results. Psychological bulletin 86, 638-641 (1979).
  12. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T. & Jennions, M. D. The extent and consequences of p-hacking in science. PLoS biology 13, e1002106 (2015).
  13. Sterne, J. A., Gavaghan, D. & Egger, M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of clinical epidemiology 53, 1119-1129 (2000).
  14. Bramley, P., López-López, J. A. & Higgins, J. P. T. Examining how meta-analytic methods perform in the presence of bias: A simulation study. Research Synthesis Methods 12, 816-830, doi:https://doi.org/10.1002/jrsm.1516 (2021). · DOI
  15. Morrissey, M. B. Meta-analysis of magnitudes, differences and variation in evolutionary parameters. Journal of Evolutionary Biology 29, 1882-1904 (2016).
  16. Viechtbauer, W. Conducting meta-analyses in R with the metafor package. Journal of statistical software 36, 1-48 (2010).
  17. Bird, G., Kaczvinsky, C., Wilson, A. E. & Hardy, N. B. When do herbivorous insects compete? A phylogenetic meta-analysis. Ecology Letters 22, 875-883 (2019).
  18. Nakagawa, S. et al. A robust and readily implementable method for the meta- analysis of response ratios with and without missing standard deviations. Ecology Letters 26, 232-244 (2023).
  19. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G. & Group, P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS med 6, e1000097 (2009).
  20. Searle, S. R. Another Look at Henderson's Methods of Estimating Variance Components. Biometrics 24, 749-787, doi:10.2307/2528870 (1968). · DOI
  21. Hartung, J. An alternative method for meta-analysis. Biometrical Journal: Journal of Mathematical Methods in Biosciences 41, 901-916 (1999).
  22. Rosenfeld, A. H. The particle data group: Growth and operations-eighteen years of particle physics. Annual Review of Nuclear Science 25, 555-598 (1975).

Cited by

Sorted by citations

See all 2

Metadata

Authors
Yang, Y., Lagisz, M., Williams, C., Pan, J., Noble, D., & Nakagawa, S.
Publisher
California Digital Library (CDL)
Published
2023-10-03
DOI
10.32942/x20g6q
Cited by
2
References
14
Type
Preprint

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review