CoChat
Sign Up
Cited by 399

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.

Methods for testing publication bias in ecological and evolutionary meta‐analyses

Nakagawa, Shinichi, Lagisz, Malgorzata, Jennions, Michael D., Koricheva, Julia, Noble, Daniel W. A., Parker, Timothy H., Sánchez‐Tójar, Alfredo, Yang, Yefeng, O'Dea, Rose E.·2022·Methods in Ecology and Evolution
DOI 10.1111/2041-210x.13724
Open this paper in CoChat — free account

Abstract

Publication bias threatens the validity of quantitative evidence from meta‐analyses as it results in some findings being overrepresented in meta‐analytic datasets because they are published more frequently or sooner (e.g. ‘positive’ results). Unfortunately, methods to test for the presence of publication bias, or assess its impact on meta‐analytic results, are unsuitable for datasets with high heterogeneity and non‐independence, as is common in ecology and evolutionary biology. We first review both classic and emerging publication bias tests (e.g. funnel plots, Egger's regression, cumulative meta‐analysis, fail‐safe N , trim‐and‐fill tests, p ‐curve and selection models), showing that some tests cannot handle heterogeneity, and, more importantly, none of the methods can deal with non‐independence. For each method, we estimate current usage in ecology and evolutionary biology, based on a representative sample of 102 meta‐analyses published in the last 10 years. Then, we propose a new method using multilevel meta‐regression, which can model both heterogeneity and non‐independence, by extending existing regression‐based methods (i.e. Egger's regression). We describe how our multilevel meta‐regression can test not only publication bias, but also time‐lag bias, and how it can be supplemented by residual funnel plots. Overall, we provide ecologists and evolutionary biologists with practical recommendations on which methods are appropriate to employ given independent and non‐independent effect sizes. No method is ideal, and more simulation studies are required to understand how Type 1 and Type 2 error rates are impacted by complex data structures. Still, the limitations of these methods do not justify ignoring publication bias in ecological and evolutionary meta‐analyses.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/2041-210x.13724

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Figures

FIGURE 1 A schematic showing the relationship among common terminology in the meta-­ analytic literature: standard error ( SE ), sampling variance, precision (the inverse of SE ) and weight (the inverse of variance). Note that the inverse of variance is the weight for a fixed-­ effect model (the weight for a random-­ effect model is the inverse of the sum of sampling variance and between-­ study variance). In the statistical literature, the inverse of variance is also referred to as precision. Importantly, 'standard error' ( SE ) can be referred to as 'standard deviation' ( SD ), which is not incorrect because standard error is 'standard deviation of a statistic'-­ not to be confounded with 'standard deviation of a sample'
FIGURE 1 A schematic showing the relationship among common terminology in the meta-­ analytic literature: standard error ( SE ), sampling variance, precision (the inverse of SE ) and weight (the inverse of variance). Note that the inverse of variance is the weight for a fixed-­ effect model (the weight for a random-­ effect model is the inverse of the sum of sampling variance and between-­ study variance). In the statistical literature, the inverse of variance is also referred to as precision. Importantly, 'standard error' ( SE ) can be referred to as 'standard deviation' ( SD ), which is not incorrect because standard error is 'standard deviation of a statistic'-­ not to be confounded with 'standard deviation of a sample'CC-BY-NC · From the PDF
Image 2
CC-BY-NC · From the PDF
FIGURE 3 Examples of funnel plots and a radial plot using the same dataset ( N effect size = N study = 100): (a) a funnel plot with sample size as a measure of uncertainty; (b) a funnel plot with precision (1/ SE ) as a measure of uncertainty, red dots representing 'expected' missing data under publication bias, and blue dots representing 'unexpected' missing data; (c) a counter enhanced funnel plot with SE as a measure of uncertainty; (d) a sunset plot showing statistical power of data using the overall effect estimate as a true effect (the black line indicates the overall effect); (e) a residual funnel plot from a meta-­ regression with one moderator; and (f) a radial plot showing the overall effect by its slope's steepness and heterogeneity with the degree of scattering of the data points (for more details, see the main text). We used the r packages metafor (panels a-­ c and e; Viechtbauer, 2010), metaviz (panel d; Kossmeier et al., 2020) and meta (panel f; Schwarzer et al., 2015) for visualizations
FIGURE 3 Examples of funnel plots and a radial plot using the same dataset ( N effect size = N study = 100): (a) a funnel plot with sample size as a measure of uncertainty; (b) a funnel plot with precision (1/ SE ) as a measure of uncertainty, red dots representing 'expected' missing data under publication bias, and blue dots representing 'unexpected' missing data; (c) a counter enhanced funnel plot with SE as a measure of uncertainty; (d) a sunset plot showing statistical power of data using the overall effect estimate as a true effect (the black line indicates the overall effect); (e) a residual funnel plot from a meta-­ regression with one moderator; and (f) a radial plot showing the overall effect by its slope's steepness and heterogeneity with the degree of scattering of the data points (for more details, see the main text). We used the r packages metafor (panels a-­ c and e; Viechtbauer, 2010), metaviz (panel d; Kossmeier et al., 2020) and meta (panel f; Schwarzer et al., 2015) for visualizationsCC-BY-NC · From the PDF
FIGURE 4 Examples of various plots (using the same dataset as Figure 3b minus 25 red datapoints; therefore, N effect size = 75): (a) a scatter plot with the height of the solid line representing the degree of funnel asymmetry (cf. the radial plot at Figure 3f); (b) a scatter plot with the steepness of the slope representing the degree of funnel asymmetry; (c) a forest plot showing results of cumulate meta-­ analyses, where only a portion of the dataset ( N effect size = 15) was used; (d) a bubble plot showing a 'decline effect' over time, where only a portion of the dataset ( N effect size = 15) was used; (e) a funnel plot with precision (1/ SE ) and with a trim-­ and-­ fill method filling missing data (red circles; using the R 0 estimator); and (f) the same as panel (e) but with SE as a measure of uncertainty. We used the r packages GGPLOT2 (panels a, b and d; Wickham, 2009) and metafor (panels e and f; Viechtbauer, 2010) for visualizations
FIGURE 4 Examples of various plots (using the same dataset as Figure 3b minus 25 red datapoints; therefore, N effect size = 75): (a) a scatter plot with the height of the solid line representing the degree of funnel asymmetry (cf. the radial plot at Figure 3f); (b) a scatter plot with the steepness of the slope representing the degree of funnel asymmetry; (c) a forest plot showing results of cumulate meta-­ analyses, where only a portion of the dataset ( N effect size = 15) was used; (d) a bubble plot showing a 'decline effect' over time, where only a portion of the dataset ( N effect size = 15) was used; (e) a funnel plot with precision (1/ SE ) and with a trim-­ and-­ fill method filling missing data (red circles; using the R 0 estimator); and (f) the same as panel (e) but with SE as a measure of uncertainty. We used the r packages GGPLOT2 (panels a, b and d; Wickham, 2009) and metafor (panels e and f; Viechtbauer, 2010) for visualizationsCC-BY-NC · From the PDF
FIGURE 5 Example plots for p -­ curves and selection models (using the same dataset as in Figure 4; N effect size = 75): (a) a line plot showing the distribution of statistically significant p values under three scenarios: (1) with the observed p values (blue solid line), (2) when there is no effect (red dotted line) and (3) when there is an effect (i.e. an observed overall effect as a true effect) with 33% statistical power (note that if a blue line increases at the α level of 0.05, this is a sign of p -­ hacking; for more details of this plot, see www.p-­ curve.com); (b) a plot showing four different weight functions that model, based on the data, the likelihood of effect sizes being selected for publication: (1) a half-­ normal function based on p values (black solid line), (2) the same function but based both on p values and precisions (black dotted line), (3) a logistic function based on p values (red solid line) and (4) the same function but based both on p values and precisions (red dotted line; these functions are based on Preston et al., 2004); and (c) a plot showing two different 'step' weight functions based on: (1) three cut-­ points ( α = 0.05, 0.1, 0.5) and (2) one cut-­ point ( α = 0.05; this model is sometimes referred to as a three-­ parameter selection model, PSM with the three parameters being an overall mean, the between-­ study variance and an index determining the likelihood of selection; e.g. Carter et al., 2019; Rodgers & Pustejovsky, 2021). We used the r packages dmetar (panel a; Harrer et al., 2021) and metafor (panels b and c; Viechtbauer, 2010) for visualizations
FIGURE 5 Example plots for p -­ curves and selection models (using the same dataset as in Figure 4; N effect size = 75): (a) a line plot showing the distribution of statistically significant p values under three scenarios: (1) with the observed p values (blue solid line), (2) when there is no effect (red dotted line) and (3) when there is an effect (i.e. an observed overall effect as a true effect) with 33% statistical power (note that if a blue line increases at the α level of 0.05, this is a sign of p -­ hacking; for more details of this plot, see www.p-­ curve.com); (b) a plot showing four different weight functions that model, based on the data, the likelihood of effect sizes being selected for publication: (1) a half-­ normal function based on p values (black solid line), (2) the same function but based both on p values and precisions (black dotted line), (3) a logistic function based on p values (red solid line) and (4) the same function but based both on p values and precisions (red dotted line; these functions are based on Preston et al., 2004); and (c) a plot showing two different 'step' weight functions based on: (1) three cut-­ points ( α = 0.05, 0.1, 0.5) and (2) one cut-­ point ( α = 0.05; this model is sometimes referred to as a three-­ parameter selection model, PSM with the three parameters being an overall mean, the between-­ study variance and an index determining the likelihood of selection; e.g. Carter et al., 2019; Rodgers & Pustejovsky, 2021). We used the r packages dmetar (panel a; Harrer et al., 2021) and metafor (panels b and c; Viechtbauer, 2010) for visualizationsCC-BY-NC · From the PDF
FIGURE 6 Examples of funnel plots from a dataset with lnRR ( N study = 70; N effect size = 271; panels a-­ d) and a different dataset with Zr ( N study = 48; N effect size = 104; panels e, f): (a) a funnel plot of raw data (the same colour indicating effect sizes from the same studies); (b) a funnel plot of marginal residuals with the fixed effects removed (as in Equation 17); (c) a funnel plot of conditional residuals with fixed effects and the between-­ study effect removed (as in Equation 18); and (d) a funnel plot of conditional residuals with all effects apart from sampling errors removed (as in Equation 19); (e) a scatterplot showing a meta-­ regression on SE (black line; the red line is the same line as in panel (f), scaled to the standard error shown on the x -­ axis of panel e). Note that an overall mean is set to be 0 in this simulated dataset along missing effect sizes imitating publication bias; and (f) a scatterplot showing a meta-­ regression on sampling variance (red line, an equivalent line as the one in panel e). The red lines, for both panel (e) and panel (f), intersect the effect size at the intercept because they are the same regression lines plotted on different x -­ axes. We used the r packages metafor (panels a-­ d; Viechtbauer, 2010) and ggplot2 (panels e, f; Wickham, 2009) for visualizations
FIGURE 6 Examples of funnel plots from a dataset with lnRR ( N study = 70; N effect size = 271; panels a-­ d) and a different dataset with Zr ( N study = 48; N effect size = 104; panels e, f): (a) a funnel plot of raw data (the same colour indicating effect sizes from the same studies); (b) a funnel plot of marginal residuals with the fixed effects removed (as in Equation 17); (c) a funnel plot of conditional residuals with fixed effects and the between-­ study effect removed (as in Equation 18); and (d) a funnel plot of conditional residuals with all effects apart from sampling errors removed (as in Equation 19); (e) a scatterplot showing a meta-­ regression on SE (black line; the red line is the same line as in panel (f), scaled to the standard error shown on the x -­ axis of panel e). Note that an overall mean is set to be 0 in this simulated dataset along missing effect sizes imitating publication bias; and (f) a scatterplot showing a meta-­ regression on sampling variance (red line, an equivalent line as the one in panel e). The red lines, for both panel (e) and panel (f), intersect the effect size at the intercept because they are the same regression lines plotted on different x -­ axes. We used the r packages metafor (panels a-­ d; Viechtbauer, 2010) and ggplot2 (panels e, f; Wickham, 2009) for visualizationsCC-BY-NC · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 58

  1. 10.1002/sim.8422 · DOI
  2. 10.1186/s12874‐020‐01137‐1 · DOI
  3. 10.1002/jrsm.1404 · DOI
  4. 10.1111/j.1600‐0706.2011.19401.x · DOI
  5. 10.1002/0470870168.ch7 · DOI
  6. 10.1002/jrsm.1441 · DOI
  7. 10.1037/met0000119 · DOI
  8. Cohen J., 1988, Statistical power analysis for the beahvioral sciences
  9. 10.1016/j.healthpol.2012.10.015 · DOI
  10. 10.1016/j.jclinepi.2005.01.016 · DOI · Paper page
  11. 10.1002/jrsm.1414 · DOI
  12. 10.1111/2041‐210X.12927 · DOI
  13. 10.1111/j.1467‐8543.2009.00723.x · DOI · Paper page
  14. 10.1136/bmj.315.7109.629 · DOI · Paper page
  15. 10.1080/00220973.2019.1582470 · DOI
  16. 10.1046/j.1365‐2656.2002.00618.x · DOI
  17. 10.1186/1471‐2288‐8‐32 · DOI
  18. 10.1002/sim.4780070807 · DOI
  19. 10.1111/2041‐210X.12758 · DOI
  20. 10.1111/j.1420‐9101.2009.01915.x · DOI
  21. 10.1016/j.jinteco.2011.07.004 · DOI
  22. Hedges L., 1985, Statistical methods for meta‐analysis
  23. 10.1002/jrsm.5 · DOI · Paper page
  24. 10.1007/s11121‐021‐01279‐8 · DOI
  25. 10.2307/3558999 · DOI
  26. 10.23943/princeton/9780691137285.003.0014 · DOI
  27. 10.1098/rspb.2001.1832 · DOI
  28. 10.1111/1365‐2745.12224 · DOI · Paper page
  29. 10.1515/9781400846184-017 · DOI
  30. 10.1016/j.tree.2019.05.006 · DOI
  31. 10.1027/2151‐2604/a000392 · DOI
  32. 10.1890/14‐2402.1 · DOI · Paper page
  33. 10.4159/9780674040243 · DOI
  34. 10.4310/SII.2020.v13.n4.a5 · DOI
  35. Marks‐Anglin A., 2021, Handbook of meta‐analysis, 283
  36. 10.1007/s10683‐021‐09716‐9 · DOI
  37. 10.1016/S0169‐5347(01)02235‐2 · DOI
  38. 10.1111/jeb.12945 · DOI
  39. 10.1002/ecy.3490 · DOI
  40. 10.1111/mec.14031 · DOI
  41. 10.1002/bimj.200610341 · DOI
  42. 10.1001/jama.295.6.676 · DOI · Paper page
  43. 10.1111/j.1365‐2753.2003.00457.x · DOI
  44. Roberts C. J., 2005, Meta‐regression analysis: Issues of publicaiton bias in economics
  45. 10.1002/sim.2971 · DOI
  46. 10.7554/eLife.37385 · DOI
  47. 10.1002/jrsm.1423 · DOI
  48. 10.1111/j.0950‐0804.2005.00250.x · DOI
  49. 10.4324/9780203111710 · DOI
  50. 10.2307/146404 · DOI
  51. 10.1016/j.jclinepi.2005.01.006 · DOI
  52. 10.1002/(SICI)1097‐0258(19991030)18:20<2693:AID‐SIM235>3.0.CO;2‐V · DOI
  53. 10.1002/0470870168.ch13 · DOI
  54. vanAert R. C. M. &vanAssen M. A. L. M.(2021).Correcting for publication bias in a meta‐analysis with the p‐uniform* Method.OSF preprint https://osf.io/ebq6m/
  55. 10.1007/BF02294384 · DOI
  56. 10.1002/jrsm.1042 · DOI
  57. 10.1007/978-0-387-98141-3 · DOI · Paper page
  58. 10.7554/eLife.24260 · DOI

Cited by

Sorted by citations

See all 399

Metadata

Authors
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E.
Venue
Methods in Ecology and Evolution
Publisher
Wiley
Published
2021-11-10
Volume
13
Issue
1
Pages
4-21
ISSN
2041-210X
DOI
10.1111/2041-210x.13724
Cited by
399
References
44
Language
en
Update notices
correction (2022-06-20)
Licence
Funders
  • Australian Research Council (DP200100367)
  • Deutsche Forschungsgemeinschaft (316099922, 396782608)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review