Correction for bias in meta‐analysis of little‐replicated studies
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Meta‐analyses conventionally weight study estimates on the inverse of their error variance, in order to maximize precision. Unbiased variability in the estimates of these study‐level error variances increases with the inverse of study‐level replication. Here, we demonstrate how this variability accumulates asymmetrically across studies in precision‐weighted meta‐analysis, to cause undervaluation of the meta‐level effect size or its error variance (the meta‐effect and meta‐variance). Small samples, typical of the ecological literature, induce big sampling errors in variance estimation, which substantially bias precision‐weighted meta‐analysis. Simulations revealed that biases differed little between random‐ and fixed‐effects tests. Meta‐estimation of a one‐sample mean from 20 studies, with sample sizes of 3–20 observations, undervalued the meta‐variance by c. 20%. Meta‐analysis of two‐sample designs from 20 studies, with sample sizes of 3–10 observations, undervalued the meta‐variance by 15%–20% for the log response ratio (ln R); it undervalued the meta‐effect by c. 10% for the standardized mean difference (SMD). For all estimators, biases were eliminated or reduced by a simple adjustment to the weighting on study precision. The study‐specific component of error variance prone to sampling error and not parametrically attributable to study‐specific replication was replaced by its cross‐study mean, on the assumptions of random sampling from the same population variance for all studies, and sufficient studies for averaging. Weighting each study by the inverse of this mean‐adjusted error variance universally improved accuracy in estimation of both the meta‐effect and its significance, regardless of number of studies. For comparison, weighting only on sample size gave the same improvement in accuracy, but could not sensibly estimate significance. For the one‐sample mean and two‐sample ln R, adjusted weighting also improved estimation of between‐study variance by DerSimonian‐Laird and REML methods. For random‐effects meta‐analysis of SMD from little‐replicated studies, the most accurate meta‐estimates obtained from adjusted weights following conventionally weighted estimation of between‐study variance. We recommend adoption of weighting by inverse adjusted‐variance for meta‐analyses of well‐ and little‐replicated studies, because it improves accuracy and significance of meta‐estimates, and it can extend the scope of the meta‐analysis to include some studies without variance estimates.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 4×
“This weighting aims to minimize the variance in the meta- estimate of effect size, thereby maximizing its precision (Hedges, 1981).”
BackgroundMethodInfluentialPrimary forests are irreplaceable for sustaining tropical biodiversity
Gibson, L., Lee, T. M., Koh, L. P., Brook, B. W., Gardner, T. A., Barlow, J., Peres, C. A., Bradshaw, C. J. A., Laurance, W. F., Lovejoy, T. E., et al. (2011) · Nature · doi:10.1038/nature10425 · Cited by 2,147 · cited 4×
“The simulation was repeated for a meta- analysis of 65 studies with the same sample sizes as those of Gibson et al. (2011) for metaestimation of bird abundance and richness in active agriculture as a standardized mean difference from primary forest: {n1, n2}i = 20 × {50, 25}, 10 × {50, 4}, 7 × {12,…”
MethodInfluentialMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · cited 1×
“Random- effects meta- analysis measured the component of error variance due to variability between studies as the parameter value τ2, in order to avoid confounding error by an estimate T2 obtained from any of the many data- based methods (reviewed in Veroniki et al., 2016).”
MethodThe power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716 · cited 2×
“Weighting still risks overvaluing littlereplicated studies, however, leading to spurious leverage on the crossstudy estimate of effect size and loss of power in random- effects tests (Hedges & Pigott, 2001; Spake, Ezard, Martin, Newton, & Doncaster, 2015).”
BackgroundUses and misuses of meta‐analysis in plant ecology
Koricheva, J., & Gurevitch, J. (2014) · Journal of Ecology · doi:10.1111/1365-2745.12224 · Cited by 460
Davies, G. M., & Gray, A. (2015) · Ecology and Evolution · doi:10.1002/ece3.1782 · Cited by 333 · cited 1×
“For behavioural and ecological studies in particular, meta- analysis can provide a solution to problems of low replication and pseudoreplication, which afflict costly field studies with little scope for replication within heterogeneous landscapes (Davies & Gray, 2015; Hargrove & Pickering, 1992).”
BackgroundA survey of the statistical power of research in behavioral ecology and animal behavior
Jennions, M. D. (2003) · Behavioral Ecology · doi:10.1093/beheco/14.3.438 · Cited by 278 · cited 1×
“Field and laboratory research in behavioural ecology is often little replicated (Jennions & Møller, 2003), and meta- analyses in ecology and evolutionary biology routinely include studies with many- fold differences in replication (Figure 1).”
BackgroundBias and correction for the log response ratio in ecological meta‐analysis
Lajeunesse, M. J. (2015) · Ecology · doi:10.1890/14-2402.1 · Cited by 244 · cited 5×
“It was effective in dealing with known issues of small- sample bias in Hedges’ g (Hedges, 1982) and in lnR (Hedges et al., 1999; Lajeunesse, 2015).”
BackgroundMethodInfluential1. Place of Meta-analysis among Other Methods of Research Synthesis
(2013) · Handbook of Meta-analysis in Ecology and Evolution · doi:10.1515/9781400846184-003 · Cited by 121
Use of meta-analysis in forest biodiversity research: key challenges and considerations
Spake, R., & Doncaster, C. P. (2017) · Forest Ecology and Management · doi:10.1016/j.foreco.2017.05.059 · Cited by 60 · cited 1×
“We assume that the primary studies (hereafter ‘studies’) for a meta- analysis have been collated by systematic review and filtering to remove biases due to differences in treatment factors and levels, scales of sampling or response types (Spake & Doncaster, 2017).”
MethodSelf-citation
Cited by
Methods for testing publication bias in ecological and evolutionary meta‐analyses
Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez‐Tójar, A., Yang, Y., & O'Dea, R. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13724 · Cited by 399
Arcsine‐based transformations for meta‐analysis of proportions: Pros, cons, and alternatives
Lin, L., & Xu, C. (2020) · Health Science Reports · doi:10.1002/hsr2.178 · Cited by 339
Evaluation of various estimators for standardized mean difference in meta‐analysis
Lin, L., & Aloe, A. M. (2021) · Statistics in Medicine · doi:10.1002/sim.8781 · Cited by 297
Guan, L., Li, S., & Gu, M. M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100323 · Cited by 72
Lam, W. N., Slade, E. M., & Wardle, D. A. (2024) · Functional Ecology · doi:10.1111/1365-2435.14520 · Cited by 17
Metadata
- Authors
- Doncaster, C. P., & Spake, R.
- Venue
- Methods in Ecology and Evolution
- Publisher
- Wiley
- Published
- 2017-11-21
- Volume
- 9
- Issue
- 3
- Pages
- 634-644
- ISSN
- 2041-210X (ISSN-L)
- OpenAlex
- W2766281042
- Cited by
- 64
- References
- 10
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.