Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 508

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses

Open access
Kontopantelis, Evangelos, Springate, David A., Reeves, David·2013·PLoS ONE
DOI 10.1371/journal.pone.0069930
Open this paper in CoChat — free account

Abstract

BACKGROUND: Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-analysis methods; c) apply the findings to published meta-analyses. METHODS AND FINDINGS: We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17-20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%. CONCLUSIONS: When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0069930

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 18

  1. SE Brockwell, 2001, A comparison of statistical methods for meta-analysis, StatMed, 20, 825
  2. E Kontopantelis, 2012, Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study, Statistical Methods in Medical Research, 21, 409, 10.1177/0962280210392008 · DOI
  3. BJ Biggerstaff, 1997, Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis, Statistics in Medicine, 16, 753, 10.1002/(SICI)1097-0258(19970415)16:7<753::AID-SIM494>3.0.CO;2-G · DOI
  4. DA Follmann, 1999, Valid inference in random effects meta-analysis, Biometrics, 55, 732, 10.1111/j.0006-341X.1999.00732.x · DOI
  5. RJ Hardy, 1996, A likelihood approach to meta-analysis with random effects, StatMed, 15, 619
  6. D Jackson, 2010, How does the DerSimonian and Laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counterparts?, Journal of Statistical Planning and Inference, 140, 961, 10.1016/j.jspi.2009.09.017 · DOI · Paper page
  7. S Greenland, 1985, Estimation of a Common Effect Parameter from Sparse Follow-up Data, Biometrics, 41, 55, 10.2307/2530643 · DOI
  8. LV Hedges, 1983, A random effects model for effect sizes, Psychological Bulletin, 93, 388, 10.1037/0033-2909.93.2.388 · DOI
  9. K Sidik, 2005, Simple heterogeneity variance estimation for meta-analysis, Journal of the Royal Statistical Society Series C-Applied Statistics, 54, 367, 10.1111/j.1467-9876.2005.00489.x · DOI
  10. Rukhin AL (2012) Estimating heterogeneity variance in meta-analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology). · DOI
  11. WG Cochran, 1937, Problems arising in the analysis of a series of similar experiments, Supplement to the Journal of the Royal Statistical Society, 4, 102, 10.2307/2984123 · DOI · Paper page
  12. RJ Hardy, 1998, Detecting and describing heterogeneity in meta-analysis, Statistics in Medicine, 17, 841, 10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D · DOI · Paper page
  13. M Mittlbock, 2006, A simulation study comparing properties of heterogeneity measures in meta-analyses, StatMed, 25, 4321
  14. The Nordic Cochrane Centre (2011) Review Manager (RevMan). 5.1 ed. Copenhagen: The Cochrane Collaboration.
  15. LV Hedges, 1983, A Random Effects Model for Effect Sizes, Psychological Bulletin, 93, 388, 10.1037/0033-2909.93.2.388 · DOI
  16. Wiley (2012) The Cochrane Library.
  17. Deeks JJ, Higgins JP (2010) Statistical algorithms in Review Manager 5. Statistical Methods Group of The Cochrane Collaboration. 1–11.
  18. E Kontopantelis, 2010, metaan: Random-effects meta-analysis, Stata Journal, 10, 395, 10.1177/1536867X1001000307 · DOI

Cited by

Metadata

Authors
Kontopantelis, E., Springate, D. A., & Reeves, D.
Venue
PLoS ONE
Publisher
Public Library of Science (PLoS)
Published
2013-07-26
Volume
8
Issue
7
Pages
e69930
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0069930
OpenAlex
W2141794277
Cited by
508
References
13
FWCI
38.36
Type
journal article
Language
en
Open access
gold
Licence
creativecommons.org/licenses/by/4.0/ (unspecified)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.