Alternative Measures of Between-Study Heterogeneity in Meta-Analysis: Reducing the Impact of Outlying Studies
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Summary: Meta-analysis has become a widely used tool to combine results from independent studies. The collected studies are homogeneous if they share a common underlying true effect size; otherwise, they are heterogeneous. A fixed-effect model is customarily used when the studies are deemed homogeneous, while a random-effects model is used for heterogeneous studies. Assessing heterogeneity in meta-analysis is critical for model selection and decision making. Ideally, if heterogeneity is present, it should permeate the entire collection of studies, instead of being limited to a small number of outlying studies. Outliers can have great impact on conventional measures of heterogeneity and the conclusions of a meta-analysis. However, no widely accepted guidelines exist for handling outliers. This article proposes several new heterogeneity measures. In the presence of outliers, the proposed measures are less affected than the conventional ones. The performance of the proposed and conventional heterogeneity measures are compared theoretically, by studying their asymptotic properties, and empirically, using simulations and case studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · cited 1×
“A fixed-effect model is customarily used for studies deemed to be homogeneous, while a random-effects model is used for heterogeneous studies (Borenstein et al., 2010; Riley et al., 2011).”
MethodThe combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · cited 1×
“The classical statistic for testing between-study heterogeneity is Cochran’s χ2 test (Cochran, 1954), also known as the Q test (Whitehead and Whitehead, 1991).”
MethodFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Outlier and influence diagnostics for meta-analysis
Viechtbauer, W., & Cheung, M. W.-L. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.11 · Cited by 2,949
Interpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884
Controlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · cited 1×
“When study-level covariates are collected in meta-analysis, meta-regression is widely applied to investigate whether study characteristics explain heterogeneity (Higgins and Thompson, 2004).”
MethodUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254 · cited 1×
“Also, the perception of heterogeneity or homogeneity helps clinicians make important decisions, such as whether the collected studies are similar enough to integrate their results and whether a treatment is applicable to all patients (Ioannidis et al., 2007).”
BackgroundCommentary: Heterogeneity in meta-analysis should be expected and appropriately quantified
Higgins, J. P. T. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn204 · Cited by 995 · cited 1×
“Some researchers believe that heterogeneity is to be expected in any meta-analysis because the collected studies were performed by different teams in different places using different methods (Higgins, 2008).”
MethodSmall study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Tschannen, B., Altman, D. G., Egger, M., & Juni, P. (2010) · BMJ · doi:10.1136/bmj.c3515 · Cited by 559 · cited 2×
“In some situations, effect sizes in small studies may be more informative than large studies due to “small study effects” (Nüesch et al., 2010); if their true within-study variances σ2i are seriously underestimated, then small studies could be outlying.”
BackgroundResultDavey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-160 · Cited by 404 · cited 1×
“However, many meta-analyses may only collect a few studies (Davey et al., 2011); these three measures need to be used with caution for small meta-analyses.”
Background
Cited by
Shi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818
Self-citationEmpirical Comparison of Publication Bias Tests in Meta-Analysis
Lin, L., Chu, H., Murad, M. H., Hong, C., Qu, Z., Cole, S. R., & Chen, Y. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4425-7 · Cited by 464
“For example, some study results could be extreme because of errors in the process of recording, analyzing, or reporting data.(33) Also, if a review did not strictly follow pre-specified inclusion and exclusion criteria, some studies may be improperly included showing extreme results (compared to other studies with pro…”
ResultSelf-citationBias caused by sampling error in meta-analysis with small sample sizes
Lin, L. (2018) · PLOS ONE · doi:10.1371/journal.pone.0204056 · Cited by 335
Self-citationObtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts
Wiernik, B. M., & Dahlke, J. A. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885611 · Cited by 135
The Relationship Between SNS Usage and Disordered Eating Behaviors: A Meta-Analysis
Zhang, J., Wang, Y., Li, Q., & Wu, C. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.641919 · Cited by 65
Burgos‐Julián, F. A., Ruiz‐Íñiguez, R., Peña‐Ibáñez, F., Montero, A. C., & Santed‐Germán, M. A. (2022) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2717 · Cited by 10
Metadata
- Authors
- Lin, L., Chu, H., & Hodges, J. S.
- Venue
- Biometrics
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-05-11
- Volume
- 73
- Issue
- 1
- Pages
- 156-166
- ISSN
- 0006-341X (ISSN-L)
- 1541-0420
- OpenAlex
- W2364933143
- Cited by
- 139
- References
- 12
- FWCI
- 5.19
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.