Hypothesis tests for population heterogeneity in meta‐analysis
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Choice of the appropriate model in meta‐analysis is often treated as an empirical question which is answered by examining the amount of variability in the effect sizes. When all of the observed variability in the effect sizes can be accounted for based on sampling error alone, a set of effect sizes is said to be homogeneous and a fixed‐effects model is typically adopted. Whether a set of effect sizes is homogeneous or not is usually tested with the so‐called Q test. In this paper, a variety of alternative homogeneity tests – the likelihood ratio, Wald and score tests – are compared with the Q test in terms of their Type I error rate and power for four different effect size measures. Monte Carlo simulations show that the Q test kept the tightest control of the Type I error rate, although the results emphasize the importance of large sample sizes within the set of studies. The results also suggest under what conditions the power of the tests can be considered adequate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 3×
“Now, ui ¼ ðmEi 2 mCi Þ=s i, which can be estimated unbiasedly by ESi ¼ cðmiÞðXEi ¼ XCi Þ=si, where cðmiÞ ¼ Gðmi=2Þðmi=2Þ1=2Gððmi 2 1Þ=2Þ and mi ¼ nEi þ nCi 2 2 (Hedges, 1981).”
MethodInfluentialThe combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Consistent Estimates Based on Partially Consistent Observations
Neyman, J., & Scott, E. L. (1948) · Econometrica · doi:10.2307/1914288 · Cited by 2,691
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667 · cited 4×
“Some of the earliest references to the Q test in the meta-analytic context can be found in Hedges (1982a, 1982b, 1983), Rosenthal and Rubin (1982), and Viana (1980).”
BackgroundMethodInfluentialSearle, S. R., Casella, G., & McCulloch, C. E. (1992) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316856 · Cited by 1,926
Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews
Rosenthal, R., & DiMatteo, M. R. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.59 · Cited by 1,780
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 7×
“However, since the sampling variability of the HE estimator generally exceeded that of the REML estimator and since the DL estimator was not always well defined asymptotically, Viechtbauer (2005) recommended the REML estimator for general use.”
BackgroundMethodInfluentialSelf-citationParametric empirical Bayes inference: theory and applications
Morris, C. N. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477920 · Cited by 1,448 · cited 1×
“…(e.g. Berkey, Hoaglin, Mosteller, & Colditz, 1995; Normand, 1999; Thompson & Sharp, 1999) is given by ŝ 2ðREMLÞ u < Pk i¼1w 2 1 kðk2 1Þ21 ESi 2 m̂ðMLÞu 2 2s 21iPk i¼1w 2 i : ð12Þ It appears that (12) originated with C. N. Morris (1983), who suggested it as an ‘approximate’ REML estimator (p. 53).”
Variance components testing in the longitudinal mixed effects model
Stram, D. O., & Lee, J. W. (1994) · Biometrics · doi:10.2307/2533455 · Cited by 992
Hunter, J. E., & Schmidt, F. L. (2000) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00156 · Cited by 833 · cited 2×
“Among the most common meta-analytic models adopted are the fixedeffects and the random-effects models (Hardy & Thompson, 1998; Hedges & Vevea, 1998; Hunter & Schmidt, 2000; Normand, 1999).”
BackgroundMethodThe power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716 · cited 1×
“Also, Hedges and Pigott (2001) derived equations for calculating the approximate power of the Q test.”
MethodElements of Large-Sample Theory
(1999) · Springer Texts in Statistics · doi:10.1007/b98855 · Cited by 677
A comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573 · cited 2×
“Typically, one starts out with an initial guess for ŝ 2ðMLÞ u and then iterates between (8) and (9) until convergence (Brockwell & Gordon, 2001; Erez, Bloom, & Wells, 1996; National Research Council, 1992).”
BackgroundResultEvaluation of old and new tests of heterogeneity in epidemiologic meta-analysis
Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009981 · Cited by 380 · cited 5×
“Takkouche et al. (1999) did not examine the REML-based LR test, but a similar finding was reported by Morrell (1998) when testing the significance of variance components in the context of repeated-measures designs.”
ResultInfluentialEREZ, A., BLOOM, M. C., & WELLS, M. T. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01801.x · Cited by 161
Detecting and describing heterogeneity in meta-analysis
Hardy, R. J., & Thompson, S. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8<841::aid-sim781>3.0.co;2-d · cited 2×
“…carried out to investigate the Type I error rateandpowerof theQ test (Alexander, Scozzaro,&Borodkin, 1989;Callender & Osburn, 1988; Field, 2001; Hardy & Thompson, 1998; Harwell, 1997; Hedges, 1982a, 1982b; Johnson, Mullen, & Salas, 1995; Koslowsky & Sagie, 1993; Morris, 2000; Rasmussen &…”
Method
Cited by
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
“A confidence interval for τ 2 can be obtained with the method described in Viechtbauer (2007a).”
MethodSelf-citationThe Effects of Vocabulary Intervention on Young Children’s Word Learning: A Meta-Analysis
Marulis, L. M., & Neuman, S. B. (2010) · Review of Educational Research · doi:10.3102/0034654310377087 · Cited by 655
Linear mixed-effects modeling approach to FMRI group analysis
Chen, G., Saad, Z. S., Britton, J. C., Pine, D. S., & Cox, R. W. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.01.047 · Cited by 447
Confidence intervals for the amount of heterogeneity in meta‐analysis
Viechtbauer, W. (2007) · Statistics in Medicine · doi:10.1002/sim.2514 · Cited by 394
Self-citationPublication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
Metadata
- Authors
- Viechtbauer, W.
- Venue
- British Journal of Mathematical and Statistical Psychology
- Publisher
- Wiley
- Published
- 2007-05
- Volume
- 60
- Issue
- 1
- Pages
- 29-60
- ISSN
- 0007-1102 (ISSN-L)
- 2044-8317
- OpenAlex
- W1996843362
- Cited by
- 135
- References
- 17
- FWCI
- 2.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.