A comparison of procedures to test for moderators in mixed-effects meta-regression models
Open accessAbstract
Several alternative methods are available when testing for moderators in mixed-effects meta-regression models. A simulation study was carried out to compare different methods in terms of their Type I error and statistical power rates. We included the standard (Wald-type) test, the method proposed by Knapp and Hartung (2003) in 2 different versions, the Huber-White method, the likelihood ratio test, and the permutation test in the simulation study. These methods were combined with 7 estimators for the amount of residual heterogeneity in the effect sizes. Our results show that the standard method, applied in most meta-analyses up to date, does not control the Type I error rate adequately, sometimes leading to overly conservative, but usually to inflated, Type I error rates. Of the different methods evaluated, only the Knapp and Hartung method and the permutation test provide adequate control of the Type I error rate across all conditions. Due to its computational simplicity, the Knapp and Hartung method is recommended as a suitable option for most meta-analyses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To assess robustness, the authors measured and controlled for outliers, evaluated potential statistical dependence between sampled discrimination ratios, and recalculated p-values of meta-regression coefficients via a permutation test.
“To further assess the robustness of our results, we (i) measured and controlled for outliers (i.e.influential cases that substantially affect the pooled discrimination ratio), (ii) evaluated potential statistical dependence between the sampled discrimination ratios, and (iii) recalculated p-values of the meta-regression coefficients via a permutation test (Borenstein et al., 2009;Higgins et al., 2”
From The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments (2023) · cites this paper as Viechtbauer et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977 · see citing works
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667 · see citing works
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374 · see citing works
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · see citing works
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 2× · see citing works
“Hunter and Schmidt (2004) proposed an estimator of 2 in the random-effects model, which, in essence, is given by ̂HS 2 wi(yi ̂)2 wi wivi wi wi(yi ̂)2 k wi , (6) where ̂ wiyi ⁄ wi and wi 1 ⁄ vi (Viechtbauer, 2005).”
BackgroundMethodSelf-citationParametric empirical Bayes inference: theory and applications
Morris, C. N. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477920 · Cited by 1,448 · cited 5× · see citing works
“This estimator can be derived based on EB methods (Morris, 1983) and will therefore be denoted by u0007̂EB 2 .”
MethodInfluentialControlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · cited 4× · see citing works
“Knapp and Hartung (2003) suggested an alternative method for testing the model coefficients that accounts for the imprecision in the estimated amount of residual heterogeneity and that yields rejection rates closer to the nominal significance level. Similar results were also found by Sidik and Jonkman (2005). Another…”
BackgroundMethodInfluentialThe Effects of School-Based Writing-to-Learn Interventions on Academic Achievement: A Meta-Analysis
Bangert-Drowns, R. L., Hurley, M. M., & Wilkinson, B. (2004) · Review of Educational Research · doi:10.3102/00346543074001029 · Cited by 802 · cited 1× · see citing works
“Data originally from Bangert-Drowns et al. (2004).”
Flagged first, then by citations
Research works printed but not linked: 27
- Andrews, D. W. K. (1991). Heteroskedasticity and autocorrelation consis- tent covariance matrix estimation. Econometrica, 59, 817-858. doi: 10.2307/2938229 · DOI
- Berkey, C. S., Hoaglin, D. C., Mosteller, F., & Colditz, G. A. (1995). A random-effects regression model for meta-analysis. Statistics in Medi- cine, 14, 395-411. doi:10.1002/sim.4780140406 · DOI
- Christensen, R. (1996). Plane answers to complex questions: The theory of linear models (2nd ed.). New York, NY: Springer. doi:10.1007/978-1- 4757-2477-6 · DOI
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
- DerSimonian, R., & Laird, N. (1986). Meta-analysis of clinical trials. Clinical Controlled Trials, 7, 177-188. doi:10.1016/0197- 2456(86)90046-2 · DOI · Paper page
- Follmann, D. A., & Proschan, M. A. (1999). Valid inference in random effects meta-analysis. Biometrics, 55, 732-737. doi:10.1111/j.0006- 341X.1999.00732.x · DOI
- Friedrich, T., & Knapp, G. (2013). Generalised interval estimation in the random effects meta regression model. Computational Statistics & Data Analysis, 64, 165-179. doi:10.1016/j.csda.2013.03.011 · DOI
- Guolo, A. (2012). Higher-order likelihood inference in meta-analysis and meta-regression. Statistics in Medicine, 31, 313-327. doi:10.1002/sim .4451 · DOI
- Hedges, L. V. (1983). A random effects model for effect sizes. Psycho- logical Bulletin, 93, 388 -395. doi:10.1037/0033-2909.93.2.388 · DOI
- Hedges, L. V. (2009). Statistical considerations. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 37-47). New York, NY: Russell Sage Foundation.
- Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
- Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Re- search Synthesis Methods, 1, 39 -65. doi:10.1002/jrsm. · DOI · Paper page
- Hogg, R. V., & Craig, A. T. (1995). Introduction to mathematical statistics (5th ed.). London, England: Prentice-Hall.
- Huber, P. (1967). The behavior of maximum-likelihood estimates under nonstandard conditions. In L. M. LeCam & J. Neyman (Eds.), Proceed- ings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (pp. 221-233). Berkeley: University of California Press.
- Huizenga, H. M., Visser, I., & Dolan, C. V. (2011). Testing overall and moderator effects in random effects meta-regression. British Journal of Mathematical and Statistical Psychology, 64, 1-19. doi:10.1348/ 000711010X522687 · DOI
- Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Cor- recting error and bias in research synthesis (2nd ed.). Thousand Oaks, CA: Sage.
- Jennrich, R. I., & Sampson, P. F. (1976). Newton-Raphson and related algorithms for maximum likelihood variance component estimation. Technometrics, 18, 11-17. doi:10.2307/1267911 · DOI
- López-López, J. A., Marín-Martínez, F., Sánchez-Meca, J., Van den Noort- gate, W., & Viechtbauer, W. (2014). Estimation of the predictive power of the model in mixed-effects meta-regression models: A simulation study. British Journal of Mathematical and Statistical Psychology, 67, 30 -48. doi:10.1111/bmsp.12002 · DOI
- Paule, R. C., & Mandel, J. (1982). Consensus values and weighting factors. Journal of Research of the National Bureau of Standards, 87, 377-385. doi:10.6028/jres.087.022 · DOI
- Raudenbush, S. W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York, NY: Russell Sage Foundation.
- Raudenbush, S. W. (2009). Analyzing effect sizes: Random-effects mod- els. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 295-315). New York, NY: Russell Sage Foundation.
- Sánchez-Meca, J., & Marín-Martínez, F. (1998). Testing continuous mod- erators in meta-analysis: A comparison of procedures. British Journal of Mathematical and Statistical Psychology, 51, 311-326. doi:10.1111/j .2044-8317.1998.tb00683.x · DOI
- Sánchez-Meca, J., & Marín-Martínez, F. (2008). Confidence intervals for the overall effect size in random-effects meta-analysis. Psychological Methods, 13, 31-48. doi:10.1037/1082-989X.13.1.31 · DOI
- Sidik, K., & Jonkman, J. N. (2005). A note on variance estimation in random effects meta-regression. Journal of Biopharmaceutical Statis- tics, 15, 823-838. doi:10.1081/BIP-200067915 · DOI
- Thompson, S. G., & Higgins, J. P. T. (2002). How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21, 1559 -1573. doi:10.1002/sim.1187 · DOI · Paper page
- Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1-48.
- White, H. (1980). A heteroskedasticity-consistent covariance matrix and a direct test for heteroscedasticity. Econometrica, 48, 817-838. doi: 10.2307/1912934 · DOI · Paper page
Cited by
Sorted by citations
Small studies are more heterogeneous than large ones: a meta-meta-analysis
IntHout, J., Ioannidis, J. P. A., Borm, G. F., & Goeman, J. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.03.017 · Cited by 314 · see citing works
Acoustic indices as proxies for biodiversity: a meta‐analysis
Alcocer, I., Lima, H., Sugai, L. S. M., & Llusia, D. (2022) · Biological Reviews · doi:10.1111/brv.12890 · Cited by 273 · see citing works
Veroniki, A. A., Jackson, D., Bender, R., Kuss, O., Langan, D., Higgins, J. P. T., Knapp, G., & Salanti, G. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1319 · Cited by 236 · see citing works
Grant, S., Colaiaco, B., Motala, A., Shanman, R., Booth, M., Sorbero, M., & Hempel, S. (2017) · Journal of Addiction Medicine · doi:10.1097/adm.0000000000000338 · Cited by 220 · see citing works
Giusti, E. M., Lacerenza, M., Manzoni, G. M., & Castelnuovo, G. (2021) · Pain · doi:10.1097/j.pain.0000000000001999 · Cited by 188 · see citing works
Work Experiences and Self-Esteem Development: A Meta-Analysis of Longitudinal Studies
Krauss, S., & Orth, U. (2022) · European Journal of Personality · doi:10.1177/08902070211027142 · Cited by 92 · see citing works
Adverse Childhood Experiences and Body Dysmorphic Symptoms: A Meta-Analysis
Longobardi, C., Badenes-Ribera, L., & Fabris, M. A. (2022) · Body Image · doi:10.1016/j.bodyim.2022.01.003 · Cited by 86 · see citing works
Bayesian estimation in random effects meta‐analysis using a non‐informative prior
Bodnar, O., Link, A., Arendacká, B., Possolo, A., & Elster, C. (2017) · Statistics in Medicine · doi:10.1002/sim.7156 · Cited by 73 · see citing works
Drivers of Consumer Adoption of e-Commerce: a Meta-Analysis
Zerbini, C., Bijmolt, T. H. A., Maestripieri, S., & Luceri, B. (2022) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2022.04.003 · Cited by 68 · see citing works
Kip, A., Schoppe, L., Arntz, A., & Morina, N. (2023) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2023.102772 · Cited by 66 · see citing works
Rubio‐Aparicio, M., Sánchez‐Meca, J., López‐López, J. A., Botella, J., & Marín‐Martínez, F. (2017) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12092 · Cited by 56 · see citing works
Self-citationOrtiz-Ruiz, A. J., Pérez-Guzmán, N., Rubio-Aparicio, M., & Sánchez-Meca, J. (2020) · Scientific Reports · doi:10.1038/s41598-020-63497-4 · Cited by 29 · see citing works
Self-citationA game-factors approach to cognitive benefits from video-game training: A meta-analysis
Smith, E. T., & Basak, C. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285925 · Cited by 28 · see citing works
Rodriguez, J. E., Williams, D. R., & Bürkner, P. (2023) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12299 · Cited by 14 · see citing works
Standardized Effect Sizes and Image-Based Meta-Analytical Approaches for fMRI Data
Bossier, H., Nichols, T. E., & Moerkerke, B. (2019) · doi:10.1101/865881 · Cited by 14 · see citing works
Danilewitz, M., & Bahji, A. (2021) · Canadian Journal of Addiction · doi:10.1097/cxa.0000000000000128 · Cited by 10 · see citing works
Bahji, A., Danilewitz, M., Vazquez, G., & Patten, S. (2021) · Canadian Journal of Addiction · doi:10.1097/cxa.0000000000000123 · Cited by 6 · see citing works
Lippens, L., Vermeiren, S., & Baert, S. (2023) · European Economic Review · doi:10.1016/j.euroecorev.2022.104315
- Authors
- Viechtbauer, W., López-López, J. A., Sánchez-Meca, J., & Marín-Martínez, F.
- Affiliations
- Maastricht University (NL)
- Corresponding author
- Viechtbauer, Wolfgang
- Venue
- Psychological Methods
- Published
- 2015
- Volume
- 20
- Issue
- 3
- Pages
- 360-374
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2325974941
- Cited by
- 241
- References
- 10
- Citations by year
- 2026: 19 · 2025: 27 · 2024: 25 · 2023: 30 · 2022: 26 · 2021: 34 · 2020: 21 · 2019: 16 · 2018: 18 · 2017: 12 · 2016: 6 · 2015: 4 · 2014: 1 · 2013: 1
- FWCI
- 6.99
- Type
- journal article
- Language
- en
- Open access
- Licence
- other-oa
- Locations
- Psychological Methods (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Lirias (submittedVersion)
- Funders
- Spanish government (PSI2012-31399)
- Fundación Séneca
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- type I error rate, statistical power, permutation tests, likelihood ratio test, Wald test, residual heterogeneity, effect size heterogeneity
- MeSH terms
- Data Interpretation, Statistical, Humans, Regression Analysis, Meta-Analysis as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
