Analysis of categorical moderators in mixed‐effects meta‐analysis: Consequences of using pooled versus separate estimates of the residual between‐studies variances
Open accessAbstract
Subgroup analyses allow us to examine the influence of a categorical moderator on the effect size in meta‐analysis. We conducted a simulation study using a dichotomous moderator, and compared the impact of pooled versus separate estimates of the residual between‐studies variance on the statistical performance of the Q B (P) and Q B (S) tests for subgroup analyses assuming a mixed‐effects model. Our results suggested that similar performance can be expected as long as there are at least 20 studies and these are approximately balanced across categories. Conversely, when subgroups were unbalanced, the practical consequences of having heterogeneous residual between‐studies variances were more evident, with both tests leading to the wrong statistical conclusion more often than in the conditions with balanced subgroups. A pooled estimate should be preferred for most scenarios, unless the residual between‐studies variances are clearly different and there are enough studies in each category to obtain precise separate estimates.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Conducting Meta-Analyses in <i>R</i> with the <b>metafor</b> Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,832 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED
WELCH, B. L. (1947) · Biometrika · doi:10.1093/biomet/34.1-2.28 · Cited by 4,135 · see citing works
Applied Statistics for the Behavioral Sciences
Witz, K. G., Hinkle, D. E., Wiersma, W., & Jurs, S. G. (1990) · Journal of Educational Statistics · doi:10.2307/1164825 · Cited by 2,927 · see citing works
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · see citing works
Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972) · Review of Educational Research · doi:10.3102/00346543042003237 · Cited by 1,826 · cited 1× · see citing works
“Note that the performance of the QB(P) test was similar to that expected for the F test in a typical ANOVA with unbalanced sample sizes, when the homoscedasticity assumption was not met (Glass et al., 1972; Hinkle et al., 2003).”
ResultBias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 2× · see citing works
“In future studies, other estimators of the residual between-studies variance could be applied, such as the restricted maximum likelihood estimator (Viechtbauer, 2005), and they might also consider alternatives to the normal distribution to generate parametric effects, in order to mimic realistic…”
BackgroundEspinosa, J. C. (2004) · Cited by 939 · see citing works
Borenstein, M., & Higgins, J. P. T. (2013) · Prevention Science · doi:10.1007/s11121-013-0377-7 · Cited by 767 · see citing works
A Method for Simulating Non-Normal Distributions
Fleishman, A. I. (1978) · Psychometrika · doi:10.1007/bf02293811 · Cited by 724 · see citing works
Davey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-160 · Cited by 404 · see citing works
Psychological treatment of panic disorder with or without agoraphobia: A meta-analysis
Sánchez-Meca, J., Rosa-Alcázar, A. I., Marín-Martínez, F., & Gómez-Conesa, A. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.08.011 · Cited by 291 · see citing works
Self-citationStatistical Methods in Education and Psychology
Schmidt, W. H., Glass, G. V., Stanley, J. C., Heermann, E. F., Braskamp, L. A., Bishir, J. W., & Drewes, D. W. (1972) · American Educational Research Journal · doi:10.2307/1162060 · Cited by 271 · see citing works
A comparison of procedures to test for moderators in mixed-effects meta-regression models
Viechtbauer, W., López-López, J. A., Sánchez-Meca, J., & Marín-Martínez, F. (2015) · Psychological Methods · doi:10.1037/met0000023 · Cited by 241 · see citing works
Self-citationMeta‐Analysis of Clinical Trials
Thompson, S. G. (2005) · Encyclopedia of Biostatistics · doi:10.1002/0470011815.b2a01034 · Cited by 52 · see citing works
Rubio‐Aparicio, M., López-López, J. A., Sánchez‐Meca, J., Marín‐Martínez, F., Viechtbauer, W., & Noortgate, W. V. D. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1312 · Cited by 43 · see citing works
Self-citationStudent t‐tests for potentially abnormal data
Shuster, J. J. (2009) · Statistics in Medicine · doi:10.1002/sim.3581 · Cited by 29 · see citing works
Senn, S. J. (2008) · Significance · doi:10.1111/j.1740-9713.2008.00280.x · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Beard, J. I. L., & Delgadillo, J. (2019) · Depression and Anxiety · doi:10.1002/da.22931 · Cited by 81 · see citing works
Guidelines for reporting systematic reviews and meta-analyses
Rubio-Aparicio, M., Sánchez-Meca, J., Marín-Martínez, F., & López-López, J. A. (2018) · Anales de Psicología · doi:10.6018/analesps.34.2.320131 · Cited by 56 · see citing works
Self-citationNeo, H. S., Tan, J. H., Ang, W. H. D., & Lau, Y. (2022) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2022.110790 · Cited by 32 · see citing works
Ortiz-Ruiz, A. J., Pérez-Guzmán, N., Rubio-Aparicio, M., & Sánchez-Meca, J. (2020) · Scientific Reports · doi:10.1038/s41598-020-63497-4 · Cited by 29 · see citing works
Self-citationRodriguez, J. E., Williams, D. R., & Bürkner, P. (2023) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12299 · Cited by 14 · see citing works
The asymmetry of working memory training transfer: A systematic review and meta-analysis
Fu, J., Zhou, Y., Yuan, M., & Cai, Y. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-026-02994-5
- Authors
- Rubio‐Aparicio, M., Sánchez‐Meca, J., López‐López, J. A., Botella, J., & Marín‐Martínez, F.
- Affiliations
- Universidad de Murcia (ES)
- School of Social and Community Medicine University of Bristol UK
- Universidad Autónoma de Madrid (ES)
- Corresponding author
- Sánchez‐Meca, Julio
- Venue
- British Journal of Mathematical and Statistical Psychology
- Publisher
- Wiley
- Published
- 2017-02-06
- Volume
- 70
- Issue
- 3
- Pages
- 439-456
- ISSN
- 0007-1102 (ISSN-L)
- 2044-8317
- OpenAlex
- W2587747265
- Cited by
- 56
- References
- 21
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 3 · 2023: 4 · 2022: 7 · 2021: 7 · 2020: 11 · 2019: 6 · 2018: 4
- FWCI
- 5.14
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- British Journal of Mathematical and Statistical Psychology (publishedVersion)
- PubMed (publishedVersion)
- Explore Bristol Research (acceptedVersion, other-oa)
- Bristol Research (University of Bristol) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- subgroup analysis, heterogeneity, Q-test, statistical power, simulation study
- MeSH terms
- Network Meta-Analysis, Analysis of Variance, Computer Simulation, Data Interpretation, Statistical, Humans, Psychology, Meta-Analysis as Topic, Models, Statistical, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
