Methods for dealing with multiple outcomes in meta-analysis:a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis
The DOI matches its Crossref record; 41 of the 44 references it cites were found in a registry or an index.
Abstract
This study investigates three methods to handle dependency among effect size estimates in meta-analysis arising from studies reporting multiple outcome measures taken on the same sample. The three-level approach is compared with the method of robust variance estimation, and with averaging effects within studies. A simulation study is performed, and the fixed and random effect estimates of the three methods are compared with each other. Both the robust variance estimation and three-level approach result in unbiased estimates of the fixed effects, corresponding standard errors and variances. Averaging effect sizes results in overestimated standard errors when the effect sizes within studies are truly independent. Although the robust variance and three-level approach are more complicated to use, they have the advantage that they do not require an estimate of the correlation between outcomes, and they still result in unbiased parameter estimates.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Under such situations, RVE results in unbiased estimates of fixed effects when dependent effect sizes are included in the analysis (Moeyaert et al. 2017).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Moeyaert et al. 2017
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,784
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Research works printed but not linked: 4
- 10.1002/jrsm.v1:1 · doi:10.1002/jrsm.v1:1
- 10.1002/jrsm.v2.1 · doi:10.1002/jrsm.v2.1
- SAS Institute Inc, 2011, SAS® 9.3 Software
- 10.1002/jrsm.v5.1 · doi:10.1002/jrsm.v5.1
Cited by
Emotional intelligence predicts academic performance: a meta-analysis
MacCann, C., Jiang, Y., Brown, L. E. R., Double, K. S., Bucich, M., & Minbashian, A. (2020) · Psychological Bulletin · doi:10.1037/bul0000219 · Cited by 1,121
A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes
Cheung, M. W.-L. (2019) · Neuropsychology Review · doi:10.1007/s11065-019-09415-6 · Cited by 610
“Similar findings were also made in Moeyaert et al. (2017).”
BackgroundInfluentialMethodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
“Meta-analysis of single-case studies (Moeyaert et al., 2017) and meta-analysis of structural equation models (Cheung, 2015) also account for the dependent nature of the meta-analytic data using multilevel”
MethodThe Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies
Double, K. S., McGrane, J. A., & Hopfenbeck, T. N. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09510-3 · Cited by 446
Differentiated Instruction in Secondary Education: A Systematic Review of Research Evidence
Smale-Jacobse, A. E., Meijer, A., Helms-Lorenz, M., & Maulana, R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02366 · Cited by 404
“When more quantitative studies are located, this enables more statistical possibilities that can be used to gain insight in differential effects and predictive characteristics of different student outcomes (Lou et al., 1996; Moeyaert et al., 2016; Deunk et al., 2018).”
Background
Metadata
- Authors
- Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W.
- Affiliations
- State University of New York (US)
- KU Leuven (BE)
- Department of Educational Psychology, University of Texas at Austin, Austin, Texas, USA
- University of South Florida (US)
- The University of Texas at Austin (US)
- Department of Psychology and Educational Sciences, University of Leuven, Leuven, Belgium
- Corresponding author
- Moeyaert, Mariola
- Venue
- International Journal of Social Research Methodology
- Publisher
- Informa UK Limited
- Published
- 2016-11-13
- Volume
- 20
- Issue
- 6
- Pages
- 559-572
- ISSN
- 1364-5579 (ISSN-L)
- 1464-5300
- OpenAlex
- W2554874129
- Semantic Scholar
- 2ec7a1fcb1362c2cf480c05f0e8c25dcb017144a
- Cited by
- 274
- References
- 44
- Citations by year
- 2026: 30 · 2025: 33 · 2024: 27 · 2023: 36 · 2022: 40 · 2021: 43 · 2020: 26 · 2019: 26 · 2018: 10 · 2017: 3
- FWCI
- 12.25
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Social Research Methodology (publishedVersion)
- Lirias (KU Leuven) (submittedVersion)
- Funders
- U.S. Department of Education (R305D150007)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- multiple outcomes, meta-analysis, robust variance estimation, multilevel meta-analysis, three-level model, standard error estimation, fixed and random effects, estimation bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Methods for dealing with multiple outcomes in meta-analysis:a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. https://app.cochat.ai/papers/10.1080/13645579.2016.1252189
- The paper
- Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017). Methods for dealing with multiple outcomes in meta-analysis:a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology. https://doi.org/10.1080/13645579.2016.1252189
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.