Methods for dealing with multiple outcomes in meta-analysis:a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis
The DOI matches its Crossref record; 41 of the 44 references it cites were found in a registry or an index.
Abstract
This study investigates three methods to handle dependency among effect size estimates in meta-analysis arising from studies reporting multiple outcome measures taken on the same sample. The three-level approach is compared with the method of robust variance estimation, and with averaging effects within studies. A simulation study is performed, and the fixed and random effect estimates of the three methods are compared with each other. Both the robust variance estimation and three-level approach result in unbiased estimates of the fixed effects, corresponding standard errors and variances. Averaging effect sizes results in overestimated standard errors when the effect sizes within studies are truly independent. Although the robust variance and three-level approach are more complicated to use, they have the advantage that they do not require an estimate of the correlation between outcomes, and they still result in unbiased parameter estimates.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Under such situations, RVE results in unbiased estimates of fixed effects when dependent effect sizes are included in the analysis (Moeyaert et al. 2017).”
From The Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies (2020) · cites this paper as Moeyaert et al. 2017
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,784
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Hox, J., Moerbeek, M., & van de Schoot, R. (2002) · doi:10.4324/9781410604118 · Cited by 2,419
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 12×
“…when distributions of effect sizes are skewed and incorrectly assumed normal the RVE procedure still provides accurate estimates of the parameters (Hedges et al., 2010) as coverage proportions of confidence intervals were found to be close to the nominal rate including scenarios that involved…”
BackgroundMethodInfluentialCheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342
Higgins (2002) · doi:10.1002/(issn)1097-0258 · Cited by 1,248
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130 · cited 6×
“Also when there are multiple outcomes in a study, a multilevel meta-analysis can be used (Cheung, 2014; Konstantopoulos, 2011; Van den Noortgate et al., 2013, 2015).”
BackgroundMethodEhri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001) · Review of Educational Research · doi:10.3102/00346543071003393 · Cited by 931
Small sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 2×
“Tipton also recently introduced small sample adjustments for RVE (Tipton, 2015).”
BackgroundRobustness Studies in Covariance Structure Modeling: An Overview and a Meta-Analysis
HOOGLAND, J. J., & BOOMSMA, A. (1998) · Sociological Methods & Research · doi:10.1177/0049124198026003003 · Cited by 734
Edmonds, M. S., Vaughn, S., Wexler, J., Reutebuch, C., Cable, A., Tackett, K. K., & Schnakenberg, J. W. (2009) · Review of Educational Research · doi:10.3102/0034654308325998 · Cited by 599 · cited 1×
“…from multiple testing occasions, multiple treatment groups, and multiple outcomes in their meta-analysis of 36 studies on Reading Recovery, while Edmonds et al. (2009) also reported 78 effect sizes calculated from multiple measures and multiple comparison groups in their meta-analysis of 21…”
BackgroundTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588
Fixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521
Stochastically dependent effect sizes.
Gleser, L. J., & Olkin, I. (1994) · Cited by 485
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450 · cited 2×
“Parameters values were chosen to be realistic for group-comparison experimental studies, and are in line with previous research (Van den Noortgate et al., 2013, 2015).”
MethodResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Cook-Cottone (2006) · doi:10.1002/(issn)1520-6807 · Cited by 313
robumeta: An R-package for robust variance estimation in meta-analysis
Fisher, Z., & Tipton, E. (2015) · arXiv · doi:10.48550/arxiv.1503.02220 · Cited by 280
Becker, B. J. (2000) · Handbook of Applied Multivariate Statistics and Mathematical Modeling · doi:10.1016/b978-012691360-6/50018-5 · Cited by 273
A multivariate mixed linear model for meta-analysis.
Kalaian, H. A., & Raudenbush, S. W. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.3.227 · Cited by 229 · cited 2×
“Van den Noortgate et al. (2013) noted a few advantages of the multilevel meta-analysis method over existing approaches such as the multivariate (Gleser & Olkin, 2009; Hedges & Olkin, 1985; Kalaian & Raudenbush, 1996; Raudenbush et al., 1988) and averaging methods (as described previously).”
MethodMultilevel Meta-analysis: a comparison with traditional meta-analytical procedures
Van Den Noortgate, W., & Onghena, P. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403251027 · Cited by 218 · cited 1×
“The concept of a multilevel meta-analysis to account for dependent studies (for instance due to the nesting of studies in countries or research groups) has already been discussed in several books and papers (Hox, 2002; Raudenbush & Bryk, 2002; Van den Noortgate & Onghena, 2003).”
BackgroundModeling multivariate effect sizes.
Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.1.111 · Cited by 195
A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses
Ahn, S., Ames, A. J., & Myers, N. D. (2012) · Review of Educational Research · doi:10.3102/0034654312458162 · Cited by 179
Evaluating single-case research data: A comparison of seven statistical methods
Parker, R. I., & Brossart, D. F. (2003) · Behavior Therapy · doi:10.1016/s0005-7894(03)80013-8 · Cited by 168
Research Synthesis and Meta-Analysis
Valentine, J. C., & Cooper, H. (2008) · doi:10.4324/9780203887882-10 · Cited by 168
Threats to the validity of research syntheses
Matt, G. E., & Cook, T. D. (1994) · Cited by 136
Statistical reform: Evidence‐based practice, meta‐analyses, and single subject designs
Jenson, W. R., Clark, E., Kircher, J. C., & Kristjansson, S. D. (2007) · Psychology in the Schools · doi:10.1002/pits.20240 · Cited by 130
Dependent Effect Sizes in Meta-Analysis: Incorporating the Degree of Interdependence.
Cheung, S. F., & Chan, D. K.-S. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.5.780 · Cited by 117
Correcting a Significance Test for Clustering
Hedges, L. V. (2007) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998606298040 · Cited by 115
A Meta-Analysis of Reading Recovery in United States Schools
D’Agostino, J. V., & Murphy, J. A. (2004) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737026001023 · Cited by 103
MULTIPLE-OUTCOME META-ANALYSIS OF CLINICAL TRIALS
BERKEY, C. S., ANDERSON, J. J., & HOAGLIN, D. C. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960315)15:5<537::aid-sim176>3.0.co;2-s · Cited by 75
A comparison of mean phase difference and generalized least squares for analyzing single-case data
Manolov, R., & Solanas, A. (2013) · Journal of School Psychology · doi:10.1016/j.jsp.2012.12.005 · Cited by 61
Research works printed but not linked: 4
- 10.1002/jrsm.v1:1 · doi:10.1002/jrsm.v1:1
- 10.1002/jrsm.v2.1 · doi:10.1002/jrsm.v2.1
- SAS Institute Inc, 2011, SAS® 9.3 Software
- 10.1002/jrsm.v5.1 · doi:10.1002/jrsm.v5.1
Cited by
Emotional intelligence predicts academic performance: a meta-analysis
MacCann, C., Jiang, Y., Brown, L. E. R., Double, K. S., Bucich, M., & Minbashian, A. (2020) · Psychological Bulletin · doi:10.1037/bul0000219 · Cited by 1,121
A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes
Cheung, M. W.-L. (2019) · Neuropsychology Review · doi:10.1007/s11065-019-09415-6 · Cited by 610
“Similar findings were also made in Moeyaert et al. (2017).”
BackgroundInfluentialMethodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
“Meta-analysis of single-case studies (Moeyaert et al., 2017) and meta-analysis of structural equation models (Cheung, 2015) also account for the dependent nature of the meta-analytic data using multilevel”
MethodThe Impact of Peer Assessment on Academic Performance: A Meta-analysis of Control Group Studies
Double, K. S., McGrane, J. A., & Hopfenbeck, T. N. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09510-3 · Cited by 446
Differentiated Instruction in Secondary Education: A Systematic Review of Research Evidence
Smale-Jacobse, A. E., Meijer, A., Helms-Lorenz, M., & Maulana, R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02366 · Cited by 404
“When more quantitative studies are located, this enables more statistical possibilities that can be used to gain insight in differential effects and predictive characteristics of different student outcomes (Lou et al., 1996; Moeyaert et al., 2016; Deunk et al., 2018).”
Background
Metadata
- Authors
- Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W.
- Affiliations
- State University of New York (US)
- KU Leuven (BE)
- Department of Educational Psychology, University of Texas at Austin, Austin, Texas, USA
- University of South Florida (US)
- The University of Texas at Austin (US)
- Department of Psychology and Educational Sciences, University of Leuven, Leuven, Belgium
- Corresponding author
- Moeyaert, Mariola
- Venue
- International Journal of Social Research Methodology
- Publisher
- Informa UK Limited
- Published
- 2016-11-13
- Volume
- 20
- Issue
- 6
- Pages
- 559-572
- ISSN
- 1364-5579 (ISSN-L)
- 1464-5300
- OpenAlex
- W2554874129
- Semantic Scholar
- 2ec7a1fcb1362c2cf480c05f0e8c25dcb017144a
- Cited by
- 274
- References
- 44
- Citations by year
- 2026: 30 · 2025: 33 · 2024: 27 · 2023: 36 · 2022: 40 · 2021: 43 · 2020: 26 · 2019: 26 · 2018: 10 · 2017: 3
- FWCI
- 12.25
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Social Research Methodology (publishedVersion)
- Lirias (KU Leuven) (submittedVersion)
- Funders
- U.S. Department of Education (R305D150007)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- multiple outcomes, meta-analysis, robust variance estimation, multilevel meta-analysis, three-level model, standard error estimation, fixed and random effects, estimation bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.