Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
Meta-analysis is an indispensable tool used to synthesize research findings in the social, educational, medical, management, and behavioral sciences. Most meta-analytic models assume independence among effect sizes. However, effect sizes can be dependent for various reasons. For example, studies might report multiple effect sizes on the same construct, and effect sizes reported by participants from the same cultural group are likely to be more similar than those reported by other cultural groups. This article reviews the problems and common methods to handle dependent effect sizes. The objective of this article is to demonstrate how 3-level meta-analyses can be used to model dependent effect sizes. The advantages of the structural equation modeling approach over the multilevel approach with regard to conducting a 3-level meta-analysis are discussed. This article also seeks to extend the key concepts of Q statistics, I2, and R2 from 2-level meta-analyses to 3-level meta-analyses. The proposed procedures are implemented using the open source metaSEM package for the R statistical environment. Two real data sets are used to illustrate these procedures. New research directions related to 3-level meta-analyses are discussed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Therefore, this study employed the three-level meta-analysis model proposed by Cheung (2014), which decomposes variance into three levels: Level 1, sampling variance; Level 2, within-study variance across multiple effect sizes from the same study; and Level 3, between-study variance across independent studies.”
From Mindfulness- and acceptance-based interventions for music performance anxiety: a three-level meta-analysis of therapeutic effects, mechanisms, and boundary conditions (2026) · cites this paper as Cheung, 2014
In the citing paper
“A major advantage of this three-level approach to meta-analysis is that all relevant effects reported in each primary study can be included, implying that all relevant information is preserved.”
From Risk Factors for School Absenteeism and Dropout: A Meta-Analytic Review (2019) · cites this paper as Cheung 2014
In the citing paper
“As we included five studies that provided two effect sizes, this violates the normal requirement of independent effect size measures in meta-analysis (Cheung, 2014).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Cheung, 2014
In the citing paper
“To account for the dependency of estimates at multiple times within studies, we fit three-level meta-analytic models with time periods (short-term or long-term) as a moderator.26”
From Adherence to PCSK9 Inhibitors in Clinical Practice (2026) · cites this paper as [26]
In the citing paper
“We thus adopted a three-level meta-analytic model in which effect sizes are nested within experiments, so we explicitly took the dependencies in our data into account in the model63,64.”
From When combinations of humans and AI are useful: A systematic review and meta-analysis (2024) · cites this paper as [64]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
mice: Multivariate Imputation by Chained Equations in R
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · cited 1×
“Second, conducting MI for multilevel data is challenging (cf. van Buuren & Groothuis-Oudshoorn, 2011).”
MethodMultilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,775
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Mixed-Effects Models in S and S-PLUS
(2001) · Technometrics · doi:10.1198/tech.2001.s574 · Cited by 9,278
(2010) · Cited by 7,034
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · cited 1×
“On the other hand, random-effects models allow for unconditional inferences to be generalized beyond the studies included under the assumption that the studies were randomly selected (see Borenstein et al., 2010; Hedges & Vevea, 1998).”
MethodMethods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · cited 1×
“Thus, the conventional Q statistic proposed by Cochran (1954) can be applied directly.”
MethodPrimary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“Keywords: meta-analysis, structural equation modeling, three-level model, maximum likelihood, R statistical environment Glass (1976) coined the term “meta-analysis” to describe “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating…”
MethodRecovery of inter-block information when block sizes are unequal
PATTERSON, H. D., & THOMPSON, R. (1971) · Biometrika · doi:10.1093/biomet/58.3.545 · Cited by 4,058
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory
Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007) · Prevention Science · doi:10.1007/s11121-007-0070-9 · Cited by 2,795 · cited 1×
“However, Graham, Olchowski, and Gilreath (2007) showed that many more imputations are required.”
BackgroundMaximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667
Self, S. G., & Liang, K.-Y. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478472 · Cited by 2,661
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 1×
“Hedges, Tipton, and Johnson (2010) showed that robust (sandwich) SEs perform well in terms of coverage probability in traditional meta-analyses when testing fixed effects with dependent effect sizes.”
BackgroundMeta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews
Rosenthal, R., & DiMatteo, M. R. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.59 · Cited by 1,780 · cited 1×
“In another example, the effect sizes reported by the same research team or authors might be more similar compared with those reported by other research teams or authors (Cooper, 2010; Rosenthal & DiMatteo, 2001; Shin, 2009).”
ResultMeta-analysis of individual participant data: rationale, conduct, and reporting
Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010) · BMJ · doi:10.1136/bmj.c221 · Cited by 1,701
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1×
“LBCIs have been used to quantify the precision of the heterogeneity variances in meta-analyses (Hardy & Thompson, 1996; Viechtbauer, 2005).”
MethodMultiple Imputation for Multivariate Missing-Data Problems: A Data Analyst's Perspective
Schafer, J. L., & Olsen, M. K. (1998) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3304_5 · Cited by 1,518
Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models
Skrondal, A., & Rabe‐Hesketh, S. (2004) · Thammasat University Digital Collections · Cited by 1,505
OpenMx: An Open Source Extended Structural Equation Modeling Framework
Boker, S., Neale, M., Maes, H., Wilde, M., Spiegel, M., Brick, T., Spies, J., Estabrook, R., Kenny, S., Bates, T., et al. (2011) · Psychometrika · doi:10.1007/s11336-010-9200-6 · Cited by 1,163
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130 · cited 1×
“The literature on three-level meta-analyses primarily focuses on parameter estimates and hypothesis testing (e.g., Konstantopoulos, 2011; Marsh, Bornmann, Mutz, Daniel, & O’Mara, 2009; Van den Noortgate et al., 2012).”
BackgroundBiculturalism and Adjustment: A Meta-Analysis
Nguyen, A.-M. D., & Benet-Martínez, V. (2013) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022111435097 · Cited by 992
The power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716 · cited 1×
“The statistical power of the tests might have been lowered because some information in the effect sizes was lost (Hedges & Pigott, 2001, 2004).”
BackgroundThe power of statistical tests for moderators in meta-analysis
Hedges, L. V., & Pigott, T. D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.426 · Cited by 672 · cited 1×
“The statistical power of the tests might have been lowered because some information in the effect sizes was lost (Hedges & Pigott, 2001, 2004).”
BackgroundMeta-analytic structural equation modeling: A two-stage approach.
Cheung, M. W.-L., & Chan, W. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.1.40 · Cited by 637
Self-citationMultivariate meta‐analysis: Potential and promise
Jackson, D., Riley, R., & White, I. R. (2011) · Statistics in Medicine · doi:10.1002/sim.4172 · Cited by 530
A LIKELIHOOD APPROACH TO META-ANALYSIS WITH RANDOM EFFECTS
Hardy, R., & Thompson, S. G. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960330)15:6<671::aid-sim205>3.0.co;2-5 · Cited by 527
Fixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521
Stochastically dependent effect sizes.
Gleser, L. J., & Olkin, I. (1994) · Cited by 485
People are variables too: Multilevel structural equations modeling.
Mehta, P. D., & Neale, M. C. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.3.259 · Cited by 467
Pigott, T. D. (2012) · doi:10.1007/978-1-4614-2278-5 · Cited by 453 · cited 1×
“Several approaches have been proposed to handle missing data (see Enders, 2010, for a general review, and Pigott, 2012, for a review in the context of meta-analyses).”
BackgroundAnalyzing effect sizes: Random-effects models.
Raudenbush, S. W. (2009) · Cited by 418
Meta-analytic procedures for combining studies with multiple effect sizes.
Rosenthal, R., & Rubin, D. B. (1986) · Psychological Bulletin · doi:10.1037//0033-2909.99.3.400 · Cited by 394
Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis
Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009981 · Cited by 380 · cited 1×
“Takkouche, Cadarso-Suárez, and Spiegelman (1999) suggested using the harmonic mean of vi, ṽHM n i 1 n 1 ⁄ vi , (13) whereas Higgins and Thompson (2002) proposed defining “typical” within-study variance using the Q statistic via ṽQ (n 1) i 1 n 1 ⁄ vi i 1 n 1 ⁄ vi 2 i 1 n 1 ⁄ vi 2 .”
MethodMechanisms of masked priming: A meta-analysis.
Van den Bussche, E., Van den Noortgate, W., & Reynvoet, B. (2009) · Psychological Bulletin · doi:10.1037/a0015329 · Cited by 350
Gender differences in grant peer review: A meta-analysis
BORNMANN, L., MUTZ, R., & DANIEL, H. (2007) · Journal of Informetrics · doi:10.1016/j.joi.2007.03.001 · Cited by 337
The use of likelihood-based confidence intervals in genetic models
Neale, M. C., & Miller, M. B. (1997) · Behavior Genetics · doi:10.1023/a:1025681223921 · Cited by 310
Fischer, R., & Boer, D. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0023663 · Cited by 251
Intra-cultural Variation: Evidence and Implications for International Business
Au, K. Y. (1999) · Journal of International Business Studies · doi:10.1057/palgrave.jibs.8490840 · Cited by 248
Using Phantom and Imaginary Latent Variables to Parameterize Constraints in Linear Structural Models
Rindskopf, D. (1984) · Psychometrika · doi:10.1007/bf02294204 · Cited by 233
Modeling multivariate effect sizes.
Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.1.111 · Cited by 195
Estimation of the predictive power of the model in mixed‐effects meta‐regression: A simulation study
López‐López, J. A., Marín‐Martínez, F., Sánchez‐Meca, J., Van den Noortgate, W., & Viechtbauer, W. (2014) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12002 · Cited by 191
Using Results from Replicated Studies to Estimate Linear Models
Becker, B. J. (1992) · Journal of Educational Statistics · doi:10.2307/1165128 · Cited by 181
A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses
Ahn, S., Ames, A. J., & Myers, N. D. (2012) · Review of Educational Research · doi:10.3102/0034654312458162 · Cited by 179
Cheung, M. W.-L. (2008) · Psychological Methods · doi:10.1037/a0013163 · Cited by 175
Self-citationMarsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009) · Review of Educational Research · doi:10.3102/0034654309334143 · Cited by 174 · cited 3×
“The second data set was based on Bornmann et al. (2007) and Marsh et al. (2009).”
BackgroundMethodStoel, R. D., Garre, F. G., Dolan, C. V., & van den Wittenboer, G. L. H. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.439 · Cited by 173
Nam, I., Mengersen, K., & Garthwaite, P. (2003) · Statistics in Medicine · doi:10.1002/sim.1410 · Cited by 153
Cheung, M. W. L., & Chan, W. (2009) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705510802561295 · Cited by 147
Self-citationCooper, H., Valentine, J. C., Charlton, K., & Melson, A. (2003) · Review of Educational Research · doi:10.3102/00346543073001001 · Cited by 138 · cited 2×
“Konstantopoulos (2011, Table 1) illustrated a three-level metaanalytic procedure with selected cases from Cooper et al. (2003).”
BackgroundMultilevel SEM Strategies for Evaluating Mediation in Three-Level Data
Preacher, K. J. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.589280 · Cited by 126
Dependent Effect Sizes in Meta-Analysis: Incorporating the Degree of Interdependence.
Cheung, S. F., & Chan, D. K. ‐. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.5.780 · Cited by 125
ML Versus MI for Missing Data With Violation of Distribution Conditions
Yuan, K., Yang‐Wallentin, F., & Bentler, P. M. (2012) · Sociological Methods & Research · doi:10.1177/0049124112460373 · Cited by 124
Multivariate Meta-Analysis as Structural Equation Models
Cheung, M. W. ‐. (2013) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2013.797827 · Cited by 120
Self-citationHierarchical Dependence in Meta-Analysis
Stevens, J. R., & Taylor, A. (2008) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998607309080 · Cited by 105
Intra-cultural Variation and Job Autonomy in 42 Countries
Au, K., & Cheung, M. W. L. (2004) · Organization Studies · doi:10.1177/0170840604046345 · Cited by 102
Self-citationAveraging Dependent Effect Sizes in Meta-Analysis: a Cautionary Note about Procedures
Marín‐Martínez, F., & Sánchez‐Meca, J. (1999) · The Spanish Journal of Psychology · doi:10.1017/s1138741600005436 · Cited by 99
Constructing Approximate Confidence Intervals for Parameters With Structural Equation Models
Cheung, M. W. ‐. (2009) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705510902751291 · Cited by 85
Self-citation17 Robust Procedures in Structural Equation Modeling
Yuan, K., & Bentler, P. M. (2007) · Handbook of computing and statistics with applications · doi:10.1016/b978-044452044-9/50020-3 · Cited by 78
Threats to the validity of generalized inferences
Matt, G. E., & Cook, T. D. (2009) · Cited by 68
Implementing Restricted Maximum Likelihood Estimation in Structural Equation Models
Cheung, M. W.-L. (2013) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2013.742404 · Cited by 56
Self-citationSynthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model
Owens, C. M., & Ferron, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0180-y · Cited by 52
Curran, P. J. (2009) · Psychological Methods · doi:10.1037/a0015972 · Cited by 51
An alternative to R2 for assessing linear models of effect size
Aloe, A. M., Becker, B. J., & Pigott, T. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.23 · Cited by 41
Cooper, H. M., & Hedges, L. V. (2009) · Cited by 37
Fixed-Effects Meta-Analyses as Multiple-Group Structural Equation Models
Cheung, M. W. ‐. (2010) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2010.489367 · Cited by 31
Self-citationHanke, K., & Fischer, R. D. (2012) · International Journal of Psychology · doi:10.1080/00207594.2011.651086 · Cited by 20
Xiong, C., Miller, J. P., & Morris, J. C. (2010) · Statistics in Biopharmaceutical Research · doi:10.1198/sbr.2009.0067 · Cited by 15
Cited by
Fitting three-level meta-analytic models in R: A step-by-step tutorial
Assink, M., & Wibbelink, C. J. M. (2016) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.12.3.p154 · Cited by 1,330
The relation between short-term emotion dynamics and psychological well-being: A meta-analysis.
Houben, M., Van Den Noortgate, W., & Kuppens, P. (2015) · Psychological Bulletin · doi:10.1037/a0038822 · Cited by 1,043
Sexual assault victimization and psychopathology: A review and meta-analysis
Dworkin, E. R., Menon, S. V., Bystrynski, J., & Allen, N. E. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.06.002 · Cited by 825
Mertens, S., Herberz, M., Hahnel, U. J. J., & Brosch, T. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2107346118 · Cited by 738
metaSEM: an R package for meta-analysis using structural equation modeling
Cheung, M. W.-L. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.01521 · Cited by 685
Self-citation
Metadata
- Authors
- Cheung, M. W.-L.
- Affiliations
- National University of Singapore (SG)
- Corresponding author
- Cheung, Mike W.-L.
- ORCID
- Cheung, Mike W.-L.
- Venue
- Psychological Methods
- Published
- 2014
- Volume
- 19
- Issue
- 2
- Pages
- 211-229
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2122962297
- Semantic Scholar
- 80381b0d516d07e42e308f983624bffd497b1d9b
- Cited by
- 1,342
- References
- 73
- Citations by year
- 2026: 294 · 2025: 203 · 2024: 174 · 2023: 157 · 2022: 143 · 2021: 91 · 2020: 80 · 2019: 69 · 2018: 41 · 2017: 29 · 2016: 27 · 2015: 22 · 2014: 8
- FWCI
- 22.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Ministry of Education (R581-000-111-112)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- dependent effect sizes, structural equation modeling, multilevel meta-analysis, q-statistics, I2 statistic, cultural similarity, heterogeneity measures
- MeSH terms
- Humans, Models, Psychological, Research Design, Meta-Analysis as Topic, Models, Statistical, Multivariate Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.