A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes
Open accessAbstract
Conventional meta-analytic procedures assume that effect sizes are independent. When effect sizes are not independent, conclusions based on these conventional procedures can be misleading or even wrong. Traditional approaches, such as averaging the effect sizes and selecting one effect size per study, are usually used to avoid the dependence of the effect sizes. These ad-hoc approaches, however, may lead to missed opportunities to utilize all available data to address the relevant research questions. Both multivariate meta-analysis and three-level meta-analysis have been proposed to handle non-independent effect sizes. This paper gives a brief introduction to these new techniques for applied researchers. The first objective is to highlight the benefits of using these methods to address non-independent effect sizes. The second objective is to illustrate how to apply these techniques with real data in R and Mplus. Researchers may modify the sample R and Mplus code to fit their data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The three-level model suits the data best because several primary studies contained information from independent sub-samples.
“This model suits our data best because several of the primary studies analyzed contained information from independent sub-samples (Cheung, 2019).”
From Do New Forms of Reading Pay Off? A Meta-Analysis on the Relationship Between Leisure Digital Reading Habits and Text Comprehension (2025) · cites this paper as Cheung, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Hox, J., Moerbeek, M., & van de Schoot, R. (2010) · doi:10.4324/9780203852279 · Cited by 3,469 · see citing works
Basics of meta‐analysis: <i>I</i> <sup>2</sup> is not an absolute measure of heterogeneity
Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1230 · Cited by 1,848 · see citing works
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709 · see citing works
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342 · see citing works
Self-citationFixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521 · see citing works
Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017) · International Journal of Social Research Methodology · doi:10.1080/13645579.2016.1252189 · Cited by 274 · cited 3× · see citing works
“Similar findings were also made in Moeyaert et al. (2017).”
BackgroundInfluentialThere’s more than one way to conduct a replication study: Beyond statistical significance.
Anderson, S. F., & Maxwell, S. E. (2016) · Psychological Methods · doi:10.1037/met0000051 · Cited by 167 · see citing works
Flagged first, then by citations
Research works printed but not linked: 41
- Abramovitch A Anholt G Raveh-Gottfried S Hamo N Abramowitz JS Meta-analysis of intelligence quotient (IQ) in obsessive-compulsive disorder Neuropsychology Review 2018 28 1 111 120 10.1007/s11065-017-9358-0 28864868 · DOI
- Albarracín D Cuijpers P Eastwick PW Johnson BT Roisman GI Sinatra GM Verhaeghen P Editorial Psychological Bulletin 2018 144 3 223 226 10.1037/bul0000147 29553772 · DOI
- Belleville S Fouquet C Hudon C Zomahoun HTV Croteau J Consortium for the Early Identification of Alzheimer’s disease-Quebec Neuropsychological measures that predict progression from mild cognitive impairment to Alzheimer’s type dementia in older adults: A systematic review and meta-analysis Neuropsychology Review 2017 27 4 328 353 10.1007/s11065-017-9361-5 29019061 PMC5754432 · DOI
- Burmester B Leathem J Merrick P Subjective cognitive complaints and objective cognitive function in aging: A systematic review and meta-analysis of recent cross-sectional findings Neuropsychology Review 2016 26 4 376 393 10.1007/s11065-016-9332-2 27714573 · DOI
- Card NA Applied meta-analysis for social science research 2012 New York The Guilford Press
- Cheung MW-L Multivariate meta-analysis as structural equation models Structural Equation Modeling: A Multidisciplinary Journal 2013 20 3 429 454 10.1080/10705511.2013.797827 · DOI
- Cheung MW-L Meta-analysis: A structural equation modeling approach 2015 Chichester, West Sussex John Wiley & Sons, Inc.
- Cheung, M. W.-L. (2015b). metaSEM: An R package for meta-analysis using structural equation modeling. Frontiers in Psychology, 5(1521). 10.3389/fpsyg.2014.01521 PMC4283449 25601849 · DOI
- Cheung, M. W.-L. (2018). Computing multivariate effect sizes and their sampling covariance matrices with structural equation modeling: Theory, examples, and computer simulations. Frontiers in Psychology, 9(1387). 10.3389/fpsyg.2018.01387 PMC6107852 30174628 · DOI
- Cheung MW-L Vijayakumar R A guide to conducting a meta-analysis Neuropsychology Review 2016 26 2 121 128 10.1007/s11065-016-9319-z 27209412 · DOI · Paper page
- Demidenko E Mixed models: Theory and applications with R 2013 2 Hoboken, N.J Wiley-Interscience
- Fernández-Castilla Belén Maes Marlies Declercq Lies Jamshidi Laleh Beretvas S. Natasha Onghena Patrick Van den Noortgate Wim A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis Behavior Research Methods 2018 51 3 1286 1304 10.3758/s13428-018-1063-2 29873036 · DOI
- Fischer R Boer D What is more important for national well-being: Money or autonomy? A meta-analysis of well-being, burnout, and anxiety across 63 societies Journal of Personality and Social Psychology 2011 101 1 164 184 10.1037/a0023663 21604894 · DOI
- Fischer R Hanke K Sibley CG Cultural and institutional determinants of social dominance orientation: A cross-cultural meta-analysis of 27 societies Political Psychology 2012 33 4 437 467 10.1111/j.1467-9221.2012.00884.x · DOI
- Goldstein H Multilevel statistical models 2011 4 Hoboken, N.J Wiley
- Hedges LV Olkin I Statistical methods for meta-analysis 1985 Orlando, FL Academic Press
- Hedges Larry V. Schauer Jacob M. Statistical analyses for studying replication: Meta-analytic perspectives Psychological Methods 2019 24 5 557 570 10.1037/met0000189 30070547 · DOI
- Hedges LV Tipton E Johnson MC Robust variance estimation in meta-regression with dependent effect size estimates Research Synthesis Methods 2010 1 1 39 65 10.1002/jrsm.5 26056092 · DOI · Paper page
- Ishak KJ Platt RW Joseph L Hanley JA Impact of approximating or ignoring within-study covariances in multivariate meta-analyses Statistics in Medicine 2008 27 5 670 686 10.1002/sim.2913 17492826 · DOI
- Jackson D Riley R White IR Multivariate meta-analysis: Potential and promise Statistics in Medicine 2011 30 20 2481 2498 10.1002/sim.4172 21268052 PMC3470931 · DOI · Paper page
- Li J Fine JP Assessing the dependence of sensitivity and specificity on prevalence in meta-analysis Biostatistics 2011 12 4 710 722 10.1093/biostatistics/kxr008 21525421 PMC4042906 · DOI
- López-López JA Van den Noortgate W Tanner-Smith EE Wilson SJ Lipsey MW Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation Research Synthesis Methods 2017 8 4 435 450 10.1002/jrsm.1245 28556477 · DOI
- Loring DW Bowden SC Editorial Neuropsychology Review 2016 26 1 1 2 10.1007/s11065-015-9314-9 26809716 · DOI
- Mauger C Lancelot C Roy A Coutant R Cantisano N Gall DL Executive functions in children and adolescents with turner syndrome: A systematic review and meta-analysis Neuropsychology Review 2018 28 2 188 215 10.1007/s11065-018-9372-x 29704077 · DOI
- Mewborn CM Lindbergh CA Stephen Miller L Cognitive interventions for cognitively healthy, mildly impaired, and mixed samples of older adults: A systematic review and meta-analysis of randomized-controlled trials Neuropsychology Review 2017 27 4 403 439 10.1007/s11065-017-9350-8 28726168 · DOI
- Moeyaert M Ugille M Ferron JM Beretvas SN Van den Noortgate W The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study Multivariate Behavioral Research 2013 48 5 719 748 10.1080/00273171.2013.816621 26741060 · DOI
- Nam I-S Mengersen K Garthwaite P Multivariate meta-analysis Statistics in Medicine 2003 22 14 2309 2333 10.1002/sim.1410 12854095 · DOI · Paper page
- Open Science Collaboration An open, large-scale, collaborative effort to estimate the reproducibility of psychological science Perspectives on Psychological Science 2012 7 6 657 660 10.1177/1745691612462588 26168127 · DOI
- Prado CE Watt S Crowe SF A meta-analysis of the effects of antidepressants on cognitive functioning in depressed and non-depressed samples Neuropsychology Review 2018 28 1 32 72 10.1007/s11065-018-9369-5 29446012 · DOI · Paper page
- Raudenbush SW Becker BJ Kalaian H Modeling multivariate effect sizes Psychological Bulletin 1988 103 1 111 120 10.1037/0033-2909.103.1.111 · DOI · Paper page
- Raudenbush SW Bryk AS Hierarchical linear models: Applications and data analysis methods 2002 Thousand Oaks Sage Publications
- Riley RD Multivariate meta-analysis: The effect of ignoring within-study correlation Journal of the Royal Statistical Society: Series A (Statistics in Society) 2009 172 4 789 811 10.1111/j.1467-985X.2008.00593.x · DOI
- Riley RD Thompson JR Abrams KR An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown Biostatistics 2008 9 1 172 186 10.1093/biostatistics/kxm023 17626226 · DOI
- Scammacca N Roberts G Stuebing KK Meta-analysis with complex research designs dealing with dependence from multiple measures and multiple group comparisons Review of Educational Research 2014 84 3 328 364 10.3102/0034654313500826 25309002 PMC4191743 · DOI
- Sherman DS Mauser J Nuno M Sherzai D The efficacy of cognitive intervention in mild cognitive impairment (MCI): A meta-analysis of outcomes on neuropsychological measures Neuropsychology Review 2017 27 4 440 484 10.1007/s11065-017-9363-3 29282641 PMC5754430 · DOI
- Stadler M Becker N Gödker M Leutner D Greiff S Complex problem solving and intelligence: A meta-analysis Intelligence 2015 53 92 101 10.1016/j.intell.2015.09.005 · DOI
- Timm NH A note on testing for multivariate effect sizes Journal of Educational and Behavioral Statistics 1999 24 2 132 145 10.3102/10769986024002132 · DOI
- Tipton E Small sample adjustments for robust variance estimation with meta-regression Psychological Methods 2015 20 3 375 393 10.1037/met0000011 24773356 · DOI · Paper page
- Van den Noortgate W López-López JA Marín-Martínez F Sánchez-Meca J Three-level meta-analysis of dependent effect sizes Behavior Research Methods 2013 45 2 576 594 10.3758/s13428-012-0261-6 23055166 · DOI · Paper page
- Viechtbauer W Conducting meta-analyses in R with the metafor package Journal of Statistical Software 2010 36 3 1 48 10.18637/jss.v036.i03 · DOI · Paper page
- Weissberger GH Strong JV Stefanidis KB Summers MJ Bondi MW Stricker NH Diagnostic accuracy of memory measures in Alzheimer’s dementia and mild cognitive impairment: A systematic review and meta-analysis Neuropsychology Review 2017 27 4 354 388 10.1007/s11065-017-9360-6 28940127 PMC5886311 · DOI
Cited by
Sorted by citations
<scp>EEG</scp> power spectral measures of cognitive workload: A meta‐analysis
Chikhi, S., Matton, N., & Blanchet, S. (2022) · Psychophysiology · doi:10.1111/psyp.14009 · Cited by 258 · see citing works
How to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218 · see citing works
Batastini, A. B., Paprzycki, P., Jones, A. C. T., & MacLean, N. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101944 · Cited by 181 · see citing works
The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis
Di Pietro, G. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100530 · Cited by 173 · see citing works
Schumann, M., Feuerbacher, J. F., Sünkeler, M., Freitag, N., Rønnestad, B. R., Doma, K., & Lundberg, T. R. (2022) · Sports Medicine · doi:10.1007/s40279-021-01587-7 · Cited by 156 · see citing works
Jukic, I., Castilla, A. P., Ramos, A. G., Van Hooren, B., McGuigan, M. R., & Helms, E. R. (2023) · Sports Medicine · doi:10.1007/s40279-022-01754-4 · Cited by 138 · see citing works
Personality and intelligence: A meta-analysis.
Anglim, J., Dunlop, P. D., Wee, S., Horwood, S., Wood, J. K., & Marty, A. (2022) · Psychological Bulletin · doi:10.1037/bul0000373 · Cited by 125 · see citing works
“In particular, there is extensive discussion in the metaanalytic literature of the implications of analyzing dependent effect sizes (Cheung, 2014, 2019; Hedges et al., 2010; Scammacca et al., 2014).”
BackgroundIntranasal oxytocin in the treatment of autism spectrum disorders: A multilevel meta-analysis
Huang, Y., Huang, X., Ebstein, R. P., & Yu, R. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.028 · Cited by 119 · see citing works
Almeida, M., Ramos, C., Maciel, L., Basto-Pereira, M., & Leal, I. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.995981 · Cited by 81 · see citing works
Multidimensional Perfectionism and Test Anxiety: a Meta-analytic Review of Two Decades of Research
Burcaş, S., & Creţu, R. Z. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09531-3 · Cited by 74 · see citing works
Lai, L., Liu, Y., McCracken, L. M., Li, Y., & Ren, Z. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104308 · Cited by 60 · see citing works
Dochat, C., Wooldridge, J. S., Herbert, M. S., Lee, M. W., & Afari, N. (2021) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2021.03.003 · Cited by 60 · see citing works
Altamura, L., Vargas, C., & Salmerón, L. (2025) · Review of Educational Research · doi:10.3102/00346543231216463 · Cited by 59 · see citing works
Spindle-dependent memory consolidation in healthy adults: a meta-analysis
Kumral, D., Matzerath, A., Leonhart, R., & Schönauer, M. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108661 · Cited by 32 · see citing works
Wood, G., Taylor, E., Ng, V., Murrell, A., Patil, A., van der Touw, T., Wolden, M., Andronicos, N., & Smart, N. A. (2023) · Sports Medicine · doi:10.1007/s40279-023-01817-0 · Cited by 31 · see citing works
Pappa, E., Baah, F., Lynch, J., Shiel, L., Blackman, G., Raihani, N., & Bell, V. (2025) · Schizophrenia Bulletin · doi:10.1093/schbul/sbae225 · Cited by 19 · see citing works
Schincariol, A., Otgaar, H., Greene, C. M., Murphy, G., Riesthuis, P., Mangiulli, I., & Battista, F. (2024) · Memory, Mind & Media · doi:10.1017/mem.2024.14 · Cited by 8 · see citing works
- Authors
- Cheung, M. W.-L.
- Affiliations
- National University of Singapore (SG)
- Corresponding author
- Cheung, Mike W.-L.
- ORCID
- Cheung, Mike W.-L.
- Venue
- Neuropsychology Review
- Published
- 2019-08-24
- Volume
- 29
- Issue
- 4
- Pages
- 387-396
- ISSN
- 1040-7308 (ISSN-L)
- 1573-6660
- PMC
- PMC6892772
- OpenAlex
- W2969304932
- Cited by
- 610
- References
- 11
- Citations by year
- 2026: 129 · 2025: 134 · 2024: 83 · 2023: 90 · 2022: 84 · 2021: 58 · 2020: 29 · 2019: 2 · 2013: 1
- FWCI
- 8.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Neuropsychology Review (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Ministry of Education, Singapore (Academic Research Fund Tier 1 grant (FY2017-FRC1-008))
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- meta-analysis, multivariate meta-analysis, R code
- MeSH terms
- Humans, Neuropsychology, Research Design, Meta-Analysis as Topic, Models, Statistical, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
