Three-level meta-analysis of dependent effect sizes
Open accessAbstract
Although dependence in effect sizes is ubiquitous, commonly used meta-analytic methods assume independent effect sizes. We describe and illustrate three-level extensions of a mixed effects meta-analytic model that accounts for various sources of dependence within and across studies, because multilevel extensions of meta-analytic models still are not well known. We also present a three-level model for the common case where, within studies, multiple effect sizes are calculated using the same sample. Whereas this approach is relatively simple and does not require imputing values for the unknown sampling covariances, it has hardly been used, and its performance has not been empirically investigated. Therefore, we set up a simulation study, showing that also in this situation, a three-level approach yields valid results: Estimates of the treatment effects and the corresponding standard errors are unbiased.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The pseudo R2 coefficient used in multilevel meta-regression analysis cannot be considered fully stable when the number of introduced studies is below 40.
“pseudo R 2 coefficient used in multilevel meta-regression analysis cannot be considered fully stable when the number of introduced studies is below 40 (Van den Noortgate et al., 2013).”
From Do New Forms of Reading Pay Off? A Meta-Analysis on the Relationship Between Leisure Digital Reading Habits and Text Comprehension (2025) · cites this paper as Van den Noortgate et al., 2013
The Conclusions (page unavailable) warn that small numbers of studies may bias study-level variance estimates, which lends support to a general concern about stability. But the paper does not identify the pseudo R² coefficient as unstable or give a 40-study cutoff; it says 30 studies is often regarded as the smallest acceptable number and did not simulate fewer than 30. (LLM, conf=0.83)
Experimental — accuracy not yet measuredA three-level meta-analytic model was used to address dependency among effect sizes from the same study.
“To address this, we employed a three-level meta-analytic model (Cheung, 2014; Van den Noortgate et al., 2013), which allows for the inclusion of all relevant effect sizes and addresses the issue of dependent effect sizes within studies by modeling three distinct sources of variation”
From The effect of acceptance and commitment therapy on the psychological flexibility and inflexibility of undergraduate students: A systematic review and three-level meta-analysis (2023) · cites this paper as Van den Noortgate et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,385 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075823 · Cited by 18,768 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820 · see citing works
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
Littell, R. C. (1996) · Medical Entomology and Zoology · Cited by 10,345 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799 · see citing works
(2000) · Cited by 9,198 · see citing works
Multilevel analysis techniques and applications
Hox, J. J., Moerbeek, M., & van de Schoot, R. (2002) · Cited by 6,005 · see citing works
Empirical Model Building and Response Surfaces
Freeny, A. E., Box, G. E. P., & Draper, N. R. (1988) · Technometrics · doi:10.2307/1270169 · Cited by 5,238 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582 · see citing works
Introducing Multilevel Modeling
Kreft, I., & de Leeuw, J. (1998) · doi:10.4135/9781849209366 · Cited by 2,997 · see citing works
Hox, J., Moerbeek, M., & van de Schoot, R. (2002) · doi:10.4324/9781410604118 · Cited by 2,419 · see citing works
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · see citing works
Empirical Model-Building and Response Surfaces.
Kshirsagar, A. M., Box, G. E. P., & Draper, N. R. (1988) · Journal of the American Statistical Association · doi:10.2307/2288890 · Cited by 1,894 · see citing works
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 2× · see citing works
“…is known from the literature on multilevel analysis (e.g., Maas & Hox, 2005) or multilevel meta-analysis (e.g., Van den Noortgate & Onghena, 2003b; Viechtbauer, 2005) that when (restricted) maximum likelihood procedures are used, smaller numbers of units at the highest level (in this case, the…”
BackgroundSystematic Reviews and Meta-Analysis
Littell, J. H., Corcoran, J., & Pillai, V. (2008) · doi:10.1093/acprof:oso/9780195326543.001.0001 · Cited by 1,293 · see citing works
Multilevel Models in Educational and Social Research
Tate, R. L., & Goldstein, H. (1988) · Journal of Educational Statistics · doi:10.2307/1164754 · Cited by 997 · see citing works
Handbook of Applied Multivariate Statistics and Mathematical Modeling
Tinsley, H. E. A., & Brown, S. D. (2000) · Elsevier eBooks · doi:10.1016/b978-0-12-691360-6.x5000-9 · Cited by 842 · see citing works
Evans, D. J., Sharp, W. R., & Ammirato, P. V. (1986) · Macmillan , Collier-Macmillan eBooks · Cited by 547 · see citing works
Stochastically dependent effect sizes.
Gleser, L. J., & Olkin, I. (1994) · Cited by 485 · see citing works
Multilevel Models in Educational and Social Research.
Bell, J. F., & Goldstein, H. (1988) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348777 · Cited by 483 · see citing works
Psychological treatment of obsessive–compulsive disorder: A meta-analysis
Rosa-Alcázar, A. I., Sánchez-Meca, J., Gómez-Conesa, A., & Marín-Martínez, F. (2008) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.07.001 · Cited by 444 · see citing works
Self-citationResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397 · see citing works
Mechanisms of masked priming: A meta-analysis.
Van den Bussche, E., Van den Noortgate, W., & Reynvoet, B. (2009) · Psychological Bulletin · doi:10.1037/a0015329 · Cited by 350 · see citing works
Self-citationSufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.85 · Cited by 309 · see citing works
Educational Applications of Hierarchical Linear Models: A Review
Raudenbush, S. W. (1988) · Journal of Educational Statistics · doi:10.3102/10769986013002085 · Cited by 301 · see citing works
Empirical bayes meta-analysis.
Raudenbush, S. W., & Bryk, A. S. (1985) · Journal of Educational Statistics · doi:10.3102/10769986010002075 · Cited by 299 · see citing works
Rosenblad, A. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_15.x · Cited by 291 · see citing works
Becker, B. J. (2000) · Elsevier eBooks · doi:10.1016/b978-012691360-6/50018-5 · Cited by 287 · see citing works
Meta‐analysis of continuous outcome data from individual patients
Higgins, J. P. T., Whitehead, A., Turner, R., Omar, R., & Thompson, S. G. (2001) · Statistics in Medicine · doi:10.1002/sim.918 · Cited by 265 · see citing works
Geeraert, L., Noortgate, W. V. D., Grietens, H. W. E., & Onghena, P. (2004) · Child Maltreatment · doi:10.1177/1077559504264265 · Cited by 238 · see citing works
A multivariate mixed linear model for meta-analysis.
Kalaian, H. A., & Raudenbush, S. W. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.3.227 · Cited by 229 · see citing works
Multivariate Meta-Analysis: The Effect of Ignoring Within-Study Correlation
Riley, R. D. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2008.00593.x · Cited by 227 · see citing works
The Effect of Choice-Making as an Intervention for Problem Behavior
Shogren, K. A., Faggella‐Luby, M. N., Bae, S. J., & Wehmeyer, M. L. (2004) · Journal of Positive Behavior Interventions · doi:10.1177/10983007040060040401 · Cited by 220 · see citing works
Multilevel Meta-analysis: a comparison with traditional meta-analytical procedures
Van Den Noortgate, W., & Onghena, P. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403251027 · Cited by 218 · see citing works
Self-citationModeling multivariate effect sizes.
Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988) · Psychological Bulletin · doi:10.1037/0033-2909.103.1.111 · Cited by 195 · see citing works
Combining single-case experimental data using hierarchical linear models.
Noortgate, W. V. D., & Onghena, P. (2003) · School Psychology Quarterly · doi:10.1521/scpq.18.3.325.22577 · Cited by 195 · see citing works
A multilevel meta-analysis of single-subject experimental design studies
Van den Noortgate, W., & Onghena, P. (2008) · Evidence-Based Communication Assessment and Intervention · doi:10.1080/17489530802505362 · Cited by 192 · see citing works
Self-citationMarsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009) · Review of Educational Research · doi:10.3102/0034654309334143 · Cited by 174 · see citing works
Research Synthesis and Meta-Analysis
Valentine, J. C., & Cooper, H. (2008) · doi:10.4324/9780203887882-10 · Cited by 169 · see citing works
Multilevel models for meta-analysis, and their application to absolute risk differences
Thompson, S. G., Turner, R., & Warn, D. E. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000602 · Cited by 99 · see citing works
Hierarchical Dependence in Meta-Analysis
Stevens, J. R., & Taylor, A. (2008) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998607309080 · Cited by 94 · see citing works
Dependent Correlations in Meta-Analysis
Cheung, S. F., & Chan, D. K. ‐. (2008) · Educational and Psychological Measurement · doi:10.1177/0013164408315263 · Cited by 45 · see citing works
The SAGE Encyclopedia of Social Science Research Methods, Volume II
Lewis‐Beck, M. S., Bryman, A. E., & Liao, T. F. (2003) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 44 · see citing works
Multilevel models for meta-analysis, and their application to absolute risk differences
Thompson, S. G., Turner, R. M., & Warn, D. E. (2001) · Statistical Methods in Medical Research · doi:10.1191/096228001682157616 · Cited by 31 · see citing works
The SAGE Encyclopedia of Social Science Research Methods, Volume II
Snijders, T. A. B. (2003) · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [12]12. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta- analysis. Orlando: Academic Press.
- [17]17. Kish, L. (1965). Survey sampling. New York: Wiley.
Cited by
Sorted by citations
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342 · see citing works
Fitting three-level meta-analytic models in R: A step-by-step tutorial
Assink, M., & Wibbelink, C. J. M. (2016) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.12.3.p154 · Cited by 1,330 · see citing works
Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., Jensen-Doss, A., Hawley, K. M., Krumholz Marchette, L. S., Chu, B. C., et al. (2017) · American Psychologist · doi:10.1037/a0040360 · Cited by 862 · see citing works
A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808 · see citing works
Heritability of personality: A meta-analysis of behavior genetic studies.
Vukasović, T., & Bratko, D. (2015) · Psychological Bulletin · doi:10.1037/bul0000017 · Cited by 716 · see citing works
Risk Factors for School Absenteeism and Dropout: A Meta-Analytic Review
Gubbels, J., van der Put, C. E., & Assink, M. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-01072-5 · Cited by 596 · see citing works
Siegel, E. H., Sands, M. K., Van den Noortgate, W., Condon, P., Chang, Y., Dy, J., Quigley, K. S., & Barrett, L. F. (2018) · Psychological Bulletin · doi:10.1037/bul0000128 · Cited by 515 · see citing works
Self-citationYeager, D. S., Fong, C. J., Lee, H. Y., & Espelage, D. L. (2015) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2014.11.005 · Cited by 472 · see citing works
Theobald, M. (2021) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2021.101976 · Cited by 478 · see citing works
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450 · see citing works
Self-citationGender differences in loneliness across the lifespan: a meta-analysis
Maes, M., Qualter, P., Vanhalst, J., Van den Noortgate, W., & Goossens, L. (2019) · European Journal of Personality · doi:10.1002/per.2220 · Cited by 405 · see citing works
Handling Complex Meta-analytic Data Structures Using Robust Variance Estimates: a Tutorial in R
Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016) · Journal of Developmental and Life-Course Criminology · doi:10.1007/s40865-016-0026-5 · Cited by 390 · see citing works
Spiegel, J. A., Goodrich, J. M., Morris, B. M., Osborne, C. M., & Lonigan, C. J. (2021) · Psychological Bulletin · doi:10.1037/bul0000322 · Cited by 362 · see citing works
Working memory training revisited: A multi-level meta-analysis of n-back training studies
Soveri, A., Antfolk, J., Karlsson, L., Salo, B., & Laine, M. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1217-0 · Cited by 350 · see citing works
Testing for funnel plot asymmetry of standardized mean differences
Pustejovsky, J. E., & Rodgers, M. A. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1332 · Cited by 354 · see citing works
Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review
Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.08.002 · Cited by 338 · see citing works
Associations between rejection sensitivity and mental health outcomes: A meta-analytic review
Gao, S., Assink, M., Cipriani, A., & Lin, K. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.08.007 · Cited by 330 · see citing works
Risk factors for child sexual abuse victimization: A meta-analytic review.
Assink, M., van der Put, C. E., Meeuwsen, M. W. C. M., de Jong, N. M., Oort, F. J., Stams, G. J. J. M., & Hoeve, M. (2019) · Psychological bulletin · doi:10.1037/bul0000188 · Cited by 322 · see citing works
Dealing with effect size multiplicity in systematic reviews and meta‐analyses
López‐López, J. A., Page, M. J., Lipsey, M. W., & Higgins, J. P. T. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1310 · Cited by 294 · see citing works
Self-citationTomaso, C. C., Johnson, A. B., & Nelson, T. D. (2021) · SLEEP · doi:10.1093/sleep/zsaa289 · Cited by 282 · see citing works
Moeyaert, M., Ugille, M., Natasha Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017) · International Journal of Social Research Methodology · doi:10.1080/13645579.2016.1252189 · Cited by 274 · see citing works
Steffens, N. K., LaRue, C. J., Haslam, C., Walter, Z. C., Cruwys, T., Munt, K. A., Haslam, S. A., Jetten, J., & Tarrant, M. (2021) · Health Psychology Review · doi:10.1080/17437199.2019.1669481 · Cited by 259 · see citing works
Kunzler, A. M., Röthke, N., Günthner, L., Stoffers-Winterling, J., Tüscher, O., Coenen, M., Rehfuess, E., Schwarzer, G., Binder, H., Schmucker, C., et al. (2021) · Globalization and Health · doi:10.1186/s12992-021-00670-y · Cited by 262 · see citing works
Spruit, A., Assink, M., van Vugt, E., van der Put, C., & Stams, G. J. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.006 · Cited by 250 · see citing works
Meta-analytic structural equation modeling with moderating effects on SEM parameters.
Jak, S., & Cheung, M. W. L. (2020) · Psychological Methods · doi:10.1037/met0000245 · Cited by 262 · see citing works
Foster care placement instability: A meta-analytic review
Konijn, C., Admiraal, S., Baart, J., van Rooij, F., Stams, G.-J., Colonnesi, C., Lindauer, R., & Assink, M. (2019) · Children and Youth Services Review · doi:10.1016/j.childyouth.2018.12.002 · Cited by 253 · see citing works
Kirby, J. N., Day, J., & Sagar, V. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.03.001 · Cited by 232 · see citing works
Creativity and Well‐being: A Meta‐analysis
Acar, S., Tadik, H., Myers, D., van der Sman, C., & Uysal, R. (2021) · The Journal of Creative Behavior · doi:10.1002/jocb.485 · Cited by 208 · see citing works
The cognitive benefits of learning computer programming: A meta-analysis of transfer effects.
Scherer, R., Siddiq, F., & Sánchez Viveros, B. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000314 · Cited by 205 · see citing works
He is a Stud, She is a Slut! A Meta-Analysis on the Continued Existence of Sexual Double Standards
Endendijk, J. J., van Baar, A. L., & Deković, M. (2020) · Personality and Social Psychology Review · doi:10.1177/1088868319891310 · Cited by 200 · see citing works
A Meta-Analytic Review of Racial Discrimination: Relationships to Health and Culture
Carter, R. T., Johnson, V. E., Kirkinis, K., Roberson, K., Muchow, C., & Galgay, C. (2019) · Race and Social Problems · doi:10.1007/s12552-018-9256-y · Cited by 176 · see citing works
The association between adversity and hair cortisol levels in humans: A meta-analysis
Khoury, J. E., Bosquet Enlow, M., Plamondon, A., & Lyons-Ruth, K. (2019) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2019.01.009 · Cited by 170 · see citing works
Automated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145 · see citing works
Jukic, I., Castilla, A. P., Ramos, A. G., Van Hooren, B., McGuigan, M. R., & Helms, E. R. (2023) · Sports Medicine · doi:10.1007/s40279-022-01754-4 · Cited by 138 · see citing works
Stice, E., Onipede, Z. A., & Marti, C. N. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102046 · Cited by 132 · see citing works
van Loon, A. W. G., Creemers, H. E., Beumer, W. Y., Okorn, A., Vogelaar, S., Saab, N., Miers, A. C., Westenberg, P. M., & Asscher, J. J. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01201-5 · Cited by 128 · see citing works
A meta-analysis of the efficacy of gabapentin for treating alcohol use disorder
Kranzler, H. R., Feinn, R., Morris, P., & Hartwell, E. E. (2019) · Addiction · doi:10.1111/add.14655 · Cited by 121 · see citing works
Siegel, R., König, C. J., & Lazar, V. (2022) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2022.100227 · Cited by 121 · see citing works
Risky decision making in attention-deficit/hyperactivity disorder: a meta-regression analysis
Dekkers, T. J., Popma, A., Agelink van Rentergem, J. A., Bexkens, A., & Huizenga, H. M. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.001 · Cited by 119 · see citing works
Intranasal oxytocin in the treatment of autism spectrum disorders: A multilevel meta-analysis
Huang, Y., Huang, X., Ebstein, R. P., & Yu, R. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.028 · Cited by 119 · see citing works
Sebri, V., Durosini, I., Triberti, S., & Pravettoni, G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.611954 · Cited by 104 · see citing works
Traut, H. J., Guild, R. M., & Munakata, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.662139 · Cited by 100 · see citing works
Oosterhoff, M., Joore, M., & Ferreira, I. (2016) · Obesity Reviews · doi:10.1111/obr.12446 · Cited by 87 · see citing works
Predicting domestic violence: A meta-analysis on the predictive validity of risk assessment tools
van der Put, C. E., Gubbels, J., & Assink, M. (2019) · Aggression and Violent Behavior · doi:10.1016/j.avb.2019.03.008 · Cited by 83 · see citing works
Adverse Childhood Experiences and Body Dysmorphic Symptoms: A Meta-Analysis
Longobardi, C., Badenes-Ribera, L., & Fabris, M. A. (2022) · Body Image · doi:10.1016/j.bodyim.2022.01.003 · Cited by 86 · see citing works
Huiberts, R. O., Wüst, R. C. I., & van der Zwaard, S. (2024) · Sports Medicine · doi:10.1007/s40279-023-01943-9 · Cited by 75 · see citing works
Vita, A., Barlati, S., Ceraso, A., Deste, G., Nibbio, G., & Wykes, T. (2023) · Psychological Medicine · doi:10.1017/s0033291722000319 · Cited by 74 · see citing works
Goense, P. B., Assink, M., Stams, G.-J., Boendermaker, L., & Hoeve, M. (2016) · Aggression and Violent Behavior · doi:10.1016/j.avb.2016.08.003 · Cited by 68 · see citing works
Robinson, Z. P., Pelland, J. C., Remmert, J. F., Refalo, M. C., Jukic, I., Steele, J., & Zourdos, M. C. (2024) · Sports Medicine · doi:10.1007/s40279-024-02069-2 · Cited by 63 · see citing works
Rameckers, S. A., Verhoef, R. E. J., Grasman, R. P. P. P., Cox, W. R., van Emmerik, A. A. P., Engelmoer, I. M., & Arntz, A. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10235622 · Cited by 63 · see citing works
Lai, L., Liu, Y., McCracken, L. M., Li, Y., & Ren, Z. (2023) · Behaviour Research and Therapy · doi:10.1016/j.brat.2023.104308 · Cited by 60 · see citing works
Sánchez, A. C., Jones, S. K., Purvis, A., Estrada-Carmona, N., & De Palma, A. (2022) · Agriculture, Ecosystems & Environment · doi:10.1016/j.agee.2022.107933 · Cited by 60 · see citing works
Altamura, L., Vargas, C., & Salmerón, L. (2025) · Review of Educational Research · doi:10.3102/00346543231216463 · Cited by 59 · see citing works
Dworschak, C., Heim, E., & Maercker, A. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100498 · Cited by 48 · see citing works
Stegen, H., Duppen, D., Savieri, P., Stas, L., Pan, H., Aartsen, M., Callewaert, H., Dierckx, E., & De Donder, L. (2024) · International Psychogeriatrics · doi:10.1017/s1041610224000425 · Cited by 42 · see citing works
Hershberger, A., Argyriou, E., & Cyders, M. (2020) · Addictive Behaviors · doi:10.1016/j.addbeh.2020.106325 · Cited by 39 · see citing works
Fradkin, I., Strauss, A. Y., Pereg, M., & Huppert, J. D. (2018) · Clinical Psychological Science · doi:10.1177/2167702618756069 · Cited by 35 · see citing works
Fang, S., & Ding, D. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.03.015 · Cited by 33 · see citing works
Hsu, T., Adamowicz, J. L., & Thomas, E. B. K. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.10.006 · Cited by 31 · see citing works
Goral, M., Norvik, M. I., Antfolk, J., Agrotou, I., & Lehtonen, M. (2023) · Brain and Language · doi:10.1016/j.bandl.2023.105326 · Cited by 27 · see citing works
Orange, S. T., Madden, L. A., & Vince, R. V. (2020) · Journal of physiotherapy · doi:10.1016/j.jphys.2020.09.009 · Cited by 23 · see citing works
A Meta-analysis of the Effect of Multimedia Technology on Creative Performance
Liu, M., Pang, W., Guo, J., & Zhang, Y. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-10981-1 · Cited by 25 · see citing works
Risk factors for juvenile cybercrime: A meta-analytic review
Wissink, I. B., Standaert, J. C. A., Stams, G. J. J. M., Asscher, J. J., & Assink, M. (2023) · Aggression and Violent Behavior · doi:10.1016/j.avb.2023.101836 · Cited by 22 · see citing works
A meta-analysis of teacher well-being: A job demands and resources perspective
Li, Y., Chen, J., Lu, J., & Wang, X. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100719 · Cited by 19 · see citing works
Rodriguez, J. E., Williams, D. R., & Bürkner, P. (2023) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12299 · Cited by 14 · see citing works
Fortunato, L., Gullo, S., Muscolino, A., La Tona, A., Rodgers, R. F., & Lo Coco, G. (2026) · Adolescent Research Review · doi:10.1007/s40894-026-00285-7 · Cited by 7 · see citing works
Arendsen, J. L., Brugman, B. C., van Koningsbruggen, G. M., & Fransen, M. L. (2025) · International Journal of Consumer Studies · doi:10.1111/ijcs.70056 · Cited by 4 · see citing works
“Compared to the traditional two-level meta-analysis (which aggregates effect sizes across studies), a main advantage of three-level meta-analysis is that it does not require only one observed effect size per study (e.g., Van den Noortgate et al. 2013).”
Yu, Y., Zhao, W., Li, A., Shanks, D. R., Hu, X., Luo, L., & Yang, C. (2025) · Educational Psychology Review · doi:10.1007/s10648-025-10024-4 · Cited by 2 · see citing works
Wu, M., & Ouyang, F. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10211-x
The asymmetry of working memory training transfer: A systematic review and meta-analysis
Fu, J., Zhou, Y., Yuan, M., & Cai, Y. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-026-02994-5
“Given that 42 of the 55 included studies contributed multiple effect sizes – violating the independence assumption of conventional meta-analysis – we adopted a three-level random-effects meta-analysis model (Van den Noortgate et al., 2013).”
Wang, C., Zhang, W., & Cheng, J. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1868415
- Authors
- Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J.
- Affiliations
- Universidad de Murcia (ES)
- Venue
- Behavior Research Methods
- Published
- 2012-10-09
- Volume
- 45
- Issue
- 2
- Pages
- 576-594
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2086645795
- Cited by
- 1,130
- References
- 52
- Citations by year
- 2026: 247 · 2025: 154 · 2024: 150 · 2023: 114 · 2022: 136 · 2021: 78 · 2020: 65 · 2019: 58 · 2018: 48 · 2017: 33 · 2016: 23 · 2015: 14 · 2014: 5 · 2013: 5
- FWCI
- 2.11
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- multilevel meta-analysis, treatment effect estimation, standard error bias
- MeSH terms
- Models, Psychological, Models, Theoretical, Sampling Studies, Meta-Analysis as Topic, Models, Statistical, Multivariate Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
