The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Fitting three-level meta-analytic models in R: A step-by-step tutorial
Open accessAbstract
Applying a multilevel approach to meta-analysis is a strong method for dealing with dependency of effect sizes.However, this method is relatively unknown among researchers and, to date, has not been widely used in meta-analytic research.Therefore, the purpose of this tutorial was to show how a three-level random effects model can be applied to meta-analytic models in R using the rma.mv function of the metafor package.This application is illustrated by taking the reader through a step-by-step guide to the multilevel analyses comprising the steps of (1) organizing a data file; (2) setting up the R environment; (3) calculating an overall effect; (4) examining heterogeneity of within-study variance and between-study variance; (5) performing categorical and continuous moderator analyses; and (6) examining a multiple moderator model.By example, the authors demonstrate how the multilevel approach can be applied to meta-analytically examining the association between mental health disorders of juveniles and juvenile offender recidivism.In our opinion, the rma.mv function of the metafor package provides an easy and flexible way of applying a multi-level structure to meta-analytic models in R. Further, the multilevel meta-analytic models can be easily extended so that the potential moderating influence of variables can be examined.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, there no information is lost and (moderator) effects can be estimated more precisely and with maximum power in the statistical analyses (Assink and Wibbelink 2016).”
From Risk Factors for School Absenteeism and Dropout: A Meta-Analytic Review (2019) · cites this paper as Assink and Wibbelink 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As recommended by Assink and Wibbelink (2016), moderation analyses should be conducted if less than 75% of the variance can be attributed to Level 1.”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink and Wibbelink (2016)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“There is no need to know the correlations between outcomes extracted from one study for estimating the covariance matrix of the effect sizes since the second level in the model accounts for sampling covariation ( Assink and Wibbelink, 2016 ).”
From The effect of contextual interference on transfer in motor learning - a systematic review and meta-analysis (2024) · cites this paper as Assink and Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“To deal with this intra-study dependency of prevalence estimates, we opted for the three-level model with random effects for data points obtained within one article to calculate pooled prevalence estimates (29, 30).”
From The global prevalence of autism spectrum disorder: A three-level meta-analysis (2023) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“we used a multilevel approach to metaanalysis (Assink & Wibbelink, 2016), which is a strong method for dealing with dependency of effect sizes, as it accounts for the hierarchical structure of the data in which effect sizes are nested within studies (Van Den Noortgate & Onghena, 2003).”
From Associations between rejection sensitivity and mental health outcomes: A meta-analytic review (2017) · cites this paper as Assink & Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“The Knapp and Hartung method (2003) has the advantage that it reduces Type I‐errors (Assink & Wibbelink, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Assink & Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“A three‐level random effects model was used to account for three levels of variance, including the sampling variance of each effect size (level 1), the variance between effect sizes extracted from the same study (level 2), and the variance between the studies (level 3).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Assink & Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Likelihood‐ratio tests were used to compare the deviance scores of the full model to the deviance of models excluding level 2 or level 3 variance parameters, making it possible to determine whether significant variance is present at the two levels (Assink & Wibbelink, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Assink & Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“This analysis considers three levels of variance components distributed across the model, including variance between effect sizes from the same study and variance between studies; this therefore allows for an examination of how effect sizes vary between participants (Level 1), outcomes (Level 2) and studies (Level 3; Assink & Wibbelink, 2016).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink & Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“The analyses suggested that there was significant variability ( p = .001) between effect sizes (Level 2) and between studies (Level 3), indicating that moderator analyses should be conducted (Assink & Wibbelink, 2016).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink & Wibbelink, 2016
The cited source says what the claim attributes to it.
In the citing paper
“Dependency of effect sizes normally means that effect sizes within studies are correlated (as these are expected to show a certain similarity); this creates an overlap of information and inflates information produced by the analysis, which can result in an overconfidence in its results (Assink & Wibbelink, 2016).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink & Wibbelink, 2016
In the citing paper
“A hierarchical multi-level multivariate meta-analytical approach for data plotting was used to account for statistical dependence in effect sizes from multiple within-study comparison arms (cannabis use frequencies) to a common control group (non-use)”
From Risk-thresholds for the association between frequency of cannabis use and the development of psychosis: a systematic review and meta-analysis (2023) · cites this paper as Assink & Wibbelink, 2016
In the citing paper
“As studies reported more than one effect size (for example, depression and suicidal ideation), we opted for a multilevel meta-analysis with three levels (for example, ref. 47), a random-effect model, which enables modelling the dependent effect sizes from the same study.”
From Prevalence and severity of mental health problems in early-career researchers: a systematic review and meta-analysis (2026) · cites this paper as [47]
In the citing paper
“For parameter estimation, restricted maximum likelihood estimation was applied, and for standard errors, P values and 95% CIs, Knapp and Hartung’s adjustment47,227 was used.”
From Prevalence and severity of mental health problems in early-career researchers: a systematic review and meta-analysis (2026) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,377
(2007) · Cited by 10,092
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,840
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
Hox, J., Moerbeek, M., & van de Schoot, R. (2010) · doi:10.4324/9780203852279 · Cited by 3,469
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Hays, W. (1983) · Contemporary Psychology · doi:10.1037/022267 · Cited by 2,399
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
Raudenbush, S. W., Hunter, J. E., & Schmidt, F. L. (1991) · Journal of the American Statistical Association · doi:10.2307/2289738 · Cited by 2,368
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075
Research Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1×
“The REML method is in some ways superior to other methods (see, for instance, Hox, 2010; Viechtbauer, 2005), but has also restrictions (e.g., Cheung, 2014; Van den Noortgate et al., 2013).”
MethodCheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130
The relation between short-term emotion dynamics and psychological well-being: A meta-analysis.
Houben, M., Van Den Noortgate, W., & Kuppens, P. (2015) · Psychological Bulletin · doi:10.1037/a0038822 · Cited by 1,043
Methodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716
Cheung, M. W. ‐. (2015) · Wiley eBooks · doi:10.1002/9781118957813 · Cited by 526
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450
Analyzing effect sizes: Random-effects models.
Raudenbush, S. W. (2009) · Cited by 418
Weisz, J. R., Kuppens, S., Eckshtain, D., Ugueto, A. M., Hawley, K. M., & Jensen-Doss, A. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.1176 · Cited by 412
Mullen, B. (2013) · Psychology Press eBooks · doi:10.4324/9780203771938 · Cited by 385
Global processing takes time: A meta-analysis on local–global visual processing in ASD.
der Hallen, R. V., Evers, K., Brewaeys, K., Noortgate, W. V. D., & Wagemans, J. P. (2014) · Psychological Bulletin · doi:10.1037/bul0000004 · Cited by 361
The effectiveness of Multisystemic Therapy (MST): A meta-analysis
van der Stouwe, T., Asscher, J. J., Stams, G. J. J. M., Deković, M., & van der Laan, P. H. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.06.006 · Cited by 352
Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review
Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.08.002 · Cited by 338
Self-citationResearch synthesis and meta-analysis: A step-by-step approach (4th ed.).
Cooper, H. M. (2010) · Cited by 335
Systematic Reviews in Health Care
Loo, L. K. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-7-200210010-00035 · Cited by 254
Spruit, A., Assink, M., van Vugt, E., van der Put, C., & Stams, G. J. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.006 · Cited by 250
Self-citationMultilevel Meta-analysis: a comparison with traditional meta-analytical procedures
Van Den Noortgate, W., & Onghena, P. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403251027 · Cited by 218
A Practical Tutorial on Conducting Meta-Analysis in R
Re, A. C. D. (2015) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.11.1.p037 · Cited by 113
Sports Participation and Juvenile Delinquency: A Meta-Analytic Review
Spruit, A., van Vugt, E., van der Put, C. E., van der Stouwe, T., & Stams, G. J. J. M. (2015) · Journal of Youth and Adolescence · doi:10.1007/s10964-015-0389-7 · Cited by 108
A meta-analysis of the association between mental disorders and juvenile recidivism
Wibbelink, C. J. M., Hoeve, M., Stams, G. J. J. M., & Oort, F. J. (2017) · Aggression and Violent Behavior · doi:10.1016/j.avb.2017.01.005 · Cited by 100
Self-citationRapp, R. C., Noortgate, W. V. D., Broekaert, E., & Vanderplasschen, W. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0036750 · Cited by 58
Martial arts participation and externalizing behavior in juveniles: A meta-analytic review
Gubbels, J., van der Stouwe, T., Spruit, A., & Stams, G. J. J. M. (2016) · Aggression and Violent Behavior · doi:10.1016/j.avb.2016.03.011 · Cited by 57
The relation between self-conscious emotions and delinquency: A meta-analysis
Spruit, A., Schalkwijk, F. W., van Vugt, E., & Stams, G. J. J. M. (2016) · Aggression and Violent Behavior · doi:10.1016/j.avb.2016.03.009 · Cited by 50
The bias of the commonly-used estimate of variance in meta-analysis
Li, Y., Shi, L., & Roth, H. D. (1994) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929408831305 · Cited by 45
Hedges, L. V., & Olkin, I. (1985) · Elsevier eBooks · doi:10.1016/b978-0-08-057065-5.50006-3 · Cited by 44
A Comparison of Methods to Detect Publication Bias for Meta-analysis of Continuous Data
Idris, N. R. N. (2012) · Journal of Applied Sciences · doi:10.3923/jas.2012.1413.1417 · Cited by 40
Deficits and Remedy of the Standard Random Effects Methods in Meta-analysis
Koch, A., Victor, N., & Ziegler, S. (2001) · Methods of Information in Medicine · doi:10.1055/s-0038-1634478 · Cited by 8
Barbero (2014) · doi:10.1016/j · Cited by 1
Flagged first, then by citations
Research works printed but not linked: 1
- Egger, M. and Davey-Smith, G. and Altman, D., 2001, Systematic Reviews in Healthcare
Cited by
Sorted by citations
How to perform a meta-analysis with R: a practical tutorial
Balduzzi, S., Rücker, G., & Schwarzer, G. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300117 · Cited by 5,827
Risk Factors for School Absenteeism and Dropout: A Meta-Analytic Review
Gubbels, J., van der Put, C. E., & Assink, M. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-01072-5 · Cited by 596
“The R syntaxes were written so that the three sources of variance were modeled (Assink and Wibbelink 2016).”
BackgroundMethodSelf-citationLi, H., Zhang, R., Lee, Y.-C., Kraut, R. E., & Mohr, D. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00979-5 · Cited by 598
Effects of music interventions on stress-related outcomes: a systematic review and two meta-analyses
de Witte, M., Spruit, A., van Hooren, S., Moonen, X., & Stams, G.-J. (2020) · Health Psychology Review · doi:10.1080/17437199.2019.1627897 · Cited by 539
Journal of Youth and Adolescence
(2013) · Journal of Youth and Adolescence · doi:10.1007/10964.1573-6601 · Cited by 502
Metadata
- Authors
- Assink, M., & Wibbelink, C. J. M.
- Affiliations
- University of Amsterdam (NL)
- Corresponding author
- Assink, Mark
- Venue
- The Quantitative Methods for Psychology
- Published
- 2016-10-01
- Volume
- 12
- Issue
- 3
- Pages
- 154-174
- ISSN
- 2292-1354 (ISSN-L)
- OpenAlex
- W2532414435
- Semantic Scholar
- 95f645af7a355ecf2b9a7958c12734ad1768e44f
- Cited by
- 1,330
- References
- 47
- Citations by year
- 2026: 345 · 2025: 244 · 2024: 208 · 2023: 146 · 2022: 134 · 2021: 93 · 2020: 81 · 2019: 44 · 2018: 22 · 2017: 9 · 2016: 2
- FWCI
- 21.40
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Quantitative Methods for Psychology (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Universiteit Maastricht
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- multilevel meta-analysis, R programming language, effect size heterogeneity, between-study variance, moderator analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
