Fitting three-level meta-analytic models in R: A step-by-step tutorial
Open accessAbstract
Applying a multilevel approach to meta-analysis is a strong method for dealing with dependency of effect sizes.However, this method is relatively unknown among researchers and, to date, has not been widely used in meta-analytic research.Therefore, the purpose of this tutorial was to show how a three-level random effects model can be applied to meta-analytic models in R using the rma.mv function of the metafor package.This application is illustrated by taking the reader through a step-by-step guide to the multilevel analyses comprising the steps of (1) organizing a data file; (2) setting up the R environment; (3) calculating an overall effect; (4) examining heterogeneity of within-study variance and between-study variance; (5) performing categorical and continuous moderator analyses; and (6) examining a multiple moderator model.By example, the authors demonstrate how the multilevel approach can be applied to meta-analytically examining the association between mental health disorders of juveniles and juvenile offender recidivism.In our opinion, the rma.mv function of the metafor package provides an easy and flexible way of applying a multi-level structure to meta-analytic models in R. Further, the multilevel meta-analytic models can be easily extended so that the potential moderating influence of variables can be examined.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The three-level approach estimates moderator effects more precisely and with maximum power.
“As a result, there no information is lost and (moderator) effects can be estimated more precisely and with maximum power in the statistical analyses (Assink and Wibbelink 2016).”
From Risk Factors for School Absenteeism and Dropout: A Meta-Analytic Review (2019) · cites this paper as Assink and Wibbelink 2016
The Introduction says that retaining all relevant effect sizes preserves information and can achieve maximum statistical power; it also explains that the three-level model permits moderator analyses. It does not state that moderator effects are estimated more precisely. The cited passages have no page number in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredModeration analyses should be conducted if less than 75% of the variance can be attributed to Level 1.
“As recommended by Assink and Wibbelink (2016), moderation analyses should be conducted if less than 75% of the variance can be attributed to Level 1.”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink and Wibbelink (2016)
In “A different approach to heterogeneity” (page not available in our copy), the paper says that less than 75% of total variance attributable to Level 1 sampling variance indicates substantial heterogeneity. It says moderator analyses “may be fruitful” in that case, rather than that they should be conducted; the citing sentence makes the recommendation more categorical. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThere is no need to know the correlations between outcomes extracted from one study for estimating the covariance matrix of the effect sizes because the second level in the model accounts for sampling covariation.
“There is no need to know the correlations between outcomes extracted from one study for estimating the covariance matrix of the effect sizes since the second level in the model accounts for sampling covariation ( Assink and Wibbelink, 2016 ).”
From The effect of contextual interference on transfer in motor learning - a systematic review and meta-analysis (2024) · cites this paper as Assink and Wibbelink, 2016
A three-level model with random effects for data points within one article was used to account for intra-study dependency of prevalence estimates.
“To deal with this intra-study dependency of prevalence estimates, we opted for the three-level model with random effects for data points obtained within one article to calculate pooled prevalence estimates (29, 30).”
From The global prevalence of autism spectrum disorder: A three-level meta-analysis (2023) · cites this paper as [30]
A multilevel approach to meta-analysis accounts for the hierarchical structure of the data in which effect sizes are nested within studies.
“we used a multilevel approach to metaanalysis (Assink & Wibbelink, 2016), which is a strong method for dealing with dependency of effect sizes, as it accounts for the hierarchical structure of the data in which effect sizes are nested within studies (Van Den Noortgate & Onghena, 2003).”
From Associations between rejection sensitivity and mental health outcomes: A meta-analytic review (2017) · cites this paper as Assink & Wibbelink, 2016
The Knapp and Hartung method reduces Type I errors.
“The Knapp and Hartung method (2003) has the advantage that it reduces Type I‐errors (Assink & Wibbelink, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Assink & Wibbelink, 2016
A three-level random effects model was used to account for three levels of variance: sampling variance of each effect size, variance between effect sizes from the same study, and variance between studies.
“A three‐level random effects model was used to account for three levels of variance, including the sampling variance of each effect size (level 1), the variance between effect sizes extracted from the same study (level 2), and the variance between the studies (level 3).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Assink & Wibbelink, 2016
Likelihood-ratio tests were used to determine whether significant variance is present at level 2 and level 3.
“Likelihood‐ratio tests were used to compare the deviance scores of the full model to the deviance of models excluding level 2 or level 3 variance parameters, making it possible to determine whether significant variance is present at the two levels (Assink & Wibbelink, 2016).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Assink & Wibbelink, 2016
A three-level meta-analytical structure allows examination of how effect sizes vary between participants (Level 1), outcomes (Level 2) and studies (Level 3).
“This analysis considers three levels of variance components distributed across the model, including variance between effect sizes from the same study and variance between studies; this therefore allows for an examination of how effect sizes vary between participants (Level 1), outcomes (Level 2) and studies (Level 3; Assink & Wibbelink, 2016).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink & Wibbelink, 2016
There was significant variability between effect sizes (Level 2) and between studies (Level 3), p = .001.
“The analyses suggested that there was significant variability ( p = .001) between effect sizes (Level 2) and between studies (Level 3), indicating that moderator analyses should be conducted (Assink & Wibbelink, 2016).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink & Wibbelink, 2016
Dependency of effect sizes within studies creates overlap of information and inflates information produced by the analysis, which can result in overconfidence in its results.
“Dependency of effect sizes normally means that effect sizes within studies are correlated (as these are expected to show a certain similarity); this creates an overlap of information and inflates information produced by the analysis, which can result in an overconfidence in its results (Assink & Wibbelink, 2016).”
From Therapeutic games to reduce anxiety and depression in young people: A systematic review and exploratory meta‐analysis of their use and effectiveness. (2024) · cites this paper as Assink & Wibbelink, 2016
A hierarchical multi-level multivariate meta-analytical approach was used to account for statistical dependence in effect sizes from multiple within-study comparison arms to a common control group.
“A hierarchical multi-level multivariate meta-analytical approach for data plotting was used to account for statistical dependence in effect sizes from multiple within-study comparison arms (cannabis use frequencies) to a common control group (non-use)”
From Risk-thresholds for the association between frequency of cannabis use and the development of psychosis: a systematic review and meta-analysis (2023) · cites this paper as Assink & Wibbelink, 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
(2007) · Cited by 10,092 · see citing works
(2000) · Cited by 9,198 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,832 · see citing works
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635 · see citing works
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · see citing works
Hox, J., Moerbeek, M., & van de Schoot, R. (2010) · doi:10.4324/9780203852279 · Cited by 3,469 · see citing works
Hunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
Hays, W. (1983) · Contemporary Psychology · doi:10.1037/022267 · Cited by 2,399 · see citing works
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
Raudenbush, S. W., Hunter, J. E., & Schmidt, F. L. (1991) · Journal of the American Statistical Association · doi:10.2307/2289738 · Cited by 2,363 · see citing works
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · see citing works
Research Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716 · see citing works
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1× · see citing works
“The REML method is in some ways superior to other methods (see, for instance, Hox, 2010; Viechtbauer, 2005), but has also restrictions (e.g., Cheung, 2014; Van den Noortgate et al., 2013).”
MethodCheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342 · see citing works
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130 · see citing works
The relation between short-term emotion dynamics and psychological well-being: A meta-analysis.
Houben, M., Van Den Noortgate, W., & Kuppens, P. (2015) · Psychological Bulletin · doi:10.1037/a0038822 · Cited by 1,043 · see citing works
Methodological issues and advances in biological meta-analysis
Nakagawa, S., & Santos, E. S. A. (2012) · Evolutionary Ecology · doi:10.1007/s10682-012-9555-5 · Cited by 1,032 · see citing works
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716 · see citing works
Cheung, M. W. ‐. (2015) · Wiley eBooks · doi:10.1002/9781118957813 · Cited by 526 · see citing works
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450 · see citing works
Analyzing effect sizes: Random-effects models.
Raudenbush, S. W. (2009) · Cited by 418 · see citing works
Weisz, J. R., Kuppens, S., Eckshtain, D., Ugueto, A. M., Hawley, K. M., & Jensen-Doss, A. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.1176 · Cited by 412 · see citing works
Mullen, B. (2013) · Psychology Press eBooks · doi:10.4324/9780203771938 · Cited by 385 · see citing works
Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review
Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.08.002 · Cited by 338 · see citing works
Self-citationResearch synthesis and meta-analysis: A step-by-step approach (4th ed.).
Cooper, H. M. (2010) · Cited by 335 · see citing works
Global processing takes time: A meta-analysis on local–global visual processing in ASD.
der Hallen, R. V., Evers, K., Brewaeys, K., Noortgate, W. V. D., & Wagemans, J. P. (2014) · Psychological Bulletin · doi:10.1037/bul0000004 · Cited by 333 · see citing works
The effectiveness of Multisystemic Therapy (MST): A meta-analysis
van der Stouwe, T., Asscher, J. J., Stams, G. J. J. M., Deković, M., & van der Laan, P. H. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.06.006 · Cited by 323 · see citing works
Systematic Reviews in Health Care
Loo, L. K. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-7-200210010-00035 · Cited by 254 · see citing works
Spruit, A., Assink, M., van Vugt, E., van der Put, C., & Stams, G. J. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.006 · Cited by 250 · see citing works
Self-citationMultilevel Meta-analysis: a comparison with traditional meta-analytical procedures
Van Den Noortgate, W., & Onghena, P. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164403251027 · Cited by 218 · see citing works
A Practical Tutorial on Conducting Meta-Analysis in R
Re, A. C. D. (2015) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.11.1.p037 · Cited by 113 · see citing works
Sports Participation and Juvenile Delinquency: A Meta-Analytic Review
Spruit, A., van Vugt, E., van der Put, C. E., van der Stouwe, T., & Stams, G. J. J. M. (2015) · Journal of Youth and Adolescence · doi:10.1007/s10964-015-0389-7 · Cited by 108 · see citing works
A meta-analysis of the association between mental disorders and juvenile recidivism
Wibbelink, C. J. M., Hoeve, M., Stams, G. J. J. M., & Oort, F. J. (2017) · Aggression and Violent Behavior · doi:10.1016/j.avb.2017.01.005 · Cited by 91 · see citing works
Self-citationRapp, R. C., Noortgate, W. V. D., Broekaert, E., & Vanderplasschen, W. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0036750 · Cited by 55 · see citing works
The relation between self-conscious emotions and delinquency: A meta-analysis
Spruit, A., Schalkwijk, F. W., van Vugt, E., & Stams, G. J. J. M. (2016) · Aggression and Violent Behavior · doi:10.1016/j.avb.2016.03.009 · Cited by 50 · see citing works
Martial arts participation and externalizing behavior in juveniles: A meta-analytic review
Gubbels, J., van der Stouwe, T., Spruit, A., & Stams, G. J. J. M. (2016) · Aggression and Violent Behavior · doi:10.1016/j.avb.2016.03.011 · Cited by 46 · see citing works
The bias of the commonly-used estimate of variance in meta-analysis
Li, Y., Shi, L., & Roth, H. D. (1994) · Communication in Statistics- Theory and Methods · doi:10.1080/03610929408831305 · Cited by 45 · see citing works
Hedges, L. V., & Olkin, I. (1985) · Elsevier eBooks · doi:10.1016/b978-0-08-057065-5.50006-3 · Cited by 44 · see citing works
A Comparison of Methods to Detect Publication Bias for Meta-analysis of Continuous Data
Idris, N. R. N. (2012) · Journal of Applied Sciences · doi:10.3923/jas.2012.1413.1417 · Cited by 40 · see citing works
Deficits and Remedy of the Standard Random Effects Methods in Meta-analysis
Koch, A., Victor, N., & Ziegler, S. (2001) · Methods of Information in Medicine · doi:10.1055/s-0038-1634478 · Cited by 8 · see citing works
Barbero (2014) · doi:10.1016/j · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Egger, M. and Davey-Smith, G. and Altman, D., 2001, Systematic Reviews in Healthcare
Cited by
Sorted by citations
How to perform a meta-analysis with R: a practical tutorial
Balduzzi, S., Rücker, G., & Schwarzer, G. (2019) · Evidence Based Mental Health · doi:10.1136/ebmental-2019-300117 · Cited by 5,827 · see citing works
Risk Factors for School Absenteeism and Dropout: A Meta-Analytic Review
Gubbels, J., van der Put, C. E., & Assink, M. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-01072-5 · Cited by 596 · see citing works
“The R syntaxes were written so that the three sources of variance were modeled (Assink and Wibbelink 2016).”
BackgroundMethodSelf-citationLi, H., Zhang, R., Lee, Y.-C., Kraut, R. E., & Mohr, D. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00979-5 · Cited by 598 · see citing works
Effects of music interventions on stress-related outcomes: a systematic review and two meta-analyses
de Witte, M., Spruit, A., van Hooren, S., Moonen, X., & Stams, G.-J. (2020) · Health Psychology Review · doi:10.1080/17437199.2019.1627897 · Cited by 539 · see citing works
Journal of Youth and Adolescence
(2013) · Journal of Youth and Adolescence · doi:10.1007/10964.1573-6601 · Cited by 502 · see citing works
Rocci, K. S., Lavallee, J. M., Stewart, C. E., & Cotrufo, M. F. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.148569 · Cited by 473 · see citing works
Effects of music in exercise and sport: A meta-analytic review.
Terry, P. C., Karageorghis, C. I., Curran, M., Martin, O., & Parsons-Smith, R. L. (2019) · Psychological Bulletin · doi:10.1037/bul0000216 · Cited by 479 · see citing works
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390 · see citing works
Visual attentional orienting by eye gaze: A meta-analytic review of the gaze-cueing effect.
McKay, K. T., Grainger, S. A., Coundouris, S. P., Skorich, D. P., Phillips, L. H., & Henry, J. D. (2021) · Psychological Bulletin · doi:10.1037/bul0000353 · Cited by 443 · see citing works
Music therapy for stress reduction: a systematic review and meta-analysis
de Witte, M., Pinho, A. D. S., Stams, G.-J., Moonen, X., Bos, A. E. R., & van Hooren, S. (2022) · Health Psychology Review · doi:10.1080/17437199.2020.1846580 · Cited by 444 · see citing works
Colich, N. L., Rosen, M. L., Williams, E. S., & McLaughlin, K. A. (2020) · Psychological Bulletin · doi:10.1037/bul0000270 · Cited by 433 · see citing works
Zeegers, M. A. J., Colonnesi, C., Stams, G. J. J. M., & Meins, E. (2017) · Psychological Bulletin · doi:10.1037/bul0000114 · Cited by 431 · see citing works
Gender differences in loneliness across the lifespan: a meta-analysis
Maes, M., Qualter, P., Vanhalst, J., Van den Noortgate, W., & Goossens, L. (2019) · European Journal of Personality · doi:10.1002/per.2220 · Cited by 405 · see citing works
The Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies
Raposa, E. B., Rhodes, J., Stams, G. J. J. M., Card, N., Burton, S., Schwartz, S., Sykes, L. A. Y., Kanchewa, S., Kupersmidt, J., & Hussain, S. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-00982-8 · Cited by 366 · see citing works
Risk factors for child neglect: A meta-analytic review.
Mulder, T., Kuiper, K., van der Put, C. E., Stams, G., & Assink, M. (2018) · International Journal of Child Abuse & Neglect · doi:10.1016/j.chiabu.2018.01.006 · Cited by 314 · see citing works
Self-citationThe association between weight stigma and mental health: A meta‐analysis
Emmer, C., Bošnjak, M., & Mata, J. (2019) · Obesity Reviews · doi:10.1111/obr.12935 · Cited by 381 · see citing works
Peverill, M., Dirks, M. A., Narvaja, T., Herts, K. L., Comer, J. S., & McLaughlin, K. A. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101933 · Cited by 348 · see citing works
Associations between rejection sensitivity and mental health outcomes: A meta-analytic review
Gao, S., Assink, M., Cipriani, A., & Lin, K. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.08.007 · Cited by 330 · see citing works
Self-citationRisk factors for child sexual abuse victimization: A meta-analytic review.
Assink, M., van der Put, C. E., Meeuwsen, M. W. C. M., de Jong, N. M., Oort, F. J., Stams, G. J. J. M., & Hoeve, M. (2019) · Psychological bulletin · doi:10.1037/bul0000188 · Cited by 322 · see citing works
Self-citationBaldwin, J. R., Wang, B., Karwatowska, L., Schoeler, T., Tsaligopoulou, A., Munafò, M. R., & Pingault, J.-B. (2023) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20220174 · Cited by 313 · see citing works
The global prevalence of autism spectrum disorder: A three-level meta-analysis
Talantseva, O. I., Romanova, R. S., Shurdova, E. M., Dolgorukova, T. A., Sologub, P. S., Titova, O. S., Kleeva, D. F., & Grigorenko, E. L. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1071181 · Cited by 318 · see citing works
van de Wetering, J., Leijten, P. H. O., Spitzer, J. E., & Thomaes, S. C. E. (2022) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2022.101782 · Cited by 299 · see citing works
Schoeler, T., Duncan, L., Cecil, C. A. M., Ploubidis, G. B., & Pingault, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000171 · Cited by 308 · see citing works
Complex agricultural landscapes host more biodiversity than simple ones: A global meta-analysis
Estrada-Carmona, N., Sánchez, A. C., Remans, R., & Jones, S. K. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2203385119 · Cited by 305 · see citing works
Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández‐Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Noortgate, W. V. D. (2019) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 292 · see citing works
Tomaso, C. C., Johnson, A. B., & Nelson, T. D. (2021) · SLEEP · doi:10.1093/sleep/zsaa289 · Cited by 282 · see citing works
Spruit, A., Goos, L., Weenink, N., Rodenburg, R., Niemeyer, H., Stams, G. J., & Colonnesi, C. (2020) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-019-00299-9 · Cited by 280 · see citing works
“Log-likelihood-ratio tests were performed to compare the deviance of the full model to the deviance of the models excluding one of the variance parameters, making it possible to determine whether significant variance is present at the second and third levels (Assink and Wibbelink 2016).”
MethodInfluentialSteffens, N. K., Munt, K. A., van Knippenberg, D., Platow, M. J., & Haslam, S. A. (2020) · Organizational Psychology Review · doi:10.1177/2041386620962569 · Cited by 287 · see citing works
McClelland, S. C., Paustian, K. H., & Schipanski, M. E. (2020) · Ecological Applications · doi:10.1002/eap.2278 · Cited by 271 · see citing works
Steffens, N. K., LaRue, C. J., Haslam, C., Walter, Z. C., Cruwys, T., Munt, K. A., Haslam, S. A., Jetten, J., & Tarrant, M. (2021) · Health Psychology Review · doi:10.1080/17437199.2019.1669481 · Cited by 259 · see citing works
Moitra, M., Santomauro, D., Degenhardt, L., Collins, P. Y., Whiteford, H., Vos, T., & Ferrari, A. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.02.053 · Cited by 252 · see citing works
van Eldik, W. M., de Haan, A. D., Parry, L. Q., Davies, P. T., Luijk, M. P. C. M., Arends, L. R., & Prinzie, P. (2020) · Psychological Bulletin · doi:10.1037/bul0000233 · Cited by 273 · see citing works
Identifying Effective Components of Child Maltreatment Interventions: A Meta-analysis
van der Put, C. E., Assink, M., Gubbels, J., & van Solinge, N. F. B. (2017) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-017-0250-5 · Cited by 230 · see citing works
Self-citationFoster care placement instability: A meta-analytic review
Konijn, C., Admiraal, S., Baart, J., van Rooij, F., Stams, G.-J., Colonnesi, C., Lindauer, R., & Assink, M. (2019) · Children and Youth Services Review · doi:10.1016/j.childyouth.2018.12.002 · Cited by 253 · see citing works
Self-citationLi, C., Wang, P., Martín-Moratinos, M., Bella-Fernández, M., & Blasco-Fontecilla, H. (2022) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-022-02128-x · Cited by 309 · see citing works
The intergenerational transmission of child maltreatment: A three-level meta-analysis
Assink, M., Spruit, A., Schuts, M., Lindauer, R., van der Put, C. E., & Stams, G.-J. J. M. (2018) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2018.07.037 · Cited by 275 · see citing works
Self-citationLi, J., Dou, K., & Liang, Y. (2020) · Journal of Happiness Studies · doi:10.1007/s10902-020-00230-y · Cited by 234 · see citing works
Virtual Reality Applications in Chronic Pain Management: Systematic Review and Meta-analysis
Goudman, L., Jansen, J., Billot, M., Vets, N., Smedt, A. D., Manuel Roulaud, B., Rigoard, P., & Moens, M. (2022) · JMIR Serious Games · doi:10.2196/34402 · Cited by 230 · see citing works
van Dijk, R., van der Valk, I. E., Deković, M., & Branje, S. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101861 · Cited by 243 · see citing works
Boele, S., Van der Graaff, J., de Wied, M., Van der Valk, I. E., Crocetti, E., & Branje, S. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-00993-5 · Cited by 216 · see citing works
van Aar, J., Leijten, P. H. O., de Castro, B. O., & Overbeek, G. J. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.11.006 · Cited by 209 · see citing works
Maes, M., Nelemans, S. A., Danneel, S., Fernández-Castilla, B., Van den Noortgate, W., Goossens, L., & Vanhalst, J. (2019) · Developmental Psychology · doi:10.1037/dev0000719 · Cited by 207 · see citing works
When Problem Solving Followed by Instruction Works: Evidence for Productive Failure
Sinha, T., & Kapur, M. (2021) · Review of Educational Research · doi:10.3102/00346543211019105 · Cited by 212 · see citing works
“A multilevel meta-analysis model was used to pool the effect sizes (Assink & Wibbelink, 2016).”
He is a Stud, She is a Slut! A Meta-Analysis on the Continued Existence of Sexual Double Standards
Endendijk, J. J., van Baar, A. L., & Deković, M. (2020) · Personality and Social Psychology Review · doi:10.1177/1088868319891310 · Cited by 200 · see citing works
de Lange, J., Baams, L., van Bergen, D. D., Bos, H., & Bosker, R. (2022) · LGBT Health · doi:10.1089/lgbt.2021.0106 · Cited by 196 · see citing works
Scionti, N., Cavallero, M., Zogmaister, C., & Marzocchi, G. M. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02812 · Cited by 193 · see citing works
Cai, Z., Mao, P., Wang, Z., Wang, D., He, J., & Fan, X. (2023) · Adolescent Research Review · doi:10.1007/s40894-022-00201-9 · Cited by 192 · see citing works
The Acute Effect of High-Intensity Exercise on Executive Function: A Meta-Analysis
Moreau, D., & Chou, E. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850568 · Cited by 190 · see citing works
Reverter, M., Tapissier‐Bontemps, N., Sarter, S., Sasal, P., & Caruso, D. (2020) · Reviews in Aquaculture · doi:10.1111/raq.12485 · Cited by 186 · see citing works
Lehmann, A., Leifheit, E. F., Gerdawischke, M., & Rillig, M. C. (2021) · Microplastics and Nanoplastics · doi:10.1186/s43591-021-00007-x · Cited by 213 · see citing works
Fernández‐Castilla, B., Jamshidi, L., Declercq, L., Beretvas, S. N., Onghena, P., & Noortgate, W. V. D. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01373-9 · Cited by 184 · see citing works
Batastini, A. B., Paprzycki, P., Jones, A. C. T., & MacLean, N. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101944 · Cited by 181 · see citing works
The effects of educational robotics in STEM education: a multilevel meta-analysis
Ouyang, F., & Xu, W. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00469-4 · Cited by 263 · see citing works
Parenting and Self-Control Across Early to Late Adolescence: A Three-Level Meta-Analysis
Li, J., Willems, Y. E., Stok, F. M., Deković, M., Bartels, M., & Finkenauer, C. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619863046 · Cited by 187 · see citing works
Johnson, D., Policelli, J., Li, M., Dharamsi, A., Hu, Q., Sheridan, M. A., McLaughlin, K. A., & Wade, M. (2021) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2021.2511 · Cited by 179 · see citing works
Van Dam, L., Smit, D., Wildschut, B., Branje, S. J. T., Rhodes, J. E., Assink, M., & Stams, G. J. J. M. (2018) · American Journal of Community Psychology · doi:10.1002/ajcp.12248 · Cited by 174 · see citing works
“In this way, multiple effect sizes can be extracted from each included primary study, so that all information in the studies can be preserved and maximum statistical power is achieved (Assink & Wibbelink, 2016).”
BackgroundMethodInfluentialSelf-citationRestoring particulate and mineral-associated organic carbon through regenerative agriculture
Prairie, A. M., King, A. E., & Cotrufo, M. F. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2217481120 · Cited by 181 · see citing works
McCartney, D., Arkell, T. R., Irwin, C., & McGregor, I. S. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.01.003 · Cited by 170 · see citing works
The heritability of self-control: A meta-analysis
Willems, Y. E., Boesen, N., Li, J., Finkenauer, C., & Bartels, M. (2019) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.02.012 · Cited by 168 · see citing works
Task characteristics influence facial emotion recognition age-effects: A meta-analytic review.
Hayes, G. S., McLennan, S. N., Henry, J. D., Phillips, L. H., Terrett, G., Rendell, P. G., Pelly, R. M., & Labuschagne, I. (2020) · Psychology and Aging · doi:10.1037/pag0000441 · Cited by 156 · see citing works
Recreational angler satisfaction: What drives it?
Birdsong, M. H., Hunt, L. M., & Arlinghaus, R. (2021) · Fish and Fisheries · doi:10.1111/faf.12545 · Cited by 161 · see citing works
Dekkers, T. J., van Rentergem, J. A. A., Meijer, B., Popma, A., Wagemaker, E., & Huizenga, H. M. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.12.004 · Cited by 147 · see citing works
Automated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145 · see citing works
“We combined the effect sizes of the included studies by applying a three-level model with random effects to take into account that several studies of our meta-analysis reported more than one effect size (Geeraert et al., 2004; Konstantopoulos, 2011; Cheung, 2014; Van den Noortgate et al., 2015; Assink and Wibbelink, 2…”
MethodTrait-dependent tolerance of bats to urbanization: a global meta-analysis
Jung, K., & Threlfall, C. G. (2018) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2018.1222 · Cited by 141 · see citing works
Effects of Acute Sleep Loss on Physical Performance: A Systematic and Meta-Analytical Review
Craven, J., McCartney, D., Desbrow, B., Sabapathy, S., Bellinger, P., Roberts, L. A., & Irwin, C. G. (2022) · Sports Medicine · doi:10.1007/s40279-022-01706-y · Cited by 162 · see citing works
The association between social anxiety and mobile phone addiction: A three-level meta-analysis
Ran, G., Li, J., Zhang, Q., & Niu, X. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107198 · Cited by 150 · see citing works
de Heus, R. A. A., Tzourio, C., Lee, E. J. L., Opozda, M., Vincent, A. D., Anstey, K. J., Hofman, A., Kario, K., Lattanzi, S., Launer, L. J., et al. (2021) · Hypertension · doi:10.1161/hypertensionaha.121.17797 · Cited by 138 · see citing works
Garavan, T. N., McCarthy, A. M., Lai, Y., Murphy, K. R., Sheehan, M., & Carbery, R. (2020) · Human Resource Management Journal · doi:10.1111/1748-8583.12284 · Cited by 138 · see citing works
Jukic, I., Castilla, A. P., Ramos, A. G., Van Hooren, B., McGuigan, M. R., & Helms, E. R. (2023) · Sports Medicine · doi:10.1007/s40279-022-01754-4 · Cited by 138 · see citing works
A quantitative meta-analysis of face recognition deficits in autism: 40 years of research.
Griffin, J. W., Bauer, R. M., & Scherf, K. S. (2020) · Psychological Bulletin · doi:10.1037/bul0000310 · Cited by 134 · see citing works
Coyne, S. M., Ward, L. M., Kroff, S. L., Davis, E. J., Holmgren, H. G., Jensen, A. C., Erickson, S. E., & Essig, L. W. (2019) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2018.11.016 · Cited by 133 · see citing works
When Is Silence Golden? A Meta-analysis on Antecedents and Outcomes of Employee Silence
Hao, L., Zhu, H., He, Y., Duan, J., Zhao, T., & Meng, H. (2022) · Journal of Business and Psychology · doi:10.1007/s10869-021-09788-7 · Cited by 132 · see citing works
van Loon, A. W. G., Creemers, H. E., Beumer, W. Y., Okorn, A., Vogelaar, S., Saab, N., Miers, A. C., Westenberg, P. M., & Asscher, J. J. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01201-5 · Cited by 128 · see citing works
Kostyrka‐Allchorne, K., Wass, S. V., & Sonuga-Barke, E. J. S. (2019) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13144 · Cited by 129 · see citing works
Microbial Responses to Biochar Soil Amendment and Influential Factors: A Three-Level Meta-Analysis
Kerner, P., Struhs, E., Mirkouei, A., Aho, K. A., Lohse, K. A., Dungan, R. S., & You, Y. (2023) · Environmental Science & Technology · doi:10.1021/acs.est.3c04201 · Cited by 134 · see citing works
Baldwin, J., Coleman, O., Francis, E. R., & Danese, A. (2024) · JAMA psychiatry · doi:10.1001/jamapsychiatry.2024.0818 · Cited by 124 · see citing works
Mallawaarachchi, S., Burley, J., Mavilidi, M., Howard, S. J., Straker, L., Kervin, L., Staton, S., Hayes, N., Machell, A., Torjinski, M., et al. (2024) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2024.2620 · Cited by 142 · see citing works
Berry, J. W., Lepshokova, Z. K., & Grigoryev, D. S. (2021) · Applied Psychology · doi:10.1111/apps.12332 · Cited by 124 · see citing works
Who believes in conspiracy theories? A meta-analysis on personality correlates
Stasielowicz, L. (2022) · Journal of Research in Personality · doi:10.1016/j.jrp.2022.104229 · Cited by 112 · see citing works
Financial Profitability of Diversified Farming Systems: A Global Meta-Analysis
Sánchez, A. C., Kamau, H., Grazioli, F., & Jones, S. K. (2022) · Ecological Economics · doi:10.1016/j.ecolecon.2022.107595 · Cited by 145 · see citing works
Tully, P. J., Yano, Y., Launer, L. J., Kario, K., Nagai, M., Mooijaart, S. P., Claassen, J. A. H. R., Lattanzi, S., Vincent, A. D., Tzourio, C., et al. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.119.013841 · Cited by 121 · see citing works
Pastor, D. A., & Lazowski, R. A. (2017) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1365684 · Cited by 117 · see citing works
van IJzendoorn, M. H., Schuengel, C., Wang, Q., & Bakermans-Kranenburg, M. J. (2023) · Development and Psychopathology · doi:10.1017/s0954579421001462 · Cited by 116 · see citing works
Mao, P., Cai, Z., He, J., Chen, X., & Fan, X. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.784068 · Cited by 145 · see citing works
van der Put, C. E., Assink, M., & van Solinge, N. F. B. (2017) · International Journal of Child Abuse & Neglect · doi:10.1016/j.chiabu.2017.09.016 · Cited by 85 · see citing works
Self-citationExecutive functions in trauma-exposed youth: a meta-analysis
den Kelder, R. O., van den Akker, A. L., Geurts, H. M., Lindauer, R., & Overbeek, G. J. (2018) · European Journal of Psychotraumatology · doi:10.1080/20008198.2018.1450595 · Cited by 113 · see citing works
Wang, B., Baldwin, J. R., Schoeler, T., Cheesman, R., Barkhuizen, W., Dudbridge, F., Bann, D., Morris, T. T., & Pingault, J.-B. (2021) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2021.07.010 · Cited by 103 · see citing works
Is Your Accent Right for the Job? A Meta-Analysis on Accent Bias in Hiring Decisions
Spence, J. L., Hornsey, M. J., Stephenson, E. M., & Imuta, K. (2022) · Personality and Social Psychology Bulletin · doi:10.1177/01461672221130595 · Cited by 112 · see citing works
Donath, J. L., Lüke, T., Graf, E., Tran, U. S., & Götz, T. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09752-2 · Cited by 199 · see citing works
Non-specific versus targeted approaches to youth mentoring: A follow-up meta-analysis
Christensen, K. M., Hagler, M. A., Stams, G.-J., Raposa, E. B., Burton, S., & Rhodes, J. E. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01233-x · Cited by 107 · see citing works
Moldes, O., & Ku, L. (2020) · Psychology and Marketing · doi:10.1002/mar.21387 · Cited by 112 · see citing works
Chevance, G., Bernard, P., Chamberland, P.-É., & Rebar, A. L. (2019) · Health Psychology Review · doi:10.1080/17437199.2019.1618726 · Cited by 110 · see citing works
Parenting and prosocial behaviors: A meta‐analysis
Wong, T. K. Y., Konishi, C., & Kong, X. (2020) · Social Development · doi:10.1111/sode.12481 · Cited by 106 · see citing works
Spaniol, M., & Danielsson, H. (2021) · Journal of Intellectual Disability Research · doi:10.1111/jir.12878 · Cited by 118 · see citing works
Sebri, V., Durosini, I., Triberti, S., & Pravettoni, G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.611954 · Cited by 104 · see citing works
Ellis, B. J., Horn, A. J., Carter, C. S., van IJzendoorn, M. H., & Bakermans‐Kranenburg, M. J. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.101985 · Cited by 103 · see citing works
Associations Between Rejection Sensitivity, Aggression, and Victimization: A Meta-Analytic Review
Gao, S., Assink, M., Liu, T., Chan, K. L., & Ip, P. (2019) · Trauma Violence & Abuse · doi:10.1177/1524838019833005 · Cited by 106 · see citing works
Self-citationWeeland, J., Moens, M. A., Beute, F., Assink, M., Staaks, J. P. C., & Overbeek, G. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.101326 · Cited by 96 · see citing works
Self-citationJanovsky, T., Rock, A. J., Thorsteinsson, E., Clark, G. I., & Murray, C. (2020) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2439 · Cited by 109 · see citing works
Traut, H. J., Guild, R. M., & Munakata, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.662139 · Cited by 100 · see citing works
- Authors
- Assink, M., & Wibbelink, C. J. M.
- Affiliations
- University of Amsterdam (NL)
- Corresponding author
- Assink, Mark
- Venue
- The Quantitative Methods for Psychology
- Published
- 2016-10-01
- Volume
- 12
- Issue
- 3
- Pages
- 154-174
- ISSN
- 2292-1354 (ISSN-L)
- OpenAlex
- W2532414435
- Semantic Scholar
- 95f645af7a355ecf2b9a7958c12734ad1768e44f
- Cited by
- 1,330
- References
- 47
- Citations by year
- 2026: 345 · 2025: 244 · 2024: 208 · 2023: 146 · 2022: 134 · 2021: 93 · 2020: 81 · 2019: 44 · 2018: 22 · 2017: 9 · 2016: 2 · 2013: 1
- FWCI
- 21.40
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The Quantitative Methods for Psychology (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Universiteit Maastricht
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- multilevel meta-analysis, R programming language, effect size heterogeneity, between-study variance, moderator analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
