Multilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures
The DOI matches its Crossref record; 64 of the 66 references it cites were found in a registry or an index.
Abstract
In this article, the authors compare the multilevel meta-analysis approach with the more traditional meta-analytical approaches. After a description and comparison of the under-lying models and some of the major techniques, the results of the multilevel approach are compared with those of the traditional approaches, using a simulation study. The results of the simulation study suggest that the maximum likelihood multilevel approach is in general superior to the fixed-effects approaches, unless only a small number of studies is available. For models without moderators, the results of the multilevel approach, however, are not substantially different from the results of the traditional random-effects approaches.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we used a multilevel approach to metaanalysis (Assink & Wibbelink, 2016), which is a strong method for dealing with dependency of effect sizes, as it accounts for the hierarchical structure of the data in which effect sizes are nested within studies (Van Den Noortgate & Onghena, 2003).”
From Associations between rejection sensitivity and mental health outcomes: A meta-analytic review (2017) · cites this paper as Van Den Noortgate & Onghena, 2003
In the citing paper
“This model is often used for multilevel meta‐analyses, and is generally considered superior to the fixed‐effects approaches used in traditional meta‐analyses (Van Den Noortgate & Onghena, 2003).”
From Does Natural Mentoring Matter? A Multilevel Meta‐analysis on the Association Between Natural Mentoring and Youth Outcomes (2018) · cites this paper as Van Den Noortgate & Onghena, 2003
In the citing paper
“Van den Noortgate and Onghena (2003) compared multilevel meta-analysis with traditional meta-analytic methods and concluded that maximum likelihood multilevel approach is in general superior to the fixed-effects approaches and that the results of the multilevel approach are not substantially different from the results of the traditional random-effects approaches for intercept only models.”
From A Meta-analysis of Attachment to Parents and Delinquency (2012) · cites this paper as Van den Noortgate and Onghena (2003)
In the citing paper
“To deal with dependency of study results, we used a multilevel random effects model for the calculation of combined effect sizes and moderator-analyses (Hox 2002; Van den Noortgate and Onghena 2003).”
From A Meta-analysis of Attachment to Parents and Delinquency (2012) · cites this paper as Van den Noortgate and Onghena 2003
In the citing paper
“The multilevel approach accounts for the hierarchical structure of the data, in which effect sizes are nested within the studies (Van Den Noortgate & Onghena, 2003).”
From The effects of physical activity interventions on psychosocial outcomes in adolescents: a meta-analytic review (2016) · cites this paper as Van Den Noortgate & Onghena, 2003
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · cited 15×
“The Qtest proposed for a multilevel meta-analysis yields comparable results with the Q-tests proposed by Hedges and Olkin (1985) and DerSimonian and Laird (1986) for testing the between-study variance, keeping the proportion of Type I errors under control.”
BackgroundMethodResultInfluentialHierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724 · cited 2×
“In estimating the standard errors, this additional variability in the estimates of the overall effect size, due to the fact they are based on estimates of the between-study variance, is not taken into account ( Bryk & Raudenbush, 1992 ).”
BackgroundMultilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,906
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 8×
“To estimate and test the parameters of the multiple regression equation, Hedges and Olkin (1985) proposed using the weighted least squares (WLS) estimation procedure, weighting each effect size by the inverse of the corre sponding sampling variance. The use of a regression equation is appealing for different reasons.…”
BackgroundMethodInfluentialMethods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · cited 4×
“A popular homogeneity test is a test described by Cochran (1954), again assuming normality:”
MethodInfluential(2006) · Technometrics · doi:10.1198/tech.2006.s417 · Cited by 4,950
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635 · cited 8×
“Rosenthal, R., & Rubin, D. B. (1982).”
MethodInfluentialMeta-Analytic Procedures for Social Research.
Metcalfe, A., & Rosenthal, R. (1994) · The Statistician · doi:10.2307/2348600 · Cited by 4,249
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853 · cited 1×
“Furthermore, Glass (1977; Glass et al., 1981) suggested looking for moderator variables by regressing the effect sizes on the study characteristics or by performing separate meta-analyses for different subgroups of studies.”
MethodThe Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Introducing Multilevel Modeling
Kreft, I., & de Leeuw, J. (1998) · doi:10.4135/9781849209366 · Cited by 2,997
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977 · cited 4×
“Finally, we note that the value of the random-effects model or the fixedeffects model in principle depends on the kind of inferences one wants to make (Hedges & Vevea, 1998; Overton, 1998).”
BackgroundInfluentialInterpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158 · cited 2×
“We mainly focused on the techniques of Hedges and Olkin (1985), although other techniques based on fixed-effects models have been proposed by Hunter et al. (1982; Hunter & Schmidt, 1990) and Rosenthal and Rubin (Rosenthal, 1991; Rosenthal & Rubin, 1978, 1982 ).”
MethodGeneralized Linear Models with Random Effects; a Gibbs Sampling Approach
Zeger, S. L., & Karim, M. R. (1991) · Journal of the American Statistical Association · doi:10.1080/01621459.1991.10475006 · Cited by 987 · cited 2×
“Maximum likelihood estimation procedures are most frequently used, although computer-intensive methods like the Gibbs sampler and the bootstrap (e.g., Meijer, van der Leeden, & Busing, 1995; Zeger & Karim, 1991) receive an increasing attention.”
MethodHierarchical linear models: Applications and data analysis methods
Rumberger, R. W. (1997) · Economics of Education Review · doi:10.1016/s0272-7757(97)80188-4 · Cited by 980
9: Integrating Findings: The Meta-Analysis of Research
Glass, G. V. (1977) · Review of Research in Education · doi:10.3102/0091732x005001351 · Cited by 832 · cited 2×
“A real breakthrough of meta-analysis came with the presentation and implementation of a set of meta-analytic techniques presented by Glass and his colleagues ( Glass, 1976, 1977; Glass, MacGaw, & Smith, 1981; Smith & Glass, 1977).”
MethodToward Reform of Program Evaluation
Gowin, D. B., Millman, J., Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1981) · Educational Evaluation and Policy Analysis · doi:10.2307/1163707 · Cited by 801 · cited 2×
“In the random-effects model for meta-analysis (Cronbach, 1980; Rubin, 1981), studies are viewed as a random sample out of a population of studies, each study investigating its own true effect size:”
MethodDevelopment of a general solution to the problem of validity generalization.
Schmidt, F. L., & Hunter, J. E. (1977) · Journal of Applied Psychology · doi:10.1037/0021-9010.62.5.529 · Cited by 769 · cited 1×
“For example, Schmidt and Hunter (1977) and Hunter and Schmidt (1990) proposed “validity generalization” as a meta-analytic method of investigating the variability of score validities across test administrations.”
Background(1990) · Cited by 561
Thompson, S. G., & Pocock, S. J. (1991) · The Lancet · doi:10.1016/0140-6736(91)91975-z · Cited by 487 · cited 2×
“Moreover, techniques for fixed-effects models and the adaptive approach are still frequently used, although using random-effects models has repeatedly been advocated (e.g., Erez, Bloom, & Wells, 1996; National Research Council, 1992) because there is evidence that the assumption of homogeneity is often implausible and…”
MethodVacha-Haase, T. (1998) · Educational and Psychological Measurement · doi:10.1177/0013164498058001002 · Cited by 449 · cited 1×
“Vacha-Haase (1998) then extended these methods in the form of the now well-known reliability generalization (RG) methods.”
MethodComparing effect sizes of independent studies.
Rosenthal, R., & Rubin, D. B. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.92.2.500 · Cited by 406 · cited 2×
“We mainly focused on the techniques of Hedges and Olkin (1985), although other techniques based on fixed-effects models have been proposed by Hunter et al. (1982; Hunter & Schmidt, 1990) and Rosenthal and Rubin (Rosenthal, 1991; Rosenthal & Rubin, 1978, 1982 ).”
MethodA random effects model for effect sizes.
Hedges, L. V. (1983) · Psychological Bulletin · doi:10.1037//0033-2909.93.2.388 · Cited by 399 · cited 4×
“Moreover, known study characteristics usually cannot fully explain the heterogeneity in effect sizes (Hedges, 1983).”
BackgroundMethodInfluentialHedges, L. V. (1994) · Cited by 397
Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis
Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009981 · Cited by 380 · cited 2×
“…random-effects approach has often been criticized due to the lack of power of the homogeneity test (e.g., Harwell, 1997; Mosteller & Colditz, 1996; Takkouche et al., 1999), our simulation results show that using the conditional procedure can be a safeguard for the strong dependence of the results…”
MethodResultOverton, R. C. (1998) · Psychological Methods · doi:10.1037//1082-989x.3.3.354 · Cited by 356 · cited 4×
“The results of some simulation studies, simulating two-level hierarchical linear models in which the (nontruncated) restricted maximum likelihood estimate of the second level variance was found negatively biased (e.g., Mok, 1995; Overton, 1998), were not replicated.”
BackgroundMethodResultInfluentialRandom-Effects Meta-Analyses Are Not Always Conservative
Poole, C., & Greenland, S. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a010035 · Cited by 338 · cited 1×
“This may be the case when there is publication bias (Poole & Greenland, 1999; Turner et al., 2000) or when the sampling distribution depends on the effect size (Van den Noortgate & Onghena, 2001).”
BackgroundA Reliability Generalization Study of the Teacher Efficacy Scale and Related Instruments
Henson, R. K., Kogan, L. R., & Vacha-Haase, T. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971284 · Cited by 307
Empirical bayes meta-analysis.
Raudenbush, S. W., & Bryk, A. S. (1985) · Journal of Educational Statistics · doi:10.3102/10769986010002075 · Cited by 299 · cited 8×
“Finally, Raudenbush and Bryk (1985) proposed to test the nonexplained variance using the statistic: ( ) ( ) d W d j s sj s S j jj k − − = = ∑ ∑ γ γ σ δ 0 1 2 2 1 , (11) which follows aχ2 distribution withk –”
BackgroundMethodInfluentialCombining Information: Statistical Issues and Opportunities for Research
Draper, D. L., Gaver, D. P., Goel, P., Greenhouse, J. B., Hedges, L. V., Morris, C., Tucker, J., & Waternaux, C. (1993) · Cited by 279
Estimation in Parallel Randomized Experiments
Rubin, D. B. (1981) · Journal of Educational Statistics · doi:10.3102/10769986006004377 · Cited by 269 · cited 2×
“In the random-effects model for meta-analysis (Cronbach, 1980; Rubin, 1981), studies are viewed as a random sample out of a population of studies, each study investigating its own true effect size:”
MethodA multilevel model framework for meta-analysis of clinical trials with binary outcomes
Turner, R. M., Omar, R. Z., Yang, M., Goldstein, H., & Thompson, S. G. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24<3417::aid-sim614>3.0.co;2-l · Cited by 265
Understanding research synthesis (meta-analysis).
Mosteller, F., & Colditz, G. A. (1996) · Annual Review of Public Health · doi:10.1146/annurev.pu.17.050196.000245 · Cited by 228 · cited 2×
“Although the conditional random-effects approach has often been criticized due to the lack of power of the homogeneity test (e.g., Harwell, 1997; Mosteller & Colditz, 1996; Takkouche et al., 1999), our simulation results show that using the conditional procedure can be a safeguard for the strong dependence of the resu…”
MethodCombining Information: Statistical Issues and Opportunities for Research
Srinivasan, C. (1993) · Technometrics · doi:10.1080/00401706.1993.10485377 · Cited by 213
Yin, P., & Fan, X. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970466 · Cited by 197 · cited 1×
“…studies include those reported by Capraro, Capraro, and Henson (2001); Caruso (2000); Caruso, Witkiewitz, Belcourt-Dittloff, and Gottlieb (2001); Helms (1999); Henson, Kogan, and Vacha-Haase (2001); Vacha-Haase, Kogan, Tani, and Woodall (2001); Viswesvaran and Ones (2000); and Yin and Fan (2000).”
BackgroundViswesvaran, C., & Ones, D. S. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970475 · Cited by 186 · cited 2×
“…studies include those reported by Capraro, Capraro, and Henson (2001); Caruso (2000); Caruso, Witkiewitz, Belcourt-Dittloff, and Gottlieb (2001); Helms (1999); Henson, Kogan, and Vacha-Haase (2001); Vacha-Haase, Kogan, Tani, and Woodall (2001); Viswesvaran and Ones (2000); and Yin and Fan (2000).”
BackgroundEREZ, A., BLOOM, M. C., & WELLS, M. T. (1996) · Personnel Psychology · doi:10.1111/j.1744-6570.1996.tb01801.x · Cited by 161 · cited 3×
“Although the choice for fixed versus random-effects models has received some attention (e.g., Erez et al., 1996; Hedges & Vevea, 1998; Overton, 1998), the choice between the estimation procedures has been largely neglected.”
BackgroundMethodViswesvaran, C., & Öneş, D. S. (2003) · Sage eBooks · doi:10.4135/9781412985789.n15 · Cited by 156
Reliability Generalization of the Neo Personality Scales
Caruso, J. C. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970484 · Cited by 153 · cited 4×
“Example RG studies include those reported by Capraro, Capraro, and Henson (2001); Caruso (2000); Caruso, Witkiewitz, Belcourt-Dittloff, and Gottlieb (2001); Helms (1999); Henson, Kogan, and Vacha-Haase (2001); Vacha-Haase, Kogan, Tani, and Woodall (2001); Viswesvaran and Ones (2000); and Yin and Fan (2000). An origina…”
BackgroundInfluentialInference and Hierarchical Modeling in the Social Sciences
Draper, D. (1995) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986020002115 · Cited by 152 · cited 1×
“Although Draper (1995) noted regarding the relation between hierarchical linear models and meta-analysis that “the former is such a natural technical tool for implementing the latter that meta-analysis may be said to have been waiting for hierarchical models to come along” (p. 133), using the…”
BackgroundMeta-Analysis Using Multilevel Models with an Application to the Study of Class Size Effects
Goldstein, H., Yang, M., Omar, R., Turner, R., & Thompson, S. (2000) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/1467-9876.00200 · Cited by 135 · cited 5×
“However, Hedges (1981) showed that the sampling distribution of the standardized mean difference is only approximately normal.”
BackgroundMethodInfluentialComparison of three major meta-analytic approaches.
Johnson, B. T., Mullen, B., & Salas, E. (1995) · Journal of Applied Psychology · doi:10.1037//0021-9010.80.1.94 · Cited by 133
Toward reform of program evaluation
Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R., Phillips, D. C., Walker, D. F., Weiner, S. S., Gowin, D. B., & Millman, J. (1980) · Cited by 113
Reliability of Scores from the Eysenck Personality Questionnaire: A Reliability Generalization Study
Caruso, J. C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J. D. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971437 · Cited by 108 · cited 1×
“Example RG studies include those reported by Capraro, Capraro, and Henson (2001); Caruso (2000); Caruso, Witkiewitz, Belcourt-Dittloff, and Gottlieb (2001) ; Helms (1999); Henson, Kogan, and Vacha-Haase (2001); Vacha-Haase, Kogan, Tani, and Woodall (2001); Viswesvaran and Ones (2000); and Yin and Fan (2000).”
Land, K. C., Wachter, K. W., & Straf, M. L. (1991) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2071894 · Cited by 97
DETECTING MODERATORS WITH META‐ANALYSIS: AN EVALUATION AND COMPARISON OF TECHNIQUES
SAGIE, A., & KOSLOWSKY, M. (1993) · Personnel Psychology · doi:10.1111/j.1744-6570.1993.tb00888.x · Cited by 92
Goldstein, H., Rasbash, J., & Plewis, I. (2000) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 90
Measurement Error of Scores on the Mathematics Anxiety Rating Scale across Studies
Capraro, M. M., Capraro, R. M., & Henson, R. K. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971266 · Cited by 79 · cited 2×
“Example RG studies include those reported by Capraro, Capraro, and Henson (2001); Caruso (2000); Caruso, Witkiewitz, Belcourt-Dittloff, and Gottlieb (2001); Helms (1999); Henson, Kogan, and Vacha-Haase (2001); Vacha-Haase, Kogan, Tani, and Woodall (2001); Viswesvaran and Ones (2000); and Yin and…”
BackgroundAn empirical study of Hedge's homogeneity test.
Harwell, M. R. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.219 · Cited by 66 · cited 4×
“…has repeatedly been advocated (e.g., Erez, Bloom, & Wells, 1996; National Research Council, 1992) because there is evidence that the assumption of homogeneity is often implausible and that the commonly used Q-test for the between-study variance lacks power (Harwell, 1997; Thompson & Pocock, 1991).”
MethodInfluentialHelms, J. E. (1999) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.1999.12068979 · Cited by 53 · cited 1×
“…RG studies include those reported by Capraro, Capraro, and Henson (2001); Caruso (2000); Caruso, Witkiewitz, Belcourt-Dittloff, and Gottlieb (2001); Helms (1999); Henson, Kogan, and Vacha-Haase (2001); Vacha-Haase, Kogan, Tani, and Woodall (2001); Viswesvaran and Ones (2000); and Yin and Fan…”
BackgroundTesting continuous moderators in meta‐analysis: A comparison of procedures
Sánchez‐Meca, J., & Marín‐Martínez, F. (1998) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1998.tb00683.x · Cited by 53
Integrating Findings: The Meta-Analysis of Research
Glass, G. V. (1977) · Review of Research in Education · doi:10.2307/1167179 · Cited by 46
Vacha-Haase, T., Kogan, L. R., Tani, C. R., & Woodall, R. A. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971059 · Cited by 45
THE IMPACT OF METHOD CHOICE ON META‐ANALYSIS
Mengersen, K. L., Tweedie, R. L., & Biggerstaff, B. (1995) · Australian Journal of Statistics · doi:10.1111/j.1467-842x.1995.tb00869.x · Cited by 40 · cited 1×
“There are some authors reporting no striking differences between iterative and noniterative parameter estimates when they compared the results for a specific meta-analytic data set (e.g., DerSimonian & Laird, 1986; Mengersen, Tweedie, & Biggerstaff, 1995 ), but a comprehensive evaluation of the relation between the es…”
ResultPower and Accuracy of the Schmidt and Hunter Meta-Analytic Procedures
Cornwell, J. M., & Ladd, R. T. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053004002 · Cited by 39
Moderator Search in Meta-Analysis: A Review and Cautionary Note on Existing Approaches
Viswesvaran, C., & Sanchez, J. I. (1998) · Educational and Psychological Measurement · doi:10.1177/0013164498058001007 · Cited by 38 · cited 1×
“This means that possible intercorrelations are taken into account, which is not the case when correlation coefficients are calculated or separate analyses are performed to investigate the moderating effects of study characteristics (Viswesvaran & Sanchez, 1998).”
Heim, U., & Pfeiffer, K. M. (1988) · Springer eBooks · doi:10.1007/978-3-642-72614-9_1 · Cited by 29
Bayesian Modelling of Prehistoric Corbelled Domes
Fan, Y., & Brooks, S. P. (2000) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.1111/1467-9884.00239 · Cited by 25
Statistische aspecten van multi-niveau onderzoek
Bosker, R., & Snijders, T. A. B. (1990) · University of Twente Research Information · Cited by 16
Meta-analyse via multi-level modellen
Hox, J. J., & de Leeuw, E. D. (1997) · Tijdschrift voor Onderwijsresearch · Cited by 4
(1985)
Research works printed but not linked: 6
- 10.1037/0033-2909.93.2.388 · doi:10.1037/0033-2909.93.2.388 · Paper page
- 10.1037/0021-9010.80.1.94 · doi:10.1037/0021-9010.80.1.94 · Paper page
- Meijer, E., van der Leeden, R. & Busing, F. M. T. A. (1995). Implementing the bootstrap for multilevel models. Multilevel Modelling Newsletter, 7(2), 7-11.
- Mok, M. (1995). Sample size requirements for 2-level designs in educational research. Multilevel Modelling Newsletter, 7(2), 11-15.
- 10.1037/1082-989x.3.3.354 · doi:10.1037/1082-989x.3.3.354 · Paper page
- 10.1037/0033-2909.92.2.500 · doi:10.1037/0033-2909.92.2.500 · Paper page
Cited by
de Ridder, D. T. D., Lensvelt-Mulders, G., Finkenauer, C., Stok, F. M., & Baumeister, R. F. (2012) · Personality and Social Psychology Review · doi:10.1177/1088868311418749 · Cited by 1,693
“…multilevel approach, which not only has the advantage of allowing for the calculation of average effect sizes across studies but also has the possibility of explaining variance at the study level (Lensvelt-Mulders, Hox, Van der Heijden, & Maas, 2005; Van den Noortgate & Onghena, 2003).”
BackgroundFitting three-level meta-analytic models in R: A step-by-step tutorial
Assink, M., & Wibbelink, C. J. M. (2016) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.12.3.p154 · Cited by 1,330
Three-level meta-analysis of dependent effect sizes
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0261-6 · Cited by 1,130
“…such as the method of moments in DerSimonian and Laird (1986), will give very similar results, even if the normality assumption that is made for the maximum likelihood procedure is violated (López-lópez, Viechtbauer, Sánchez-Meca, & Marín-Martínez, 2010; Van den Noortgate & Onghena, 2003b).”
BackgroundMethodPrati, G., & Pietrantoni, L. (2009) · Journal of Loss and Trauma · doi:10.1080/15325020902724271 · Cited by 978
“A Monte Carlo simulation showed that this approach is more effective with unreliable between-study variance estimates if the number and size of the studies are small (Van den Noortgate & Onghena, 2003 ).”
MethodWeisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., Jensen-Doss, A., Hawley, K. M., Krumholz Marchette, L. S., Chu, B. C., et al. (2017) · American Psychologist · doi:10.1037/a0040360 · Cited by 862
“This was not likely to be a major factor; multilevel models have been found to yield ES estimates comparable with traditional random-effects models (Van den Noortgate & Onghena, 2003).”
Result
Metadata
- Authors
- Van Den Noortgate, W., & Onghena, P.
- Affiliations
- Katholieke Universiteit Leuven, Belgium
- KU Leuven (BE)
- ORCID
- Onghena, Patrick
- Venue
- Educational and Psychological Measurement
- Publisher
- SAGE Publications
- Published
- 2003-10
- Volume
- 63
- Issue
- 5
- Pages
- 765-790
- ISSN
- 0013-1644 (ISSN-L)
- 1552-3888
- OpenAlex
- W2085396959
- Cited by
- 218
- References
- 66
- Citations by year
- 2026: 11 · 2025: 10 · 2024: 15 · 2023: 15 · 2022: 26 · 2021: 9 · 2020: 16 · 2019: 10 · 2018: 17 · 2017: 14 · 2016: 12 · 2015: 5 · 2014: 12 · 2013: 8 · 2012: 5
- FWCI
- 2.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational and Psychological Measurement (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- multilevel meta-analysis, fixed effects model, random effects models, moderators, simulation study, maximum likelihood estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.