Random-effects meta-analysis: The number of studies matters
Abstract
This paper investigates the impact of the number of studies on meta-analysis and meta-regression within the random-effects model framework. It is frequently neglected that inference in random-effects models requires a substantial number of studies included in meta-analysis to guarantee reliable conclusions. Several authors warn about the risk of inaccurate results of the traditional DerSimonian and Laird approach especially in the common case of meta-analysis involving a limited number of studies. This paper presents a selection of likelihood and non-likelihood methods for inference in meta-analysis proposed to overcome the limitations of the DerSimonian and Laird procedure, with a focus on the effect of the number of studies. The applicability and the performance of the methods are investigated in terms of Type I error rates and empirical power to detect effects, according to scenarios of practical interest. Simulation studies and applications to real meta-analyses highlight that it is not possible to identify an approach uniformly superior to alternatives. The overall recommendation is to avoid the DerSimonian and Laird method when the number of meta-analysis studies is modest and prefer a more comprehensive procedure that compares alternative inferential approaches. R code for meta-analysis according to all of the inferential methods examined in the paper is provided.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The small number of studies included in the random-effects meta-regression (k = 8) may have resulted in a Type I error.
“However, the small number of studies included in this random-effects meta-regression (k = 8) may have resulted in a Type I error (Guolo & Varin, 2017).”
From Taking stock: A meta-analysis of the effects of foreign language enjoyment (2022) · cites this paper as Guolo & Varin, 2017
Cites
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · see citing works
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977 · see citing works
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · see citing works
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374 · see citing works
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · see citing works
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1× · see citing works
“the fixed effects.(51) Restricted maximum likelihood is a popular method to correct for the degrees of freedom lost in the estimation of variance components.”
MethodControlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · see citing works
Hunter, J. E., & Schmidt, F. L. (2000) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00156 · Cited by 833 · see citing works
Meta‐Analysis Of Controlled Clinical Trials
Whitehead, A. (2002) · doi:10.1002/0470854200 · Cited by 827 · see citing works
A refined method for the meta-analysis of controlled clinical trials with binary outcome
Hartung, J., & Knapp, G. (2001) · Statistics in Medicine · doi:10.1002/sim.1009 · Cited by 775 · see citing works
Meat Intake and Mortality: A Prospective Study of Over Half a Million People
Sinha, R., Cross, A. J., Graubard, B. I., Leitzmann, M. F., & Schatzkin, A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.6 · Cited by 543 · see citing works
A simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Taking stock: A meta-analysis of the effects of foreign language enjoyment
Botes, E., Dewaele, J.-M., & Greiff, S. (2022) · Studies in Second Language Learning and Teaching · doi:10.14746/ssllt.2022.12.2.3 · Cited by 266 · see citing works
Methods for evidence synthesis in the case of very few studies
Bender, R., Friede, T., Koch, A., Kuss, O., Schlattmann, P., Schwarzer, G., & Skipka, G. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1297 · Cited by 250 · see citing works
Veroniki, A. A., Jackson, D., Bender, R., Kuss, O., Langan, D., Higgins, J. P. T., Knapp, G., & Salanti, G. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1319 · Cited by 236 · see citing works
Kim, N. J., Belland, B. R., & Walker, A. E. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9419-1 · Cited by 220 · see citing works
Forgrave, L. M., Ma, M., Best, J. R., & DeMarco, M. L. (2019) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2019.08.009 · Cited by 177 · see citing works
Ruiz-Goikoetxea, M., Cortese, S., Aznarez-Sanado, M., Magallón, S., Alvarez Zallo, N., Luis, E. O., de Castro-Manglano, P., Soutullo, C., & Arrondo, G. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.007 · Cited by 175 · see citing works
Howarth, E. J., O'Connor, D. B., Panagioti, M., Hodkinson, A., Wilding, S., & Johnson, J. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.01.171 · Cited by 134 · see citing works
Bayesian Meta-Analysis with Weakly Informative Prior Distributions
Williams, D. R., Rast, P., & Bürkner, P. C. (2018) · doi:10.31234/osf.io/7tbrm · Cited by 124 · see citing works
Keynejad, R. C., Hanlon, C., & Howard, L. M. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30510-3 · Cited by 114 · see citing works
Koo, C.-H., Park, J.-W., Ryu, J.-H., & Han, S.-H. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9103151 · Cited by 96 · see citing works
de Mello, R. G. B., Dalla Corte, R. R., Gioscia, J., & Moriguchi, E. H. (2019) · Journal of Aging Research · doi:10.1155/2019/1959486 · Cited by 79 · see citing works
Bayesian estimation in random effects meta‐analysis using a non‐informative prior
Bodnar, O., Link, A., Arendacká, B., Possolo, A., & Elster, C. (2017) · Statistics in Medicine · doi:10.1002/sim.7156 · Cited by 73 · see citing works
Ferrell, E. L., Russin, S. E., & Flint, D. D. (2022) · Journal of Aggression, Maltreatment & Trauma · doi:10.1080/10926771.2020.1832168 · Cited by 68 · see citing works
Systematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017.
Malički, M., Jerončić, A., Aalbersberg, I. J., Bouter, L., & ter Riet, G. (2021) · Nature Communications · doi:10.1038/s41467-021-26027-y · Cited by 53 · see citing works
- Authors
- Guolo, A., & Varin, C.
- Affiliations
- University of Padua (IT)
- Ca' Foscari University of Venice (IT)
- Corresponding author
- Guolo, Annamaria
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2015-05-07
- Volume
- 26
- Issue
- 3
- Pages
- 1500-1518
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W1885620792
- Cited by
- 203
- References
- 12
- Citations by year
- 2026: 27 · 2025: 19 · 2024: 20 · 2023: 11 · 2022: 16 · 2021: 21 · 2020: 30 · 2019: 17 · 2018: 16 · 2017: 17 · 2016: 8 · 2015: 1
- FWCI
- 11.67
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- ARCA (Università Ca' Foscari Venezia) (submittedVersion)
- Padua Research Archive (University of Padova) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- random-effects meta-analysis, meta-regression, likelihood methods, type I error rate, statistical power, small sample inference, heterogeneity
- MeSH terms
- Red Meat, Anesthesia, Diet, Humans, Meat Products, Pain, Research Design, Software, Meta-Analysis as Topic, Hysteroscopy, Likelihood Functions, Pain Management
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
