Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Standard random-effects meta-analysis methods perform poorly when applied to few studies only. Such settings however are commonly encountered in practice. It is unclear, whether or to what extent small-sample-size behaviour can be improved by more sophisticated modeling. METHODS: We consider likelihood-based methods, the DerSimonian-Laird approach, Empirical Bayes, several adjustment methods and a fully Bayesian approach. Confidence intervals are based on a normal approximation, or on adjustments based on the Student-t-distribution. In addition, a linear mixed model and two generalized linear mixed models (GLMMs) assuming binomial or Poisson distributed numbers of events per study arm are considered for pairwise binary meta-analyses. We extract an empirical data set of 40 meta-analyses from recent reviews published by the German Institute for Quality and Efficiency in Health Care (IQWiG). Methods are then compared empirically as well as in a simulation study, based on few studies, imbalanced study sizes, and considering odds-ratio (OR) and risk ratio (RR) effect sizes. Coverage probabilities and interval widths for the combined effect estimate are evaluated to compare the different approaches. RESULTS: Empirically, a majority of the identified meta-analyses include only 2 studies. Variation of methods or effect measures affects the estimation results. In the simulation study, coverage probability is, in the presence of heterogeneity and few studies, mostly below the nominal level for all frequentist methods based on normal approximation, in particular when sizes in meta-analyses are not balanced, but improve when confidence intervals are adjusted. Bayesian methods result in better coverage than the frequentist methods with normal approximation in all scenarios, except for some cases of very large heterogeneity where the coverage is slightly lower. Credible intervals are empirically and in the simulation study wider than unadjusted confidence intervals, but considerably narrower than adjusted ones, with some exceptions when considering RRs and small numbers of patients per trial-arm. Confidence intervals based on the GLMMs are, in general, slightly narrower than those from other frequentist methods. Some methods turned out impractical due to frequent numerical problems. CONCLUSIONS: In the presence of between-study heterogeneity, especially with unbalanced study sizes, caution is needed in applying meta-analytical methods to few studies, as either coverage probabilities might be compromised, or intervals are inconclusively wide. Bayesian estimation with a sensibly chosen prior for between-trial heterogeneity may offer a promising compromise.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · cited 1×
“Bayesian random-effects metaanalyses with weakly informative priors for the betweenstudy heterogeneity have been proposed for this setting [4] and their performance has been found to be satisfactory in numerical applications and simulations [3, 5].”
BackgroundIntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419 · cited 9×
“[14] observe in an empirical study, such unequal study sizes are common in meta-analyses.”
InfluentialImproved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · cited 10×
“Röver C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies.”
BackgroundMethodInfluentialBasics of meta‐analysis: I 2 is not an absolute measure of heterogeneity
Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1230 · Cited by 1,848 · cited 1×
“As the I2 expresses the between-study variation relative to the total variation, the same values of τ may lead to different values of I2, depending on the precision of the underlying studies and should therefore always be interpreted as a relative measure [36].”
BackgroundBias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 8×
“With only two studies included, three methods coincide: the DL, the REML and the EB estimation [45].”
InfluentialTurner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837
A refined method for the meta-analysis of controlled clinical trials with binary outcome
Hartung, J., & Knapp, G. (2001) · Statistics in Medicine · doi:10.1002/sim.1009 · Cited by 775
Stijnen, T., Hamza, T. H., & Özdemir, P. (2010) · Statistics in Medicine · doi:10.1002/sim.4040 · Cited by 757 · cited 1×
“Fisher’s non-central hypergeometric distribution may be approximated by a binomial distribution, if the number of cases is small compared to the overall participants in that study [29]; this model specification will be denoted by “CM.”
Burke, D. L., Ensor, J., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7141 · Cited by 538
Cited by
Valenzuela, P. L., Saco-Ledo, G., Morales, J. S., Gallardo-Gómez, D., Morales-Palomo, F., López-Ortiz, S., Rivas-Baeza, B., Castillo-García, A., Jiménez-Pavón, D., Santos-Lozano, A., et al. (2023) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(23)00057-0 · Cited by 173
Peripheral blood cellular immunophenotype in depression: a systematic review and meta-analysis
Foley, É. M., Parkinson, J. T., Mitchell, R. E., Turner, L., & Khandaker, G. M. (2023) · Molecular Psychiatry · doi:10.1038/s41380-022-01919-7 · Cited by 138
Koo, C.-H., Park, J.-W., Ryu, J.-H., & Han, S.-H. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9103151 · Cited by 96
Ju, K., Lin, L., Chu, H., Cheng, L.-L., & Xu, C. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01035-6 · Cited by 61
Gutman, J. R., Khairallah, C., Stepniewska, K., Tagbor, H., Madanitsa, M., Cairns, M., L'lanziva, A. J., Kalilani, L., Otieno, K., Mwapasa, V., et al. (2021) · eClinicalMedicine · doi:10.1016/j.eclinm.2021.101160 · Cited by 33
Metadata
- Authors
- Seide, S. E., Röver, C., & Friede, T.
- Venue
- BMC Medical Research Methodology
- Published
- 2019-01-11
- Volume
- 19
- Issue
- 1
- Pages
- 16-16
- ISSN
- 1471-2288 (ISSN-L)
- OpenAlex
- W2909849393
- Cited by
- 156
- References
- 19
- FWCI
- 11.35
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies. https://app.cochat.ai/papers/10.1186/s12874-018-0618-3
- The paper
- Seide, S. E., Röver, C., & Friede, T. (2019). Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-018-0618-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.