Random effects meta‐analysis: Coverage performance of 95 % confidence and prediction intervals following REML estimation
Open accessAbstract
A random effects meta‐analysis combines the results of several independent studies to summarise the evidence about a particular measure of interest, such as a treatment effect. The approach allows for unexplained between‐study heterogeneity in the true treatment effect by incorporating random study effects about the overall mean. The variance of the mean effect estimate is conventionally calculated by assuming that the between study variance is known; however, it has been demonstrated that this approach may be inappropriate, especially when there are few studies. Alternative methods that aim to account for this uncertainty, such as Hartung–Knapp, Sidik–Jonkman and Kenward–Roger, have been proposed and shown to improve upon the conventional approach in some situations. In this paper, we use a simulation study to examine the performance of several of these methods in terms of the coverage of the 95 % confidence and prediction intervals derived from a random effects meta‐analysis estimated using restricted maximum likelihood. We show that, in terms of the confidence intervals, the Hartung–Knapp correction performs well across a wide‐range of scenarios and outperforms other methods when heterogeneity was large and/or study sizes were similar. However, the coverage of the Hartung–Knapp method is slightly too low when the heterogeneity is low ( I 2 < 30 % ) and the study sizes are quite varied. In terms of prediction intervals, the conventional approach is only valid when heterogeneity is large ( I 2 > 30 % ) and study sizes are similar. In other situations, especially when heterogeneity is small and the study sizes are quite varied, the coverage is far too low and could not be consistently improved by either increasing the number of studies, altering the degrees of freedom or using variance inflation methods. Therefore, researchers should be cautious in deriving 95 % prediction intervals following a frequentist random‐effects meta‐analysis until a more reliable solution is identified. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Partlett and Riley showed that the equation for the 95% prediction interval has poor coverage in many situations and requires improvement.
“Recently, Partlett and Riley showed that this equation has poor coverage in many situations, and requires improvement.46”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
Many frequentist estimation methods (including REML) sometimes fail to produce reliable confidence or prediction intervals.
“Furthermore, many frequentist estimation methods (including REML) sometimes fail to produce reliable confidence or prediction intervals.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
The coverage of Hartung-Knapp confidence intervals based on REML estimation is too narrow for meta-analyses with less than five studies.
“Partlett and Riley showed that the coverage of Hartung–Knapp confidence intervals based on REML estimation is too narrow for meta-analyses with less than five studies.46”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
Results from previous studies suggest that these methods are most likely to yield discordance when there is low between-study heterogeneity, when there are few studies for meta-analysis, or when studies are small or based on different sample sizes.
“Results from previous studies suggest that these methods are most likely to yield discordance when there is low between-study heterogeneity, when there are few studies for meta-analysis, or when studies are small or based on different sample sizes.46,65,66”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
Prediction intervals have poor coverage when the extent of heterogeneity is limited, when few (less than 5) studies are included, or when there are a mixture of large and small studies.
“prediction intervals have poor coverage when the extent of heterogeneity is limited, when few (less than 5) studies are included, or when there are a mixture of large and small studies.46”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
When heterogeneity is small and study sizes are mixed, the Hartung-Knapp method produces confidence intervals that are too wide.
“when the heterogeneity is small and the study sizes are mixed, the Hartung–Knapp method produces confidence intervals that are too wide.46,47”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
A Bayesian estimation framework more naturally accounts for all parameter uncertainty in the derivation of credibility and probability intervals.
“which more naturally accounts for all parameter uncertainty in the derivation of credibility and probability intervals.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · see citing works
Interpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884 · see citing works
Self-citationA Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · see citing works
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419 · see citing works
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837 · see citing works
A general parametric approach to the meta‐analysis of randomized clinical trials
Whitehead, A., & Whitehead, J. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101105 · Cited by 750 · see citing works
On tests of the overall treatment effect in meta-analysis with normally distributed responses
Hartung, J., & Knapp, G. (2001) · Statistics in Medicine · doi:10.1002/sim.791 · Cited by 543 · see citing works
Röver, C., Knapp, G., & Friede, T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0091-1 · Cited by 527 · see citing works
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453 · see citing works
A simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · cited 7× · see citing works
“Sidik and Jonkman [5] also independently suggested using (3) to calculate confidence intervals for 𝜃 and subsequently proposed an alternative modification to the HK method, which uses a more robust estimator of the variance.”
MethodInfluentialMeta‐analysis of continuous outcomes combining individual patient data and aggregate data
Riley, R. D., Lambert, P. C., Staessen, J. A., Wang, J., Gueyffier, F., Thijs, L., & Boutitie, F. (2007) · Statistics in Medicine · doi:10.1002/sim.3165 · Cited by 280 · see citing works
Self-citationSystolic and Diastolic Blood Pressure Lowering as Determinants of Cardiovascular Outcome
Wang, J.-G., Staessen, J. A., Franklin, S. S., Fagard, R., & Gueyffier, F. (2005) · Hypertension · doi:10.1161/01.hyp.0000165020.14745.79 · Cited by 275 · see citing works
Confidence intervals for the overall effect size in random-effects meta-analysis.
Sánchez‐Meca, J., & Marín‐Martínez, F. (2008) · Psychological Methods · doi:10.1037/1082-989x.13.1.31 · Cited by 242 · see citing works
Flexible parametric models for random‐effects distributions
Lee, K. J., & Thompson, S. G. (2007) · Statistics in Medicine · doi:10.1002/sim.2897 · Cited by 148 · see citing works
Robust variance estimation for random effects meta-analysis
Sidik, K., & Jonkman, J. N. (2006) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2005.07.019 · Cited by 138 · see citing works
Two-stage Individual Participant Data Meta-analysis and Generalized Forest Plots
Fisher, D. J. (2015) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1501500203 · Cited by 127 · see citing works
A new approach to outliers in meta-analysis
Baker, R., & Jackson, D. (2007) · Health Care Management Science · doi:10.1007/s10729-007-9041-8 · Cited by 90 · see citing works
An empirical comparison of heterogeneity variance estimators in 12 894 meta-analyses
Langan, D., Higgins, J. P. T., & Simmonds, M. C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1140 · Cited by 51 · see citing works
A Bayesian nonparametric meta‐analysis model
Karabatsos, G., Talbott, E., & Walker, S. G. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1117 · Cited by 24 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [4]Savin A, Wimmer G, Witkovskỳ V. On Kenward–Roger confidence intervals for common mean in interlaboratory trials In Measurement 2003. Proceedings of the 4th International Conference on Measurement, Frollo I, Tysler M, Plackova A (eds). Institute of Measurement Science, Slovak Academy of Sciences: Bratislava 2003; 79– 82.
Cited by
Sorted by citations
Predictors of adverse prognosis in COVID‐19: A systematic review and meta‐analysis
Figliozzi, S., Masci, P. G., Ahmadi, N., Tondi, L., Koutli, E., Aimo, A., Stamatelopoulos, K., Dimopoulos, M., Caforio, A. L. P., & Georgiopoulos, G. (2020) · European Journal of Clinical Investigation · doi:10.1111/eci.13362 · Cited by 361 · see citing works
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278 · see citing works
“Recently, Partlett and Riley showed that this equation has poor coverage in many situations, and requires improvement.(46) This motivates the Bayesian approach below.”
BackgroundMethodResultInfluentialSelf-citationVeroniki, A. A., Jackson, D., Bender, R., Kuss, O., Langan, D., Higgins, J. P. T., Knapp, G., & Salanti, G. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1319 · Cited by 236 · see citing works
Willis, B. H., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7372 · Cited by 171 · see citing works
Self-citationThe Effect of Exercise Training on Blood Lipids: A Systematic Review and Meta-analysis
Smart, N. A., Downes, D., van der Touw, T., Hada, S., Dieberg, G., Pearson, M. J., Wolden, M., King, N., & Goodman, S. P. J. (2025) · Sports Medicine · doi:10.1007/s40279-024-02115-z · Cited by 100 · see citing works
“We supplemented 95% confidence intervals (95% CIs) with 95% prediction intervals (95% PIs) so we could assess the distribution of effect sizes across all studies [23].”
Saco‐Ledo, G., Valenzuela, P. L., Castillo‐García, A., Arenas, J., León‐Sanz, M., Ruilope, L. M., & Lucia, A. (2021) · Obesity Reviews · doi:10.1111/obr.13103 · Cited by 47 · see citing works
Brini, S., Brudasca, N. I., Hodkinson, A., Kaluzinska, K., Wach, A., Storman, D., Prokop-Dorner, A., Jemioło, P., & Bala, M. M. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102236 · Cited by 44 · see citing works
Maric, M., Bexkens, A., & Bögels, S. M. (2018) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-018-0255-8 · Cited by 42 · see citing works
“It can be particularly informative to inspect PIs when there is high heterogeneity between studies included in a meta-analysis (IntHout et al. 2016; Partlett and Riley 2017).”
BackgroundPfefferbaum, B., Nitiéma, P., & Newman, E. (2019) · Journal of Traumatic Stress · doi:10.1002/jts.22391 · Cited by 41 · see citing works
Naudet, F., Pellen, C., Fodor, L. A., Gastaldon, C., Barbui, C., Turner, E. H., Le Pabic, E., & Cristea, I. A. (2025) · BMC Medicine · doi:10.1186/s12916-025-04435-x · Cited by 6 · see citing works
- Authors
- Partlett, C., & Riley, R. D.
- Affiliations
- University of Oxford (GB)
- Keele University (GB)
- Corresponding author
- Partlett, Christopher
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2016-10-07
- Volume
- 36
- Issue
- 2
- Pages
- 301-317
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- PMC
- PMC5157768
- OpenAlex
- W2527786681
- Cited by
- 240
- References
- 20
- Citations by year
- 2026: 61 · 2025: 34 · 2024: 30 · 2023: 20 · 2022: 20 · 2021: 21 · 2020: 19 · 2019: 16 · 2018: 16 · 2017: 3
- FWCI
- 11.21
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Statistics in Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, other-oa)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- random-effects meta-analysis, confidence interval coverage, prediction intervals, restricted maximum likelihood, study heterogeneity, variance inflation, I2 statistic
- MeSH terms
- Antihypertensive Agents, Blood Pressure, Computer Simulation, Humans, Hypertension, Meta-Analysis as Topic, Models, Statistical, Confidence Intervals, Likelihood Functions, Randomized Controlled Trials as Topic, Treatment Outcome, Sample Size, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
