CoChat
Sign Up
Cited by 2,569Meta-analysis and systematic reviewsStatistical Methods and Bayesian InferenceStatistical Methods in Epidemiology

A Re-Evaluation of Random-Effects Meta-Analysis

Open access
Higgins, Julian P. T., Thompson, Simon G., Spiegelhalter, David J.·2009·Journal of the Royal Statistical Society Series A: Statistics in Society
DOI 10.1111/j.1467-985x.2008.00552.xPMC PMC2667312
Open this paper in CoChat — free account

Abstract

Summary: Meta-analysis in the presence of unexplained heterogeneity is frequently undertaken by using a random-effects model, in which the effects underlying different studies are assumed to be drawn from a normal distribution. Here we discuss the justification and interpretation of such models, by addressing in turn the aims of estimation, prediction and hypothesis testing. A particular issue that we consider is the distinction between inference on the mean of the random-effects distribution and inference on the whole distribution. We suggest that random-effects meta-analyses as currently conducted often fail to provide the key results, and we investigate the extent to which distribution-free, classical and Bayesian approaches can provide satisfactory methods. We conclude that the Bayesian approach has the advantage of naturally allowing for full uncertainty, especially for prediction. However, it is not without problems, including computational intensity and sensitivity to a priori judgements. We propose a simple prediction interval for classical meta-analysis and offer extensions to standard practice of Bayesian meta-analysis, making use of an example of studies of ‘set shifting’ ability in people with eating disorders.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper re-evaluates random-effects meta-analysis, arguing that inference should address the whole distribution of effects—especially prediction—rather than only the mean effect. It proposes a simple classical prediction interval and illustrates the issues with 14 studies of set-shifting ability in eating disorders.

Key findings

  • Focusing only on the mean of the random-effects distribution is insufficient: two meta-analytic data sets can produce identical mean estimates and standard errors while differing greatly in heterogeneity, so τ² and the whole distribution matter.
  • Distribution-free classical methods can estimate μ and τ² but cannot satisfactorily address prediction; the paper proposes an approximate prediction interval using a t-distribution with k−2 degrees of freedom.
  • In the 14-study set-shifting example, heterogeneity was Q=16.73, df=13, P=0.02, I²=22%, and τ²=0.022; the classical random-effects interval was 0.19–0.53, while the approximate prediction interval for a new study ranged from −0.01 to 0.74.
  • Bayesian normal random-effects analysis gave μ median 0.36 (95% credible interval 0.18–0.55), θ_new median 0.36 (−0.12–0.84), P(θ_new<0)=0.950, and τ² median 0.023 (0.000024–0.196); Bayesian credible intervals were wider because uncertainty in τ² was properly accounted for.

Methods

  • The paper is a methodological re-evaluation structured around estimation, prediction, and hypothesis testing, comparing distribution-free, classical parametric, Bayesian, and non-normal/flexible random-effects approaches.
  • The example uses 14 comparative studies of set-shifting ability in people with eating disorders versus healthy controls, taken from Roberts et al. (2007), with effect sizes as standardized differences in means and their standard errors.
  • Analyses include classical inverse-variance random-effects meta-analysis with a moment estimate of between-study variance, and Bayesian normal random-effects meta-analysis via MCMC in WinBUGS with a uniform prior for τ (>0) and a vague N(0,10³) prior for μ; sensitivity analyses used a t-distribution with 4 degrees of freedom, an inverse gamma prior, and a shrinkage prior.

Limitations

  • Stated: the Bayesian approach has problems including computational intensity and sensitivity to a priori judgements; prior choice can substantially affect results; model uncertainty is not accounted for; the normal random-effects assumption is strong and usually made without convincing evidence; few studies make τ² and μ imprecise; separating diversity from bias is difficult; and caution is warranted with funnel plot asymmetry.
  • Stated: the example data were available only as effect sizes and confidence intervals, from which standard errors were calculated; more detailed data should be explicitly modelled when available.
  • Evident: the example has only 14 studies and some large within-study standard errors; the paper notes that meta-analyses with very small numbers of studies (2–4) pose particular problems, and prediction intervals depend on the number of studies and within-study standard errors.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1111/j.1467-985x.2008.00552.x

Cites

94 of 94 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 31 cited sources checked · 0 supported · 0 flagged

  • Findingreasoning

    In the 14-study set shifting meta-analysis, the approximate prediction interval for the true effect in a new study ranges from −0.01 to 0.74.

    “The approximate prediction interval (12) for the true effect in a new study, however, ranges from −0.01 to 0.74, which is slightly less convincing.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.
  • Findingcitation

    Using an inverse gamma distribution for τ2 with parameters 0.001 and 0.001 produces a prediction interval from −0.008 to 0.73 (P(θnew<0)=0.973).

    “Alternatively using an inverse gamma distribution for τ2 with parameters 0.001 and 0.001 (Smith et al., 1995) produces a prediction interval from −0.008 to 0.73 (P(θnew<0)=0.973)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Smith et al., 1995 source

      Not checked in the free run.

  • Findingown data

    In the Bayesian normal random-effects meta-analysis of the set shifting data, the posterior median for τ2 is 0.023 with 95% credible interval (0.000024, 0.196).

    “| τ2 | 0.023 | (0.000024, 0.196) | — |”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.
  • Findingcitation

    In the 14-study set shifting meta-analysis, there is some heterogeneity between effect sizes, accounting for 22% of the variation in point estimates by using I2.

    “There is some heterogeneity between the effect sizes, accounting for 22% of the variation in point estimates by using the statistic I2 (Higgins and Thompson, 2002).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Higgins and Thompson, 2002

      Not checked in the free run.

    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.
  • Findingreasoning

    In the 14-study set shifting meta-analysis, a random-effects meta-analysis reveals a statistically significant benefit on average.

    “A random-effects meta-analysis reveals a statistically significant benefit on average, based on the inference in equation (13) regarding μ alone.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.
  • May combine two claims. 25
  • Classified as probably not a substantial claim. 29

  • May combine two claims.

    Findingown data

    Various sensitivity analyses have small effects on the results and do not materially affect the conclusions from the Bayesian meta-analysis.

    “Various sensitivity analyses have small effects on the results, though they do not materially affect the conclusions from the Bayesian meta-analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Using the shrinkage prior advocated by DuMouchel and Normand (2000) produces a prediction interval from −0.009 to 0.72 (P(θnew<0)=0.973).

    “using the shrinkage prior that was advocated by DuMouchel and Normand (2000) produces a prediction interval from −0.009 to 0.72 (P(θnew<0)=0.973)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites DuMouchel and Normand (2000)

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the 14-study set shifting meta-analysis, the estimate of the heterogeneity variance is 0.022 and the test for heterogeneity is statistically significant at conventional levels.

    “The estimate of the heterogeneity variance is 0.022 and the test for heterogeneity is statistically significant at conventional levels of significance.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.

  • May combine two claims.

    Findingnone given

    Using a t-distribution with 4 degrees of freedom for the random effects yields a prediction interval from −0.16 to 0.89 (P(θnew<0)=0.947).

    “Using a t-distribution with 4 degrees of freedom for the random effects yields a prediction interval from −0.16 to 0.89 (P(θnew<0)=0.947).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Bayesian normal random-effects meta-analysis of the set shifting data, results for μ and θnew are comparable with the classical results, with credible intervals being wider because the Bayesian analyses account for uncertainty in the estimation of τ2.

    “Results for μ and θnew are comparable with the classical results, with credible intervals being wider because the Bayesian analyses properly account for uncertainty in the estimation of τ2.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.

  • May combine two claims.

    Findingown data

    In the Bayesian normal random-effects meta-analysis of the set shifting data, the posterior median for μ is 0.36 with 95% credible interval (0.18, 0.55) and P(μ>0)=0.999.

    “| μ | 0.36 | (0.18, 0.55) | 0.999 |”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.

  • May combine two claims.

    Findingown data

    In the Bayesian normal random-effects meta-analysis of the set shifting data, the posterior median for θnew is 0.36 with 95% credible interval (−0.12, 0.84) and P(θnew>0)=0.950.

    “| θnew | 0.36 | (−0.12, 0.84) | 0.950 |”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The random effects in the meta-analysis are normally distributed.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Selection of covariates to explain heterogeneity can be difficult both practically and theoretically.

    “Although sources of heterogeneity should ideally be investigated (Thompson, 1994), selection of covariates to explain heterogeneity can be difficult, both practically (Higgins et al., 2002) and theoretically (Thompson and Higgins, 2002).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Thompson, 1994 source

      Not checked in the free run.

    • cites Higgins et al., 2002

      Not checked in the free run.

    • cites Thompson and Higgins, 2002 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Realistic interpretation of predictions from a random-effects model can be difficult.

    “Realistic interpretation of predictions from a random-effects model can, however, be difficult.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The heterogeneity variance τ2 describes the extent of heterogeneity and has a crucial role in assessing the degree of consistency of effects across studies.

    “This variance explicitly describes the extent of the heterogeneity and has a crucial role in assessing the degree of consistency of effects across studies, which is an element of random-effects meta-analysis that often receives too little attention.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    A single parameter cannot adequately summarize heterogeneous effects, so focusing on the mean effect alone is usually insufficient.

    “However, a single parameter cannot adequately summarize heterogeneous effects (Raudenbush and Bryk, 1985) and so focusing on μ alone is usually insufficient.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Raudenbush and Bryk, 1985 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The conventional test from a common effect meta-analysis may be misleading if there is heterogeneity and will not reject hypothesis (1) if effects are variable and centred on 0.

    “Although often useful, the test may be misleading if there is heterogeneity and in particular will not reject hypothesis (1) if effects are variable and centred on 0.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Meta-analyses are often built on an assumption that directions of effect are consistent across studies, yet formal tests of this supposition are not routinely undertaken.

    “Meta-analyses are often built on an assumption that directions of effect are consistent across studies (Peto, 1987), yet formal tests of this important supposition are not routinely undertaken.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Peto, 1987 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    An important rationale for meta-analyses is the possibility of increased power compared with individual studies to detect moderate effects.

    “An important rationale for meta-analyses is the possibility of increased power, compared with the individual studies, to detect moderate effects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationcitation

    The usual null hypothesis that all effects are equal is typically untenable, and homogeneity tests should be discouraged in favour of quantifying the extent of heterogeneity.

    “We have argued that such a null hypothesis is typically untenable and have discouraged the use of homogeneity tests in favour of quantifying the extent of heterogeneity (Higgins and Thompson, 2002).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Higgins and Thompson, 2002

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    In the absence of a specific distributional assumption for the random effects, confidence intervals rely on approximating distributions, sometimes based on the central limit theorem, which may not be realistic when there are few studies.

    “Point estimation of μ and τ2 poses no problem, although confidence intervals rely on approximating distributions, sometimes based on the central limit theorem, which may not be realistic when there are few studies.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The available effect estimates are unbiased for the study-specific effects θi.
    • The authors state this as an assumption: The estimated variances may be used in place of the true variances.

  • May combine two claims.

    Premisecitation

    NPML has been noted to be unstable and has not been widely adopted.

    “However, NPML has been noted to be unstable (Van Houwelingen et al., 1993) and has not been widely adopted.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van Houwelingen et al., 1993 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    NPML procedures provide a discrete distribution based on a finite number of mass points.

    “NPML procedures provide a discrete distribution that is based on a finite number of mass points (Laird, 1978; Böhning, 2005).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Laird, 1978 source

      Not checked in the free run.

    • cites Böhning, 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The assumption of a normal distribution for the random-effects distribution allows the calculation of confidence intervals and prediction intervals.

    “The assumption of a normal distribution for the random-effects distribution allows the calculation of confidence intervals and prediction intervals.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The random-effects distribution is normal.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    A second potential advantage of a Bayesian approach is the intuitive interpretation of posterior distributions in terms of probability, including the ready availability of probabilities that effects exceed specified levels.

    “A second potential advantage of a Bayesian approach is the intuitive interpretation of posterior distributions in terms of probability, including the ready availability of probabilities that effects exceed specified levels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    A particular advantage of the NPML approach is its ability to detect and incorporate outliers.

    “A particular advantage of the NPML approach is its ability to detect and incorporate outliers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Bayesian probabilities can address a variety of questions more readily than classical hypothesis tests.

    “Bayesian probabilities can address a variety of questions more readily than classical hypothesis tests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    An asymmetric funnel plot leads necessarily to a difference in point estimates between common effect and random-effects meta-analyses (using standard inverse variance methods), because random-effects methods give relatively more weight to less precise studies.

    “Whatever the reason for it, an asymmetric funnel plot leads necessarily to a difference in point estimates between common effect and random-effects meta-analyses (using standard inverse variance methods), because random-effects methods give relatively more weight to less precise studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Meta-analyses of a large number of large studies lend themselves to relatively complex models, since there may be strong evidence against a normal distribution assumption for the random effect.

    “Meta-analyses of a large number of large studies lend themselves to relatively complex models, since there may be strong evidence against a normal distribution assumption for the random effect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The posterior distribution for the underlying mean μ is readily available from a Bayesian meta-analysis.

    “The posterior distribution for the underlying mean μ is readily available from a Bayesian meta-analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    In practice, it can be difficult to separate diversity from bias, and this lack of identifiability often necessitates the use of the conventional random-effects model.

    “In practice, it can be difficult to separate diversity from bias, and it is partly this lack of identifiability that often necessitates the use of the conventional random-effects model.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The true (unbiased) effects Φi follow a normal distribution across studies.
    • The authors state this as an assumption: The study-specific bias components βi follow a normal distribution.
    • The authors state this as an assumption: The diversity and bias components are uncorrelated.
    • The authors state this as an assumption: The effect being evaluated in the ith study is the sum of the true effect and bias: θi=Φi+βi.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The posterior distributions for the study-specific effects θi are readily available from a Bayesian analysis and provide fully Bayesian shrunken estimates.

    “The posterior distributions for the study-specific effects θi are readily available from a Bayesian analysis. These provide fully Bayesian shrunken estimates, in which inference for each particular study is performed by ‘borrowing strength’ from the other studies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    An important distinction when considering heterogeneity of effect sizes is between the effects of diversity and bias.

    “An important distinction to make when considering heterogeneity of effect sizes is between the effects of diversity (variation in populations, interventions, exposures, outcome measures etc.) and bias (variation in the quality of the design and conduct of the studies) (Glasziou and Sanders, 2002; Greenland and O'Rourke, 2001).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Glasziou and Sanders, 2002

      Not checked in the free run.

    • cites Greenland and O'Rourke, 2001

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    No model for the random-effects distribution will be true.

    “No model for the random-effects distribution will be ‘true’, and we might seek a parsimonious model that adequately fits the observed data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Bayesian uncertainty does not extend to the choice of model for the random effects, which is more difficult to defend empirically when there are few studies.

    “However, this uncertainty does not extend to the choice of model for the random effects, which is more difficult to defend empirically when there are few studies.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The random-effects distribution is normal.

  • May combine two claims.

    Premisenone given

    Tests for heterogeneity have low power when there are few studies, and heterogeneity may be just as likely for small numbers of studies as it is for large numbers of studies.

    “Tests for heterogeneity have low power when there are few studies, and heterogeneity may be just as likely for small numbers of studies as it is for large numbers of studies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    When 2×2 contingency tables are available for binary outcomes, meta-analyses are appropriately performed by using binomial likelihoods.

    “For example, when 2×2 contingency tables are available for binary outcomes, meta-analyses are appropriately performed by using binomial likelihoods (Smith et al., 1995; Warn et al., 2002).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Smith et al., 1995 source

      Not checked in the free run.

    • cites Warn et al., 2002 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Bayesian posterior probabilities can depend heavily on the distribution of the assumed likelihood and prior specification.

    “Bayesian posterior probabilities in particular can depend heavily on the distribution of the assumed likelihood and prior specification.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The prior distributions of μ and τ2 are independent.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The derivation of predictive distributions for the effect θnew in a future study has received little methodological attention in a frequentist context.

    “The derivation of the predictive distributions for the effect θnew in a future study has received little methodological attention in a frequentist context.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    It is unclear how prior distributions should be specified, and Bayes factors can be very sensitive to them.

    “However, it is unclear how prior distributions should be specified, which is an important limitation since Bayes factors can be very sensitive to them (Kass and Raftery, 1995).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kass and Raftery, 1995 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Such an infinite population does not in reality exist, although the construct allows inference about treatment effects to be extended to a broader population than the studies at hand.

    “Such an infinite population does not in reality exist, although the construct does allow inference about treatment effects to be extended to a broader population than the studies at hand.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The true predictive distribution is a complex function of the degree of heterogeneity, the number of studies, and the within-study standard errors.

    “The true predictive distribution is a complex function of the degree of heterogeneity, the number of studies and the within-study standard errors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Bayes factors may be used to provide evidence specifically in favour of a null hypothesis.

    “A particular advantage is that they may be used to provide evidence specifically in favour of a null hypothesis.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    From a classical point of view, parametric random-effects meta-analysis is formally interpreted in the context of an imaginary universe or population of similar effects from which the effects in observed studies are independently and identically sampled.

    “From a classical point of view, interpretation of parametric random-effects meta-analysis is formally in the context of an imaginary ‘universe’, or ‘population’ of similar effects from which the effects in the observed studies are independently and identically sampled.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    A realistic predictive distribution must recognize that τ2 is estimated, usually from a small number of studies, and that imprecision in τ2 affects both terms in the variance of θnew.

    “A realistic predictive distribution must recognize, however, that τ2 is estimated, usually from a small number of studies, and imprecision in affects both terms in the variance of θnew.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Approximately, the predictive distribution of θnew follows a t-distribution with k−2 degrees of freedom.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The most parsimonious choice for the random-effects distribution is the normal distribution, requiring only two parameters for full specification.

    “Requiring only two parameters for its full specification, the most parsimonious choice is also the conventional choice, where f(Φ) is chosen to be a normal distribution: (10)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    It remains difficult to implement non-normal distributions outside an MCMC approach to the analysis.

    “Flexible distributions such as Bayesian semiparametric models are available, although it remains difficult to implement non-normal distributions outside an MCMC approach to the analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    A Bayesian approach has the additional advantages of flexibility, allowing incorporation of full uncertainty in all parameters (but not uncertainty in the model) and of yielding more readily interpretable inferences.

    “A Bayesian approach has the additional advantages of flexibility, allowing incorporation of full uncertainty in all parameters (but not uncertainty in the model) and of yielding more readily interpretable inferences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    In many situations, the usual normal distribution assumption for random effects is the most feasible assumption given the limited number of studies and the lack of strong evidence for or against the assumption.

    “In many situations this is the most feasible assumption given the limited number of studies and the lack of strong evidence for or against the assumption.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The rationale is not strong for opting for a common effect model when there are very few studies.

    “Thus the rationale is not strong for opting for a common effect model in this situation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Meta-analyses of very few studies need not default to a common effect model, since informative prior distributions for the extent of heterogeneity can be incorporated in a simple Bayesian approach.

    “Meta-analyses of very few studies need not ‘default’ to a common effect model, since informative prior distributions for the extent of heterogeneity can be incorporated in a simple Bayesian approach.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Certain common effect approaches have practical advantages over random-effects methods, such as being easier to implement, and the one-step 'Peto' method has desirable properties when binary events are rare.

    “Furthermore, certain common effect approaches have practical advantages over random-effects methods. For example, they are easier to implement, and the one-step ‘Peto’ method has been observed to have desirable properties when binary events are rare (Bradburn et al., 2006).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bradburn et al., 2006 source

      Not checked in the free run.

47 of 47

  • The authors state this as an assumption.

    If an effect exists then heterogeneity exists, although it may be negligible.

    “We assume a priori that if an effect exists then heterogeneity exists, although it may be negligible.”

  • The authors state this as an assumption.

    The available effect estimates are unbiased for the study-specific effects θi.

    “For generality, the situation that we address assumes only the availability of effect estimates that are unbiased for θi (i =1,…,k), with sampling variance (i.e. squared standard error) .”

  • The authors state this as an assumption.

    The estimated variances may be used in place of the true variances.

    “We shall follow the conventional assumption that estimates (which are independent of ) may be used in place of the true variances, and we use the two interchangeably.”

  • The authors state this as an assumption.

    The effect estimates from each study are normally distributed.

    “We are willing to assume normality of the effect estimates from each study, so(6)”

  • The authors state this as an assumption.

    The prior distributions of μ and τ2 are independent.

    “It is standard to assume prior independence of μ and τ2.”

  • The authors state this as an assumption.

    Approximately, the predictive distribution of θnew follows a t-distribution with k−2 degrees of freedom.

    “Again taking a t-distribution with k−2 degrees of freedom, we assume that, approximately, (11)”

  • The authors state this as an assumption.

    The between-study variance is common to each study.

    “The assumption of a common between-study variance for each study is a strong one that we return to in Section 8.1.”

  • The authors state this as an assumption.

    The underlying effects are exchangeable.

    “Fundamental to a Bayesian approach to random-effects meta-analysis is the concept of exchangeability. Exchangeability represents a judgement that the treatment effects may be non-identical but their magnitudes cannot be differentiated a priori (Skene and Wakefield, 1990; Gelman et al., 1995).”

  • The authors state this as an assumption.

    The random-effects distribution is normal.

    “Requiring only two parameters for its full specification, the most parsimonious choice is also the conventional choice, where f(Φ) is chosen to be a normal distribution: (10)”

  • The authors state this as an assumption.

    The random effects in the meta-analysis are normally distributed.

    “by assuming a normal distribution for the random effects”
    “Using a normal distribution for the underlying effects in different studies is a strong assumption”

  • The authors state this as an assumption.

    The true (unbiased) effects Φi follow a normal distribution across studies.

    “we represent this by using a normal distribution across the ‘true’ (unbiased) effects”

  • The authors state this as an assumption.

    The study-specific bias components βi follow a normal distribution.

    “Suppose that the study-specific bias components follow a normal distribution such that .”

  • The authors state this as an assumption.

    The diversity and bias components are uncorrelated.

    “assuming that the diversity and bias components are uncorrelated”

  • The authors state this as an assumption.

    The effect being evaluated in the ith study is the sum of the true effect and bias: θi=Φi+βi.

    “The effect being evaluated in the ith study, θi, is the sum of the true effect and bias: θi=Φi+βi.”
14 of 14

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Probability of a negative predicted effect has the wrong direction

    Minor

    The text says the probability that θnew is below zero is 0.950, but Table 2 assigns 0.950 to the probability that it is above zero. The same sentence describes the cumulative probability at zero as approximately 5%, supporting the table's direction.

  • Classical interval is labelled a credible interval

    Minor

    Table 1 labels the reported classical random-effects interval a credible interval, whereas the text identifies the results as a classical analysis, for which it elsewhere uses confidence-interval terminology.

2 of 2

Cites

Flagged first, then by citations

6 of the 100 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. DuMouchel, 1994, Hierarchical Bayes linear models for meta-analysis
  2. Morris, 1992, Hierachical models for combining information and for meta-analysis, Bayes. Statist., 4, 321
  3. Stein, 1956, Proc. 3rd Berkeley Symp, 197

Cited by

Sorted by citations

Related

Authors
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J.
Affiliations
MRC Biostatistics Unit (GB)
Corresponding author
Higgins, Julian P. T.
ORCID
Venue
Journal of the Royal Statistical Society Series A: Statistics in Society
Publisher
Oxford University Press (OUP)
Published
2008-07-24
Volume
172
Issue
1
Pages
137-159
ISSN
  • 0964-1998 (ISSN-L)
  • 1467-985X
DOI
10.1111/j.1467-985x.2008.00552.x
PMC
PMC2667312
OpenAlex
W2108116635
Semantic Scholar
b42717775886e436d97964fca187c3248684112e
Cited by
2,569
References
100
Citations by year
2026: 222 · 2025: 197 · 2024: 219 · 2023: 186 · 2022: 208 · 2021: 246 · 2020: 224 · 2019: 160 · 2018: 153 · 2017: 124 · 2016: 128 · 2015: 117 · 2014: 93 · 2013: 96 · 2012: 89
FWCI
38.18
Type
journal article
Language
en
Open access
Licence
academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
Locations
  • Journal of the Royal Statistical Society Series A (Statistics in Society) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Explore Bristol Research (publishedVersion)
  • Bristol Research (University of Bristol) (submittedVersion)
Funders
UK Medical Research Council (U.1052.00.011, U.1052.00.001, U.1052.00.005)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
random-effects meta-analysis, heterogeneity, Bayesian meta-analysis, prediction intervals, nonparametric inference, hypothesis testing, set-shifting, eating disorders, prior sensitivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review