CoChat
Sign Up
Cited by 400Climate variability and modelsMeteorological Phenomena and SimulationsClimate Change Policy and Economics

Risks of Model Weighting in Multimodel Climate Projections

Open access
Weigel, Andreas P., Knutti, Reto, Liniger, Mark A., Appenzeller, Christof·2010·Journal of Climate
DOI 10.1175/2010jcli3594.1
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract Multimodel combination is a pragmatic approach to estimating model uncertainties and to making climate projections more reliable. The simplest way of constructing a multimodel is to give one vote to each model (“equal weighting”), while more sophisticated approaches suggest applying model weights according to some measure of performance (“optimum weighting”). In this study, a simple conceptual model of climate change projections is introduced and applied to discuss the effects of model weighting in more generic terms. The results confirm that equally weighted multimodels on average outperform the single models, and that projection errors can in principle be further reduced by optimum weighting. However, this not only requires accurate knowledge of the single model skill, but the relative contributions of the joint model error and unpredictable noise also need to be known to avoid biased weights. If weights are applied that do not appropriately represent the true underlying uncertainties, weighted multimodels perform on average worse than equally weighted ones, which is a scenario that is not unlikely, given that at present there is no consensus on how skill-based weights can be obtained. Particularly when internal variability is large, more information may be lost by inappropriate weighting than could potentially be gained by optimum weighting. These results indicate that for many applications equal weighting may be the safer and more transparent way to combine models. However, also within the presented framework eliminating models from an ensemble can be justified if they are known to lack key mechanisms that are indispensable for meaningful climate projections.

Integrity signals

Registry

Crossref · 10.1175/2010jcli3594.1

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 32 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Knutti et al. (2010) found that half of typical surface temperature biases of climate models would remain even if an infinite number of models of the same quality were combined.

    “For example, Knutti et al. (2010) have shown that half of the typical surface temperature biases of climate models would remain, even if an infinite number of models of the same quality were combined.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Knutti et al. (2010) source
  • Premisecitation

    A weak link between current model performance and future projections has been reported in many studies.

    “many studies have found only a weak relation between present-day model performance and future projections (Ra ¨isa ¨nen 2007; Whetton et al. 2007;Jun et al. 2008;Knutti et al. 2010;Scherrer 2010)”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Ra ¨isa ¨nen 2007 source

      Section 3.3 finds little evidence of a relationship between simulated present-day mean temperatures and projected temperature changes outside high-latitude sea areas, and says earlier simulations support that conclusion. The paper describes the broader connection between present-day model performance and future-change skill as poorly understood, rather than establishing that many studies found a weak relationship across future projections generally. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Whetton et al. 2007 source
    • cites Jun et al. 2008 source
  • Premisecitation

    For weather and seasonal forecasts, model errors can be measured empirically from forecast error statistics over many independent verification cases.

    “On the shorter time scales of weather and seasonal forecasting, model errors exist equally, but their effects can be empirically quantified by sampling the forecast error statistics over a sufficiently large set of independent verification data (e.g., Raftery et al. 2005;Doblas-Reyes et al. 2005;Weigel et al. 2009).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Raftery et al. 2005 source

      The Introduction reports forecast-error statistics from weather-forecast verifications, and “Length of training period” describes estimating calibration parameters from historical data and evaluating them against separate verifications. The paper supports empirical quantification for weather forecasts, but the cited passages do not establish the claim for seasonal forecasts or specify a sufficiently large set of independent verification cases. Page numbers are unavailable in our copy. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Doblas-Reyes et al. 2005 source
    • cites Weigel et al. 2009 source
  • Premisecitation

    The full error cancellation limit is inconsistent with what has been observed in reality.

    “However, this limit of full error cancellation is not consistent with what has been observed in reality.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Knutti et al. (2010) source
  • Premisecitation

    Correlations between present-day model performance measured by alternative variables and future climate changes are weak.

    “Whetton et al. (2007), Jun et al. (2008), Knutti et al. (2010), and other studies, for example, show that the correlations between presentday model performance (in terms of such alternative variables) and future changes are in fact weak”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Whetton et al. (2007) source
    • cites Jun et al. (2008) source
    • cites Knutti et al. (2010) source
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 10
  • The quote only partly states this claim. 3

  • May combine two claims.

    Premisecitation

    In seasonal forecasting, performance-based weighting schemes have been successfully implemented and improve average prediction skill.

    “Given that, in seasonal forecasting, performance-based weighting schemes have been successfully implemented and have been demonstrated to improve the average prediction skill (e.g., Rajagopalan et al. 2002;Robertson et al. 2004;Stephenson et al. 2005;Weigel et al. 2008b)”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Rajagopalan et al. 2002 source
    • cites Robertson et al. 2004 source
    • cites Stephenson et al. 2005 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some studies have found more consistent projections when poor models are eliminated.

    “In fact, some studies have found more consistent projections when eliminating poor models (e.g., Walsh et al. 2008;Perkins and Pitman 2009;Scherrer 2010).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Walsh et al. 2008 source

      The Conclusions say that eliminating models with larger biases might narrow uncertainty in Arctic projections. They describe this as a suggested approach for future assessments, not as a demonstrated finding that projections became more consistent after poor models were eliminated. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Perkins and Pitman 2009 source
    • cites Scherrer 2010 source

  • May combine two claims. The quote only partly states this claim.

    Findingreasoning

    For r equal to 0.6, the optimal weight is 0.26 when R is 0, 0.33 when R is 0.5, 0.40 when R is 1, and approaches 0.5 for all r as R tends to infinity.

    “For instance, if noise is negligible (i.e., R 5 0), r 5 0.6 corresponds to w opt 5 0.26.However, for R 5 0.5 one has w opt 5 0.33, while for R 5 1 one has”
    • Also stated in: § c. The effect of unpredictable noise · plain
      “w opt 5 0.40, and for R / ' the optimum weight approaches 0.5 for all r.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingreasoning

    Optimally and randomly weighted multimodels converge to the same limit as equal weighting, but the optimally weighted one converges faster and the randomly weighted one slower.

    “Optimally and randomly weighted multimodels can be shown to approach the same limit.The only difference is that the optimally weighted multimodel would converge more quickly than the equally weighted one, while the randomly weighted multimodel would converge more slowly.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The model errors of M1 and M2 are independent of each other.
    • The authors state this as an assumption: The unpredictable noise terms n_M and n_x can be ignored.

  • Classified as probably not a substantial claim.

    Findingreasoning

    The expected MSE of a single model M equals 2 sigma_n^2 + sigma_M^2 under the framework assumptions.

    “S M 5 h(Dy M À Dx) 2 i 5 h(n M 1 M 1 n x ) 2 i 5 2s 2 n 1 s 2 M .(2)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The climate change signal Dx can be decomposed into a predictable signal Dm and unpredictable noise n_x.
    • The authors state this as an assumption: A model's projection Dy_M can be decomposed into Dm, internal variability n_M, and model error M.
    • The authors state this as an assumption: The noise terms n_x and n_M follow the same distribution with expectation 0 and standard deviation sigma_n.
    • The authors state this as an assumption: The noise and error terms n_x, n_M, and M are statistically independent from each other.
    • The authors state this as an assumption: The terms n_x, n_M, and M have expectation 0.

  • May combine two claims.

    Findingreasoning

    As R increases, the difference between single-model expected squared errors S_M1 and S_M2 decreases and the models become more similar in net skill.

    “As R increases, the difference between S M1 and S M2 decreases and the two models become more similar in terms of their net skill, because the individual model error terms M1 and M2 lose are diminished in relative importance with respect to the unpredictable noise.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingreasoning

    For r equal to 0.6, the optimal weight of M1 is about 0.26 when j is 0, about 0.19 when j is 0.4, and zero when j equals r equals 0.6.

    “For example, if the model errors M1 and M2 are fully independent ( j 5 0), an error ratio of r 5 0.6 would correspond to an optimum weight of approximately 0.26.However, w opt would drop to 0.19 if j 5 0.4, that is if 40% of the root-mean-squared error of M1 contributes to the root-mean-square error of M2 ; and w opt would be zero if j 5 r 5 0.6.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The joint error and the independent residual errors are mutually independent.

  • May combine two claims.

    Findingreasoning

    With infinitely many equally weighted models under noise, the expected multimodel projection approaches Dm and the expected MSE approaches s_n squared.

    “lim m!' S (m) eq 5 lim m!' s 2 n 1 1 m 2 å m i51 (s 2 M,i 1 s 2 n )”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingreasoning

    Without noise, equal weighting outperforms the better single model only if r is at least about 0.58; with R equal to 0.5, for all r at least about 0.40; and if R is at least about 0.71, equal weighting outperforms both single models for any r in [0,1].

    “Indeed, in section 3a it has been noted that, under the absence of unpredictable noise, S (2) eq # S M2 only if r $ ffiffiffiffiffiffi ffi 1/3 p ' 0.58.However, if R 5 0.5, then S (2) eq # S M2 for all r $ ffiffiffiffiffiffi ffi 1/6 p ' 0.40; and if R $ ffiffiffiffiffiffi ffi 0.5 p ' 0.71, then the equally weighted multimodel outperforms both single models for any r 2 [0, 1].”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingown data

    For two models, the optimally weighted multimodel has a lower expected mean squared error than either single model and than equal weighting for every model error ratio r.

    “The curve of S opt as a function of r has been included in Fig. 2 (solid gray line), showing that the optimally weighed multimodel clearly outperforms S M1 , S M2 , and S (2) eq for all r.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The model errors of M1 and M2 are independent of each other.
    • The authors state this as an assumption: The unpredictable noise terms n_M and n_x can be ignored.

  • May combine two claims.

    Findingreasoning

    For very large or very small error ratios, w_opt approaches 1 or 0, implying all weight is put on M1 or M2.

    “For very large (very small) error ratios, on the other hand, w opt approaches 1 (0), implying that all weight is put on M1 (M2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Using simplistic weights from Eq. 9 instead of Eq. 16 yields squared errors larger than optimal weighting but still lower than equal weighting.

    “Finally, note that the application of ''simplistic'' weights derived from Eq. ( 9) rather than Eq. ( 16) implies squared errors, which are larger than S (2) opt , but still lower than S eq .”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The joint error and the independent residual errors are mutually independent.

  • May combine two claims.

    Findingreasoning

    Under noise, optimally and randomly weighted multimodels converge to the same limit, with optimal faster and random slower.

    “Optimally and randomly weighted multimodels can be shown to approach the same limit.The only difference is that the optimally weighted multimodel converges more quickly than the equally weighted one, while the randomly weighted multimodel converges more slowly.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingown data

    If the amount of independent training data is systematically reduced, weight estimates become more uncertain and average prediction skill drops.

    “However, if the amount of independent training data is systematically reduced, the weight estimates become more uncertain and the average prediction skill drops (see Table 1 for skill values).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Averaging ensemble members before model combination reduces the noise contribution to total projection uncertainty but leaves the model error contribution.

    “Thus, in comparison to Eq. ( 2) the contribution of noise to the total projection uncertainty is strongly reduced, but the contribution of model error remains.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingreasoning

    Under noise and joint errors, the optimal multimodel mean converges to Dm plus j rather than Dm, and the MSE limit is s_j squared plus s_n squared rather than s_n squared.

    “In this case, the optimum multimodel mean would converge to (Dm 1 j ) rather than Dm, and the limit of the MSE would be (s j 2 1 s n 2 ) rather than s n 2 .”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Findingreasoning

    Having many ensemble members increases the weight given to the better model and reduces the weight given to the weaker model, as observed in seasonal forecasting by Weigel et al. (2007).

    “In other words, the availability of many ensemble members increases (reduces) the weight to be put on the better (weaker) of the two models-a pattern of behavior that has already been observed and discussed within the context of seasonal forecasting by Weigel et al. (2007).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Weigel et al. (2007) source
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Interpretationreasoning

    Model combination can cancel noise terms from internal model variability but cannot cancel observational noise n_x.

    “This is plausible, because model combination can only cancel out the noise terms n M stemming from internal model variability; the unpredictable noise of the observations, n x , remains.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Interpretationreasoning

    As R increases, optimal compensation of noise errors becomes more important for minimizing total projection error, and under the assumptions noise errors are optimally reduced by equal weighting.

    “Thus, as R increases, the optimum compensation of noise errors becomes more and more important for the minimization of the total projection error, and under the assumptions made, the noise errors are optimally reduced by equal weighting.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Premisereasoning

    In short-range forecasting, an objective estimate of forecast uncertainty and model quality is possible, backed by past measurements of model performance in comparable cases.

    “In this way, an objective estimate of the forecast uncertainty and thus of model quality is possible; the confidence we put into the accuracy of a model projection is backed up by past measurements of model performance in comparable cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A multimodel can at best provide an unbiased estimate of the predictable signal Dm, but not the actual outcome Dx.

    “A multimodel can thus at best provide an unbiased estimate of the predictable signal Dm, but not the actual outcome Dx.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • Classified as probably not a substantial claim.

    Premisereasoning

    In short-range forecasting, uncertainty is defined by repeated reproducible hindcast experiments and follows a frequentist/physical perception as a measurable quantity.

    “In the former, uncertainty is defined by long series of repeated and reproducible hindcast experiments and thus follows the relative frequentists' or physical perception of uncertainty, in the sense of a measurable quantity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Model weighting is relatively straightforward in short-range forecasting but controversial on climate change time scales because it requires a robust definition of model uncertainty.

    “It is for exactly this reason that the concept of model weighting, which requires a robust definition of model uncertainty, is relatively straightforward in short-range forecasting but so controversial on climate change time scales.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The effective skill difference between M1 and M2 is characterized by the ratio of independent residual errors rather than total model errors, and this ratio grows as j increases for r at most 1.

    “That is, the error ratio characterizing the effective skill difference between M1 and M2 is no longer given by (s M2 /s M1 ), but rather by (s9 M2 /s9 M1 ), which grows as j is increased (for r # 1).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The joint error and the independent residual errors are mutually independent.

  • May combine two claims.

    Interpretationreasoning

    Neglecting joint model errors when formulating optimal weights biases w_opt, assigning too little weight to the better model and too much to the poorer one.

    “In summary, when the existence of joint model errors is neglected in the formulation of the optimum model weights, then the resulting estimates of w opt would be implicitly biased.Too little weight would be assigned to the better one of the two models, and too much weight to the poorer one.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The joint error and the independent residual errors are mutually independent.

  • May combine two claims.

    Interpretationreasoning

    Neglecting noise effects when formulating optimal weights biases w_opt, with the bias growing quickly as R increases, assigning too much weight to the better model and too little to the poorer one.

    “In summary, when the effects of noise are neglected in the formulation of optimum model weights, then the resulting estimates of w opt are implicitly biased, with the bias growing quickly as R becomes larger.The bias is such that too much weight would be given to the better one of the two models, and too little weight to the poorer one.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.
    • The authors state this as an assumption: The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

  • May combine two claims.

    Interpretationreasoning

    Any uncertainty estimate of M is necessarily based on unprovable assumptions and is inherently subjective and volatile.

    “Any uncertainty estimate of M is therefore necessarily based on an array of unprovable assumptions and thus is inherently subjective-and volatile.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Without a uniquely determined model error uncertainty, defining optimum weights and the concept of model weighting would be ill-posed by principle.

    “Without the existence of a uniquely determined model error uncertainty, the task of defining optimum weights and thus the concept of model weighting in general would be ill-posed by principle.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The inability to sample M is due to the sample size problem and the out-of-sample problem.

    “This is for two reasons: (i) the ''sample size problem,'' that is, the fact that the long time scales involved reduce our sample size of independent past observations, and (ii) the ''out of sample problem,'' that is, the fact that any conclusion drawn on the basis of past and present-day observations needs to be extrapolated to so far unexperienced climate conditions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Correlated model errors have only a minor impact on the relative performance of weighted versus unweighted multimodels, but a major impact on absolute multimodel performance compared with single models.

    “In summary, correlated model errors have only a minor impact on the results in section 3a concerning the relative performance of weighted versus unweighted multimodels; however, they have a major impact on the absolute multimodel performance in comparison to the single models.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The joint error and the independent residual errors are mutually independent.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The central conceptual difference between short-range forecasts and long-range climate projections is their different definitions of uncertainty.

    “Thus, the central conceptual difference between the interpretation of short-range forecasts of weeks and seasons and long-range projections of climate change is in their different definitions of ''uncertainty.''”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    In long-range climate projections, uncertainty is partially subjective and depends on prior assumptions and expert opinion, following a Bayesian perception.

    “In the latter, uncertainty is partially subjective and depends on prior assumptions as well as expert opinion, thus following the Bayesian perception of uncertainty.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    Multimodels can only reduce independent error components, whose magnitude decreases as the joint error fraction increases.

    “This is plausible, since multimodels can only reduce the independent error components, whose magnitude decreases as j is increased.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Each model error decomposes into a joint error common to all models and an independent residual error.
    • The authors state this as an assumption: The joint error and the independent residual errors are mutually independent.

  • Classified as probably not a substantial claim.

    Premisenone given

    Weights can be judged correct or false only when the uncertainties they represent are uniquely defined.

    “However, a weight can only be called correct or false if the underlying uncertainties to be represented by the weights are well defined and uniquely determined.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Given these problems, equal-weight model combination cannot be considered a final solution but rather a starting point for further discussion and research.

    “Given these problems and challenges, model combination with equal weights cannot be considered to be a final solution, either, but rather a starting point for further discussion and research.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The results do not imply that deriving performance-based weights is impossible in principle.

    “These results do not imply that the derivation of performance-based weights is impossible by principle.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    If a model M1 is known a priori to lack key mechanisms indispensable for correct climate projections while another model M2 includes them, it may be justifiable to assume that M2's error variance is smaller than M1's and that their error correlation is zero.

    “If we know a priori that a given model M1 cannot provide a meaningful estimate of future climate while another model M2 can (e.g., because M1 is known to lack important key mechanisms that are indispensable to providing correct climate projections, while M2 has them included), then it may be justifiable to assume that s M2 ( s M1 and thus r 5 0.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Any estimate of r is necessarily based on past and present model performance and must be extrapolated into the future to be useful for weighting.

    “Any estimate of r is to some degree necessarily based on the assessment of past and present model performance, and it needs to be extrapolated into the future to be of use for model weighting.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    The relevance of large R and equal weighting under large noise is particularly high for variables with low signal-to-noise ratios, small spatial and temporal aggregations or high detail, short lead times, and when no ensembles are available to sample initial-condition uncertainty.

    “In practice, this is particularly relevant if variables with low signal-to-noise ratios are considered (e.g., precipitation rather than temperature), if high spatial and temporal detail is required, if the lead times are short, and if no ensemble members are available to sample the uncertainty of the initial conditions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The results are valid only to the degree that the underlying assumptions hold.

    “As all results presented above are based on a simple conceptual framework, they are as such only valid to the degree that the underlying assumptions hold.”
    Quote in the paperSubstantial claim
40 of 40

  • The paper relies on this without support.

    There is no consensus on how model weights should be obtained and it is unclear whether appropriate weights can be obtained with current data and methods.

    “So far there is no consensus on what is the best method of combining the output of several climate models.”
    “it seems that at present there is no consensus on how model weights should be obtained, nor is it clear that appropriate weights can be obtained at all with the data and methods at hand.”

  • The authors state this as an assumption.

    The climate change signal Dx can be decomposed into a predictable signal Dm and unpredictable noise n_x.

    “We decompose Dx, the predictand, into the sum of a potentially predictable signal, Dm, and an unpredictable ''noise'' term, n x : Dx 5 Dm 1 n x .”

  • The authors state this as an assumption.

    A model's projection Dy_M can be decomposed into Dm, internal variability n_M, and model error M.

    “Formally, Dy M can then be decomposed into the sum of the predictable signal Dm, a random noise term n M (often referred to as internal variability), and a residual error term M .”

  • The authors state this as an assumption.

    There is no scenario uncertainty; the model is subject to the same external forcing as reality.

    “Assume that there is no scenario uncertainty, that is, that M is subject to the same changes in external forcing as reality.”

  • The authors state this as an assumption.

    The noise terms n_x and n_M follow the same distribution with expectation 0 and standard deviation sigma_n.

    “For simplicity, we assume that both n x and n M follow the same (not necessarily Gaussian) distribution with expectation 0 and standard deviation s n”

  • The authors state this as an assumption.

    Enough data or many universes are available so that the uncertainty range of M can be fully sampled and defined in a frequentist sense, and frequentist and Bayesian interpretations converge.

    “we simply assume that enough data were available, or, as Smith (2002) and Stainforth et al. (2007) put it, that we had access to many universes so that the uncertainty range of M can be fully sampled and defined in a relative frequentists' sense; that is, we assume that enough information was available such that the relative frequentists' and Bayesian interpretations of model uncertainty converge.”

  • The authors state this as an assumption.

    The true model uncertainty sigma_M exists as a unique physical propensity.

    “we assume that s M exists in the sense of a unique physical propensity as defined by Popper (1959).”

  • The authors state this as an assumption.

    The noise and error terms n_x, n_M, and M are statistically independent from each other.

    “we assume that (i) the noise and error terms n x , n M , and M are statistically independent from each other”

  • The authors state this as an assumption.

    The terms n_x, n_M, and M have expectation 0.

    “and (ii) that not only n x and n M , but also M , have expectation 0.”

  • The authors state this as an assumption.

    After removing internal variability, a model's expected mean bias during the scenario period equals its observed mean bias during the control period (constant biases).

    “The latter assumption implies that, after removing the effects of internal variability, the expected mean bias of a model during the scenario period is the same as the observed mean bias during the control period (otherwise a nonzero M would be expected).This assumption of ''constant biases'' has recently been questioned”

  • The authors state this as an assumption.

    The unpredictable noise can be ignored (n_x = n_M = 0) in section 3a.

    “Assume that the unpredictable noise can be ignored (i.e., n x 5 n M 5 0).”

  • The authors state this as an assumption.

    The model errors M1 and M2 are independent.

    “Assuming independence of M1 and M2”

  • The authors state this as an assumption.

    The model errors of M1 and M2 are independent of each other.

    “So far, we have assumed that the model errors M1 and M2 are independent of each other”
    “we return to the assumption of independent model errors; that is, j 5 0.”

  • The authors state this as an assumption.

    The unpredictable noise terms n_M and n_x can be ignored.

    “and that the unpredictable noise n M and n x can be ignored.”

  • The authors state this as an assumption.

    Each model error decomposes into a joint error common to all models and an independent residual error.

    “Assume now that for each climate model M contributing to the multimodel, the model error M can be decomposed into a joint error contribution j , which is common to all models, and an independent residual error term 9 M ; that is, M 5 j 1 9 M .”

  • The authors state this as an assumption.

    The joint error and the independent residual errors are mutually independent.

    “Assuming mutual independence of 9 M1 , 9 M2 and j”

  • The authors state this as an assumption.

    The noise terms n_x, n_M1, and n_M2 are independent samples from a distribution with mean zero and standard deviation s_n.

    “we assume that the noise terms n x , n M1 , and n M2 are independent samples from a distribution with expectation 0 and standard deviation s n .”

  • The authors state this as an assumption.

    The noise terms n_x, n_M1, and n_M2 are samples from the same distribution with variance s_n squared.

    “we have made the simplifying assumption that the noise terms n x , n M1 , and n M2 are samples from the same distribution with variance s n 2 .”

  • The authors state this as an assumption.

    Emission uncertainty is entirely ignored.

    “our most unrealistic assumption is that the emission uncertainty has been entirely ignored.”

  • The authors state this as an assumption.

    The model error ratio r is stationary under a changing climate.

    “the assumption must be made that r is stationary under a changing climate”

  • The paper relies on this without support.

    The idealized framework used in the study is representative enough of real climate change projections that its generic results apply to actual multimodel climate projections.

    “This framework has been designed such that it allows us to assess, in generic terms, the effects of multimodel combination independently of the model error magnitudes, the degree of model error correlation, and the amount of unpredictable noise (internal variability).”

  • The paper relies on this without support.

    Seasonal forecasting results are a valid analogue for climate change projection weighting.

    “Also this aspect finds its equivalent in the context of seasonal forecasting.”
22 of 22

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Joint-error ratio is said to move in the wrong direction

    Minor

    The residual-error ratio falls, rather than grows, as the shared-error fraction increases when r is below 1. The paper's own example moves toward giving all weight to M2 as j reaches r; at that point M2's residual error, and thus the ratio, is zero.

1 of 1

Cites

Flagged first, then by citations

1 of the 57 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. Roulston, 2002, Evaluating probabilistic forecasts using information theory., Mon. Wea. Rev., 130, 1653, 10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2 · DOI

Cited by

Sorted by citations

Related

Authors
Weigel, A. P., Knutti, R., Liniger, M. A., & Appenzeller, C.
Affiliations
  • Federal Office of Meteorology and Climatology MeteoSwiss (CH)
  • ETH Zurich (CH)
ORCID
Venue
Journal of Climate
Publisher
American Meteorological Society
Published
2010-08-01
Volume
23
Issue
15
Pages
4175-4191
ISSN
  • 0894-8755 (ISSN-L)
  • 1520-0442
DOI
10.1175/2010jcli3594.1
OpenAlex
W2085893612
Cited by
386
References
57
Citations by year
2026: 13 · 2025: 24 · 2024: 23 · 2023: 33 · 2022: 26 · 2021: 37 · 2020: 22 · 2019: 22 · 2018: 25 · 2017: 19 · 2016: 17 · 2015: 17 · 2014: 20 · 2013: 36 · 2012: 27
FWCI
25.32
Type
journal article
Language
en
Open access
Locations
Journal of Climate (publishedVersion)
Funders
  • National Science Foundation
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
Topics
  • Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
  • Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Climate Change Policy and Economics
Keywords
model weighting, equal weighting, climate change projections, model uncertainty, natural climate variability, conceptual climate model, projection error, model elimination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review