Skill of Multimodel ENSO Probability Forecasts

Classified as probably not a substantial claim.

Earlier work has shown that forecasts combining several models usually outperform forecasts from one model.

“Numerous studies have demonstrated that multimodel forecasts are generally more skillful than single-model forecasts (Krishnamurti et al. 1999;Kharin and Zwiers 2002;Palmer et al. 2004;van Oldenborgh et al. 2005;Kug et al. 2007).”

• cites Krishnamurti et al. 1999Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kharin and Zwiers 2002Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Palmer et al. 2004 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The result that ensemble-mean variations dominate is consistent with conclusions from other recent studies of other forecast variables.

“This finding is consistent with the conclusions of other recent studies of other forecast variables (Kharin and Zwiers 2003;Hamill et al. 2004;Tippett et al. 2007;Tang et al. 2007).”

• cites Kharin and Zwiers 2003Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Hamill et al. 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Tippett et al. 2007 sourceSupported

May combine two claims.

Hagedorn et al. (2005) showed with the ECMWF model that increasing single-model ensemble size raised skill, but the multimodel ensemble still had a slight advantage.

“Using the ECMWF model, Hagedorn et al. (2005) demonstrated that while increasing the ensemble size of a single model did increase skill, the multimodel ensemble retained a slight advantage over the single model ensemble.”

• cites Hagedorn et al. (2005) sourcePartly supported

  The passage labeled Abstract in our indexed copy says the DEMETER analyses found improvements from multi-model over single-model ensembles. It does not establish the claimed ECMWF-specific comparison, that increasing the single-model ensemble size raised skill, or that the remaining multi-model advantage was slight; searches for those details returned no relevant results passage. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

May combine two claims.

Methods that build probabilities solely from the ensemble mean (MM-c and MM-glr) perform nearly as well as MM-vc but usually not better.

“Two schemes that use only the ensemble means to construct the probability distribution (MM-c and MM-glr) have skill very close to that of MM-vc, but generally do not exceed it.”

May combine two claims.

Equalizing the interensemble variances across models tends to add a small further gain in RPSS and ROR, but the effect is not always positive.

“Making the interensemble variances of each model identical (MM-vc) tends to slightly further increase RPSS and ROR skills, although the effect is not consistently positive.”

May combine two claims.

In that 1983 case, MM-vc assigned slightly weaker cold-category probabilities than MM-c and therefore received a smaller penalty.

“The MM-vc forecast had slightly weaker probabilities for the cold category than did MM-c and so was penalized less.”

May combine two claims.

Independent calibration methods generally have comparable skill, with grsep performing best overall and its ROR slightly above MM-vc.

“For the most part, the independent calibration methods have comparable skill with grsep having the best overall performance, with its ROR slightly exceeding that of MM-vc.”

May combine two claims.

The average RPSS of the individual models across all starts and leads is 0.47, or 0.50 when MPI is excluded.

“The average RPSS for the individual models over all starts and leads is 0.47 (0.50 excluding MPI).”

May combine two claims.

Bayesian regression methods achieved comparable performance and are designed to avoid overfitting.

“Comparable performance was achieved by the Bayesian regression methods, which also are designed to avoid overfitting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In some cases, the numerical method used to estimate glro parameters does not converge.

“Moreover, in some cases the numerical method for estimating the glro parameters fails to converge.”

May combine two claims.

MM-bow has similar problems and somewhat poor skill relative to the other independent calibration methods.

“The MM-bow method suffers from similar problems and has somewhat poor skill compared to the other independent calibration methods.”

May combine two claims.

Joint calibration methods mostly have comparable skill, with only occasional improvement over MM-vc, especially for November starts.

“The joint calibration methods have mostly comparable skills with only occasional improvement on that of MM-vc especially in the November starts.”

May combine two claims.

Setting the expected RPSS for ensemble size 9 to 0.5 yields an infinite-ensemble RPSS of 0.55 and an ensemble-size-63 RPSS of 0.54.

“Setting ͗RPSS(9)͘ ϭ 0.5, gives ͗RPSS(ϱ)͘ ϭ 0.55, and ͗RPSS(63)͘ ϭ 0.54.”

May combine two claims.

When cross-validation is not used, multiple linear regression results are excellent and usually exceed MM-vc, indicating overfitting to the short sample.

“Apparently, the period of data ( 22 OCTOBER 2008 training sample (i.e., when cross validation is not used), results (not shown) using multiple linear regression are excellent, and usually exceed those of MM-vc.This indicates overfitting to the short sample data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In one forecast shown in Fig. 3, multimodel probabilities are less extreme than those from a single model.

“Figure 3 shows a particular forecast where multimodel probabilities are less extreme than those of a single model.”

May combine two claims.

Much of the useful skill in these multimodel ENSO probability forecasts comes from ensemble-mean variations rather than higher-order statistics, since MM-c and MM-glr are only slightly less skillful than MM-vc.

“Figure 2 indicates that much of the useful forecast skill can be attributed to variations in the ensemble mean rather than variations in higher-order statistics, as MM-c and MM-glr skill tends to be only very slightly lower than MM-vc.”

May combine two claims.

Although ENSO predictive skill is generally low in boreal summer, the advantage of model forecasts over historical-frequency forecasts is greatest then.

“Interestingly, while ENSO predictive skill is generally low during boreal summer relative to other times of the year, the advantage of the model forecasts over forecasts based on historical frequencies is greatest at this time.”

May combine two claims.

Forecast distributions built from the multimodel ensemble mean alone had nearly as much skill as those built from the full forecast ensemble, indicating that much useful forecast information is in the forecast-distribution mean.

“Forecast distributions constructed from the multimodel ensemble mean alone had nearly as much skill as ones constructed from the forecast ensemble, indicating that much of the useful forecast information is contained in the mean of the forecast distribution.”

May combine two claims.

Some models in this study appeared generally somewhat more skillful than others, but none had unusually low skill.

“While some of the models used in this study appeared to have generally somewhat higher skills than others, none had unusually low skill.”

May combine two claims.

Reducing predictors or incorporating prior information about weights were effective ways to prevent overfitting.

“Reducing the number of predictors or incorporating prior information about the weights were effective ways to prevent overfitting.”

May combine two claims.

The rrmmr method is slightly better overall than MM-g and is consistently better when the penalty weight is chosen using all the data.

“The Bayesian regression method that penalizes unequal weights, rrmmr, is slightly better over all than MM-g.It is consistently better than MM-g if the penalty weight is chosen using all the data.”

May combine two claims.

Overall, independent calibration outperformed joint calibration on both RPSS and ROR unless overfitting-prevention steps were used.

“Overall, independent calibration performed better than joint calibration for both the RPSS and for the ROR score unless steps were take to avoid overfitting.”

May combine two claims.

MM-vc is taken as the reference ensembling method because it broadly corrects each model's mean and interannual-variance biases while keeping its own ensemble spread and shape.

“We therefore consider the MM-vc as the benchmark among the ensembling methods to be tested below, as it represents a most general and basic calibration of the models' individual biases in mean and interannual variability, while retaining the individual ensemble distributions with their year-to-year variations of spreads and shapes.”

Classified as probably not a substantial claim.

The lower skill of February lead-1 forecasts may be related to existing ENSO episodes weakening toward neutral at that time, a tendency that simple frequency forecasts can capture.

“This could relate to the fact that it is a time of year when existing ENSO episodes are in a process of weakening toward neutral, a tendency that can be captured by the simple frequency forecasts.”

May combine two claims.

The frequency forecast treats both ENSO transition possibilities indiscriminately using all historical data, whereas dynamical models can use initial conditions, such as subsurface tropical Pacific temperatures, to help identify the direction of ENSO change.

“The frequency forecast incorporates both possibilities indiscriminately from the entirety of the data history, while the dynamical models have the opportunity to respond to initial conditions that may help identify the direction of change in the ENSO condition (e.g., from subsurface tropical Pacific sea temperature structure).”

• AssumptionRests on an assumption: May is the northern spring ENSO predictability barrier, when warm or cold ENSO episodes are roughly equally likely to be dissipating or growing toward later maturity.

Classified as probably not a substantial claim.

Reducing the number of predictors, and therefore the number of parameters to estimate, is one way to avoid overfitting.

“One way to avoid overfitting is to reduce the number of predictors and hence the number of parameters that need to be estimated.”

May combine two claims.

Since glro probabilities sometimes approach zero or one, ROR penalizes them heavily when they are incorrect.

“Because the glro probabilities sometimes approach zero or one, the ROR scores them harshly when they are wrong.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The conditional-climatology benchmark uses frequencies from the separate 1900-79 period.

“The historical frequencies are computed using the independent period 1900-79.”

Classified as probably not a substantial claim.

The penalty weight for a particular forecast is chosen in a second level of cross-validation using data that excludes that forecast.

“The choice of the weight given to the penalty is chosen in a second level of cross validation, that is, the penalty parameter for a particular forecast is chosen using data that does not include that forecast.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Historical records of model performance can sometimes help improve model output.

“Past model performance data can sometimes be used to improve model output.”

• cites Glahn and Lowry 1972Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The glro behavior appears to result from the small sample size and from using 0/1 predictands.

“This behavior appears to be an effect of the small sample size and the fact that the predictands used to develop the model are 0 or 1.”

Classified as probably not a substantial claim.

For each of the three correction schemes, the bias and variance adjustments for a forecast exclude that forecast itself.

“In all three correction schemes, the bias and/or variance used to correct a particular forecast are computed without using that forecast (i.e., cross validation is always used).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In rrmm and rrmmr, a second cross-validation step sets the penalty weight.

“For both the rrmm and rrmmr methods, the parameter determining the relative weight of the penalty term is computed using a second level of cross validation.”

May combine two claims.

Coupled models differ in how they reproduce interannual variability and in their oceanic and atmospheric forecasts.

“Therefore, different models have differing biases in their reproduction of interannual variability, as well as in their forecasts of oceanic and atmospheric climate (Shukla et al. 2000).”

• cites Shukla et al. 2000 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The quartile thresholds are derived by cross-validation, leaving out the forecast being verified while using all others.

“Quartile boundaries are computed in a cross-validated fashion so that the current forecast is not included but all the other ones are.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Kang and Yoo (2006) found that the superensemble method overfits more severely when skill is low.

“Similar behavior was seen by Kang and Yoo (2006) who, using an idealized climate models and deterministic skill scores, noted that the superensemble method leads to overfitting that was more severe when skill is low.”

• cites Kang and Yoo (2006)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Because of uncertainty, forecasts must be probabilistic, and a probability distribution gives the fullest description of a forecast.

“Therefore, forecast information is necessarily probabilistic and the most complete description of a forecast is its probability distribution.”

• AssumptionRests on an assumption: Uncertainty in ENSO forecasts comes from errors in models and observations and from chaos inherent in the ocean-atmosphere system.

May combine two claims.

The penalty parameter does not appear to be optimally estimated, because if it were, rrmm would always beat MM-g, which it does not.

“This parameter does not seem to be optimally estimated since, if it were, rrmm would always be better than MM-g (it is not) since MM-g is a special case of rrmm with infinite weight given to the penalty term.”

May combine two claims.

Bayesian regression methods penalize deviations from equal weighting and are less likely to overfit.

“The Bayesian regression methods penalize deviations from equal weighting and are less likely to overfit.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MOS has a long history of removing bias and systematic error from deterministic forecasts.

“For instance, model output statistics (MOS; in its simplest form, a regression between forecast and verifying observations) has long been used to correct biases and systematic errors in deterministic forecasts (Glahn and Lowry 1972).”

• cites Glahn and Lowry 1972Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

More recently, ensemble MOS approaches have been introduced to adjust the whole forecast distribution.

“Recently “ensemble MOS” methods have been developed to correct the entire forecast distribution (Wilks 2006;Wilks and Hamill 2007).”

• cites Wilks 2006 sourceNot checked — no full text of the source
• cites Wilks and Hamill 2007Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Independent calibration and joint calibration with overfitting prevention can be used without sacrificing skill when forecasting outside the training sample.

“Rather, independent calibration, as well as joint calibration with overfitting prevention measures, can be used without skill sacrifice when forecasting outside of the training sample.”

May combine two claims.

Joint calibration estimates seven parameters simultaneously, whereas independent calibration estimates one parameter at a time.

“Joint calibration estimates seven parameters simultaneously from the data while independent calibration estimates a single parameter at a time from the data.”

May combine two claims.

Skill results for grsep, rrmm, rrmmr, and cca are relatively resistant to overfitting and can represent expected skill on independent data, but they are generally not superior to equal-weighting MM-vc despite apparent model skill differences.

“Thus, while skill results for some of the methods are relatively resistant to overfitting (grsep, rrmm, rrmmr, cca) and can be considered representative of expected skill on truly independent (e.g., future) data, a disappointment is that they are generally not superior to the skill of methods with equal weighting (MM-vc) despite apparent skill differences among the models (Fig. 4).”

May combine two claims.

Weigel et al. (2007) used different assumptions, but their debiased RPSS agrees to leading order with the result in this paper.

“Although Weigel et al. (2007) use a different set of assumptions, their debiased RPSS agrees to leading order with the result here.”

• cites Weigel et al. (2007) sourceNot checked — no full text of the source

May combine two claims.

Using the most general joint calibration schemes for seven models with 20-25 years of model history gains little or nothing and could lose skill.

“The conclusion and recommendation are that little or nothing is gained, and something could be lost, in attempting to use the most general joint calibration schemes for seven models contributing to multimodel ensembles based on 20-25 yr of model history.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ROR attaches an investment value to forecast information and ranks forecasts the same way as the ignorance or log-likelihood skill score.

“ROR is a skill measure that attaches an investment value to the forecast information and has the same ranking as the ignorance or log-likelihood skill score (Roulston and Smith 2002).”

• cites Roulston and Smith 2002Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The multimodel ensemble's advantage over the small single-model ensemble is consistent with both increased ensemble size and reduced model error.

“Therefore, the advantage of the multimodel ensemble over the small singlemodel ensemble is consistent with both increased ensemble size and reduced model error.”