Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Presentation and Evaluation of the IPSL‐CM6A‐LR Climate Model
Boucher, O., Servonnat, J., Albright, A. L., Aumont, O., Balkanski, Y., Bastrikov, V., Bekki, S., Bonnet, R., Bony, S., Bopp, L., et al. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms002010 · Cited by 1,694
“We have used the PCMDI Metrics Package (PMP) to calculate a set of large‐scale performance metrics (Gleckler et al., 2008), also called summary statistics, to summarize the agreement between the climate simulated by the model over the recent period and a set of references (observations and…”
MethodInfluentialClimate extremes indices in the CMIP5 multimodel ensemble: Part 2. Future climate projections
Sillmann, J., Kharin, V. V., Zwiers, F. W., Zhang, X., & Bronaugh, D. (2013) · Journal of Geophysical Research: Atmospheres · doi:10.1002/jgrd.50188 · Cited by 1,466
“Multimodel ensemble simulations have been shown to outperform individual models and are also expected to provide more robust estimates of future changes and model related uncertainties [e.g.,Gleckler et al., 2008; Sillmann et al., 2013].”
BackgroundSillmann, J., Kharin, V. V., Zhang, X., Zwiers, F. W., & Bronaugh, D. (2013) · Journal of Geophysical Research: Atmospheres · doi:10.1002/jgrd.50203 · Cited by 1,461
“[19] Given the large number of indices and models analyzed in this study, we use a metric approach to assess model performance similar to that advocated by Gleckler et al. [2008] (hereafter referred to as G08), which is based on the root-mean-square errors (RMSEs) of model climatologies.”
MethodChallenges in combining projections from multiple climate models.
Knutti, R., Furrer, R., Tebaldi, C., Cermak, J., & Meehl, G. A. (2010) · Journal of Climate · doi:10.1175/2009jcli3361.1 · Cited by 1,270
“…with observations of presentday climate than any single model, and that the average also consistently scores high in almost all diagnostics (Lambert and Boer 2001; Phillips and Gleckler 2006; Randall et al. 2007; Gleckler et al. 2008; Pincus et al. 2008; Reichler and Kim 2008; Pierce et al. 2009).”
BackgroundInfluentialThe Concept of Essential Climate Variables in Support of Climate Research, Applications, and Policy
Bojinski, S., Verstraete, M., Peterson, T. C., Richter, C., Simmons, A., & Zemp, M. (2014) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-13-00047.1 · Cited by 1,026
“…past observations of atmosphere, ocean and land using data assimilation methods as developed for numerical weather prediction and seasonal forecasting has become an important information source on recent climate variations (Dee et al. 2014) and for assessing climate models (Gleckler et al. 2008).”
MethodClimate model genealogy: Generation CMIP5 and how we got there
Knutti, R., Masson, D., & Gettelman, A. (2013) · Geophysical Research Letters · doi:10.1002/grl.50256 · Cited by 918
“[12] We can also use the distance metric of Figure 1 as one example to quantify distance from observations for different CMIP generations (shown in Figure 3), but with the strong caveat that linking model performance metrics to model quality or skill is difficult, subjective, and strongly metric dependent [Gleckler et…”
MethodENSO representation in climate models: from CMIP3 to CMIP5
Bellenger, H., Guilyardi, E., Leloup, J., Lengaigne, M., & Vialard, J. (2014) · Climate Dynamics · doi:10.1007/s00382-013-1783-z · Cited by 872
“Simulated and observed ENSO characteristics may depend on the mean state in the 277 tropical Pacific Ocean (e.g. Wang and An 2002, Guilyardi 2006, Sun et al 2009).”
BackgroundA global assessment of the impact of climate change on water scarcity
Gosling, S. N., & Arnell, N. W. (2016) · Climatic Change · doi:10.1007/s10584-013-0853-x · Cited by 860
“Moreover, no single GCM may consistently out-perform all others when variables beyond precipitation only are considered (Gleckler et al. 2008).”
The DOE E3SM Coupled Model Version 1: Overview and Evaluation at Standard Resolution
Golaz, J., Caldwell, P. M., Van Roekel, L. P., Petersen, M. R., Tang, Q., Wolfe, J. D., Abeshu, G., Anantharaj, V., Asay‐Davis, X. S., Bader, D. C., et al. (2019) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2018ms001603 · Cited by 825
“For a more exhaustive evaluation, we turn to a comparison with an ensemble of 45 CMIP5 models using metrics computed with the PCMDI Metrics Package (PMP; Gleckler et al., 2008, 2016).”
MethodRegional Dynamical Downscaling and the CORDEX Initiative
Giorgi, F., & Gutowski, W. J. (2015) · Annual Review of Environment and Resources · doi:10.1146/annurev-environ-102014-021217 · Cited by 792
Space can substitute for time in predicting climate-change effects on biodiversity
Blois, J. L., Williams, J. W., Fitzpatrick, M. C., Jackson, S. T., & Ferrier, S. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1220228110 · Cited by 786
“, climate models may be inaccurate (50) and pollen data represent a biased sample of fossil plant taxa (51), but there is no reason to expect that these issues are differential in space vs.”
BackgroundCOSP: Satellite simulation software for model assessment
Bodas-Salcedo, A., Webb, M. J., Bony, S., Chepfer, H., Dufresne, J.-L., Klein, S. A., Zhang, Y., Marchand, R., Haynes, J. M., Pincus, R., et al. (2011) · Bulletin of the American Meteorological Society · doi:10.1175/2011bams2856.1 · Cited by 701
A sea ice free summer Arctic within 30 years?
Wang, M., & Overland, J. E. (2009) · Geophysical Research Letters · doi:10.1029/2009gl037820 · Cited by 699
“models perform better than others is difficult [Gleckler et al., 2008], we do note that among the six selected models, three (CCSM3, CNRM-CM3 and UKMO-HadGEM1) include a Figure 1.”
MethodInfluentialHESS Opinions "Should we apply bias correction to global and regional climate model data?"
Ehret, U., Zehe, E., Wulfmeyer, V., Warrach-Sagi, K., & Liebert, J. (2012) · Hydrology and Earth System Sciences · doi:10.5194/hess-16-3391-2012 · Cited by 680
Selecting global climate models for regional climate change studies
Pierce, D. W., Barnett, T. P., Santer, B. D., & Gleckler, P. J. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0900094106 · Cited by 640
“The atmospheric resolution for the models varies (12).”
BackgroundMethodSelf-citationPark, S., & Bretherton, C. S. (2009) · Journal of Climate · doi:10.1175/2008jcli2557.1 · Cited by 598
“Gleckler et al. (2008) have also experimented with simple metrics of climate model performance.”
BackgroundUnderstanding El Niño in Ocean–Atmosphere General Circulation Models: Progress and Challenges
Guilyardi, E., Wittenberg, A., Fedorov, A., Collins, M., Wang, C., Capotondi, A., van Oldenborgh, G. J., & Stockdale, T. (2009) · Bulletin of the American Meteorological Society · doi:10.1175/2008bams2387.1 · Cited by 525
“…a few examples of ENSO metrics : • Mean and seasonal spatial root-mean-square errors, computed in specific regions (e.g.. in the entire tropics, east Pacific, warm pool, equatorial wave guide, etc.) and for key fields: SST, precipitation, wind stress, cloudiness, etc. (see Gleckler et al. 2008).”
InfluentialEvaluating the Land and Ocean Components of the Global Carbon Cycle in the CMIP5 Earth System Models
Anav, A., Friedlingstein, P., Kidston, M., Bopp, L., Ciais, P., Cox, P., Jones, C., Jung, M., Myneni, R., & Zhu, Z. (2013) · Journal of Climate · doi:10.1175/jcli-d-12-00417.1 · Cited by 502
“ESMs are routinely subjected to a variety of tests to assess their capabilities, and several papers provide extensive model evaluation (e.g., Tebaldi et al. 2006; Lin 2007; Lucarini et al. 2007; Santer et al. 2007; Gillett et al. 2008; Gleckler et al. 2008; Reichler and Kim 2008; Schneider et al. 2008; Santer et al. 2…”
InfluentialThe end of model democracy?: An editorial comment
Knutti, R. (2010) · Climatic Change · doi:10.1007/s10584-010-9800-2 · Cited by 499
“While an overall metric of skill can be defined (Gleckler et al. 2008; Reichler and Kim 2008), specific projections (e.”
BackgroundBrown, C., Ghile, Y., Laverty, M., & Li, K. (2012) · Water Resources Research · doi:10.1029/2011wr011212 · Cited by 492
“Based on the evidence of various studies [Brekke et al., 2009; Gleckler et al., 2008], we use multimodel super ensembles to estimate Prð i Þ and limit the use of projections to variables and scales for which skill is indicated in historical runs.”
MethodA climate model projection weighting scheme accounting for performance and interdependence
Knutti, R., Sedláček, J., Sanderson, B. M., Lorenz, R., Fischer, E. M., & Eyring, V. (2017) · Geophysical Research Letters · doi:10.1002/2016gl072012 · Cited by 491
Climate change risks for African agriculture
Müller, C., Cramer, W., Hare, W. L., & Lotze-Campen, H. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1015078108 · Cited by 490
The North American Regional Climate Change Assessment Program: Overview of Phase I Results
Mearns, L. O., Arritt, R., Biner, S., Bukovsky, M. S., McGinnis, S., Sain, S., Caya, D., Correia, J., Flory, D., Gutowski, W., et al. (2012) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-11-00223.1 · Cited by 476
“Failure to discern a best model has been documented in recent papers on model evaluation (e.g., Gleckler et al. 2008; Reichler and Kim 2008; Walsh et al. 2008).”
BackgroundMethodThe International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation
Collier, N., Hoffman, F. M., Lawrence, D. M., Keppel‐Aleks, G., Koven, C. D., Riley, W. J., Mu, M., & Randerson, J. T. (2018) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2018ms001354 · Cited by 447
“Early work on coupled model evaluation and establishing formal metrics focused primarily on atmospheric variables (Gleckler et al., 2008; Reichler & Kim, 2008).”
BackgroundHigh-resolution and bias-corrected CMIP5 projections for climate change impact assessments
Navarro-Racines, C., Tarapues, J., Thornton, P., Jarvis, A., & Ramirez-Villegas, J. (2020) · Scientific Data · doi:10.1038/s41597-019-0343-8 · Cited by 446
“To avoid this, we made two adjustments: (1) we set a threshold of 0.”
MethodProgressing emergent constraints on future climate change
Hall, A., Cox, P., Huntingford, C., & Klein, S. (2019) · Nature Climate Change · doi:10.1038/s41558-019-0436-6 · Cited by 443
(2020) · Clouds and Climate · doi:10.1017/9781107447738.012 · Cited by 439
A Representative Democracy to Reduce Interdependency in a Multimodel Ensemble
Sanderson, B. M., Knutti, R., & Caldwell, P. (2015) · Journal of Climate · doi:10.1175/jcli-d-14-00362.1 · Cited by 407
“Some models are closer to observations than others (Gleckler et al. 2008; Knutti and Sed á (cid:1) cek 2013).”
MethodModel projections of rapid sea-level rise on the northeast coast of the United States
Yin, J., Schlesinger, M. E., & Stouffer, R. J. (2009) · Nature Geoscience · doi:10.1038/ngeo462 · Cited by 401
“It realistically simulates many features of the climate system and has been assessed systematicall”
MethodRisks of Model Weighting in Multimodel Climate Projections
Weigel, A. P., Knutti, R., Liniger, M. A., & Appenzeller, C. (2010) · Journal of Climate · doi:10.1175/2010jcli3594.1 · Cited by 386
“, Gleckler et al. 2008). In fact, given that all models have essentially zero weight relative to the real world, Stainforth et al. (2007) go a step further and claim that any attempts to assign weights are, by principle, futile.”
BackgroundJiang, J. H., Su, H., Zhai, C., Perun, V. S., Del Genio, A., Nazarenko, L. S., Donner, L. J., Horowitz, L., Seman, C., Cole, J., et al. (2012) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2011jd017237 · Cited by 383
Weaver, C. P., Lempert, R. J., Brown, C., Hall, J. A., Revell, D., & Sarewitz, D. (2013) · WIREs Climate Change · doi:10.1002/wcc.202 · Cited by 370
Deryng, D., Sacks, W. J., Barford, C. C., & Ramankutty, N. (2011) · Global Biogeochemical Cycles · doi:10.1029/2009gb003765 · Cited by 366
Reduced global warming from CMIP6 projections when weighting models by performance and independence
Brunner, L., Pendergrass, A. G., Lehner, F., Merrifield, A. L., Lorenz, R., & Knutti, R. (2020) · Earth System Dynamics · doi:10.5194/esd-11-995-2020 · Cited by 362
Feng, Z., De Marco, A., Anav, A., Gualtieri, M., Sicard, P., Tian, H., Fornasier, F., Tao, F., Guo, A., & Paoletti, E. (2019) · Environment International · doi:10.1016/j.envint.2019.104966 · Cited by 361
Comparing estimates of climate change impacts from process-based and statistical crop models
Lobell, D. B., & Asseng, S. (2017) · Environmental Research Letters · doi:10.1088/1748-9326/aa518a · Cited by 360
“This finding reflects a ‘wisdom of the crowds’ often seen in model intercomparisons of other fields such as climate or weather forecasting (Pierce et al 2009, Gleckler et al 2008, Doblas-Reyes et al 2005), which derives from the independence of errors in different individual models.”
BackgroundCMIP6 Evaluation and Projection of Temperature and Precipitation over China
Yang, X., Zhou, B., Xu, Y., & Han, Z. (2021) · Advances in Atmospheric Sciences · doi:10.1007/s00376-021-0351-4 · Cited by 360
“The interannual variability of the simulations relative to the observations is assessed by the interannual variability skill score (IVS) (Gleckler et al., 2008; Scherrer, 2011), which is calculated as”
MethodMaurer, E. P., Hidalgo, H. G., Das, T., Dettinger, M. D., & Cayan, D. R. (2010) · Hydrology and Earth System Sciences · doi:10.5194/hess-14-1125-2010 · Cited by 345
Use of models in detection and attribution of climate change
Hegerl, G., & Zwiers, F. (2011) · WIREs Climate Change · doi:10.1002/wcc.121 · Cited by 345
An Overview of the Atmospheric Component of the Energy Exascale Earth System Model
Rasch, P. J., Xie, S., Ma, P. ‐., Lin, W., Wang, H., Tang, Q., Burrows, S. M., Caldwell, P., Zhang, K., Easter, R. C., et al. (2019) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001629 · Cited by 345
“Turbulence, Shallow Convection, and Stratiform Clouds Turbulence, shallow cumulus clouds, and stratocumulus clouds are parameterized with a single equation set describing the evolution of a set of probability density functions (PDFs) of fields, using the Cloud Layers Unified By Binormals (CLUBB) parameterization (Gola…”
BackgroundMethodResultInfluentialPincus, R., Platnick, S., Ackerman, S. A., Hemler, R. S., & Patrick Hofmann, R. J. (2012) · Journal of Climate · doi:10.1175/jcli-d-11-00267.1 · Cited by 342
“…the differences between these observational datasets in the way climate model simulations are normally evaluated against observations (see, e.g., Gleckler et al. 2008; Pincus et al. 2008): by summarizing the components of the total RMS difference between multiple datasets and a single reference.”
Rainfall trends in the African Sahel: Characteristics, processes, and causes
Biasutti, M. (2019) · WIREs Climate Change · doi:10.1002/wcc.591 · Cited by 336
Beyond equilibrium climate sensitivity
Knutti, R., Rugenstein, M. A. A., & Hegerl, G. C. (2017) · Nature Geoscience · doi:10.1038/ngeo3017 · Cited by 333
Grotjahn, R., Black, R., Leung, R., Wehner, M. F., Barlow, M., Bosilovich, M., Gershunov, A., Gutowski, W. J., Gyakum, J. R., Katz, R. W., et al. (2016) · Climate Dynamics · doi:10.1007/s00382-015-2638-6 · Cited by 331
“Both global and regional climate models have been used to elucidate processes contributing to ETEs and evaluate As in Gleckler et al. (2008), to plot errors from multiple variables on the same scale, they are normalized by the median error of the CMIP5 models using the formula Here, E median is the…”
Xin, X., Wu, T., Zhang, J., Yao, J., & Fang, Y. (2020) · International Journal of Climatology · doi:10.1002/joc.6590 · Cited by 325
“This is consistent with previous studies (Lambert and Boer, 2001; Gleckler et al., 2008; Reichler and Kim, 2008).”
Earth system models: an overview
Flato, G. M. (2011) · WIREs Climate Change · doi:10.1002/wcc.148 · Cited by 316
Zhao, M., Golaz, J. ‐., Held, I. M., Guo, H., Balaji, V., Benson, R., Chen, J. ‐., Chen, X., Donner, L. J., Dunne, J. P., et al. (2018) · Journal of Advances in Modeling Earth Systems · doi:10.1002/2017ms001208 · Cited by 314
“In addition, we have also recently utilized the portrait plots described by Gleckler et al. (2008, 2016) to monitor aspects of simulation quality along our development trajectory.”
MethodThe shifting influence of drought and heat stress for crops in northeast Australia
Lobell, D. B., Hammer, G. L., Chenu, K., Zheng, B., McLean, G., & Chapman, S. C. (2015) · Global Change Biology · doi:10.1111/gcb.13022 · Cited by 310
“…the multimodel average as the most likely projection, based on evidence that it tends to outperform individual GCMs when simulating past climate (Gleckler et al., 2008), treating each projection as equally likely is more conservative, in that it assumes past observations cannot adequately…”
BackgroundAhmed, K., Sachindra, D. A., Shahid, S., Demirel, M. C., & Chung, E.-S. (2019) · Hydrology and Earth System Sciences · doi:10.5194/hess-23-4803-2019 · Cited by 309
“…error (RMSE), mean absolute error, correlation coefficient, and skill scores into one performance index to assess the accuracy of GCMs in reproducing past climate (Gu et al., 2015; Barfus and Bernhofer, 2015; Gleckler et al., 2008; Wu et al., 2016; Ahmadalipour et al., 2017; Raju et al., 2017).”
BackgroundMethodEvaluation of CMIP5 20th century climate simulations for the Pacific Northwest USA
Rupp, D. E., Abatzoglou, J. T., Hegewisch, K. C., & Mote, P. W. (2013) · Journal of Geophysical Research: Atmospheres · doi:10.1002/jgrd.50843 · Cited by 308
“[13] Lacking any established standard methodology for evaluating climate models at a regional scale [Gleckler et al., 2008], we proceeded in the spirit of Brekke et al.”
MethodHanasaki, N., Fujimori, S., Yamamoto, T., Yoshikawa, S., Masaki, Y., Hijioka, Y., Kainuma, M., Kanamori, Y., Masui, T., Takahashi, K., et al. (2013) · Hydrology and Earth System Sciences · doi:10.5194/hess-17-2393-2013 · Cited by 302
“Gleckler et al., 2008) would be useful to ensure that the selection process is less subjective. When preparing the climate scenario, we adopted a shifting and scaling method (Lehner et al., 2006) to remove the biases of the GCMs. This was computationally efficient, because only the mean monthly difference was added or…”
MethodEarly onset of significant local warming in low latitude countries
Mahlstein, I., Knutti, R., Solomon, S., & Portmann, R. W. (2011) · Environmental Research Letters · doi:10.1088/1748-9326/6/3/034009 · Cited by 299
Evaluation of CMIP5 palaeo-simulations to improve climate projections
Harrison, S. P., Bartlein, P. J., Izumi, K., Li, G., Annan, J., Hargreaves, J., Braconnot, P., & Kageyama, M. (2015) · Nature Climate Change · doi:10.1038/nclimate2649 · Cited by 297
Description and basic evaluation of Beijing Normal University Earth System Model (BNU-ESM) version 1
Ji, D., Wang, L., Feng, J., Wu, Q., Cheng, H., Zhang, Q., Yang, J., Dong, W., Dai, Y., Gong, D., et al. (2014) · Geoscientific Model Development · doi:10.5194/gmd-7-2039-2014 · Cited by 294
“To systematically evaluate the general performance of BNUESM, we use the Taylor diagram (Taylor, 2001; Gleckler et al., 2008), which relates the “centered” root-mean square (RMS) error, the pattern correlation and the standard deviation of particular climate fields.”
BackgroundMethodInfluentialReanalyses and Observations: What’s the Difference?
Parker, W. S. (2016) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-14-00226.1 · Cited by 292
The BRIDGE HadCM3 family of climate models: HadCM3@Bristol v1.0
Valdes, P. J., Armstrong, E., Badger, M. P. S., Bradshaw, C. D., Bragg, F., Crucifix, M., Davies-Barnard, T., Day, J. J., Farnsworth, A., Gordon, C., et al. (2017) · Geoscientific Model Development · doi:10.5194/gmd-10-3715-2017 · Cited by 286
Collins, M., Booth, B. B. B., Bhaskaran, B., Harris, G. R., Murphy, J. M., Sexton, D. M. H., & Webb, M. J. (2011) · Climate Dynamics · doi:10.1007/s00382-010-0808-0 · Cited by 282
“Nevertheless, this does represent an advance on previous studies (e.g. Gleckler et al. 2008).”
BackgroundResultOrbital forcing of tree-ring data
Esper, J., Frank, D. C., Timonen, M., Zorita, E., Wilson, R. J. S., Luterbacher, J., Holzkämper, S., Fischer, N., Wagner, S., Nievergelt, D., et al. (2012) · Nature Climate Change · doi:10.1038/nclimate1589 · Cited by 280
Should we believe model predictions of future climate change?
Knutti, R. (2008) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2008.0169 · Cited by 279
“2007 and references therein; Gleckler et al .”
BackgroundResultInfluentialValidating CHIRPS-based satellite precipitation estimates in Northeast Brazil
Paredes-Trejo, F. J., Barbosa, H. A., & Lakshmi Kumar, T. V. (2017) · Journal of Arid Environments · doi:10.1016/j.jaridenv.2016.12.009 · Cited by 279
Attribution of polar warming to human influence
Gillett, N. P., Stone, D. A., Stott, P. A., Nozawa, T., Karpechko, A. Y., Hegerl, G. C., Wehner, M. F., & Jones, P. D. (2008) · Nature Geoscience · doi:10.1038/ngeo338 · Cited by 275
“, climate and variability in the northern extratropic”
BackgroundRivera, J. A., & Arnould, G. (2020) · Atmospheric Research · doi:10.1016/j.atmosres.2020.104953 · Cited by 275
Sheffield, J., Barrett, A. P., Colle, B., Nelun Fernando, D., Fu, R., Geil, K. L., Hu, Q., Kinter, J., Kumar, S., Langenbrunner, B., et al. (2013) · Journal of Climate · doi:10.1175/jcli-d-12-00592.1 · Cited by 270
“Evaluations of model performance are not straightforward because of the broad range of uses of climate model data (Gleckler et al. 2008) and therefore there is not an accepted universal set of performance metrics.”
BackgroundUsing palaeo-climate comparisons to constrain future projections in CMIP5
Schmidt, G. A., Annan, J. D., Bartlein, P. J., Cook, B. I., Guilyardi, E., Hargreaves, J. C., Harrison, S. P., Kageyama, M., LeGrande, A. N., Konecky, B., et al. (2014) · Climate of the Past · doi:10.5194/cp-10-221-2014 · Cited by 265
Maloney, E. D., Camargo, S. J., Chang, E., Colle, B., Fu, R., Geil, K. L., Hu, Q., Jiang, X., Johnson, N., Karnauskas, K. B., et al. (2014) · Journal of Climate · doi:10.1175/jcli-d-13-00273.1 · Cited by 260
“Multimodel ensemble mean (MEM) differences are highlighted for most of the analyses, as the MEM produces demonstrably superior results in historical climate assessment to those from an individual model (e.g., Gleckler et al. 2008; Pierce et al. 2009).”
ResultSpatial Variability of Sea Level Rise in Twenty-First Century Projections
Yin, J., Griffies, S. M., & Stouffer, R. J. (2010) · Journal of Climate · doi:10.1175/2010jcli3533.1 · Cited by 254
“1 realistically simulates many features of the climate system such as ocean temperature and salinity (Gleckler et al. 2008).”
BackgroundEvaluation of historical CMIP6 model simulations of extreme precipitation over contiguous US regions
Srivastava, A., Grotjahn, R., & Ullrich, P. A. (2020) · Weather and Climate Extremes · doi:10.1016/j.wace.2020.100268 · Cited by 251
Evaluating Climate Models with the CLIVAR 2020 ENSO Metrics Package
Planton, Y. Y., Guilyardi, E., Wittenberg, A. T., Lee, J., Gleckler, P. J., Bayr, T., McGregor, S., McPhaden, M. J., Power, S., Roehrig, R., et al. (2021) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-19-0337.1 · Cited by 246
“2) that is similar to those shown in Gleckler et al. (2008), Flato et al. (2013), Sillmann et al. (2013), and Lee et al. (2019).”
ResultSelf-citationPichelli, E., Coppola, E., Sobolowski, S., Ban, N., Giorgi, F., Stocchi, P., Alias, A., Belušić, D., Berthou, S., Caillaud, C., et al. (2021) · Climate Dynamics · doi:10.1007/s00382-021-05657-4 · Cited by 244
Ensemble modeling, uncertainty and robust predictions
Parker, W. S. (2013) · WIREs Climate Change · doi:10.1002/wcc.220 · Cited by 244
Are climate model simulations of clouds improving? An evaluation using the ISCCP simulator
Klein, S. A., Zhang, Y., Zelinka, M. D., Pincus, R., Boyle, J., & Gleckler, P. J. (2013) · Journal of Geophysical Research: Atmospheres · doi:10.1002/jgrd.50141 · Cited by 242
“However, multimodel means are sensitive to the addition of new models (especially given the small sizes of the model ensembles), and changes in the multimodel mean may not reveal individual model error reductions when the spread of model results is centered on the observed value, as is often the case [Gleckler et al.,…”
BackgroundResultSelf-citationQuantifying Progress Across Different CMIP Phases With the ESMValTool
Bock, L., Lauer, A., Schlund, M., Barreiro, M., Bellouin, N., Jones, C., Meehl, G. A., Predoi, V., Roberts, M. J., & Eyring, V. (2020) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2019jd032321 · Cited by 242
“Figure 6 is an extended and updated version of Figure 9.7 of Flato et al. (2013) that is based on Gleckler et al. (2008).”
BackgroundInfluentialMehran, A., AghaKouchak, A., & Phillips, T. J. (2014) · Journal of Geophysical Research: Atmospheres · doi:10.1002/2013jd021152 · Cited by 240
Mauser, W., Klepper, G., Zabel, F., Delzeit, R., Hank, T., Putzenlechner, B., & Calzadilla, A. (2015) · Nature Communications · doi:10.1038/ncomms9946 · Cited by 236
On the Effective Number of Climate Models
Pennell, C., & Reichler, T. (2011) · Journal of Climate · doi:10.1175/2010jcli3814.1 · Cited by 235
“Previous studies have provided convincing evidence that averaging over the outcomes from many models (multimodel mean) generally outperforms any individual model (e.g., Reichler and Kim 2008a; Gleckler et al. 2008).”
BackgroundProbabilistic Projections of Climate Change over China under the SRES A1B Scenario Using 28 AOGCMs
Chen, W., Jiang, Z., & Li, L. (2011) · Journal of Climate · doi:10.1175/2011jcli4102.1 · Cited by 235
“As in the study by Gleckler et al. (2008) and Santer et al.”
BackgroundResultInfluentialIncorporating Climate Uncertainty into Estimates of Climate Change Impacts
Burke, M., Dykema, J., Lobell, D. B., Miguel, E., & Satyanath, S. (2015) · Review of Economics and Statistics · doi:10.1162/rest_a_00478 · Cited by 234
Climate model benchmarking with glacial and mid-Holocene climates
Harrison, S. P., Bartlein, P. J., Brewer, S., Prentice, I. C., Boyd, M., Hessler, I., Holmgren, K., Izumi, K., & Willis, K. (2014) · Climate Dynamics · doi:10.1007/s00382-013-1922-6 · Cited by 231
“Following Gleckler et al. (2008) the metrics are normalized by the median model error to yield an evaluation of how well a given model compares to the typical model error.”
MethodClimate Change and Water Resources Management: A Federal Perspective
Brekke, L. D., Kiang, J. E., Olsen, J. R., Pulwarty, R. S., Raff, D. A., Turnipseed, D. P., Webb, R. S., & White, K. D. (2009) · Circular · doi:10.3133/cir1331 · Cited by 228
Gosling, S. N., Taylor, R. G., Arnell, N. W., & Todd, M. C. (2011) · Hydrology and Earth System Sciences · doi:10.5194/hess-15-279-2011 · Cited by 228
“…uncertainty from climate model structural uncertainty could be reduced by comparing the GCM simulations of baseline climate with observations (e.g. Gleckler et al., 2008) but the calculation of single indices of model performance can be misleading because it hides a more complex picture of the…”
BackgroundMethodEvaluating the present‐day simulation of clouds, precipitation, and radiation in climate models
Pincus, R., Batstone, C. P., Hofmann, R. J. P., Taylor, K. E., & Glecker, P. J. (2008) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2007jd009334 · Cited by 228
“This is one motivation for developing a suite of metrics for evaluating a model’s climatology [e.g., Gleckler et al., 2007] (hereafter GTD2007).”
BackgroundSelf-citationComparison of CMIP6 and CMIP5 models in simulating mean and extreme precipitation over East Africa
Ayugi, B., Zhihong, J., Zhu, H., Ngoma, H., Babaousmail, H., Rizwan, K., & Dike, V. (2021) · International Journal of Climatology · doi:10.1002/joc.7207 · Cited by 220
“Following the approach proposed by Gleckler et al., 2008) and later modified by Sillmann et al. (2013), the metrics technique based on RMSEs of the model climatology pattern is employed in this study.”
MethodHow well can CMIP5 simulate precipitation and its controlling processes over tropical South America?
Yin, L., Fu, R., Shevliakova, E., & Dickinson, R. E. (2013) · Climate Dynamics · doi:10.1007/s00382-012-1582-y · Cited by 217
“One is the root-mean square error (RMSE) for a simulated field M corresponding to a reference O (Gleckler et al. 2008).”
BackgroundMishra, A. K., & Singh, V. P. (2009) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2008jd010986 · Cited by 216
“5.3. Major Historical Droughts and Projected SAF Curves [48] Several researchers [e.g., Gleckler et al., 2008;”
BackgroundInfluentialReview of approaches for selection and ensembling of GCMs
Raju, K. S., & Kumar, D. N. (2020) · Journal of Water and Climate Change · doi:10.2166/wcc.2020.128 · Cited by 215
Jose, D. M., Vincent, A. M., & Dwarakish, G. S. (2022) · Scientific Reports · doi:10.1038/s41598-022-08786-w · Cited by 213
“In order to calculate the weights of the GCMs according to their performance in the past, multi-criteria decision making methods or other matrices are often used 3,20 .”
MethodProjection of climate extremes in China, an incremental exercise from CMIP5 to CMIP6
Zhu, H., Jiang, Z., & Li, L. (2021) · Science Bulletin · doi:10.1016/j.scib.2021.07.026 · Cited by 205
Evaluation of the CORDEX-Africa multi-RCM hindcast: systematic model errors
Kim, J., Waliser, D. E., Mattmann, C. A., Goodale, C. E., Hart, A. F., Zimdars, P. A., Crichton, D. J., Jones, C., Nikulin, G., Hewitson, B., et al. (2014) · Climate Dynamics · doi:10.1007/s00382-013-1751-7 · Cited by 204
Abramowitz, G., Herger, N., Gutmann, E., Hammerling, D., Knutti, R., Leduc, M., Lorenz, R., Pincus, R., & Schmidt, G. A. (2019) · Earth System Dynamics · doi:10.5194/esd-10-91-2019 · Cited by 201
“Gleckler et al. (2008) also show that the variability of the ensemble mean is much less than individual models or observations – and so does not represent a potentially real climate.”
BackgroundBrands, S., Herrera, S., Fernández, J., & Gutiérrez, J. M. (2013) · Climate Dynamics · doi:10.1007/s00382-013-1742-8 · Cited by 197
“This questions the basic downscaling assumption of ‘real’ or ‘perfect’ reanalysis data and hinders the objective evaluation of ESM performance (Gleckler et al. 2008) in these regions.”
BackgroundResultTaiwan Earth System Model Version 1: description and evaluation of mean state
Lee, W., Wang, Y., Shiu, C., Tsai, I., Tu, C., Lan, Y.-Y., Chen, J., Pan, H., & Hsu, H. (2020) · Geoscientific model development · doi:10.5194/gmd-13-3887-2020 · Cited by 196
Salman, S. A., Shahid, S., Ismail, T., Ahmed, K., & Wang, X.-J. (2018) · Atmospheric Research · doi:10.1016/j.atmosres.2018.07.008 · Cited by 195
Climate model dependence and the replicate Earth paradigm
Bishop, C. H., & Abramowitz, G. (2013) · Climate Dynamics · doi:10.1007/s00382-012-1610-y · Cited by 193
“2005) or sub-selecting (e.g. Perkins et al. 2007; Gleckler et al. 2008) different simulations or models, based on their”
BackgroundInfluentialMartin, G. M., Milton, S. F., Senior, C. A., Brooks, M. E., Ineson, S., Reichler, T., & Kim, J. (2010) · Journal of Climate · doi:10.1175/2010jcli3541.1 · Cited by 192
“vations or reanalyses now exist in the literature (e.g., Murphy et al. 2004; Reichler and Kim 2008a; Gleckler et al. 2008).”
BackgroundGhorbani, M. A., Deo, R. C., Yaseen, Z. M., H. Kashani, M., & Mohammadi, B. (2018) · Theoretical and Applied Climatology · doi:10.1007/s00704-017-2244-0 · Cited by 192
“Moreover, the normalized RMSE can be resolved into a part due to differences in the overall means and a part due to errors in the pattern of variations (Taylor 2001; IPCC 2007; Gleckler et al. 2008).”
BackgroundEyring, V., Righi, M., Lauer, A., Evaldsson, M., Wenzel, S., Jones, C., Anav, A., Andrews, O., Cionni, I., Davin, E. L., et al. (2016) · Geoscientific Model Development · doi:10.5194/gmd-9-1747-2016 · Cited by 191
Self-citationAn asynchronous regional regression model for statistical downscaling of daily climate variables
Stoner, A. M. K., Hayhoe, K., Yang, X., & Wuebbles, D. J. (2013) · International Journal of Climatology · doi:10.1002/joc.3603 · Cited by 190
“Previous studies (e.g. Gleckler et al., 2008; Stoner et al., 2009; Rusticucci et al., 2010) show that these models are able to represent key features of atmospheric variability including teleconnection patterns, extreme temperature and precipitation, as well as other climate metrics.”
BackgroundA simple lumped model to convert air temperature into surface water temperature in lakes
Piccolroaz, S., Toffolon, M., & Majone, B. (2013) · Hydrology and Earth System Sciences · doi:10.5194/hess-17-3323-2013 · Cited by 186
Radić, V., & Hock, R. (2014) · Surveys in Geophysics · doi:10.1007/s10712-013-9262-y · Cited by 184
“, Gleckler et al. 2008; Pierce et al. 2009). Surv Geophys (2014) 35:813–837 819”
BackgroundBrutel-Vuilmet, C., Ménégoz, M., & Krinner, G. (2013) · The Cryosphere · doi:10.5194/tc-7-67-2013 · Cited by 184
“The “superiority” of the “mean model” in this type of model intercomparison exercise has been clearly shown before for a large range of climate variables (Gleckler et al., 2008) and is therefore no surprise.”
Background
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.