Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Items 101–200 of 1,001
Khan, N., Shahid, S., Ahmed, K., Ismail, T., Nawaz, N., & Son, M. (2018) · Water · doi:10.3390/w10121793 · Cited by 183
Law, R. M., Ziehn, T., Matear, R. J., Lenton, A., Chamberlain, M. A., Stevens, L. E., Wang, Y.-P., Srbinovsky, J., Bi, D., Yan, H., et al. (2017) · Geoscientific Model Development · doi:10.5194/gmd-10-2567-2017 · Cited by 183
“We use monthly mean values averaged across 100 years from pre-industrial climate simulations and and calculate the root mean square difference (RMSD), similar to Gleckler et al. (2008), between each modelled field 5”
Addressing Interdependency in a Multimodel Ensemble by Interpolation of Model Properties
Sanderson, B. M., Knutti, R., & Caldwell, P. (2015) · Journal of Climate · doi:10.1175/jcli-d-14-00361.1 · Cited by 183
“1 2015 American Meteorological Society Unauthenticated | Downloaded 12/24/21 10:15 AM UTC 2011) and that some models exhibit more skill than others in reproducing past and present climate (Gleckler et al. 2008; Reichler and Kim 2008; Knutti et al. 2013).”
BackgroundBador, M., Boé, J., Terray, L., Alexander, L. V., Baker, A., Bellucci, A., Haarsma, R., Koenigk, T., Moine, M., Lohmann, K., et al. (2020) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2019jd032184 · Cited by 181
“The majority of studies evaluating global climate models focus on mean precipitation (e.g., Gleckler et al., 2008; Reichler & Kim, 2008; Watterson, 2015).”
BackgroundMJO Propagation Across the Maritime Continent: Are CMIP6 Models Better Than CMIP5 Models?
Ahn, M., Kim, D., Kang, D., Lee, J., Sperber, K. R., Gleckler, P. J., Jiang, X., Ham, Y., & Kim, H. (2020) · Geophysical Research Letters · doi:10.1029/2020gl087250 · Cited by 180
“…objective tests to gauge how well climate models represent various observed characteristics (e.g., Baker & Taylor, 2016; Flato et al., 2013; Gleckler et al., 2008; Gleckler et al., 2016; Lee et al., 2019), a diverse collection of community‐based MJO simulation diagnostics and metrics has…”
BackgroundSelf-citationA Decision-Analytic Approach to Managing Climate Risks: Application to the Upper Great Lakes1
Brown, C., Werick, W., Leger, W., & Fay, D. (2011) · JAWRA Journal of the American Water Resources Association · doi:10.1111/j.1752-1688.2011.00552.x · Cited by 179
Philosophy and Climate Science
Winsberg, E. (2018) · Cambridge University Press eBooks · doi:10.1017/9781108164290 · Cited by 178
Gualdi, S., Somot, S., Li, L., Artale, V., Adani, M., Bellucci, A., Braun, A., Calmanti, S., Carillo, A., Dell'Aquila, A., et al. (2013) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-11-00136.1 · Cited by 178
Wang, Y., Zhou, B., Qin, D., Wu, J., Gao, R., & Song, L. (2017) · Advances in Atmospheric Sciences · doi:10.1007/s00376-016-6160-5 · Cited by 178
“…ensemble mean (MME) has been emonstrated to outperform individual models and is ex- pected to provide more robust estimates of future changes (e.g., Gleckler et al., 2008; Jiang et al., 2009b, 2016; Chen, 2013; Gupta et al., 2013; Sillmann et al., 2013; Zhou et al., 2014), we focus on the…”
MethodRanking of CMIP5-based global climate models for India using compromise programming
Srinivasa Raju, K., Sonali, P., & Nagesh Kumar, D. (2017) · Theoretical and Applied Climatology · doi:10.1007/s00704-015-1721-6 · Cited by 174
Estimation of the climate change impact on a catchment water balance using an ensemble of GCMs
Reshmidevi, T. V., Nagesh Kumar, D., Mehrotra, R., & Sharma, A. (2018) · Journal of Hydrology · doi:10.1016/j.jhydrol.2017.02.016 · Cited by 173
“Multi-model mean has been found to be giving better simulation of the climate variables compared to the individual models (Gleckler et al., 2008; Knutti et al. , 2010; Zhang and Huang, 2013).”
MethodSelecting a climate model subset to optimise key ensemble properties
Herger, N., Abramowitz, G., Knutti, R., Angélil, O., Lehmann, K., & Sanderson, B. M. (2018) · Earth System Dynamics · doi:10.5194/esd-9-135-2018 · Cited by 172
“(2001); Gleckler et al. (2008); Baker and Taylor (2016)) ::: that ::: we :::::: might :::::: choose :: to”
InfluentialClimate Models: An Assessment of Strengths and Limitations
Bader, D. C., Covey, C., Gutowski, W. J., Held, I. M., Kunkel, K. E., Miller, R. L., Tokmakian, R. T., & Zhang, M. H. (2008) · Iowa State University Digital Repository (Iowa State University) · Cited by 172
Hamlet, A. F., Elsner, M. M., Mauger, G., Lee, S.-Y., Tohver, I., & Norheim, R. A. (2013) · ATMOSPHERE-OCEAN · doi:10.1080/07055900.2013.819555 · Cited by 172
Barlow, M., Gutowski, W. J., Gyakum, J. R., Katz, R. W., Lim, Y., Schumacher, R. S., Wehner, M. F., Agel, L., Bosilovich, M. G., Collow, A. B. M., et al. (2019) · Climate Dynamics · doi:10.1007/s00382-019-04958-z · Cited by 172
Hantson, S., Kelley, D. I., Arneth, A., Harrison, S. P., Archibald, S., Bachelet, D., Forrest, M., Hickler, T., Lasslop, G., Li, F., et al. (2020) · Geoscientific Model Development · doi:10.5194/gmd-13-3299-2020 · Cited by 171
“…depending on their overall performance or performance against a minimum set of specific criteria (e.g. Eyring et al., 2005; Tebaldi et al., 2005; Gleckler et al., 2008; Weigel et al., 2008; Santer et al., 2009; Parker, 2013; Abramowitz et al., 2019), it is clear that results from systematic…”
BackgroundOn the use and misuse of climate change projections in international development
Nissan, H., Goddard, L., de Perez, E. C., Furlow, J., Baethgen, W., Thomson, M. C., & Mason, S. J. (2019) · Wiley Interdisciplinary Reviews Climate Change · doi:10.1002/wcc.579 · Cited by 170
“capture the basics cannot responsibly be used to assess how an extreme weather event may change decades in the future (see Endris et al., 2013; Gleckler et al., 2008; Kalognomou et al., 2013; Muñoz, Yang, Vecchi, Robertson, and Cooke, 2017).”
BackgroundHydrologic extremes – an intercomparison of multiple gridded statistical downscaling methods
Werner, A. T., & Cannon, A. J. (2016) · Hydrology and Earth System Sciences · doi:10.5194/hess-20-1483-2016 · Cited by 169
Model robustness as a confirmatory virtue: The case of climate science
Lloyd, E. A. (2015) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2014.12.002 · Cited by 169
Values and Uncertainties in the Predictions of Global Climate Models
Winsberg, E. (2012) · Kennedy Institute of Ethics Journal · doi:10.1353/ken.2012.0008 · Cited by 168
A quantitative performance assessment of cloud regimes in climate models
Williams, K. D., & Webb, M. J. (2009) · Climate Dynamics · doi:10.1007/s00382-008-0443-1 · Cited by 168
“Of course, GCMs have many more applications than providing a global temperature response, hence we propose that the metrics should be part of a basket of metrics similar to Murphy et al. (2004) or Gleckler et al. (2008) , with CREMpd used in present-day evaluation of GCMs and CREMcc used as part of a basket to assess…”
BackgroundMethodInfluentialReliability of climate models for China through the IPCC Third to Fifth Assessment Reports
Jiang, D., Tian, Z., & Lang, X. (2016) · International Journal of Climatology · doi:10.1002/joc.4406 · Cited by 168
“The multi-GCM mean outperforms most of individual GCMs for the annual and seasonal temperatures over China, as already found in global studies examining the mean climate (e.g. Lambert and Boer, 2001; Gleckler et al., 2008).”
BackgroundEvaluation of extreme precipitation over Asia in CMIP6 models
Dong, T., & Dong, W. (2021) · Climate Dynamics · doi:10.1007/s00382-021-05773-1 · Cited by 167
II—Wendy S. Parker: Confirmation and adequacy-for-Purpose in Climate Modelling
Parker, W. S. (2009) · Aristotelian Society Supplementary Volume · doi:10.1111/j.1467-8349.2009.00180.x · Cited by 167
Challenges in the Quest for Added Value of Regional Climate Dynamical Downscaling
Luca, A. D., de Elía, R., & Laprise, R. (2015) · Current Climate Change Reports · doi:10.1007/s40641-015-0003-9 · Cited by 167
“[34], the choice of metric is arbitrary and hence there is no objective way to evaluate the overall performance of climate models and to identify the “best” climate model.”
BackgroundIncorporating model quality information in climate change detection and attribution studies
Santer, B. D., Taylor, K. E., Gleckler, P. J., Bonfils, C., Barnett, T. P., Pierce, D. W., Wigley, T. M. L., Mears, C., Wentz, F. J., Brüggemann, W., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0901736106 · Cited by 166
“Similar findings have been obtained in related studies (19, 20, 22, 23).”
BackgroundResultSelf-citationMeasurement of GCM Skill in Predicting Variables Relevant for Hydroclimatological Assessments
Johnson, F., & Sharma, A. (2009) · Journal of Climate · doi:10.1175/2009jcli2681.1 · Cited by 166
When Climate Models Agree: The Significance of Robust Model Predictions
Parker, W. S. (2011) · Philosophy of Science · doi:10.1086/661566 · Cited by 164
“…to determine what a model’s performance in simulating past and present climate indicates about its accuracy in predicting various quantities of interest (see Randall et al. 2007; Gleckler, Taylor, and Doutriaux 2008; Parker 2009). values are chosen from a range identified by expert judgment.”
BackgroundUnique challenges and opportunities for northeastern US crop production in a changing climate
Wolfe, D. W., DeGaetano, A. T., Peck, G. M., Carey, M., Ziska, L. H., Lea-Cox, J., Kemanian, A. R., Hoffmann, M. P., & Hollinger, D. Y. (2018) · Climatic Change · doi:10.1007/s10584-017-2109-7 · Cited by 164
“Here, we use downscaled simulations from a subset of 10 CMIP5 models (Supplemental Table S2) based in part on: (1) their skill in simulating global short and long wave radiation, temperature, wind, and precipitation (Gleckler et al. 2008); (2) their ability to simulate historical climate conditions in eastern North Am…”
MethodGarcia, R. A., Burgess, N. D., Cabeza, M., Rahbek, C., & Araújo, M. B. (2012) · Global Change Biology · doi:10.1111/j.1365-2486.2011.02605.x · Cited by 164
“Similarities were assessed separately for each variable because the performance of climate models varies for different variables (Lambert & Boer, 2001; Gleckler et al., 2008).”
BackgroundInfluentialAn Introduction to the E3SM Special Collection: Goals, Science Drivers, Development, and Analysis
Leung, L. R., Bader, D. C., Taylor, M. A., & McCoy, R. B. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001821 · Cited by 164
Gain, A. K., Immerzeel, W. W., Weiland, F. S., & Bierkens, M. F. P. (2011) · Hydrology and earth system sciences · doi:10.5194/hess-15-1537-2011 · Cited by 164
Evaluation of downscaled, gridded climate data for the conterminous United States
Behnke, R., Vavrus, S., Allstadt, A., Albright, T., Thogmartin, W. E., & Radeloff, V. C. (2016) · Ecological Applications · doi:10.1002/15-1061 · Cited by 163
A comprehensive benchmarking system for evaluating global vegetation models
Kelley, D. I., Prentice, I. C., Harrison, S. P., Wang, H., Simard, M., Fisher, J. B., & Willis, K. O. (2013) · Biogeosciences · doi:10.5194/bg-10-3313-2013 · Cited by 163
“This approach has been used by the climate-modelling community to evaluate performance against modern climate observations (e.g. Gleckler et al., 2008) and is used here for CO 2 inter-annual comparisons.”
MethodA Unified Modeling Approach to Climate System Prediction
Hurrell, J., Meehl, G. A., Bader, D., Delworth, T. L., Kirtman, B., & Wielicki, B. (2009) · Bulletin of the American Meteorological Society · doi:10.1175/2009bams2752.1 · Cited by 163
Comparing Large-Scale Hydrological Model Simulations to Observed Runoff Percentiles in Europe
Gudmundsson, L., Tallaksen, L. M., Stahl, K., Clark, D. B., Dumont, E., Hagemann, S., Bertrand, N., Gerten, D., Heinke, J., Hanasaki, N., et al. (2012) · Journal of Hydrometeorology · doi:10.1175/jhm-d-11-083.1 · Cited by 163
Warming of Central European lakes and their response to the 1980s climate regime shift
Woolway, R. I., Dokulil, M. T., Marszelewski, W., Schmid, M., Bouffard, D., & Merchant, C. J. (2017) · Climatic Change · doi:10.1007/s10584-017-1966-4 · Cited by 162
“The advantage of using a model that simulates LSWT from SAT alone is that future SAT projections based on general circulation models or regional climate models are commonly more reliable than those of other meteorological variables (Gleckler et al. 2008), and downscaling is typically associated with smaller uncertaint…”
Geoscientific Model Development
Hoffman, F. M., Hassler, B., Swaminathan, R., Lewis, J., Andela, B., Collier, N., Hegedūs, D., Lee, J., Pascoe, C., Pflüger, M., et al. (2018) · doi:10.5194/gmd-volume10-issue2-editorial · Cited by 161
Perez, J., Menendez, M., Mendez, F. J., & Losada, I. J. (2014) · Climate Dynamics · doi:10.1007/s00382-014-2078-8 · Cited by 159
“There are even some studies that aggregate several statistical measures to form a single metric (e.g. Gleckler et al. 2008).”
BackgroundCardoso, R. M., Soares, P. M. M., Lima, D. C. A., & Miranda, P. M. A. (2019) · Climate Dynamics · doi:10.1007/s00382-018-4124-4 · Cited by 159
Ge, F., Zhu, S., Luo, H., Zhi, X., & Wang, H. (2021) · Environmental Research Letters · doi:10.1088/1748-9326/abd7ad · Cited by 159
“Given the substantial uncertainties of projected changes in precipitation, the RMSE′ used to evaluate the climate simulation capability of each individual CMIP6 model is defined as follows (Gleckler et al 2008, Dong et al 2015).”
MethodThe CNRM Global Atmosphere Model ARPEGE‐Climat 6.3: Description and Evaluation
Roehrig, R., Beau, I., Saint‐Martin, D., Alias, A., Decharme, B., Guérémy, J., Voldoire, A., Abdel‐Lathif, A. Y., Bazile, E., Belamari, S., et al. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2020ms002075 · Cited by 158
“The diagnostic, proposed by Gleckler et al. (2008), provides a synthetic comparison of a set of simulations.”
BackgroundRegional Climate–Weather Research and Forecasting Model
Liang, X.-Z., Xu, M., Yuan, X., Ling, T., Choi, H. I., Zhang, F., Chen, L., Liu, S., Su, S., Qiao, F., et al. (2012) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-11-00180.1 · Cited by 158
“…predictions based on an ensemble of multiple physics configurations of a model or multiple models have recently been highlighted for their superior skills over those using a single configuration or model (Krishnamurti et al. 2000; Palmer et al. 2004; Gleckler et al. 2008; Kirtman and Min 2009).”
BackgroundKlocke, D., Pincus, R. L., & Quaas, J. (2011) · Journal of Climate · doi:10.1175/2011jcli4193.1 · Cited by 158
“Most measures of skill compare models to observations in global domains (e.g., Gleckler et al. 2008; Pincus et al. 2008; Reichler and Kim 2008).”
BackgroundImplications of regional improvement in global climate models for agricultural impact research
Ramírez-Villegas, J., Challinor, A. J., Thornton, P. K., & Jarvis, A. (2013) · Environmental Research Letters · doi:10.1088/1748-9326/8/2/024018 · Cited by 157
“Several questions arise from the results presented here and elsewhere [19, 32, 39].”
BackgroundMethodInfluentialVerification of the First 11 Years of IRI’s Seasonal Climate Forecasts
Barnston, A. G., Li, S., Mason, S. J., DeWitt, D. G., Goddard, L., & Gong, X. (2009) · Journal of Applied Meteorology and Climatology · doi:10.1175/2009jamc2325.1 · Cited by 156
“…consist of a deemed optimal superensemble of regionally specific Intergovernmental Panel on Climate Change (IPCC) model forecasts (e.g., Tebaldi et al. 2005; Greene et al. 2006; Furrer et al. 2007; Christensen et al. 2007; Gleckler et al. 2008) averaged over a period centered on the current year.”
BackgroundDynamical downscaling: Fundamental issues from an NWP point of view and recommendations
Hong, S., & Kanamitsu, M. (2014) · Asia-Pacific Journal of Atmospheric Sciences · doi:10.1007/s13143-014-0029-2 · Cited by 156
Evaluation and Projection of Surface Wind Speed Over China Based on CMIP6 GCMs
Wu, J., Shi, Y., & Xu, Y. (2020) · Journal of Geophysical Research Atmospheres · doi:10.1029/2020jd033611 · Cited by 153
“In assessing the monthly, seasonal, and annual mean wind speed over China, a normalized RMSE, similar to the relative error defined by Gleckler et al. (2008), is applied in this study.”
MethodKamruzzaman, M., Shahid, S., Islam, A. R. M. T., Hwang, S., Cho, J., Zaman, M. A. U., Ahmed, M., Rahman, M. M., & Hossain, M. B. (2021) · Theoretical and Applied Climatology · doi:10.1007/s00704-021-03691-0 · Cited by 153
“It is also suggested that GCMs should be selected on the basis of their performance in simulating the climate variable of interest to reduce the uncertainty in the projection of that variable (Gleckler et al. 2008; McMahon et al. 2015; Lutz et al. 2016; Sa’adi et al. 2017; Salman et al. 2018) .”
BackgroundA framework for testing the ability of models to project climate change and its impacts
Refsgaard, J. C., Madsen, H., Andréassian, V., Arnbjerg‐Nielsen, K., Davidson, T. A., Drews, M., Hamilton, D. P., Jeppesen, E., Kjellström, E., Olesen, J. E., et al. (2013) · Climatic Change · doi:10.1007/s10584-013-0990-2 · Cited by 154
“From different ensemble model experiments, it is generally concluded that the ensemble average provides a better representation of the current climate than any single climate model (Gleckler et al. 2008).”
BackgroundGlobal assessment of agreement among streamflow projections using CMIP5 model outputs
Koirala, S., Hirabayashi, Y., Roobavannan, M., & Kanae, S. (2014) · Environmental Research Letters · doi:10.1088/1748-9326/9/6/064017 · Cited by 152
“Further, the projections of each AOGCM were assumed to be equally plausible and performance metrics (as in Gleckler et al 2008) were not used to filter out models with weak performance.”
MethodConsiderations for parameter optimization and sensitivity in climate models
Neelin, J. D., Bracco, A., Luo, H., McWilliams, J. C., & Meyerson, J. E. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1015473107 · Cited by 135
“(iii) Objective functions that could be used for optimization procedures are poorly agreed on, although there has been considerable attempt to define scalar “metrics” to condense information (17, 18).”
BackgroundModelling and quantifying tomorrow's risks from natural hazards
Cremen, G., Galasso, C., & McCloskey, J. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.152552 · Cited by 150
Projections of leaf area index in earth system models
Mahowald, N. M., Lo, F., Zheng, Y., Harrison, L., Funk, C. C., Lombardozzi, D., & Goodale, C. L. (2016) · Earth System Dynamics · doi:10.5194/esd-7-211-2016 · Cited by 151
Towards improved and more routine Earth system model evaluation in CMIP
Eyring, V., Gleckler, P. J., Heinze, C., Stouffer, R. J., Taylor, K. E., Balaji, V., Guilyardi, E., Joussaume, S., Kindermann, S., Lawrence, B. N., et al. (2016) · Earth System Dynamics · doi:10.5194/esd-7-813-2016 · Cited by 133
“The PMP is implementing a diverse suite of summary statistics to objectively gauge the level of agreement between model simulations and observations across a broad range of space and time scales (Gleckler et al., 2008).”
Self-citationBrunner, L., Lorenz, R., Zumwald, M., & Knutti, R. (2019) · Environmental Research Letters · doi:10.1088/1748-9326/ab492f · Cited by 147
“The aim is to select diagnostics which are, on the one hand, relevant for the target and, on the other hand, add additional information to the already used diagnostics (Gleckler et al 2008, Lorenz et al 2018).”
BackgroundGulizia, C. N., & Camilloni, I. A. (2014) · International Journal of Climatology · doi:10.1002/joc.4005 · Cited by 146
“A relative error (E′mt) for a given model m and time period t (season/year) is computed according to Gleckler et al. (2008) as E′mt = Emt − Et Et (2) where Emt is the RMSE of the model m at the time period t and Et is the median of the RMSE of the N GCMs considered (N = 8 or 19 for each…”
MethodThe causes, effects and challenges of Sahelian droughts: a critical review
Épule, T. É., Peng, C. H., Lepage, L., & Chen, Z. (2013) · Regional Environmental Change · doi:10.1007/s10113-013-0473-z · Cited by 147
“…based climatological analysis that reveals that with the onset of Sahelian droughts in the late 1960s, a particular configuration of global SST characterized by warm anomalies in the Southern Hemisphere and cool anomalies in the Northern Hemisphere Oceans was observed (Gleckler et al. 2008).”
BackgroundBayesian estimation of local signal and noise in multimodel simulations of climate change
Duan, Q., & Phillips, T. J. (2010) · Journal of Geophysical Research Atmospheres · doi:10.1029/2009jd013654 · Cited by 144
“For historical climate simulations, such simple model averaging (SMA) is often found to replicate observational data better than even the best performing individual simulation [IPCC, 2001; Lambert and Boer, 2001; Phillips and Gleckler, 2006; Gleckler et al., 2008; Pincus et al., 2008; Reichler and Kim, 2008].”
BackgroundMethodENSO in the CMIP5 Simulations: Life Cycles, Diversity, and Responses to Climate Change
Chen, C., Cane, M. A., Wittenberg, A. T., & Chen, D. (2017) · Journal of Climate · doi:10.1175/jcli-d-15-0901.1 · Cited by 126
“Studies of CMIP models often identify subsets of good models based on various metrics (e.g., Gleckler et al. 2008; Kim and Yu 2012; Bellenger et al. 2014).”
MethodEvaluating Emergent Constraints on Equilibrium Climate Sensitivity
Caldwell, P. M., Zelinka, M. D., & Klein, S. A. (2018) · Journal of Climate · doi:10.1175/jcli-d-17-0631.1 · Cited by 132
“Unfortunately, models that perform well for some metrics may perform poorly for others (Gleckler et al. 2008), climate predictions from skillful models do not always agree (Waugh and Eyring 2008), and the ability to reproduce current climate does not necessarily imply predictive skill.”
BackgroundGuidelines for constructing climate scenarios
Mote, P. W., Brekke, L. D., Duffy, P. B., & Maurer, E. P. (2011) · Eos · doi:10.1029/2011eo310001 · Cited by 143
Future changes in precipitation extremes across China based on CMIP6 models
Xu, H., Chen, H., & Wang, H. (2021) · International Journal of Climatology · doi:10.1002/joc.7264 · Cited by 141
“This “portrait” diagram is based on calculations of the relative magnitudes of the area-weighted and temporal mean RMSEs for each model and each precipitation index, which are represented by the different colours shown in Figure 5 (Gleckler et al., 2008).”
BackgroundMethodRanking of global climate models for India using multicriterion analysis
Raju, K., & Nagesh Kumar, D. (2014) · Climate Research · doi:10.3354/cr01222 · Cited by 140
Building confidence in climate model projections: an analysis of inferences from fit
Baumberger, C., Knutti, R., & Hadorn, G. H. (2017) · Wiley Interdisciplinary Reviews Climate Change · doi:10.1002/wcc.454 · Cited by 140
Future Projections of Antarctic Ice Shelf Melting Based on CMIP5 Scenarios
Naughten, K. A., Meißner, K. J., Galton-Fenzi, B. K., England, M. H., Timmermann, R., & Hellmer, H. H. (2018) · Journal of Climate · doi:10.1175/jcli-d-17-0854.1 · Cited by 140
“Following the equations of Gleckler et al. (2008), we calculate the rms error for each model and variable with respect to ERA-Interim over 308–808S, as well as the ‘‘relative error,’’ which normalizes units across different variables.”
BackgroundMethodInfluentialShould we apply bias correction to global and regional climate model data
Ehret, U., Zehe, E., Wulfmeyer, V., Warrach‐Sagi, K., & Liebert, J. (2012) · Cited by 140
Payne, A. E., & Magnúsdóttir, G. (2015) · Journal of Geophysical Research Atmospheres · doi:10.1002/2015jd023586 · Cited by 140
Lafaysse, M., Hingray, B., Mezghani, A., Gailhard, J., & Terray, L. (2014) · Water Resources Research · doi:10.1002/2013wr014897 · Cited by 138
“The evaluation can focus on the climatological mean state of some variables, their daily distribution, seasonal cycle, interannual variability, or long-term trends [Gleckler et al., 2008; Santer et al., 2008; Errasti et al., 2011; Brands et al., 2011].”
BackgroundQasem, S. N., Samadianfard, S., Kheshtgar, S., Jarhan, S., Kişi, Ö., Shamshirband, S., & Chau, K. (2019) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2018.1564702 · Cited by 138
Comparison of observed and simulated tropical climate trends using a forward model of coralδ18O
Thompson, D. M., Ault, T. R., Evans, M. N., Cole, J. E., & Emile‐Geay, J. (2011) · Geophysical Research Letters · doi:10.1029/2011gl048224 · Cited by 139
“The average CGCM correlation pattern, weighted by the model’s ability to capture tropical climate variability [Gleckler et al., 2008], displayed a region of significant positive SST‐SSS correlations in the central to Figure 2.”
MethodRuane, A. C., & McDermid, S. (2017) · Earth Perspectives · doi:10.1186/s40322-017-0036-4 · Cited by 137
“…of this study each GCM is considered to be equally likely and exchangeable as there is no clear method to evaluate GCM performance in a climate that has not yet occurred (Gleckler et al. 2008; IPCC 2010), and the approach would likely not make sense for perturbed physics ensembles of a single GCM.”
MethodAssessment of CMIP6 Cloud Fraction and Comparison with Satellite Observations
Vignesh, P. P., Jiang, J. H., Kishore, P., Su, H., Smay, T., Brighton, N., & Velicogna, I. (2020) · Earth and Space Science · doi:10.1029/2019ea000975 · Cited by 137
“The Taylor diagram (Gleckler et al., 2008; Taylor, 2001) can provide a quantitative statistical summary of how well the spatial or temporal patterns match between the ensemble simulations and observations in terms of their correlations coefficients, their root mean square difference, and the…”
BackgroundWang, Y., Hsu, H., Chen, C., Tseng, W., Hsu, P., Lin, C., Chen, Y., Jiang, L., Lee, Y., Liang, H., et al. (2021) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2020ms002353 · Cited by 138
GCMeval – An interactive tool for evaluation and selection of climate model ensembles
Parding, K. M., Dobler, A., McSweeney, C., Landgren, O., Benestad, R. E., Erlandsen, H. B., Mezghani, A., Gregow, H., Räty, O., Viktor, E., et al. (2020) · Climate Services · doi:10.1016/j.cliser.2020.100167 · Cited by 133
A review of observed and projected changes in climate for the islands in the Caribbean
Karmalkar, A. V., Taylor, M. A., Campbell, J., Stephenson, T. S., New, M. G., Centella, A., Benzanilla, A., & Charlery, J. L. (2013) · Atmósfera · doi:10.1016/s0187-6236(13)71076-2 · Cited by 133
“The ensemble mean and median have been shown to match observations better than any individual models (Gleckler et al., 2008), but using equal weights in ensemble mean calculations is problematic due to intermodel relationships (Masson and Kuntti, 2011).”
BackgroundMethodZamani, Y., Monfared, S. A. H., Moghaddam, M. A., & Hamidianpour, M. (2020) · Theoretical and Applied Climatology · doi:10.1007/s00704-020-03406-x · Cited by 132
“According to Gleckler et al. (2008), the relative error for a model P over the time t, which is season per year in this study, is calculated by Eq.”
BackgroundMethodInfluentialResponding to climate change in national forests: a guidebook for developing adaptation options
Peterson, D. L., Millar, C. I., Joyce, L. A., Furniss, M. J., Halofsky, J. E., Neilson, R. P., & Morelli, T. L. (2011) · doi:10.2737/pnw-gtr-855 · Cited by 119
“In addition, the " mean model, " or average of a group of climate models, consistently outperforms any single model in most respects; the averaging seems to cancel out errors in individual models (Gleckler et al. 2008).”
BackgroundParedes‐Trejo, F., Barbosa, H. Á., Peñaloza‐Murillo, M. A., Moreno-Pizani, M. A., & Farías, A. J. (2016) · Atmósfera · doi:10.20937/atm.2016.29.04.04 · Cited by 129
Jakob, C. (2010) · Bulletin of the American Meteorological Society · doi:10.1175/2009bams2898.1 · Cited by 117
Sun, Q., Miao, C., & Duan, Q. (2015) · Journal of Geophysical Research Atmospheres · doi:10.1002/2014jd022994 · Cited by 127
“comprehensive analysis of climate model performance should be emphasized [Gleckler et al., 2008].”
BackgroundEvaluation of Soil Thermal Conductivity Schemes for Use in Land Surface Modeling
Dai, Y., Wei, N., Yuan, H., Zhang, S., Shangguan, W., Liu, S., Lu, X., & Xin, Y. (2019) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001723 · Cited by 127
“The relative error depicts the relative performance of each STC scheme, indicating how much better or worse the performance is compared to the median level of the scheme predictions (Gleckler et al., 2008).”
BackgroundMethodENERGETICS OF CLIMATE MODELS: NET ENERGY BALANCE AND MERIDIONAL ENTHALPY TRANSPORT
Lucarini, V., & Ragone, F. (2011) · Reviews of Geophysics · doi:10.1029/2009rg000323 · Cited by 126
He, C., Wang, Z., Zhou, T., & Li, T. (2019) · Journal of Climate · doi:10.1175/jcli-d-18-0427.1 · Cited by 125
Evaluating climate model outputs for hydrological applications
Wilby, R. L. (2010) · Hydrological Sciences Journal · doi:10.1080/02626667.2010.513212 · Cited by 124
“ple diagnostics because inter-comparison projects show that no single model consistently outperforms all others (Gleckler et al., 2008).”
BackgroundObjective calibration of regional climate models
Bellprat, O., Kotlarski, S., Lüthi, D., & Schär, C. (2012) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2012jd018262 · Cited by 106
“surface radiation balance, mean sea level pressure and total cloud cover [Gleckler et al., 2008].”
BackgroundChen, A., Huang, L., Liu, Q., & Piao, S. L. (2021) · Global Change Biology · doi:10.1111/gcb.15542 · Cited by 123
Multi-model climate projections for biodiversity risk assessments
Fordham, D. A., Wigley, T. M. L., & Brook, B. W. (2011) · Ecological Applications · doi:10.1890/11-0314.1 · Cited by 122
Metrics for the Diurnal Cycle of Precipitation: Toward Routine Benchmarks for Climate Models
Covey, C., Gleckler, P. J., Doutriaux, C. M., Williams, D. N., Dai, A., Fasullo, J., Trenberth, K. E., & Berg, A. (2016) · Journal of Climate · doi:10.1175/jcli-d-15-0664.1 · Cited by 122
“By ‘‘metrics’’ we mean a few numbers that condense large amounts of output into summary statistics (Gleckler et al. 2008).”
BackgroundSelf-citationNowicki, S., Bindschadler, R., Abe‐Ouchi, A., Aschwanden, A., Bueler, E., Choi, H., Fastook, J., Granzow, G. D., Greve, R., Gutowski, G., et al. (2013) · Journal of Geophysical Research Earth Surface · doi:10.1002/jgrf.20081 · Cited by 123
Piccolroaz, S., Woolway, R. I., & Merchant, C. J. (2020) · Climatic Change · doi:10.1007/s10584-020-02663-z · Cited by 120
“These require as input only air temperature, a variable that is more often available and commonly more reliable than other meteorological variables (Gleckler et al. 2008).”
BackgroundStatistical multi-model climate projections of surface ocean waves in Europe
Pérez, J., Menéndez, M., Camus, P., Mendez, F. J., & Losada, Í. J. (2015) · Ocean Modelling · doi:10.1016/j.ocemod.2015.06.001 · Cited by 120
AghaKouchak, A., & Mehran, A. (2013) · Water Resources Research · doi:10.1002/wrcr.20498 · Cited by 120
“, 2011b], and relative error [Gleckler et al., 2008].”
BackgroundMethodPerformance of the general circulation models in simulating temperature and precipitation over Iran
Abbasian, M. S., Moghim, S., & Abrishamchi, A. (2018) · Theoretical and Applied Climatology · doi:10.1007/s00704-018-2456-y · Cited by 119
“Gleckler et al. (2008) evaluated the performance of 22 CMIP3 GCMs in simulating several atmospheric variables relative to NCEP/NCAR reanalysis datasets.”
MethodAdachi, S. A., & Tomita, H. (2020) · Journal of Geophysical Research Atmospheres · doi:10.1029/2019jd032166 · Cited by 121
“The model democracy can be an efficient approach; however, the plausibility of the projected climate remains a major research subject (Collins et al., 2013; Flato et al., 2013; Gleckler et al., 2008; Knutti et al., 2010; Knutti, 2010; Stephenson et al., 2012).”
BackgroundAssessing relevant climate data for agricultural applications
Ramírez-Villegas, J., & Challinor, A. J. (2012) · Agricultural and Forest Meteorology · doi:10.1016/j.agrformet.2012.03.015 · Cited by 121
Ge, F., Zhu, S., Peng, T., Zhao, Y., Sielmann, F., Fraedrich, K., Zhi, X., Liu, X., Tang, W., & Ji, L. (2019) · Environmental Research Letters · doi:10.1088/1748-9326/aaff7e · Cited by 118
“…In the study, the relative model error is employed to evaluate the climate simulation capability of each individual RCM with regard to various climate parameters, which is represented by the relative root mean squared error (RMSE′; Gleckler et al 2008, Dong et al 2015) defined as follows.”
MethodColle, B. A., Booth, J. F., & Chang, E. K. M. (2015) · Current Climate Change Reports · doi:10.1007/s40641-015-0013-7 · Cited by 119
Bennett, K. E., Werner, A. T., & Schnorbus, M. (2012) · Journal of Climate · doi:10.1175/jcli-d-11-00417.1 · Cited by 118
“Additionally, selected models were ranked within the top 10 according to the Model Climate Performance Index (MCPI) in the Northern Hemisphere, which combines comparisons of multiple variables as simulated by GCMs to observed values (Gleckler et al. 2008).”
MethodOverland, J. E., Wang, M., Bond, N. A., Walsh, J. E., Катцов, В. М., & Chapman, W. L. (2011) · Journal of Climate · doi:10.1175/2010jcli3462.1 · Cited by 120
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.