Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Items 201–300 of 1,001
Kamworapan, S., & Surussavadee, C. (2019) · Advances in Meteorology · doi:10.1155/2019/1067365 · Cited by 117
Uncertainty in the relationship between climate forcing and hydrological response in UK catchments
Arnell, N. W. (2011) · Hydrology and earth system sciences · doi:10.5194/hess-15-897-2011 · Cited by 117
“Ability to simulate past behaviour is not necessarily a good guide to a model’s ability to project future changes, and there are many potential indicators of model skill (Gleckler et al., 2008).”
Projected future duration of the sea-ice-free season in the Alaskan Arctic
Wang, M., & Overland, J. E. (2015) · Progress In Oceanography · doi:10.1016/j.pocean.2015.01.001 · Cited by 116
Warnatzsch, E. A., & Reay, D. (2018) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2018.11.098 · Cited by 115
Cui, D., Liang, S., Wang, D., & Liu, Z. (2021) · Earth system science data · doi:10.5194/essd-13-5087-2021 · Cited by 115
Piccolroaz, S., Healey, N. C., Lenters, J. D., Schladow, S. G., Hook, S. J., Sahoo, G. B., & Toffolon, M. (2017) · Limnology and Oceanography · doi:10.1002/lno.10626 · Cited by 114
“AT predictions based on General Circulation Models (GCMs) or Regional Climate Models (RCMs) are usually more reliable than other meteorological variables (e.g., Gleckler et al. 2008), and downscaling is typically associated with smaller uncertainties (Dettinger 2013).”
BackgroundUsing Proper Divergence Functions to Evaluate Climate Models
Thorarinsdottir, T. L., Gneiting, T., & Gissibl, N. (2013) · SIAM/ASA Journal on Uncertainty Quantification · doi:10.1137/130907550 · Cited by 113
“Traditionally, climate models have been assessed by comparing summary statistics or point estimates that derive from the simulated model output to the corresponding observed quantities [1, 2].”
BackgroundInfluentialAbbas, A., Ullah, S., Ullah, W., Waseem, M., Dou, X., Zhao, C., Karim, A., Zhu, J., Hagan, D. F. T., Bhatti, A. S., et al. (2022) · International Journal of Climatology · doi:10.1002/joc.7602 · Cited by 113
Harding, B. L., Wood, A. W., & Prairie, J. R. (2012) · Hydrology and earth system sciences · doi:10.5194/hess-16-3989-2012 · Cited by 113
Radić, V., Cannon, A. J., Menounos, B., & Gi, N. (2015) · Journal of Geophysical Research Atmospheres · doi:10.1002/2015jd023279 · Cited by 113
“Nevertheless, a successful model performance over one region of interest and a set of climate variables does not guarantee similarly successful performance over another region or/and other set of variables [Gleckler et al., 2008].”
BackgroundMethodOptimally choosing small ensemble members to produce robust climate simulations
Evans, J. P., Ji, F., Abramowitz, G., & Ekström, M. (2013) · Environmental Research Letters · doi:10.1088/1748-9326/8/4/044050 · Cited by 112
“The ensemble mean is often used to synthesize the information present, and has been found to perform better than any individual ensemble member where the mean, rather than variance quantities, is of interest (Gleckler et al 2008, Pierce et al 2009, Pincus et al 2008, Reichler and Kim 2008).”
MethodRanking general circulation models for India using TOPSIS
Srinivasa Raju, K., & Nagesh Kumar, D. (2015) · Journal of Water and Climate Change · doi:10.2166/wcc.2014.074 · Cited by 100
Strengers, B. J., Müller, C., Schaeffer, M., Haarsma, R. J., Severijns, C., Gerten, D., Schaphoff, S., van den Houdt, R., & Oostenrijk, R. (2010) · International Journal of Climatology · doi:10.1002/joc.2132 · Cited by 91
“The latter deviations, especially in the precipitation pattern (Figure 2(c)), are common for many GCMs operating at the same grid size as ours (Solomon et al., 2007; Gleckler et al., 2008).”
BackgroundGirard, C., Pulido-Velázquez, M., Rinaudo, J., Pagé, C., & Caballero, Y. (2015) · Global Environmental Change · doi:10.1016/j.gloenvcha.2015.07.002 · Cited by 110
Schaller, N., Mahlstein, I., Cermak, J., & Knutti, R. (2011) · Journal of Geophysical Research · doi:10.1029/2010jd014963 · Cited by 90
“Gleckler et al. [2008] and Pincus et al. [2008] further state that all models have distinctive weaknesses in simulating specific variables.”
BackgroundMethodResultInfluentialAhmed, K., Shahid, S., Sachindra, D. A., Nawaz, N., & Chung, E.-S. (2019) · Journal of Hydrology · doi:10.1016/j.jhydrol.2019.03.092 · Cited by 110
Towards multi-resolution global climate modeling with ECHAM6-FESOM. Part II: climate variability
Rackow, T., Goessling, H., Jung, T. J., Sidorenko, D., Semmler, T., Barbi, D., & Handorf, D. (2016) · Climate Dynamics · doi:10.1007/s00382-016-3192-6 · Cited by 108
“Appendix 1: Computation of performance indices To obtain an objective view of the skill of a model to simulate basic quantities compared to observations, performance indices (PI) have been developed (e.g. Reichler and Kim 2008; Gleckler et al. 2008).”
MethodEvaluating Climate Models with an African Lens
James, R., Washington, R., Abiodun, B. J., Kay, G., Mutemi, J. N., Pokam, W. M., Hart, N. C. G., Artan, G. A., & Senior, C. A. (2017) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-16-0090.1 · Cited by 107
“These “skill scores” have important applications, for tracking model development over time (Reichler and Kim 2008) and for comparing or ranking models (Gleckler et al. 2008; Schaller et al. 2011; Watterson et al. 2014).”
BackgroundResultXie, S., Ma, H.-Y., Boyle, J. S., Klein, S. A., & Zhang, Y. (2012) · Journal of Climate · doi:10.1175/jcli-d-12-00134.1 · Cited by 103
“We have started to apply those standard metrics that are commonly used to systematically assess climate model performance skill (Gleckler et al. 2008) to the hindcast runs with emphasis given”
MethodBias correcting climate model multi-member ensembles to assess climate change impacts on hydrology
Chen, J., Brissette, F. P., Zhang, X., Chen, H., Guo, S., & Zhao, Y. (2019) · Climatic Change · doi:10.1007/s10584-019-02393-x · Cited by 106
The Response of the Northwest Atlantic Ocean to Climate Change
Alexander, M. A., Shin, S., Scott, J. D., Curchitser, E., & Stock, C. A. A. (2019) · Journal of Climate · doi:10.1175/jcli-d-19-0117.1 · Cited by 106
Comparing Methods to Constrain Future European Climate Projections Using a Consistent Framework
Brunner, L., McSweeney, C., Ballinger, A. P., Befort, D. J., Benassi, M., Booth, B. B. B., Coppola, E., de Vries, H., Harris, G. R., Hegerl, G. C., et al. (2020) · Journal of Climate · doi:10.1175/jcli-d-19-0953.1 · Cited by 106
Weighting of model results for improving best estimates of climate change
Räisänen, J., Ruokolainen, L., & Ylhäisi, J. S. (2009) · Climate Dynamics · doi:10.1007/s00382-009-0659-8 · Cited by 106
“The dependence on the predictor reflects the well-known fact that the relative performance of the models varies with the aspect of climate considered (Gleckler et al. 2008).”
BackgroundJuneng, L., Tangang, F., Chung, J. X., Ngai, S. T., Tay, T., Narisma, G. T., Cruz, F. A. T., Phan‐Van, T., Ngo‐Duc, T., Santisirisomboon, J., et al. (2016) · Climate Research · doi:10.3354/cr01386 · Cited by 106
Santos, J. A., Belo‐Pereira, M., Fraga, H., & Pinto, J. G. (2016) · Journal of Geophysical Research Atmospheres · doi:10.1002/2015jd024399 · Cited by 105
“In fact, multimodel ensembles have been shown to consistently outperform individual model experiments for different atmospheric fi elds and spatial domains [e.g., Gleckler et al., 2008].”
Chen, J., Li, C., Brissette, F. P., Chen, H., Wang, M., & Essou, G. R. C. (2018) · Journal of Hydrology · doi:10.1016/j.jhydrol.2018.03.040 · Cited by 105
Kumar, D., Kodra, E., & Ganguly, A. R. (2014) · Climate Dynamics · doi:10.1007/s00382-014-2070-3 · Cited by 105
“Reliable and complete observational record is available from 1979 (Gleckler et al. 2008), and data from most of the CMIP3 models from ‘20c3m’ experiment is available until 1999.”
MethodA Hierarchical Statistical Framework for Emergent Constraints: Application to Snow‐Albedo Feedback
Bowman, K. W., Cressie, N., Qu, X., & Hall, A. D. (2018) · Geophysical Research Letters · doi:10.1029/2018gl080082 · Cited by 102
“While historic and current observations can be used to benchmark climate models (e.g., Gleckler et al., 2008; Teixeira et al., 2014), the establishment of robust relationships between contemporary performance and the credibility in future response has proven difficult.”
BackgroundImpact of increased resolution on long-standing biases in HighResMIP-PRIMAVERA climate models
Moreno‐Chamarro, E., Caron, L., Tomas, S. L., Vegas-Regidor, J., Gutjahr, O., Moine, M., Putrasahan, D. A., Roberts, C. D., Roberts, M. J., Senan, R., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-15-269-2022 · Cited by 102
On the interpretation of constrained climate model ensembles
Sanderson, B. M., & Knutti, R. (2012) · Geophysical Research Letters · doi:10.1029/2012gl052665 · Cited by 94
“mance, resolution, as well as the degree of complexity by which the different models describe various processes vary substantially [Gleckler et al., 2008; Reichler and Kim, 2008].”
BackgroundGoosse, H., Lefebvre, W., de Montety, A., Crespin, E., & Orsi, A. H. (2008) · Climate Dynamics · doi:10.1007/s00382-008-0500-9 · Cited by 102
“Lefebvre and Goosse 2008a; Gleckler et al. 2008 ).”
BackgroundAre CMIP5 Models Better than CMIP3 Models in Simulating Precipitation over East Asia?
Kusunoki, S., & Arakawa, O. (2015) · Journal of Climate · doi:10.1175/jcli-d-14-00585.1 · Cited by 102
Can a Regional Climate Model Improve the Ability to Forecast the North American Monsoon?
Castro, C. L., Chang, H.-I., Domínguez, F., Carrillo, C. M., Schemm, J.-K. E., & Juang, H. H. (2012) · Journal of Climate · doi:10.1175/jcli-d-11-00441.1 · Cited by 101
“This would follow similar strategies that have been used to differentially weight GCMs from the Intergovernmental Panel on Climate Change Fourth Assessment report (IPCC AR4; e.g., Gleckler et al. 2008; Maxino et al. 2008; Dominguez et al. 2010).”
MethodExploiting large ensembles for a better yet simpler climate model evaluation
Suarez‐Gutierrez, L., Milinski, S., & Maher, N. (2021) · Climate Dynamics · doi:10.1007/s00382-021-05821-w · Cited by 101
The Role of the Sahara Low in Summertime Sahel Rainfall Variability and Change in the CMIP3 Models
Biasutti, M., Sobel, A. H., & Camargo, S. J. (2009) · Journal of Climate · doi:10.1175/2009jcli2969.1 · Cited by 100
“6, left), the strengthening of the Sahara low is limited both in intensity and in vertical extent; conversely, the ‘‘wet’’ models (Fig.”
BackgroundMethodBrown, C. (2010) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)wr.1943-5452.65 · Cited by 101
Bukovsky, M., Gochis, D., & Mearns, L. O. (2013) · Journal of Climate · doi:10.1175/jcli-d-12-00538.1 · Cited by 100
“This study presents another example of how difficult it is to determine how to evaluate climate models (regional or global) to determine their credibility (e.g., Gleckler et al. 2008; Knutti et al. 2010).”
BackgroundEvaluation of Soil Moisture in CMIP6 Simulations
Qiao, L., Zuo, Z., & Xiao, D. (2021) · Journal of Climate · doi:10.1175/jcli-d-20-0827.1 · Cited by 100
“In addition to systematic bias, CMIP models employ various structures and physical parameterization schemes (Gleckler et al. 2008; Jamison and Kravt-sov 2010), with the result that data output can be highly variable among different models.”
MethodQuantitative performance metrics for stratospheric-resolving chemistry-climate models
Waugh, D. W., & Eyring, V. (2008) · Atmospheric Chemistry and Physics · doi:10.5194/acp-8-5699-2008 · Cited by 87
“A different approach was used by Reichler and Kim (2008) and Gleckler et al. (2008), who normalized the error by the “typical” model error for each quantity.”
Nowicki, S., Bindschadler, R., Abe‐Ouchi, A., Aschwanden, A., Bueler, E., Choi, H., Fastook, J., Granzow, G. D., Greve, R., Gutowski, G., et al. (2013) · Journal of Geophysical Research Earth Surface · doi:10.1002/jgrf.20076 · Cited by 100
“, 2005], evaluating the skill of models is difficult, as different metrics produce different rankings of climate models considered [Gleckler et al., 2008].”
Wu, Y., Liu, S., & Gallant, A. L. (2012) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2012.04.058 · Cited by 99
“Combining output across multiple GCMs can provide more reliable representation of regional changes and uncertainties than result from single models by reducing the influences of weaknesses or biases inherited from individual models (Gleckler et al., 2008; Lambert and Boer, 2001; Pierce et al., 2009; Tebaldi and Knutti…”
BackgroundUrban warming and future air-conditioning use in an Asian megacity: importance of positive feedback
Takane, Y., Kikegawa, Y., 原, 政., & Grimmond, S. (2019) · npj Climate and Atmospheric Science · doi:10.1038/s41612-019-0096-2 · Cited by 99
Fasullo, J. (2020) · Geoscientific model development · doi:10.5194/gmd-13-3627-2020 · Cited by 99
Understanding the CMIP3 Multimodel Ensemble
Annan, J. D., & Hargreaves, J. C. (2011) · Journal of Climate · doi:10.1175/2011jcli3873.1 · Cited by 98
Computer Simulation, Measurement, and Data Assimilation
Parker, W. S. (2017) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axv037 · Cited by 99
Rousseau, A. N., Klein, I. M., Freudiger, D., Gagnon, P., Frigon, A., & Ratté-Fortin, C. (2014) · Journal of Hydrology · doi:10.1016/j.jhydrol.2014.10.053 · Cited by 98
An Assessment of GCM Skill in Simulating Persistence across Multiple Time Scales
Johnson, F. M., Westra, S., Sharma, A., & Pitman, A. J. (2011) · Journal of Climate · doi:10.1175/2011jcli3732.1 · Cited by 98
“Gleckler et al. (2008) address this by proposing a model variability index, which is based on the ratio of the observed and modeled monthly variances, relative to the monthly climatology.”
BackgroundMethodInfluentialOn climate change skepticism and denial in tourism
Hall, C. M., Amelung, B., Cohen, S., Eijgelaar, E., Gößling, S., Higham, J. E. S., Leemans, R., Peeters, P., Ram, Y., & Scott, D. J. (2014) · Journal of Sustainable Tourism · doi:10.1080/09669582.2014.953544 · Cited by 97
Gao, J., Li, Y., Zhu, B., Hu, B., Wang, L., & Bao, F. (2020) · Atmospheric Chemistry and Physics · doi:10.5194/acp-20-10831-2020 · Cited by 89
Tefera, G. W., Teferi, E., Bantider, A., Dile, Y. T., & Taye, M. T. (2018) · Dynamics of Atmospheres and Oceans · doi:10.1016/j.dynatmoce.2018.06.002 · Cited by 96
CLIMBra - Climate Change Dataset for Brazil
Ballarin, A. S., Sone, J. S., Gesualdo, G. C., Schwamback, D., Reis, A., Almagro, A., & Wendland, E. C. (2023) · Scientific Data · doi:10.1038/s41597-023-01956-z · Cited by 96
Atmospheric rivers in 20 year weather and climate simulations: A multimodel, global evaluation
Guan, B., & Waliser, D. E. (2017) · Journal of Geophysical Research Atmospheres · doi:10.1002/2016jd026174 · Cited by 95
“, median) model within the ensemble (in this case, 24 models) for a given Fr and can be compactly shown in a “portrait diagram” [Gleckler et al., 2008] for a collection of models and evaluation metrics.”
BackgroundMethodInfluentialSamadianfard, S., Asadi, E., Jarhan, S., Kazemi, H., Kheshtgar, S., Kişi, Ö., Sajjadi, S., & Manaf, A. A. (2017) · Soil and Tillage Research · doi:10.1016/j.still.2017.08.012 · Cited by 95
Du, Y., Wang, D., Zhu, J., Wang, D., Qi, X., & Cai, J. (2022) · International Journal of Climatology · doi:10.1002/joc.7616 · Cited by 94
“IVS is a symmetric variability statistic that is used to measure the similarity of interannual variation between GCM output and observation (Gleckler et al., 2008; Scherrer, 2011), which is calculated as”
MethodTowards Prediction of Decadal Climate Variability and Change
Murphy, J. M., Катцов, В. М., Keenlyside, N. S., Kimoto, M., Meehl, G. A., Mehta, V. M., Pohlmann, H., Scaife, A. A., & Smith, D. M. (2010) · Procedia Environmental Sciences · doi:10.1016/j.proenv.2010.09.018 · Cited by 93
Selection of GCMs for the projection of spatial distribution of heat waves in Pakistan
Khan, N., Shahid, S., Ahmed, K., Wang, X., Ali, R., Ismail, T., & Nawaz, N. (2020) · Atmospheric Research · doi:10.1016/j.atmosres.2019.104688 · Cited by 89
Ma, P., Harrop, B. E., Larson, V. E., Neale, R., Gettelman, A., Morrison, H., Wang, H., Zhang, K., Klein, S. A., Zelinka, M. D., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-15-2881-2022 · Cited by 93
Climate change multi-model projections in CMIP6 scenarios in Central Hokkaido, Japan
Peng, S., Wang, C., Li, Z., Mihara, K., Kuramochi, K., Toma, Y., & Hatano, R. (2023) · Scientific Reports · doi:10.1038/s41598-022-27357-7 · Cited by 93
“Gleckler et al. (2008) created a set of measurements to accurately measure the relative advantages of CMIP3 models 28 .”
MethodOcean Heat Transport as a Cause for Model Uncertainty in Projected Arctic Warming
Mahlstein, I., & Knutti, R. (2010) · Journal of Climate · doi:10.1175/2010jcli3713.1 · Cited by 93
“Hall (2004) shows that the strength of snow albedo feedback in spring is coupled to the strength of future temperature increase in each model.”
BackgroundDoes global warming amplify interannual climate variability?
He, C., & Li, T. (2018) · Climate Dynamics · doi:10.1007/s00382-018-4286-0 · Cited by 92
“Multi-model median is superior to multi-model mean in representing the forced response since it is robust to outliers (Gleckler et al. 2008).”
BackgroundRapid flips between warm and cold extremes in a warming world
Wu, S., Luo, M., Lau, G. N., Zhang, W., Wang, L., Liu, Z., Lin, L., Wang, Y., Ge, E., Li, J., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58544-5 · Cited by 92
“This approach effectively mitigates the uncertainties from inter-model differences as it has been shown to outperform a single model in simulating 64 – 66 .”
Pirtle, Z., Meyer, R., & Hamilton, A. L. (2010) · Environmental Science & Policy · doi:10.1016/j.envsci.2010.04.004 · Cited by 91
Seasonal and regional biases in CMIP5 precipitation simulations
Liu, Z., Mehran, A., Phillips, T. J., & AghaKouchak, A. (2014) · Climate Research · doi:10.3354/cr01221 · Cited by 91
“Gleckler et al. (2008) has introduced a number of performance metrics for evaluation of climate model simulations against observations.”
BackgroundHe, C., Lin, A., Gu, D., Li, C., Zheng, B., Wu, B., & Zhou, T. (2016) · Theoretical and Applied Climatology · doi:10.1007/s00704-016-2001-9 · Cited by 91
“Following the recommendation by Gleckler et al. (2008), we extract the forced response by using the multi-model median (MMM), which effectively suppresses the bias and internal variability of individual models.”
MethodKling, C. L., Arritt, R. W., Calhoun, G., & Keiser, D. A. (2017) · Annual Review of Resource Economics · doi:10.1146/annurev-resource-100516-033533 · Cited by 90
Filahi, S., Tramblay, Y., Mouhir, L., & Diaconescu, E. P. (2017) · International Journal of Climatology · doi:10.1002/joc.5127 · Cited by 90
“To assess model performance, we use Taylor diagrams (Taylor, 2001) and Portrait diagrams (Gleckler et al., 2008).”
BackgroundMethodGao, J., Li, Y., Zhu, B., Hu, B., Wang, L., & Bao, F. (2020) · doi:10.5194/acp-2020-140 · Cited by 89
McMahon, T. A., Peel, M. C., & Karoly, D. J. (2015) · Hydrology and Earth System Sciences · doi:10.5194/hess-19-361-2015 · Cited by 89
“Table 2a indicates that only two papers (Räisänen, 2007; Gleckler et al., 2008) detail numerical measures for both mean annual precipitation and temperature for 21 and 22 CMIP3 GCMs, respectively, at a global scale. Reifen and Toumi (2009) (17 GCMs) and Knutti et al. (2010) (23 GCMs) address, inter alia only mean annu…”
BackgroundMethodInfluentialRadić, V., & Clarke, G. K. C. (2011) · Journal of Climate · doi:10.1175/jcli-d-11-00011.1 · Cited by 81
“Continuing to follow Gleckler et al. (2008), we calculate the model variability index (MVI), which serves as an overall index of model performance for simulating the interannual variability.”
BackgroundMethodResultInfluentialWhat Influences the Skill of Climate Models over the Continents?
Watterson, I. G., Bathols, J., & Heady, C. (2013) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-12-00136.1 · Cited by 88
“Applying objective metrics to the comparisons of such data, Gleckler et al. (2008) found a considerable range of skill across the CMIP3 ensemble in representing atmospheric and surface features over the globe. Reichler and Kim (2008) accumulated skill scores into a single metric, based on which it was argued that mode…”
BackgroundMethodResultInfluentialStriking stationarity of large-scale climate model bias patterns under strong climate change
Krinner, G., & Flanner, M. G. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807912115 · Cited by 88
“F climate projections are based on numerical simulations from global climate models that are grounded in first principles but exhibit well-documented biases in their simulation of the current climate state (1, 2), thus raising questions about their fitness for climate projections.”
BackgroundDorado‐Liñán, I., Büntgen, U., González-Rouco, J. F., Zorita, E., Montávez, J. P., Gómez‐Navarro, J. J., Brunet, M., Heinrich, I., Helle, G., & Gutiérrez, E. (2012) · Climate of the past · doi:10.5194/cp-8-919-2012 · Cited by 87
An evaluation of the consistency of extremes in gridded precipitation data sets
Timmermans, B. W., Wehner, M. F., Cooley, D., O'Brien, T. A., & Krishnan, H. (2019) · Climate Dynamics · doi:10.1007/s00382-018-4537-0 · Cited by 87
“An ensemble approach would seem natural, analogous to studies of numerical simulations (Gleckler et al. 2008; Sanderson et al. 2017), where member products can be treated equally or weighted according to prior information, perhaps based upon some measure of performance.”
BackgroundDownscaled and debiased climate simulations for North America from 21,000 years ago to 2100AD
Lorenz, D. J., Nieto‐Lugilde, D., Blois, J. L., Fitzpatrick, M. C., & Williams, J. W. (2016) · Scientific Data · doi:10.1038/sdata.2016.48 · Cited by 87
Strengths and challenges for transient Mid- to Late Holocene simulations with dynamical vegetation
Braconnot, P., Zhu, D., Marti, O., & Servonnat, J. (2019) · Climate of the past · doi:10.5194/cp-15-997-2019 · Cited by 91
A novel method for ranking CMIP6 global climate models over the southeast Asian region
Desmet, Q., & Ngo‐Duc, T. (2021) · International Journal of Climatology · doi:10.1002/joc.7234 · Cited by 87
Climate simulations of major estuarine watersheds in the Mid-Atlantic region of the US
Najjar, R. G., Patterson, L., & Graham, S. (2008) · Climatic Change · doi:10.1007/s10584-008-9521-y · Cited by 86
“To summarize the overall model evaluation, we find that the multi-model mean is superior to any individual model, a finding similar to numerous climate model intercomparison studies conducted at global and continental scales ( Gleckler et al. 2008; Lambert and Boer 2001; Phillips and Gleckler 2006 ;R eichler and Kim 2…”
ResultA framework for assessing flood frequency based on climate projection information
Raff, D. A., Pruitt, T., & Brekke, L. D. (2009) · Hydrology and earth system sciences · doi:10.5194/hess-13-2119-2009 · Cited by 86
Grose, M., Brown, J. N., Narsey, S., Brown, J. R., Murphy, B. F., Langlais, C., Gupta, A. S., Moise, A. F., & Irving, D. (2014) · International Journal of Climatology · doi:10.1002/joc.3916 · Cited by 86
“We also suggest that the evaluation is not be used to weight the model results since this is only appropriate for targeted applications, and there are risks in doing this overall (Gleckler et al., 2008; Weigel et al., 2010).”
MethodSamadianfard, S., Ghorbani, M. A., & Mohammadi, B. (2018) · Information Processing in Agriculture · doi:10.1016/j.inpa.2018.06.005 · Cited by 86
Zounemat‐Kermani, M., Seo, Y., Kim, S., Ghorbani, M. A., Samadianfard, S., Naghshara, S., Kim, N. W., & Singh, V. P. (2019) · Applied Sciences · doi:10.3390/app9122534 · Cited by 86
Evaluating the Performance of Climate Models Based on Wasserstein Distance
Vissio, G., Lembo, V., Lucarini, V., & Ghil, M. (2020) · Geophysical Research Letters · doi:10.1029/2020gl089385 · Cited by 85
“Gleckler et al. (2008) proposed a multidimensional metric based on the comparison of the spatiotemporal variability of many climatic fields with respect to reference data sets and found that creating a scalar comprehensive metric is nontrivial.”
BackgroundA regional climate simulation over the Iberian Peninsula for the last millennium
Gómez‐Navarro, J. J., Montávez, J. P., Jerez, S., Jiménez‐Guerrero, P., Lorente‐Plazas, R., González-Rouco, J. F., & Zorita, E. (2011) · Climate of the past · doi:10.5194/cp-7-451-2011 · Cited by 85
“Here we discuss briefly some general aspects of these forcings; for a full description of this simul tion and their xternal forcings the reader is referr d to González-Rouco et al. (2003); Zorita t al.”
BackgroundInfluentialLi, H., Chen, H., Wang, H., & Yu, E. (2018) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2018.05.324 · Cited by 84
Observed trends and climate projections affecting marine ecosystems in the Canadian Arctic
Steiner, N. S., Azetsu‐Scott, K., Hamilton, J., Hedges, K. J., Hu, X., Janjua, M. Y., Lavoie, D., Loder, J. W., Melling, H., Merzouk, A., et al. (2015) · Environmental Reviews · doi:10.1139/er-2014-0066 · Cited by 83
Kişi, Ö., Choubin, B., Deo, R. C., & Yaseen, Z. M. (2019) · Hydrological Sciences Journal · doi:10.1080/02626667.2019.1632460 · Cited by 82
“2005, Gleckler et al .”
Lee, T., Waliser, D. E., Li, J. F., Landerer, F. W., & Gierach, M. M. (2013) · Journal of Climate · doi:10.1175/jcli-d-12-00591.1 · Cited by 82
Noor, M., Ismail, T., Shahid, S., Ahmed, K., Chung, ., & Nawaz, N. (2019) · Theoretical and Applied Climatology · doi:10.1007/s00704-019-02874-0 · Cited by 82
“This approach can be justified at global scale for assessing the mean climate that is efficiently projected by the average of models (Gleckler et al. 2008; Reichler and Kim 2008).”
BackgroundRepresentation of the terrestrial carbon cycle in CMIP6
Gier, B. K., Schlund, M., Friedlingstein, P., JONES, C. D., Jones, C., Zaehle, S., & Eyring, V. (2024) · Biogeosciences · doi:10.5194/bg-21-5321-2024 · Cited by 83
The dynamics of the Indian Ocean sea surface temperature forcing of Sahel drought
Lu, J. (2009) · Climate Dynamics · doi:10.1007/s00382-009-0596-6 · Cited by 81
“Gleckler et al. (2008) ranks CM2.0 as one of those having the most realistic climatological tropical precipitation pattern among the CMIP3 participating models.”
BackgroundLee, J., Sperber, K. R., Gleckler, P. J., Bonfils, C., & Taylor, K. E. (2018) · Climate Dynamics · doi:10.1007/s00382-018-4355-4 · Cited by 81
Self-citationMamalakis, A., Langousis, A., Deidda, R., & Marrocu, M. (2017) · Water Resources Research · doi:10.1002/2016wr019578 · Cited by 80
Cool roof strategies for urban thermal resilience to extreme heatwaves in tropical cities
Khorat, S., Das, D., Khatun, R., Aziz, S. M., Anand, P., Khan, A., Santamouris, M., & Niyogi, D. (2023) · Energy and Buildings · doi:10.1016/j.enbuild.2023.113751 · Cited by 80
Evaluation of extreme sub-daily precipitation in high-resolution global climate model simulations
Wehner, M. F., Lee, J., Risser, M. D., Ullrich, P. A., Gleckler, P. J., & Collins, W. D. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2019.0545 · Cited by 79
“In this paper, we use standard practice model evaluation techniques [17,18] to analyse the quality of seasonal 3-hourly precipitation extremes produced by available HighResMIP models.”
MethodSelf-citationRobust Evaluation of ENSO in Climate Models: How Many Ensemble Members Are Needed?
Lee, J., Planton, Y., Gleckler, P. J., Sperber, K. R., Guilyardi, É., Wittenberg, A. T., McPhaden, M. J., & Pallotta, G. (2021) · Geophysical Research Letters · doi:10.1029/2021gl095041 · Cited by 78
“Figure 1 provides a quick-look summary of CMIP6 results using a portrait plot (Gleckler et al., 2008) for each of the three metrics collections defined as part of the CEM2021.”
BackgroundMethodSelf-citationWhat have we learnt from palaeoclimate simulations?
Harrison, S. P., Bartlein, P. J., & Prentice, I. C. (2016) · Journal of Quaternary Science · doi:10.1002/jqs.2842 · Cited by 79
“Benchmarking is a measurement tool, whereby model outputs are compared with a pre-defined set of observations using appropriate metrics to define the degree of agreement quantitatively (Taylor, 2001; Gleckler et al., 2008).”
MethodChen, H., Sun, J., & Li, H. (2017) · Atmospheric and Oceanic Science Letters · doi:10.1080/16742834.2017.1367625 · Cited by 78
“The MRE provides a measure of how well a given model simulates the patterns when compared with the typical model error (MME; Gleckler, Taylor, and Doutriaux 2008).”
MethodMulti-Decadal Variability and Future Changes in Precipitation over Southern Africa
Sian, K. T. C. L. K., Wang, J., Ayugi, B. O., Nooni, I. K., & Ongoma, V. (2021) · Atmosphere · doi:10.3390/atmos12060742 · Cited by 77
“The RMSD shows how far the model data are from the reference, and portrays uncertainties in both spatial pattern and annual cycle [60].”
BackgroundMethodObservations for Model Intercomparison Project (Obs4MIPs): status for CMIP6
Waliser, D. E., Gleckler, P. J., Ferraro, R. D., Taylor, K. E., Ames, S. K., Biard, J. C., Bosilovich, M. G., Brown, O. B., Chepfer, H., Cinquini, L., et al. (2020) · Geoscientific model development · doi:10.5194/gmd-13-2945-2020 · Cited by 77
Self-citationAssessment of CMIP6 and CMIP5 model performance for extreme temperature in China
Luo, N., Guo, Y., Gao, Z., Kexin, C. H. E. N., & Chou, J. (2020) · Atmospheric and Oceanic Science Letters · doi:10.1080/16742834.2020.1808430 · Cited by 76
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.