Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Items 601–700 of 1,001
Bhowmik, R. D., Sharma, A., & Sankarasubramanian, A. (2017) · Journal of Climate · doi:10.1175/jcli-d-17-0225.1 · Cited by 27
Which climate change path are we following? Bad news from Scots pine
Bombi, P., D’Andrea, E., Rezaie, N., Cammarano, M., & Matteucci, G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0189468 · Cited by 27
Iyer, G. C., Brown, M. L., Cohen, S., Macknick, J., Patel, P., Wise, M., Binsted, M. T., & Voisin, N. (2019) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2019.109416 · Cited by 27
A Review on Evaluation Methods of Climate Modeling
Zongci, Z., Luo, Y., & Huang, J. (2013) · Advances in Climate Change Research · doi:10.3724/sp.j.1248.2013.137 · Cited by 27
Non-stationary return levels of CMIP5 multi-model temperature extremes
Cheng, L., Phillips, T. J., & AghaKouchak, A. (2015) · Climate Dynamics · doi:10.1007/s00382-015-2625-y · Cited by 27
“…simulations are known to have high uncertainty (Kharin et al. 2013; Mehran et al. 2014; Liu et al. 2014; Sillmann et al. 2013), and different methods have been developed to assess their consistency against observations (Gleckler et al. 2008; AghaKouchak and Mehran 2013; Phillips and Gleckler 2006).”
MethodAssessing the Effects of Climate Change on Precipitation and Flood Damage in Wisconsin
Schuster, Z., Potter, K. W., & Liebl, D. S. (2011) · Journal of Hydrologic Engineering · doi:10.1061/(asce)he.1943-5584.0000513 · Cited by 27
Twenty‐first century changes in daily temperature variability in CMIP3 climate models
Ylhäisi, J. S., & Räisänen, J. (2013) · International Journal of Climatology · doi:10.1002/joc.3773 · Cited by 27
“, 2007a), different sets of metrics for model evaluation (Gleckler et al., 2008; Reichler and Kim, 2008; Räisänen et al., 2010) or low-frequency variability (Räisänen, 2002; Kravtsov and Spannangle, 2007).”
BackgroundLoikith, P. C., & Broccoli, A. J. (2014) · Journal of Climate · doi:10.1175/jcli-d-13-00544.1 · Cited by 27
“…used in this section aims to summarize the overall ability of the models to simulate the characteristics of observed temperature extremes given the very large number of dimensions inherent to this analysis. a. Composite pattern correlations A portrait diagram (Gleckler et al. 2008) is used in Fig.”
MethodBao, Y., Gao, Y., Lü, S., Wang, Q., Zhang, S., Jianwei, X., Li, R., Li, S., Ma, D., Meng, X., et al. (2014) · 气象学报:英文版 · doi:10.1007/s13351-014-4023-5 · Cited by 27
“(6) In some model evaluation studies, an MVI value equal to 0.5 is considered as a good representation of IAV (Scherrer, 2011; Anav et al., 2013b).”
BackgroundMethodInfluentialWójcik, R. (2014) · International Journal of Climatology · doi:10.1002/joc.4015 · Cited by 26
“Recent studies have shown a measurable improvement in GCM performance over the past few years (Gleckler et al., 2008; Reichler and Kim, 2008).”
BackgroundResultDuchesne, L., Houle, D., Ouimet, R., Lambert, M.-C., & Logan, T. (2016) · PeerJ · doi:10.7717/peerj.1767 · Cited by 26
Yimer, S. M., Bouanani, A., Kumar, N., Tischbein, B., & Borgemeister, C. (2022) · Climate · doi:10.3390/cli10070095 · Cited by 26
“Secondly, climate model performance was assessed at the grid and basin level using CRU data as a reference dataset.”
InfluentialBotero‐Acosta, A., Ficklin, D. L., Ehsani, N., & Knouft, J. H. (2022) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2022.101202 · Cited by 26
Leveraging global climate models to assess multi-year hydrologic drought
Vieira, M. J. F., & Stadnyk, T. (2023) · npj Climate and Atmospheric Science · doi:10.1038/s41612-023-00496-y · Cited by 26
Desta, H. B., Melesse, A. M., & Tegegne, G. (2024) · Theoretical and Applied Climatology · doi:10.1007/s00704-024-05187-z · Cited by 26
Radiation budget biases in AMIP5 models over the East Asian monsoon region
Wang, F., Yang, S., & Wu, T. (2014) · Journal of Geophysical Research Atmospheres · doi:10.1002/2014jd022243 · Cited by 26
Markus, M., Angel, J. R., Byard, G., McConkey, S. A., Zhang, C., Cai, X., Notaro, M., & Ashfaq, M. (2018) · Journal of Hydrologic Engineering · doi:10.1061/(asce)he.1943-5584.0001614 · Cited by 26
Dhakal, K., Kakani, V. G., & Linde, E. (2018) · Atmospheric and Climate Sciences · doi:10.4236/acs.2018.82011 · Cited by 26
“Multi-model ensemble averages are found to outperform individual models when tested for the historical period [33].”
BackgroundHow well do climate models simulate regional atmospheric circulation over East Asia?
Zhao, C., Jiang, Z., Sun, X., Li, W., & Li, L. (2019) · International Journal of Climatology · doi:10.1002/joc.6205 · Cited by 26
“This result seems to be in agreement with a few previous studies (Gleckler et al., 2008; Radi c and Clarke, 2011; Sheffield et al., 2013a, 2013b; Su et al., 2013).”
ResultTamoffo, A. T., Amekudzi, L. K., Weber, T., Vondou, D. A., Yamba, E. I., & Jacob, D. (2021) · Journal of Climate · doi:10.1175/jcli-d-21-0487.1 · Cited by 26
Longmate, J. M., Risser, M. D., & Feldman, D. (2023) · Climate Dynamics · doi:10.1007/s00382-023-06846-z · Cited by 26
“While Taylor diagrams are helpful for visualizing a large amount of information about IC ensemble behavior between models and within models, the units of distance between points are not uniform or easily interpretable (Gleckler et al. 2008).”
BackgroundUncertainties in Projections of Future Changes in Extremes
Brekke, L. D., & Barsugli, J. J. (2012) · Water science and technology library · doi:10.1007/978-94-007-4479-0_11 · Cited by 26
Li, J. ‐. F., Lee, W., Lee, T., Fetzer, E. J., Yu, J., Kubar, T. L., & Böening, C. (2015) · Journal of Geophysical Research Atmospheres · doi:10.1002/2014jd022538 · Cited by 26
“Representation of the present climate by climate models is important to the reliability of climate projections using thesemodels [e.g., Pierce et al., 2006; Gleckler et al., 2008].”
BackgroundClimate Change Projections for Mediterranean Region with Focus over Alpine Region and Italy
Faggian, P. (2015) · Journal of Environmental Science and Engineering B · doi:10.17265/2162-5263/2015.09.004 · Cited by 26
“Many metrics have been proposed by the Program for Climate Model Diagnosis and Intercomparison [20-22].”
MethodThe hindcast skill of the CMIP ensembles for the surface air temperature trend
Sakaguchi, K., Zeng, X., & Brunke, M. A. (2012) · Journal of Geophysical Research Atmospheres · doi:10.1029/2012jd017765 · Cited by 25
DECISION-MAKING UNDER DEEP UNCERTAINTY
(2023) · Anthem Press eBooks · doi:10.2307/jj.4418217.19 · Cited by 25
Spatial Patterns of Human Thermal Comfort Conditions in Russia: Present Climate and Trends
Varentsov, M. I., Shartova, N. V., Grischenko, M., & Konstantinov, P. (2020) · Weather Climate and Society · doi:10.1175/wcas-d-19-0138.1 · Cited by 25
Evaluation of trends in extreme temperatures simulated by HighResMIP models across Europe
Squintu, A., van der Schrier, G., van den Besselaar, E. J. M., van der Linden, E. C., Putrasahan, D. A., Roberts, C. D., Roberts, M. J., Scoccimarro, E., Senan, R., & Tank, A. K. (2021) · Climate Dynamics · doi:10.1007/s00382-020-05596-6 · Cited by 25
Evaluating Global Climate Models for Hydrological Studies of the Upper Colorado River Basin
Pierce, D. W., Cayan, D. R., Goodrich, J. P., Das, T., & Munévar, A. (2021) · JAWRA Journal of the American Water Resources Association · doi:10.1111/1752-1688.12974 · Cited by 25
More than six billion people encountering more exposure to extremes with 1.5 °C and 2.0 °C warming
Qin, P. (2022) · Atmospheric Research · doi:10.1016/j.atmosres.2022.106165 · Cited by 25
Hsiao, S., Fu, H.-S., Chen, W., Chang, T.-Y., Wu, H.-L., & Liang, T.-Y. (2022) · Environmental Research Communications · doi:10.1088/2515-7620/ac8f15 · Cited by 25
How much of the interannual variability of East Asian summer rainfall is forced by SST?
He, C., Wu, B., Li, C., Lin, A., Gu, D., Zheng, B., & Zhou, T. (2015) · Climate Dynamics · doi:10.1007/s00382-015-2855-z · Cited by 25
“2b), the MMM is not substantially misled by these outlier models, suggesting the superiority of multi-model median to multi-model average (Gleckler et al. 2008).”
BackgroundEconomic impacts of climate change on water resources in the coterminous United States
Henderson, J. M., Rodgers, C. A., Jones, R., Smith, J. B., Strzepek, K. M., & Martinich, J. A. (2013) · Mitigation and Adaptation Strategies for Global Change · doi:10.1007/s11027-013-9483-x · Cited by 25
“Since the outputs of specific GCMs can vary widely within the same emission scenario (Burke et al. 2006), employing ensemble means has become common practice in climate science research, because ensembles have been shown to best replicate past climatological averages (Gleckler et al. 2008).”
BackgroundMurphy, J. M., Booth, B. B. B., Boulton, C. A., Clark, R. T., Harris, G. R., Lowe, J. A., & Sexton, D. M. H. (2014) · Climate Dynamics · doi:10.1007/s00382-014-2097-5 · Cited by 25
“There is scope to achieve the latter by considering expanded sets of physical variables (e.g. Murphy et al. 2004; Gleckler et al. 2008; Martin et al. 2010), biogeochemical variables (Anav et al.”
BackgroundMean Biases, Variability, and Trends in Air–Sea Fluxes and Sea Surface Temperature in the CCSM4
Bates, S. C., Fox‐Kemper, B., Jayne, S. R., Large, W. G., Stevenson, S. L., & Yeager, S. G. (2012) · Journal of Climate · doi:10.1175/jcli-d-11-00442.1 · Cited by 25
“Our approach to the problemswith observed fluxes is to compare model fields with the best available component flux data obtained frommultiple sources as advocated by Gleckler et al. (2008). The flux dataset computed from the forcing developed for the Coordinated Ocean-Ice Reference Experiments (CORE) combined with the…”
MethodInfluentialImpacts of climate change on nutrient losses from the Pike River watershed of southern Québec
Gombault, C., Madramootoo, C. A., Michaud, A. R., Beaudin, I., Sottile, M.-F., Chikhaoui, M., & Ngwa, F. F. (2015) · Canadian Journal of Soil Science · doi:10.4141/cjss-2014-012 · Cited by 25
Liu, W., Ullrich, P. A., Li, J., Zarzycki, C. M., Caldwell, P., Leung, L.-Y. R., & Qian, Y. (2023) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2022ms003358 · Cited by 25
“In the case of climate models, evaluation has generally focused on average behavior over large regions or long time periods (Gleckler et al., 2008; Eyring et al., 2019).”
BackgroundMethodCan model weighting improve probabilistic projections of climate change?
Räisänen, J., & Ylhäisi, J. S. (2011) · Climate Dynamics · doi:10.1007/s00382-011-1217-8 · Cited by 25
“This choice is seemingly supported by the fact that, in the simulation of present-day climate, the multi-model mean (hereafter MMM) performs better than most if not all individual models (Lambert and Boer 2001; Gleckler et al. 2008).”
BackgroundGreene, A. M., Hellmuth, M., & Lumsden, T. (2012) · Water Resources Research · doi:10.1029/2011wr011152 · Cited by 25
Smith, M. B., Mahowald, N. M., Albani, S., Perry, A. T., Losno, R., Qu, Z., Marticorena, B., Ridley, D. A., & Heald, C. L. (2017) · Atmospheric chemistry and physics · doi:10.5194/acp-17-3253-2017 · Cited by 25
Javadinejad, S. (2016) · University of Birmingham Institutional Research Archive (University of Birmingham) · Cited by 25
Ghulami, M. (2017) · theses.fr (ABES) · doi:10.70675/decb5d23z63f1z4a57zbb3fz869f7b888727 · Cited by 25
“The CMIP5 highlights the quantitative test method, and emphasizes the model performance metrics (quantitative measures) for the model simulations and projections of climate change (Taylor et al., 2012; Gleckler et al., 2008).”
MethodEvaluation and Projection of Extreme Precipitation over Northern China in CMIP5 Models
Rao, X., Lu, X., & Dong, W. (2019) · Atmosphere · doi:10.3390/atmos10110691 · Cited by 24
“In order to assess the model performance on the time variation, a score skill IVS [10,34,35] has been defined as follows.”
BackgroundHaslebacher, C., Demory, M., Demory, B.-O., Sarazin, M. S., & Vidale, P. L. (2022) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202142493 · Cited by 24
Downscaling CORDEX Through Deep Learning to Daily 1 km Multivariate Ensemble in Complex Terrain
Quesada‐Chacón, D., Baño‐Medina, J., Barfus, K., & Bernhofer, C. (2023) · Earth s Future · doi:10.1029/2023ef003531 · Cited by 24
“To evaluate and show the performance of the bias-correction method, we produced portrait plots (Gleckler et al., 2008) of the relative mean bias, the ratio of the variances, and, Kolmogorov–Smirnov statistic and p-values, for both bias-corrected and raw EUR25 data (see Figure 4).”
MethodStratospheric Sulfate Aerosols Impacts on West African Monsoon Precipitation Using GeoMIP Models
Bonou, F., Da‐Allada, C. Y., Baloïtcha, E., Alamou, E., Biao, E. I., Zandagba, J., Obada, E., Pomalegni, Y. W., Irvine, P. J., Boucher, O., et al. (2023) · Earth s Future · doi:10.1029/2023ef003779 · Cited by 25
Warren, R., Yu, R., Osborn, T. J., & Santos, S. D. L. N. (2011) · Climate Research · doi:10.3354/cr01042 · Cited by 24
“Evaluating GCM simulations of presentday climate (e.g. Gleckler et al. 2008) is one indication of model performance, though of course it is no guarantee that the response to forcings will be simulated well and it is difficult to convert a measure of similarity into some quantitative indication of…”
BackgroundXia, J., Yan, Z., Jia, G., Zeng, H., Jones, P. D., Zhou, W., & Zhang, A. (2015) · Advances in Atmospheric Sciences · doi:10.1007/s00376-014-4125-0 · Cited by 24
Multi-scale modelling to synergise Plant Systems Biology and Crop Science
Chew, Y. H., Seaton, D. D., & Millar, A. J. (2016) · Field Crops Research · doi:10.1016/j.fcr.2016.02.012 · Cited by 24
A multi-diagnostic approach to cloud evaluation
Williams, K. D., & Bodas‐Salcedo, A. (2017) · Geoscientific model development · doi:10.5194/gmd-10-2547-2017 · Cited by 24
Global Climate Models and 20th and 21st Century Arctic Climate Change
Bitz, C. M., Ridley, J., Holland, M. M., & Cattle, H. (2011) · Atmospheric and oceanographic sciences library · doi:10.1007/978-94-007-2027-5_11 · Cited by 24
“Model intercomparisons usually find that an across-model ensemble mean performs better than any individual model, especially for large-scale performance metrics (Gleckler et al., 2008, e.g.,).”
BackgroundLoikith, P. C., Waliser, D. E., Lee, H., Kim, J., Neelin, J. D., Lintner, B. R., McGinnis, S., Mattmann, C. A., & Mearns, L. O. (2014) · Journal of Climate · doi:10.1175/jcli-d-13-00457.1 · Cited by 24
“Comprehensive evaluation of GCMs archived as part of phase 3 of the Coupled Model Intercomparison Project (CMIP3) was performed by Gleckler et al. (2008); however, the demand for more geographically specific climate projections has increased the prominence of limited domain regional climate models…”
MethodHadley cell bias in climate models linked to extratropical eddy stress
Caballero, R. (2008) · Geophysical Research Letters · doi:10.1029/2008gl035084 · Cited by 24
“Both stationary and transient MMEM eddy stresses agree well with the observational estimates—better in fact than any single model, as is commonly found [Gleckler et al., 2008; Reichler and Kim, 2008].”
Documenting Climate Models and Their Simulations
Guilyardi, É., Balaji, V., Lawrence, B., Callaghan, S., DeLuca, C., Denvil, S., Lautenschlager, M., Morgan, M., Murphy, S. J., & Taylor, K. E. (2012) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-11-00035.1 · Cited by 24
“This information is not only important for scientific interpretation but, under increasing scrutiny from society, it is also demanded of a science that purports to be mature, credible, and open (Carlson, 2011; Kleiner, 2011).”
BackgroundSelf-citationLoutre, M., Mouchet, A., Fichefet, T., Goosse, H., Goelzer, H., & Huybrechts, P. (2011) · Climate of the past · doi:10.5194/cp-7-511-2011 · Cited by 24
Brekke, L. D. (2011) · Cited by 24
Climate Change Impact on Potential Distribution of an Endemic Species Abies marocana Trabut
Moukrim, S., Lahssini, S., Rhazi, M., Menzou, K., Madihi, M. E., Rifai, N., Bouziani, Y., Azedou, A., Boukhris, I., & Rhazi, L. (2022) · Ochrana prírody Slovenska/Ekológia · doi:10.2478/eko-2022-0034 · Cited by 23
Muñoz, L. E., Campozano, L., Guevara, D. C., Parra, R., Tonato, D., Suntaxi, A., Maisincho, L., Páez, C., Villacís, M., Córdova, J., et al. (2023) · Atmosphere · doi:10.3390/atmos14020264 · Cited by 23
Assessing decadal variability of subseasonal forecasts of opportunity using explainable AI
Arcodia, M. C., Barnes, E. A., Mayer, K. J., Lee, J., Ordoñez, A. C., & Ahn, M. (2023) · Environmental Research Climate · doi:10.1088/2752-5295/aced60 · Cited by 23
Rapid Acceleration of Dangerous Compound Heatwaves and Their Impacts in a Warmer China
Dong, W. G., Jia, X., Qian, Q., & Li, X. (2023) · Geophysical Research Letters · doi:10.1029/2023gl104850 · Cited by 23
“In the current work, the multi-model ensemble of the 12 CMIP6 models which can represent the “best guess” scenarios (Bishop & Abramowitz, 2013; Gleckler et al., 2008) is used to perform the analysis.”
MethodDommenget, D. (2015) · Climate Dynamics · doi:10.1007/s00382-015-2591-4 · Cited by 23
“The models of CMIP are different in their structures and physical parameterizations and have shown significant disagreement and uncertainties in their performance (e.g. Stott and Kettleborough 2002; Reichler and Kim 2008; Gleckler et al. 2008; Hawkins and Sutton 2009; Shiogama et al. 2012).”
BackgroundPerformance of CMIP5 wind speed from global climate models for the Bay of Bengal region
Krishnan, A., & Bhaskaran, P. K. (2019) · International Journal of Climatology · doi:10.1002/joc.6404 · Cited by 23
“Another study by Gleckler et al. (2008) observed uncertainties in climate models attributed due to choice of reference data, spatial resolutions, and selection of domain size.”
BackgroundEvaluation of merged multi-sensor ocean-color chlorophyll products in the Northern Persian Gulf
Moradi, M. (2021) · Continental Shelf Research · doi:10.1016/j.csr.2021.104415 · Cited by 23
Repercusiones del cambio climático global en el estado de Sinaloa, México
Campaña, L. M. F., Arzola–González, J. F., Ramírez-Soto, M., & Osorio-Pérez, A. (2012) · Cuadernos de Geografía Revista Colombiana de Geografía · doi:10.15446/rcdg.v21n1.25562 · Cited by 23
Targeted model evaluations for climate services: A case study on heat waves in Bangladesh
Nissan, H., Muñoz, Á. G., & Mason, S. J. (2020) · Climate Risk Management · doi:10.1016/j.crm.2020.100213 · Cited by 23
“This result is consistent with the weak relationship between model performance regarding the climatology and interannual variability over the tropics (Gleckler et al., 2008).”
BackgroundResultInfluentialThe Energy Exascale Earth System Model Version 3: 2. Overview of the Coupled System
Golaz, J., Lin, W., Zheng, X., Xie, S., Roberts, A., Roekel, L. P. V., THORNTON, P. E., Barthel, A. M., Bradley, A., Wolfe, J. D., et al. (2026) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2025ms005302 · Cited by 22
On the estimation of systematic error in regression-based predictions of climate sensitivity
Sanderson, B. M. (2013) · Climatic Change · doi:10.1007/s10584-012-0671-6 · Cited by 22
“Firstly, it has been shown by Rougier (2007) and others that rigorous inter-model weighting within a PPE is not a trivial task, and is potentially sensitive to the choice of metrics considered (Gleckler et al. 2008).”
BackgroundLu, Z., Fu, Z., Hua, L., Yuan, N., & Chen, L. (2018) · Scientific Reports · doi:10.1038/s41598-018-33340-y · Cited by 22
Srivastava, A., Grotjahn, R., Ullrich, P. A., & Sadegh, M. (2021) · Journal of Hydrometeorology · doi:10.1175/jhm-d-20-0180.1 · Cited by 22
Kalashnikova, O. Y., Niyazov, J., Nurbatsina, A., Kodirov, S., Radchenko, I., & Kretova, Z. (2023) · Central Asian Journal of Water Research · doi:10.29258/cajwr/2023-r1.v9-1/59-88.eng · Cited by 22
“(Flato et al., 2013; Gleckler et al., 2008; Knutti et al., 2010).”
Chen, X., Zhang, Y., Zhang, M., Feng, Y., Wu, Z., Qiao, F., & Huang, N. E. (2012) · Climate Dynamics · doi:10.1007/s00382-012-1554-2 · Cited by 22
Anandhi, A., Frei, A., Pradhanang, S. M., Zion, M. S., Pierson, D. C., & Schneiderman, E. M. (2011) · Hydrological Processes · doi:10.1002/hyp.8230 · Cited by 22
“A number of studies have used SS and other criteria statistics for evaluating the different meteorological variables available from GCMs simulations (Taylor, 2001; Giorgi and Mearns, 2003; Tebaldi et al., 2004; Murphy et al., 2007; Perkins et al., 2007; Randall et al., 2007; Gleckler et al., 2008; Maxino et al., 2008;…”
BackgroundMethodMultivariable integrated evaluation of model performance with the vector field evaluation diagram
Xu, Z., Han, Y., & Fu, C. (2017) · Geoscientific model development · doi:10.5194/gmd-10-3805-2017 · Cited by 22
Abe, M., Shiogama, H., Nozawa, T., & Emori, S. (2011) · Journal of Geophysical Research Atmospheres · doi:10.1029/2010jd015111 · Cited by 22
“It has been assumed that equal‐weight multimodel mean (MMM) performs better than most, if not all, individual models in simulating present‐day climate [ Gleckler et al., 2008 ].”
Plavcová, E., & Kyselý, J. (2019) · Advances in Meteorology · doi:10.1155/2019/2178321 · Cited by 21
“,e ensemble-mean results show that by averaging the model biases, one can get the best reproduction of observed patterns [20].”
BackgroundTaylor, G. P., Loikith, P. C., Aragon, C. M., Lee, H., & Waliser, D. E. (2022) · Climate Dynamics · doi:10.1007/s00382-022-06410-1 · Cited by 21
“parameters (for example mean temperature; Boucher et al. 2020; Gleckler et al. 2008; Mote and Salathé 2010).”
BackgroundAn evaluation of wildland fire simulators used operationally in Australia
Fox‐Hughes, P., Bridge, C., Faggian, N., Jolly, C., Matthews, S., Ebert, E. E., Jacobs, H., Brown, B. G., & Bally, J. (2024) · International Journal of Wildland Fire · doi:10.1071/wf23028 · Cited by 21
Screening CMIP6 models for Chile based on past performance and code genealogy
Gateño, F., Mendoza, P. A., Vasquéz, N., Lagos-Zúñiga, M. Á., Jiménez, H., Jerez, C., Vargas, X., Rubio‐Álvarez, E., & Montserrat, S. (2024) · Climatic Change · doi:10.1007/s10584-024-03742-1 · Cited by 21
“Several studies have considered the annual cycle of precipitation in GCM diagnostics (e.g., Taylor 2001; Phillips and Gleckler 2006; Gleckler et al. 2008; Pierce et al. 2009; McSweeney et al. 2015; Chhin and Yoden 2018; Karmalkar et al. 2019; Lovino et al. 2021; Ortega et al. 2021), as it is…”
Rakhmatova, N., Nishonov, B., Kholmatjanov, B. M., Rakhmatova, V., Toderich, K. N., Khasankhanova, G. M., Shardakova, L., Khujanazarov, T., Navruzovich, U. A., & Belikov, D. (2024) · Atmosphere · doi:10.3390/atmos15070866 · Cited by 21
“In the context of climate modeling, a bias correction in CMIP5/6 refers to a statistical adjustment applied to correct systematic errors or biases in the output of climate models [8,11,60,61].”
BackgroundMethodHay, L. E., LaFontaine, J. H., & Markstrom, S. L. (2014) · Earth Interactions · doi:10.1175/2013ei000554.1 · Cited by 21
“Previous studies have found that the more metrics used to evaluate model skill, the harder it becomes to identify a ‘‘better model’’ (Bureau of Reclamation 2011; Brekke et al. 2008; Reichler and Kim 2008; Gleckler et al. 2008).”
BackgroundDasgupta, A., Burrows, S. M., Han, K., & Rasch, P. J. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025882 · Cited by 21
“…many metrics ( v 1 , v 2 , v 3 , v 4 , etc.). frequently summarize aspects of the model’s fidelity , using statistical metrics such as (but not limited to) the root-mean-square error, correlation, and relative variance of the model output variables compared to observations of the same variable [12].”
MethodHow Climate Models Gain and Exercise Authority
Munk, A. K. (2012) · Routledge eBooks · doi:10.4324/9780203093870-8 · Cited by 21
A Markov chain method for weighting climate model ensembles
Kulinich, M., Fan, Y., Penev, S. I., Evans, J. P., & Olson, R. (2021) · Geoscientific model development · doi:10.5194/gmd-14-3539-2021 · Cited by 21
“The first approach is the widely used average of individual climate model outputs (Lambert and Boer, 2001; Gleckler et al., 2008): EAVEt = 1/N N∑ j=1 Mj,t , (3) for each 1≤ t ≤ T .”
BackgroundMethodInfluentialSchmith, T., Thejll, P. A., Berg, P., Boberg, F., Christensen, O. B., Christiansen, B., Christensen, J. H., Steger, C., & Madsen, M. S. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-273-2021 · Cited by 21
“Any discrimination of GCMs depends on area, season, and the meteorological field and property being investigated (Gleckler et al., 2008; e.g. their Fig.”
Temperature projections over Iran during the twenty-first century using CMIP5 models
Usta, D. F. B., Teymouri, M., Chatterjee, U., & Koley, B. (2021) · Modeling Earth Systems and Environment · doi:10.1007/s40808-021-01115-6 · Cited by 21
Dai, D., Chen, L., Ma, Z., & Xu, Z. (2021) · Climate Dynamics · doi:10.1007/s00382-021-05723-x · Cited by 21
“Furthermore, a variety of measures, such as metrics (Willmott et al. 1985; Taylor 2001; Gleckler et al. 2008) and statistical approaches (Perkins et al.”
BackgroundZhang, R., Cheng, L., Liu, P., Huang, K., Gong, Y., Qin, S., & Liu, D. (2021) · Advances in Water Resources · doi:10.1016/j.advwatres.2021.104063 · Cited by 21
Abe, M., Shiogama, H., Hargreaves, J. C., Annan, J. D., Nozawa, T., & Emori, S. (2009) · SOLA · doi:10.2151/sola.2009-034 · Cited by 21
Izumi, K., Bartlein, P. J., & Harrison, S. P. (2014) · Climate Dynamics · doi:10.1007/s00382-014-2189-2 · Cited by 21
“We therefore calculated the ensemble-mean anomalies of each variable across the six models (Table 1) because the ensemble mean results are generally closer to observations than individual model results under the current climate (Gleckler et al. 2008).”
ResultClimate model pluralism beyond dynamical ensembles
Mazzocchi, F., & Pasini, A. (2017) · Wiley Interdisciplinary Reviews Climate Change · doi:10.1002/wcc.477 · Cited by 21
“Interestingly, when model outputs are weighed against available climatic datasets, different models exhibit a better performance for different fields and measures of fit.(12,25,26) Such models are, hence, used together as complementary devices,(27) evaluating their degree of agreement and attempting to understand (thr…”
BackgroundThe Significance of Robust Climate Projections
Parker, W. S. (2018) · Springer eBooks · doi:10.1007/978-3-319-65058-6_9 · Cited by 21
A multi-model analysis of moisture changes during the last glacial maximum
Liu, S., Jiang, D., & Lang, X. (2018) · Quaternary Science Reviews · doi:10.1016/j.quascirev.2018.05.029 · Cited by 21
Krinner, G., Beaumet, J., Favier, V., Déqué, M., & Brutel-Vuilmet, C. (2018) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2018ms001438 · Cited by 21
Karmalkar, A. V., Sexton, D. M. H., Murphy, J. M., Booth, B. B. B., Rostron, J. W., & McNeall, D. (2019) · Climate Dynamics · doi:10.1007/s00382-019-04617-3 · Cited by 21
“The performance metrics often used are based on (root) mean square errors for global fields (e.g., Reichler and Kim 2008; Gleckler et al. 2008) and therefore we chose to emulate the global MSEs.”
MethodHe, Y., Wang, B., Liu, L., Huang, W., Xu, S., Liu, J., Wang, Y., Li, L., Huang, X., Peng, Y., et al. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001768 · Cited by 21
“The area‐weighted bias and RMSE, which represent systematic and random errors as those defined in Taylor (2001), Gleckler et al. (2008) and Goddard et al. (2013), are essential metrics used to assess the accuracy of the CDA system and are respectively defined as bias ¼ ∑ n i¼1diff ið Þ × area ið Þ…”
MethodTopál, D., Hatvani, I. G., & Kern, Z. (2020) · Theoretical and Applied Climatology · doi:10.1007/s00704-020-03361-7 · Cited by 21
“Action taken in this direction introduced the application of diverse model ranking methodologies, ranging from studies using correlation, root-mean-square error, and variance ratio (Boer and Lambert 2001; Gleckler et al. 2008) to the application of prediction indices (Murphy et al.”
MethodCatchment response to climate change under CMIP6 scenarios: a case study of the Krishna River Basin
Anil, S., P, A. R., & Vema, V. K. (2024) · Journal of Water and Climate Change · doi:10.2166/wcc.2024.442 · Cited by 20
Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions
Haßler, B., Hoffman, F. M., Beadling, R. L., Blockley, E., Huang, B., Lee, J., Lembo, V., Lewis, J., Lü, J., Madaus, L. E., et al. (2026) · Reviews of Geophysics · doi:10.1029/2025rg000891 · Cited by 20
Pan, H., Jin, Y., & Zhu, X. (2022) · Water · doi:10.3390/w14121888 · Cited by 20
Raj, N., & Brown, J. (2023) · Remote Sensing · doi:10.3390/rs15112881 · Cited by 20
“The computation of more than one error metric for evaluation was as important as the performance metrics to ensure a comprehensive evaluation [43].”
Background
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.