Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Items 701–800 of 1,001
Dong, W. G., Jia, X., Li, X., & Wu, R. (2024) · npj Climate and Atmospheric Science · doi:10.1038/s41612-024-00703-4 · Cited by 20
“Notably, the MME demonstrates superior performance in replicating the temperature anomalies observed, which aligns with previous studies 50,51 .”
Rapid Evaluation Framework for the CMIP7 Assessment Fast Track
Hoffman, F. M., Haßler, B., Swaminathan, R., Lewis, J., Andela, B., Collier, N., Hegedūs, D., Lee, J., Pascoe, C., Pflüger, M., et al. (2026) · Geoscientific model development · doi:10.5194/gmd-19-7415-2026 · Cited by 20
“These diagnostics, with their unique IDs, 5.1 (High amplitude Rossby waves) and 5.2 (Internal variability or ensemble spread for individual models), were then removed from the list of those to be integrated in the initial REF version, following consultation and agreement from the Impacts &…”
InfluentialPilcher, D. J., Brody, S. R., Johnson, L., & Bronselaer, B. (2015) · Journal of Geophysical Research Oceans · doi:10.1002/2015jc010759 · Cited by 20
“CMIP5 MODEL SKILL METRICS FOR pCO2 4625 base model skill on the accuracy of hindcast simulations [Lin, 2007; Gleckler et al., 2008; Radić and Clarke, 2011].”
MethodPausata, F. S. R., Gaetani, M., Messori, G., Kloster, S., & Dentener, F. J. (2015) · Atmospheric chemistry and physics · doi:10.5194/acp-15-1725-2015 · Cited by 20
Tsushima, Y., Brient, F., Klein, S. A., Konsta, D., Nam, C., Qu, X., Williams, K. D., Sherwood, S. C., Suzuki, K., & Zelinka, M. D. (2017) · Geoscientific model development · doi:10.5194/gmd-10-4285-2017 · Cited by 20
Li, J., Chen, H., Rong, X., Su, J., Xin, Y., Furtado, K., Milton, S. F., & Li, N. (2018) · Journal of Climate · doi:10.1175/jcli-d-17-0801.1 · Cited by 20
Evaluation of CORDEX Africa multi-model precipitation simulations over the Pra River Basin, Ghana
Gyamfi, C., Tindan, J. Z.-O., & Kifanyi, G. E. (2021) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2021.100815 · Cited by 20
Pomee, M. S., & Hertig, E. (2021) · International Journal of Climatology · doi:10.1002/joc.7244 · Cited by 20
“Model ensembles encompassing a broader uncertainty (e.g., Seperber et al., 2013; Gleckler et al., 2008) are more useful for such complex regions.”
BackgroundSrivastava, A., Grotjahn, R., Ullrich, P. A., & Zarzycki, C. M. (2021) · Climate Dynamics · doi:10.1007/s00382-021-05980-w · Cited by 20
Top-down estimates of biomass burning emissions of black carbon in the Western United States
Mao, Y., Li, Q. B., Randerson, J. T., Chen, D., Zhang, L., Hao, W. M., & Liou, K. N. (2014) · Atmospheric chemistry and physics · doi:10.5194/acp-14-7195-2014 · Cited by 20
Physics-guided probabilistic modeling of extreme precipitation under climate change
Kodra, E., Bhatia, U., Chatterjee, S., Chen, S., & Ganguly, A. R. (2020) · Scientific Reports · doi:10.1038/s41598-020-67088-1 · Cited by 20
“. Literature has pointed out the difficulty of measuring the “skill” of an ESM (Tebaldi and Knutti 2007; Knutti et al. 2010; Knutti 2010; Weigel et al. 2010), despite a multitude of attempts to do so (Gleckler et al. 2008; Pierce et al. 2009; Santer et al. 2009). Furthermore, in many cases common skill metrics such as…”
BackgroundPoan, E., Gachon, P., Dueymes, G., Diaconescu, E. P., Laprise, R., & Sanda, I. S. (2016) · Climate Dynamics · doi:10.1007/s00382-016-3016-8 · Cited by 20
“The Root Mean Square Error (RMSE, against ARC2 used as reference dataset) is used as the metric and the information is summarized using a condensed performance matrix (e.g. Gleckler et al. 2008).”
BackgroundMethodMulti-Fidelity Residual Neural Processes for Scalable Surrogate ModelingPreprint
Niu, R., Wu, D., Kim, K., Ma, Y.-A., Watson‐Parris, D., & Yu, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.18846 · Cited by 20
“Averaging these models often leads to improved results compared to using individual models (Lambert & Boer, 2001; Gleckler et al., 2008; Knutti et al., 2010).”
ResultSmith, K., Barthel, A. M., Conlon, L. M., Roekel, L. P. V., Bartoletti, A., Golaz, J., Zhang, C., Begeman, C., Benedict, J. J., Bisht, G., et al. (2025) · Geoscientific model development · doi:10.5194/gmd-18-1613-2025 · Cited by 19
Self-citationLee, K.-T., Jeon, H.-W., Park, S., Cho, J., & Kim, K. (2022) · Climatic Change · doi:10.1007/s10584-022-03410-2 · Cited by 19
Dong, W., Krasting, J. P., & Guo, H. (2023) · Journal of Climate · doi:10.1175/jcli-d-23-0268.1 · Cited by 19
Earth system models: A tool to understand changes in the Earth system
Scholze, M., Allen, J. I., Collins, W. J., Cornell, S., Huntingford, C., Joshi, M., Lowe, J. A., Smith, R. S., & Wild, O. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921155.008 · Cited by 19
Tian, B., Lee, H., Waliser, D. E., Ferraro, R. D., Kim, J., Case, J. L., Iguchi, T., Kemp, E., Wu, D., Putman, W., et al. (2017) · Journal of Hydrometeorology · doi:10.1175/jhm-d-17-0045.1 · Cited by 19
“Various performance metrics for climate model evaluations that rely on the MSE and/or the linear correlation coefficient have been proposed previously (Table 3; e.g., Taylor 2001; Murphy et al. 2004; Gleckler et al. 2008; Reichler and Kim 2008; Hirota et al. 2011; Watterson et al. 2014).”
BackgroundMethodZhang, T., Zhang, M., Lin, W., Lin, Y., Xue, W., Yu, H., He, J., Xin, X., Ma, H., Xie, S., et al. (2018) · Geoscientific model development · doi:10.5194/gmd-11-5189-2018 · Cited by 19
Advanced cyberinfrastructure for intercomparison and validation of climate models
Sun, Z., Di, L., Cash, B. A., & Gaigalas, J. (2019) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2019.104559 · Cited by 19
Yang, Z., & Villarini, G. (2020) · International Journal of Climatology · doi:10.1002/joc.6612 · Cited by 19
“Their performance also varies widely across different regions (e.g., Ou et al., 2013) and variables (e.g., Gleckler et al., 2008), and the uncertainties of the climate models are still large (Knutti and Sedláček, 2013).”
BackgroundXue, Z., & Ullrich, P. A. (2021) · Journal of Hydrometeorology · doi:10.1175/jhm-d-20-0314.1 · Cited by 19
Model assessments and future projections of spring climate extremes in China based on CMIP6 models
Ai, Y., Chen, H., & Sun, J. (2021) · International Journal of Climatology · doi:10.1002/joc.7492 · Cited by 19
“A model climate performance index (MCPI) and a model variability index (MVI) are constructed to evaluate the overall model skill (Gleckler et al., 2008).”
BackgroundMethodRisser, M. D., Collins, W. D., Wehner, M. F., O'Brien, T. A., Paciorek, C. J., O’Brien, J. P., Patricola, C. M., Huang, H., Ullrich, P. A., & Loring, B. (2022) · Climate Dynamics · doi:10.1007/s00382-022-06321-1 · Cited by 19
Loew, A. (2013) · Theoretical and Applied Climatology · doi:10.1007/s00704-013-0880-6 · Cited by 19
“Applications in climate modeling comprise the evaluation, initialization, or parameterization of climate models (Hall and Qu 2006; Gleckler et al. 2008; Hagemann et al. 2013; Brovkin et al. 2013).”
BackgroundSachindra, D. A., Huang, F., Barton, A., & Perera, B. J. C. (2014) · Journal of Water and Climate Change · doi:10.2166/wcc.2014.056 · Cited by 19
Evaluating climate model simulations of drought for the northwestern United States
Abatzoglou, J. T., & Rupp, D. E. (2017) · International Journal of Climatology · doi:10.1002/joc.5046 · Cited by 19
“Evaluation metrics Gleckler et al. (2008) introduced several metrics for evaluating climate models and emphasized the need to go beyond the mean statistics.”
BackgroundFuture precipitation changes over Panama projected with the atmospheric global model MRI-AGCM3.2
Kusunoki, S., Nakaegawa, T., Pinzón, R., Sánchez-Galán, J. E., & Fábrega, J. (2019) · Climate Dynamics · doi:10.1007/s00382-019-04842-w · Cited by 19
Gulizia, C. N., Camilloni, I. A., & Doyle, M. E. (2012) · Theoretical and Applied Climatology · doi:10.1007/s00704-012-0729-4 · Cited by 19
Quantification of Uncertainties of Future Climate Change: Challenges and Applications
Mearns, L. O. (2010) · Philosophy of Science · doi:10.1086/656817 · Cited by 19
Challenges for Observing and Modeling the Global Water Cycle
Trenberth, K. E. (2014) · Geophysical monograph · doi:10.1002/9781118872086.ch32 · Cited by 18
Moayedi, H., Dehrashid, A. A., & Le, B. N. (2024) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2023.2300456 · Cited by 18
“This diagram presents correlation coefficient, standard deviation and RMSE values among predicted and observed values for better recognition of variations (Gleckler et al., 2008; Taylor, 2001).”
Winderlich, K., Dalelane, C., & Walter, A. (2024) · Earth System Dynamics · doi:10.5194/esd-15-607-2024 · Cited by 18
Campbell, I., Gibson, P. B., Stuart, S. J., Broadbent, A. M., Sood, A., Pirooz, A. A. S., & Rampal, N. (2024) · International Journal of Climatology · doi:10.1002/joc.8578 · Cited by 18
“To summarize RCM performance across climatological mean fields, we present skill scores relative to the median model in a portrait plot (Figure 4), as implemented in previous model evaluation studies (e.g., Gleckler et al., 2008).”
Assessing high-resolution precipitation extremes in Central Asia: evaluation and future projections
Gummadi, S., Samineni, S., & López‐Lavalle, L. A. B. (2025) · Climatic Change · doi:10.1007/s10584-025-03872-0 · Cited by 18
“Furthermore, the comparison between the interannual variability of the simulations and the observations was evaluated using the Interannual Variability Skill Score (IVS) (Gleckler et al. 2008; Scherrer 2011 ).”
InfluentialMejía, J. F., Koracin, D. R., & Wilcox, E. M. (2018) · International Journal of Climatology · doi:10.1002/joc.5817 · Cited by 18
“Ensemble subsample selection, or weighting depending on ensemble member performance, is highly sensitive to the intended application, region of interest, and metrics implemented (Gleckler et al., 2008).”
BackgroundLuneva, M. V., Wakelin, S. L., Holt, J., Inall, M., Евгеньевич, К. И., Palmer, M. R., Toberman, M., Zubkova, E. V., & Polton, J. A. (2019) · Journal of Geophysical Research Oceans · doi:10.1029/2018jc014307 · Cited by 18
“The evaluation of model skill is not a trivial task and depends on themetrics, databases chosen, and length of observations (Allen et al., 2007; Gleckler et al., 2008).”
BackgroundXu, H., Wang, T., Wang, H., Miao, J., Chen, J., & Chen, S. (2020) · Acta Geologica Sinica - English Edition · doi:10.1111/1755-6724.14542 · Cited by 18
Niu, Z., Feng, L., Chen, X., & Yi, X. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18116029 · Cited by 18
“where RMSEj is the RMSE of the j-th model and RMSEmedian is the median RMSE of the 14 and 10 CMIP5 models for extreme temperature and precipitation indices, respectively [20,86].”
MethodDakhlaoui, H., & Djebbi, K. (2021) · Journal of Water and Climate Change · doi:10.2166/wcc.2021.004 · Cited by 18
Climate change effects on offshore wind turbines
James, M., Haldar, S., Varghese, R., Bhattacharya, S., & Pakrashi, V. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-99353-1.00030-x · Cited by 18
Ganguly, T., & Arya, D. S. (2023) · International Journal of Climatology · doi:10.1002/joc.8266 · Cited by 20
“A large number of data sets from GCMs, their varied physical and statistical properties and related uncertainties form a major impediment to arrive at a one-size-fits-all model evaluation metric (Gleckler et al., 2008).”
BackgroundHow do weather characteristics change in a warming climate?
Kim, O.-Y., Wang, B., & Shin, S. (2013) · Climate Dynamics · doi:10.1007/s00382-013-1795-8 · Cited by 18
“climate processes and phenomena in climate models (Gleckler et al. 2008; Pincus et al. 2008; Taylor 2001).”
BackgroundMethodKara, F., Yücel, İ., & Akyürek, Z. (2016) · Hydrological Sciences Journal · doi:10.1080/02626667.2015.1133911 · Cited by 18
“It is important to assess model skill using multiple diagnostics because inter-comparison projects show that no single model consistently outperforms all others (Gleckler et al. 2008, Wilby 2010).”
Zhang, M., Xu, Z., Han, Y., & Guo, W. (2021) · Geoscientific model development · doi:10.5194/gmd-14-3079-2021 · Cited by 18
“Objective performance metrics (e.g., relative error and portrait diagrams) have been proposed for the evaluation of various variables (Gleckler et al., 2008).”
BackgroundResultVariations in glacier peak water timing and its influencing factors in High-Mountain Asia
Lyu, H., Fang, G., Chen, Y., Liang, W., Qiu, Z., Li, Y., Duan, W., & Li, Z. (2024) · Fundamental Research · doi:10.1016/j.fmre.2024.12.006 · Cited by 17
“Runoff was projected under the four SSPs and averaged to obtain a robust future glacier runoff [ 31 ].”
Hamadalnel, M., Zhu, Z., Lu, R. M., Almazroui, M., & Shahid, S. (2021) · International Journal of Climatology · doi:10.1002/joc.7387 · Cited by 17
“It assures the models' capacity to realistically represent the features of the climate system, their potential, and their limitations (Gleckler et al., 2008).”
BackgroundWeighting climate models for hydrological projections: effects on contrasting hydroclimatic regions
Castañeda-González, M., Poulin, A., Romero-López, R., & Turcotte, R. (2023) · Climatic Change · doi:10.1007/s10584-023-03643-9 · Cited by 19
“…(Abramowitz et al. 2019; Eyring et al. 2019; Knutti et al. 2017); (2) some climate models have shown more difficulties in representing mean regional climate than other climate models (Bracon-not et al. 2012; Gleckler et al. 2008), especially over extremes (Do et al. 2020; Giuntoli et al. 2018).”
Liu, S., Jiang, D., & Lang, X. (2019) · The Holocene · doi:10.1177/0959683619854512 · Cited by 17
“We emphasize the multi-model medians in the following analysis, as the median field can avoid individual outliers and generally outperforms individual models in a collection of models (Gleckler et al., 2008).”
Methoddel Carmen, M. A. A., Estrada, F., & Gay-García, C. (2021) · Journal of Climate · doi:10.1175/jcli-d-20-0510.1 · Cited by 17
“The effect that in the univariate setting may be commonly attributed to a particular natural variability mode can be better represented by a combination of other modes in the multivariate setting (this is sometimes referred to as the omitted variable problem; see Greene 2012).”
BackgroundMethodYokohata, T., Iwahana, G., Sone, T., Saitô, K., Ishizaki, N. N., Kubo, T., Oguma, H., & Uchida, M. (2021) · Scientific Reports · doi:10.1038/s41598-021-94222-4 · Cited by 17
Zegeye, M. K., Ture, K., & Nedaw, D. (2023) · ENVIRONMENTAL SYSTEMS RESEARCH · doi:10.1186/s40068-023-00325-4 · Cited by 17
“Smaller values of RMSE close to zero had good model performance and vice versa (Gleckler et al. 2008; Chai and Draxler 2014).”
BackgroundEstimates of black carbon emissions in the western United States using the GEOS-Chem adjoint model
Mao, Y., Li, Q. B., Henze, D. K., Jiang, Z., Jones, D. B. A., Kopacz, M., He, C., Qi, L., Gao, M. C., Hao, W. M., et al. (2015) · Atmospheric chemistry and physics · doi:10.5194/acp-15-7685-2015 · Cited by 17
Quantifying the sources of uncertainty in upper air climate variables
Eghdamirad, S., Johnson, F. M., Woldemeskel, F., & Sharma, A. (2016) · Journal of Geophysical Research Atmospheres · doi:10.1002/2015jd024341 · Cited by 17
“For example, model agreement is often considered as a way of weighting the contributions toward a multimodel ensemble and implicitly considers the variation in future projections [Giorgi and Mearns, 2000; Gleckler et al., 2008; Johnson and Sharma, 2009].”
BackgroundCampozano, L., Vázquez‐Patiño, A., Tenelanda, D., Feyen, J., Samaniego, E. P., & Sánchez, E. M. (2017) · International Journal of Climatology · doi:10.1002/joc.5008 · Cited by 17
“For the global interpretation of the performance of all CMIP3&5 models and RAD at the level of each of the six climate stations a portrait diagram (Gleckler et al., 2008) was designed.”
MethodKwon, S., Kim, J., Boo, K., Shim, S., Kim, Y., & Byun, Y. (2018) · International Journal of Climatology · doi:10.1002/joc.5954 · Cited by 17
“The Taylor score, in conjunction with the Taylor diagram, has been widely used in model evaluation studies (e.g., Taylor, 2001; Gleckler et al., 2008).”
BackgroundMethodLi, W., Zou, T., Li, L., Deng, Y., Sun, V. T., Zhang, Q., Layton, J. B., & Setoguchi, S. (2019) · Climate Dynamics · doi:10.1007/s00382-019-04708-1 · Cited by 17
Boé, J., & Terray, L. (2015) · Climate Dynamics · doi:10.1007/s00382-014-2445-5 · Cited by 17
“1 3 root mean square error (RMSE) than the vast majority of individual models (Lambert and Boer 2001; Gleckler et al. 2008).”
BackgroundCui, D., Liang, S., Wang, D., & Liu, Z. (2021) · doi:10.5194/essd-2021-53 · Cited by 17
Aryal, Y. N., & Zhu, J. (2021) · Theoretical and Applied Climatology · doi:10.1007/s00704-021-03704-y · Cited by 16
Lin, Z., Ge, F., Chen, Q., Fraedrich, K., & Jin, Z. (2024) · Climate Dynamics · doi:10.1007/s00382-024-07491-w · Cited by 16
“Generally, a negative RMSE′ indicates a better performance than other models, and accordingly models with a positive RMSE′ perform poorly (Gleckler et al. 2008 ; Zhu et al. 2020a; Sun et al. 2022).”
Monitoring and benchmarking Earth system model simulations with ESMValTool v2.12.0
Lauer, A., Bock, L., Haßler, B., Jöckel, P., Ruhe, L., & Schlund, M. (2025) · Geoscientific model development · doi:10.5194/gmd-18-1169-2025 · Cited by 16
“Portrait diagrams (Gleckler et al., 2008) of the different models and variables, where red corresponds to a higher RMSE (= worse performance), and blue to a lower RMSE (= better performance) than the median.”
Zhang, L., Yuan, F., Wang, B., Ren, L., Zhao, C., Shi, J., Liu, Y., Jiang, S. S., Yang, X., Chen, T., et al. (2020) · Atmospheric Research · doi:10.1016/j.atmosres.2020.105331 · Cited by 16
Saranya, M. S., & Nair, V. V. (2021) · Journal of Water and Climate Change · doi:10.2166/wcc.2021.139 · Cited by 16
Are Cut-off Lows simulated better in CMIP6 compared to CMIP5?
Pinheiro, H. R., Ambrizzi, T., Hodges, K. I., Gan, M. A., Andrade, K. M., & Garcia, J. R. M. (2022) · Climate Dynamics · doi:10.1007/s00382-022-06200-9 · Cited by 16
“The performance of individual climate models in simulating the spatial distribution of COLs (in terms of track density) is evaluated using the Taylor diagram (Taylor 2001), which has been used in other model intercomparison and evaluation studies (Gleckler et al. 2008; Walsh et al. 2013; Langenbrunner and Neelin 2013;…”
MethodEvaluation of Extreme Precipitation Indices in Brazil’s Semiarid Region from Satellite Data
dos Santos, A. L. M., Gonçalves, W. A., Rodrigues, D. T., Andrade, L. D. M. B., & Silva, C. M. S. E. (2022) · Atmosphere · doi:10.3390/atmos13101598 · Cited by 16
A Finite-Time Ensemble Method for Mixed Layer Model Comparison
Johnson, L., Fox‐Kemper, B., Li, Q., Pham, H. T., & Sarkar, S. (2023) · Journal of Physical Oceanography · doi:10.1175/jpo-d-22-0107.1 · Cited by 16
“A rule-of-thumb that ensemble means tend to outperform individual models has long been noted in model ensembles where every model has good reason to be included (e.g., Gleckler et al. 2008), but the rule can be violated with pathological choices of models to include.”
BackgroundAssessment of GCM capabilities to simulate tropospheric stability on the Arabian Peninsula
Barfus, K., & Bernhofer, C. (2014) · International Journal of Climatology · doi:10.1002/joc.4092 · Cited by 16
“These range from RMSE and Taylor diagrams (e.g. Taylor, 2001; Gleckler et al., 2008) over comparing probability density functions (Brands et al., 2011) to the * Correspondence to: K. Barfus, Institute of Hydrology and Meteorology, Technical University of Dresden, Pienner Str. 23, 01737 Tharandt,…”
MethodMenge, E. O., Stobo‐Wilson, A. M., Oliveira, S. L. J., & Lawes, M. J. (2016) · The Rangeland Journal · doi:10.1071/rj15081 · Cited by 16
“5 emission scenario (Gleckler et al. 2008; Pierce et al. 2009).”
Climate Change Impacts on the Ou?m? River, Benin, West Africa
Essou, G. R. C. (2013) · Journal of Earth Science & Climatic Change · doi:10.4172/2157-7617.1000161 · Cited by 16
Kusunoki, S. (2017) · Atmosphere · doi:10.3390/atmos8050093 · Cited by 16
Dasgupta, A., Wang, H., O'Brien, N. D., & Burrows, S. M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934540 · Cited by 16
“Statistical fidelity metrics for climate models [9] measure the degree to which model outputs match observation (Figure 1).”
BackgroundMethodMa, H., Köhler, M., Li, J.-L. F., Farrara, J. D., Mechoso, C. R., FORBES, R. M., & Waliser, D. E. (2012) · Journal of Geophysical Research Atmospheres · doi:10.1029/2011jd016275 · Cited by 16
“We also plot in Figure 5b a portrait diagram of relative error metrics [Gleckler et al., 2008] for the multilayer cloud IWC.”
MethodAssessing Surface Solar Radiation Fluxes in the CMIP Ensembles
Loew, A., Andersson, A., Trentmann, J., & Schröder, M. (2016) · Journal of Climate · doi:10.1175/jcli-d-14-00503.1 · Cited by 16
“Model evaluation using quantitative model performance metrics has become a widely used practice in different research communities (Abramowitz et al. 2008; Gleckler et al. 2008; Reichler and Kim 2008; Blyth et al. 2011; Abramowitz 2012; Luo et al. 2012; Gettelman et al. 2012; Luo et al. 2012;Anav et al. 2013;Goddard et…”
BackgroundMethodInfluentialExploration of diverse solutions for the calibration of imperfect climate models
Peatier, S., Sanderson, B. M., & Terray, L. (2024) · Earth System Dynamics · doi:10.5194/esd-15-987-2024 · Cited by 15
Calibrating large-ensemble European climate projections using observational data
O’Reilly, C., Befort, D. J., & Weisheimer, A. (2020) · Earth System Dynamics · doi:10.5194/esd-11-1033-2020 · Cited by 15
Chen, A., Zhang, S., & Li, Z. (2021) · Climate Research · doi:10.3354/cr01634 · Cited by 15
Bayissa, Y. A., Melesse, A. M., Bhat, M. G., Tadesse, T., & Shiferaw, A. S. (2021) · Water · doi:10.3390/w13172411 · Cited by 15
Russotto, R. D., Strong, J. D. O., Camargo, S. J., Sobel, A. H., Elsaesser, G. S., Kelley, M., Genio, A. D. D., Moon, Y., & Kim, D. (2021) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2021ms002601 · Cited by 15
“The latter normalization is similar to the method described by Gleckler et al. (2008) and used, for example, by Zhao et al. (2018) for visualizing relative error metrics across model experiments, except that we normalize RMSE by the IQR instead of the median in order to better draw out inter-model differences when the…”
MethodZhu, C., Yue, Q., & Huang, J. (2023) · Water · doi:10.3390/w15173043 · Cited by 15
Evaluation and correction analysis of the regional rainfall simulation by CMIP6 over Sudan
Babiker, W., Tan, G., Abdallah, A., & Elameen, A. M. (2024) · Geographica Pannonica · doi:10.5937/gp28-46565 · Cited by 15
“…is carried out by comparing the simulations to the observations (Trigo & Palutikof, 2001), since it guarantees that the models can accurately represent some aspects of the climate system while employing model performance indicators to assess their potential and limitations (Gleckler et al., 2008).”
MethodParker, W. S. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009619301 · Cited by 15
Nury, A. H., & Alam, M. J. B. (2013) · Journal of Scientific Research · doi:10.3329/jsr.v6i1.16511 · Cited by 15
How reliable are GCM simulations for different atmospheric variables?
Eghdamirad, S., Johnson, F. M., & Sharma, A. (2017) · Climatic Change · doi:10.1007/s10584-017-2086-x · Cited by 15
“Gleckler et al. (2008) ranked models based on their relative error in the twentieth century simulations.”
BackgroundSensitivity of future climate change and uncertainty over India to performance-based model weighting
Singh, R., & AchutaRao, K. M. (2020) · Climatic Change · doi:10.1007/s10584-019-02643-y · Cited by 15
“Gleckler et al. (2008) used robust performance metrics to rank the climate models and concluded that models ranked higher in one field are not necessarily reliable for other variables and the multi-model mean outperforms individual models in many respects.”
MethodOdom, R. H., & Ford, W. M. (2020) · Environmental Management · doi:10.1007/s00267-020-01331-3 · Cited by 15
“…using average values from an ensemble of 16 GCMs (Girvetz et al. 2009; see Appendix, Table 3) because climate-change studies have shown that using averaged values from multiple models produces more reliable estimates of climatic means than any single GCM (Gleckler et al. 2008; Pierce et al. 2009).”
BackgroundLu, X., Rao, X., & Dong, W. (2020) · International Journal of Climatology · doi:10.1002/joc.6907 · Cited by 15
“There is obvious wet bias along the north edge of Tibetan Plateau from AMME and the precipitation has the similar pattern that is the typical GCM/RCM bias problem because the resolution is too coarse to simulate the complex terrain (Gao et al., 2008; Gleckler et al., 2008; Hawkins and Sutton, 2011).”
BackgroundNishii, K., Miyasaka, T., Kosaka, Y., & Nakamura, H. (2009) · Journal of the Meteorological Society of Japan Ser II · doi:10.2151/jmsj.87.581 · Cited by 15
The epistemological status of general circulation models
Loehle, C. (2017) · Climate Dynamics · doi:10.1007/s00382-017-3717-7 · Cited by 15
Bias‐Adjustment Methods for Future Subdaily Precipitation Extremes Consistent Across Durations
de Vyver, H. V., Schaeybroeck, B. V., Cruz, L. D., Hamdi, R., & Termonia, P. (2023) · Earth and Space Science · doi:10.1029/2022ea002798 · Cited by 14
“Finally, since a multi-model ensemble mean generally provides a better estimate than each constituent ensemble member (Gleckler et al., 2008; Reichler & Kim, 2008; Tebaldi & Knutti, 2007), the validation effort was repeated in an ensemble framework by comparing the pseudo future IDF-values with the mean of the member-…”
MethodKonca-Kędzierska, K., Wibig, J., & Gruszczynska, M. (2023) · Meteorology Hydrology and Water Management · doi:10.26491/mhwm/171699 · Cited by 14
Chen, R., Duan, K., Shi, P., Shang, W., Yang, J., He, J., Meng, Y., Dou, M., & Li, L. (2024) · Climatic Change · doi:10.1007/s10584-024-03834-y · Cited by 14
“However, all models overestimated the precipitation in Xinjiang, ranging from 0.56 to 2.02 mm/d. IVS values of most models were 0.3 in temperature and 0.5 in precipitation, indicating the CMIP6 models could reproduce the interannual variation of temperature and precipitation in Xin-1 3 jiang.”
InfluentialWang, Y., Shen, Y., Wang, L., Guo, Y., Cheng, Y., & Zhang, X. (2025) · Remote Sensing · doi:10.3390/rs17040582 · Cited by 14
Sakaguchi, K., Zeng, X., & Brunke, M. A. (2011) · Journal of Climate · doi:10.1175/jcli-d-11-00106.1 · Cited by 14
“The MME mean consistently shows smaller RMSE than individual ensemble members (gray lines) at all the scales and agrees with the better performance of MME means in previous studies (Gleckler et al. 2008; Reichler and Kim 2008).”
ResultRing, C., Pollinger, F., Kaspar‐Ott, I., Hertig, E., Jacobeit, J., & Paeth, H. (2017) · Climate Dynamics · doi:10.1007/s00382-017-3737-3 · Cited by 14
Selection of representative GCM scenarios preserving uncertainties
Lee, J.-K., & Kim, Y.-O. (2017) · Journal of Water and Climate Change · doi:10.2166/wcc.2017.101 · Cited by 14
CMIP5 model performance of significant wave heights over the Indian Ocean using COWCLIP datasets
Krishnan, A., Bhaskaran, P. K., & Kumar, P. (2021) · Theoretical and Applied Climatology · doi:10.1007/s00704-021-03642-9 · Cited by 14
“The smaller the values of MCPI and MVI, the better the model performance (Gleckler et al. 2008).”
BackgroundMethodInfluentialBenchmarking Simulated Precipitation Variability Amplitude across Time Scales
Ahn, M., Gleckler, P. J., Lee, J., Pendergrass, A. G., & Jakob, C. (2022) · Journal of Climate · doi:10.1175/jcli-d-21-0542.1 · Cited by 14
Self-citationOn the Mechanisms of Variability of the Atlantic Meridional Overturning Circulation (AMOC)
Kuznetsova, D. A., & Bashmachnikov, I. (2021) · Oceanology · doi:10.1134/s0001437021060072 · Cited by 14
Holtanová, E., Belda, M., & Halenka, T. (2022) · Frontiers in Earth Science · doi:10.3389/feart.2022.1018661 · Cited by 14
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.