Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Items 501–600 of 1,001
Riboust, P., & Brissette, F. P. (2015) · JAWRA Journal of the American Water Resources Association · doi:10.1111/jawr.12271 · Cited by 38
Application of Cluster Analysis to Climate Model Performance Metrics
Yokoi, S., Takayabu, Y. N., Nishii, K., Nakamura, H., Endo, H., Ichikawa, H., Inoue, T., Kimoto, M., Kosaka, Y., Miyasaka, T., et al. (2011) · Journal of Applied Meteorology and Climatology · doi:10.1175/2011jamc2643.1 · Cited by 38
“In addition, because the magnitudes of the observed surface turbulent fluxes (LHF and SHF) tend to be considerably less accurate than their horizontal patterns (Gleckler et al. 2008), we consider only the centered RMSE for these variables (LHFc and SHFc).”
BackgroundMethodResultInfluentialBiases and Model Agreement in Projections of Climate Extremes over the Tropical Pacific
Perkins, S. E. (2011) · Earth Interactions · doi:10.1175/2011ei395.1 · Cited by 33
Influence of Ocean and Atmosphere Components on Simulated Climate Sensitivities
Winton, M., Adcroft, A., Griffies, S. M., Hallberg, R., Horowitz, L. W., & Stouffer, R. J. (2012) · Journal of Climate · doi:10.1175/jcli-d-12-00121.1 · Cited by 37
“The step response function can also be used to predict the model’s global temperature response to more complex forcing (Hasselmann et al. 1993; Held et al. 2010; Good et al. 2011).”
MethodPauci ex tanto numero: reduce redundancy in multi-model ensembles
Solazzo, E., Riccio, A., Kioutsioukis, I., & Galmarini, S. (2013) · Atmospheric chemistry and physics · doi:10.5194/acp-13-8315-2013 · Cited by 37
Climate Model Confirmation: From Philosophy to Predicting Climate in the Real World
Knutti, R. (2018) · Springer eBooks · doi:10.1007/978-3-319-65058-6_11 · Cited by 37
Atlas de scénarios climatiques pour la forêt québécoise
Logan, T., Charron, I., Chaumont, D., & Houle, D. (2011) · Cited by 36
Seasonal predictability of Kiremt rainfall in coupled general circulation models
Gleixner, S., Keenlyside, N. S., Demissie, T. D., Counillon, F., Wang, Y., & Viste, E. (2017) · Environmental Research Letters · doi:10.1088/1748-9326/aa8cfa · Cited by 36
“This comes as no surprise, as multi-model predictions based on systems of similar skill usually score higher skills than individual models and the result is more robust (Palmer et al 2016, Hagedorn et al 2005, Gleckler et al 2008, Kirtman and Pirani 2008).”
BackgroundRusso, E., Kirchner, I., Pfahl, S., Schaap, M., & Cubasch, U. (2019) · Geoscientific model development · doi:10.5194/gmd-12-5229-2019 · Cited by 36
“The analyses of MAE for the mean seasonal cycle are conducted for the entire domain and single subregions.”
MethodAssessment of extreme rainfall estimates from satellite-based: Regional analysis
Palharini, R. S. A., Vila, D. A., Rodrigues, D. T., Palharini, R. C., Mattos, E. V., & Pedra, G. U. (2021) · Remote Sensing Applications Society and Environment · doi:10.1016/j.rsase.2021.100603 · Cited by 36
Assessment and prediction of regional climate based on a multimodel ensemble machine learning method
Fu, Y., Zhuang, H., Shen, X., & Li, W. (2023) · Climate Dynamics · doi:10.1007/s00382-023-06787-7 · Cited by 36
“② Taylor diagram (Gleckler et al. 2008) can visually compare differences between datasets.”
BackgroundInflated Uncertainty in Multimodel‐Based Regional Climate Projections
Madsen, M. S., Langen, P. L., Boberg, F., & Christensen, J. H. (2017) · Geophysical Research Letters · doi:10.1002/2017gl075627 · Cited by 36
“…change patterns together with the observation that a multimodel average often outperforms any individual model when compared to observations (Gleckler et al., 2008) suggests that the multimodel mean of a particular climate variable is indicative of a best estimate of the future pattern of…”
BackgroundEvaluation of CORDEX- South Asia regional climate models for heat wave simulations over India
Singh, S., Mall, R. K., Dadich, J., Verma, S., Singh, J. V., & Gupta, A. (2021) · Atmospheric Research · doi:10.1016/j.atmosres.2020.105228 · Cited by 35
A new diagram for performance evaluation of complex models
Izzaddin, A., Langousis, A., Totaro, V., Yaseen, M., & Iacobellis, V. (2024) · Stochastic Environmental Research and Risk Assessment · doi:10.1007/s00477-024-02678-3 · Cited by 35
“In particular, (1) it does not explicitly consider the overall bias of model outputs (see e.g. Gleckler et al. 2008; Hu et al. 2019) and (2) it exploits the root-mean-squared error (RMSE), which is a measure of fit conditional on correlation.”
A Decade of Climate Scenarios: The Ouranos Consortium Modus Operandi
Huard, D., Chaumont, D., Logan, T., Sottile, M.-F., Brown, R. D., St-Denis, B. G., Grenier, P., & Braun, M. (2014) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-12-00163.1 · Cited by 35
Qiao, L., Hong, Y. E., McPherson, R. A., Shafer, M. A., Gade, D. R., Williams, D., Chen, S., & Lilly, D. (2014) · Water Resources Management · doi:10.1007/s11269-014-0678-z · Cited by 35
“Some studies have tried to use weighted model average to improve climate realization (Gleckler et al. 2008; Christensen et al. 2010; Sobolowski and Pavelsky 2012), however, metrics usage could further bias climate change impacts because models performances vary from different evaluation angle.”
BackgroundWatterson, I. G. (2015) · Journal of Climate · doi:10.1175/jcli-d-14-00702.1 · Cited by 35
“Gleckler et al. (2008) noted the potential dependence of the relative ranking of models for pre-”
BackgroundMethodLee, J., Sperber, K. R., Gleckler, P. J., Taylor, K. E., & Bonfils, C. (2021) · Journal of Climate · doi:10.1175/jcli-d-20-0832.1 · Cited by 35
“This effort has facilitated an exponential growth in the community of scientists involved with model development, diagnosis, and evaluation (e.g., Gleckler et al. 2008, 2016; Eyring et al. 2020; Fasullo 2020; Bock et al. 2020).”
BackgroundMethodInfluentialSelf-citationOduro, C., Bi, S., Ayugi, B. O., Li, B., Babaousmail, H., Sarfo, I., Ullah, S., & Ngoma, H. (2021) · International Journal of Climatology · doi:10.1002/joc.7439 · Cited by 35
“A detailed description of the portrait diagram is well enumerated in previous studies (e.g., Gleckler et al., 2008; Sillmann et al., 2013).”
MethodTime of emergence in climate extremes corresponding to Köppen-Geiger classification
Zhang, M., & Gao, Y. (2023) · Weather and Climate Extremes · doi:10.1016/j.wace.2023.100593 · Cited by 35
Wu, W., Liu, Y., & Betts, A. K. (2012) · Journal of Geophysical Research Atmospheres · doi:10.1029/2011jd016971 · Cited by 35
“This technique has been widely used in climate researches and IPCC assessment [e.g., IPCC, 2001; Anderson et al., 2004; Martin et al., 2006; Miller et al., 2006; Bosilovich et al., 2008; Gleckler et al., 2008; Pincus et al., 2008].”
MethodAnnual and seasonal mean tropical and subtropical precipitation bias in CMIP5 and CMIP6 models
Li, J. F., Xu, K., Richardson, M., Lee, W., Jiang, J. H., Yu, J., Wang, Y., Fetzer, E. J., Wang, L., Stephens, G. L., et al. (2020) · Environmental Research Letters · doi:10.1088/1748-9326/abc7dd · Cited by 34
Twentieth century global glacier mass change: an ensemble-based model reconstruction
Malles, J., & Marzeion, B. (2021) · The Cryosphere · doi:10.5194/tc-15-3135-2021 · Cited by 36
“Often, a single metric is chosen to be minimized (e.g., the model’s RMSE with respect to observed in situ mass balances).”
MethodInfluentialUncertainties in calculating precipitation climatology in East Asia
Kim, J., & Park, S. K. (2016) · Hydrology and earth system sciences · doi:10.5194/hess-20-651-2016 · Cited by 34
Homogenization of Portuguese long-term temperature data series: Lisbon, Coimbra and Porto
Morozova, A. L., & Valente, M. A. (2012) · Earth system science data · doi:10.5194/essd-4-187-2012 · Cited by 34
Jayasankar, C. B., Rajendran, K., & Surendran, S. (2018) · Journal of Geophysical Research Atmospheres · doi:10.1029/2018jd028677 · Cited by 34
Sgubin, G., Swingedouw, D., Atauri, I. G. D. C., Ollat, N., & van Leeuwen, C. (2019) · Agronomy · doi:10.3390/agronomy9070397 · Cited by 34
“Indeed, comparisons between historical simulations and observational data demonstrated that multi-model mean generally outperforms most of, if not all, individual models [35,36].”
BackgroundKrakauer, N. Y., & Fekete, B. M. (2014) · Environmental Research Letters · doi:10.1088/1748-9326/9/2/024009 · Cited by 33
“Using the multimodel mean is consistent with previous findings that multimodel averages typically performed better than individual GCMs in comparisons of observations with climate model ensembles [20–23].”
ResultMcIntyre, N. E., Liu, G., Gorzo, J. M., Wright, C. K., Guntenspergen, G. R., & Schwartz, F. W. (2019) · Ecosphere · doi:10.1002/ecs2.2711 · Cited by 33
Leung, F., Williams, K., Sitch, S. A., Tai, A. P. K., Wiltshire, A., Gornall, J. L., Ainsworth, E. A., Arkebauer, T. J., & Scoby, D. (2020) · Geoscientific model development · doi:10.5194/gmd-13-6201-2020 · Cited by 33
Anil, S., & Raj, P. A. (2022) · Journal of Water and Climate Change · doi:10.2166/wcc.2022.399 · Cited by 34
“Apart from these metrics, a composite metric is also used by collecting the metrics in evaluating climate models (Taylor 2001; Murphy et al. 2004; Gleckler et al. 2008; Reichler & Kim 2008; Fu et al. 2013; Raju & Nagesh kumar 2014; Ahmadalipour et al. 2017; Raju et al. 2017).”
MethodLawal, I. M., Bertram, D., White, C. J., Kutty, S. R. M., Hassan, I. T. M., & Jagaba, A. H. (2023) · Theoretical and Applied Climatology · doi:10.1007/s00704-023-04466-5 · Cited by 33
“…time scales and the ability of a GCM capturing the spatial structure of a catchment’s key climate feature may give crucial information for water resource management accurately, especially at basins with high climate variability (Gleckler, Taylor, and Doutriaux 2008; Ahmed, Sachindra, et al.”
BackgroundEfficient screening of climate model sensitivity to a large number of perturbed input parameters
Covey, C., Lucas, D. D., Tannahill, J., Garaizar, X., & Klein, R. (2013) · Journal of Advances in Modeling Earth Systems · doi:10.1002/jame.20040 · Cited by 33
“These fields comprise most of a proposed set of standard metrics of climate model performance [Gleckler et al., 2008, Table 2] and the set used by Jackson et al. [2008] as observational constraints on their CAM perturbed input parameter ensemble.”
BackgroundOperational assimilation of GPS slant path delay measurements into the MM5 4DVAR system
Bauer, H., Wulfmeyer, V., Schwitalla, T., Zus, F., & Grzeschik, M. (2010) · Tellus A Dynamic Meteorology and Oceanography · doi:10.1111/j.1600-0870.2010.00489.x · Cited by 33
“…initial and boundary conditions, (2) LAM numerics, (3) LAM physics and its consistency with the physics of the driving global model, and (4) the predictability of the weather situation (e.g. Gleckler et al., 2008; Skamarock and Klemp, 2007; Küll and Bott, 2009; Hohenegger and Schär, 2007).”
BackgroundAssessing climate model projections: State of the art and philosophical reflections
Katzav, J., Dijkstra, H. A., & de Laat, J. (2012) · Studies in History and Philosophy of Science Part B Studies in History and Philosophy of Modern Physics · doi:10.1016/j.shpsb.2012.07.002 · Cited by 33
Climate change impact on Sub-Saharan Africa : An overview and analysis of scenarios and models
Müller, C. (2009) · Fachinformationen für Politikwissenschaft, Verwaltungswissenschaft und Kommunalwissenschaften (Institut für Friedensforschung und Sicherheitspolitik) · Cited by 33
Lembo, V., Bordi, I., & Speranza, A. (2017) · Earth System Dynamics · doi:10.5194/esd-8-295-2017 · Cited by 32
O’Loughlin, R., Li, D., Neale, R., & O'Brien, T. A. (2025) · Geoscientific model development · doi:10.5194/gmd-18-787-2025 · Cited by 32
When 1+1 can be >2: Uncertainties compound when simulating climate, fisheries and marine ecosystems
Evans, K., Brown, J. N., Gupta, A. S., Nicol, S., Hoyle, S., Matear, R. J., & Arrizabalaga, H. (2014) · Deep Sea Research Part II Topical Studies in Oceanography · doi:10.1016/j.dsr2.2014.04.006 · Cited by 32
Ocean-forced evolution of the Amundsen Sea catchment, West Antarctica, by 2100
Alevropoulos-Borrill, A., Nias, I., Payne, A. J., Golledge, N. R., & Bingham, R. J. (2020) · The Cryosphere · doi:10.5194/tc-14-1245-2020 · Cited by 32
Labe, Z. M., & Barnes, E. A. (2022) · Earth and Space Science · doi:10.1029/2022ea002348 · Cited by 32
“There are many existing methods for ranking the skill of climate models against observations (Eyring et al., 2019; Gleckler et al., 2008).”
BackgroundMethodHobbi, S., Papalexiou, S. M., Rajulapati, C. R., Nerantzaki, S. D., Markonis, Y., Tang, G., & Clark, M. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.128121 · Cited by 32
Nishant, N., Virgilio, G. D., Ji, F., Tam, E., Beyer, K., & Riley, M. (2022) · Atmosphere · doi:10.3390/atmos13091478 · Cited by 32
Anav, A., Antonelli, M., Calmanti, S., Carillo, A., Catalano, F., Dell’Aquila, A., Iacono, R., Marullo, S., Napolitano, E., Palma, M., et al. (2024) · Climate Dynamics · doi:10.1007/s00382-023-07064-3 · Cited by 32
“…procedure with 1000 repetitions Model performance in reproducing inter-annual variability has been assessed by the Model Variability Index (MVI, Gleckler et al. 2008); this metric provides a good measure to assess standard deviation differences between model and reference data and allows to…”
Leibowitz, S. G., Comeleo, R. L., Wigington, P. J., Weaver, C., Morefield, P. E., Sproles, E. A., & Ebersole, J. L. (2014) · Hydrology and earth system sciences · doi:10.5194/hess-18-3367-2014 · Cited by 32
The role of reduced aerosol precursor emissions in driving near-term warming
Gillett, N. P., & von Salzen, K. (2013) · Environmental Research Letters · doi:10.1088/1748-9326/8/3/034008 · Cited by 32
Kim, J., Sanjay, J., Mattmann, C. A., Boustani, M., Ramarao, M. V. S., Krishnan, R., & Waliser, D. E. (2015) · International Journal of Climatology · doi:10.1002/joc.4306 · Cited by 32
Liu, Y., Shi, Z., Liu, R., & Xing, M. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103888 · Cited by 31
Foulon, É., Rousseau, A. N., & Gagnon, P. (2017) · Journal of Hydrology · doi:10.1016/j.jhydrol.2017.12.064 · Cited by 31
Communicating Uncertainty to Policymakers: The Ineliminable Role of Values
Winsberg, E. B. (2018) · Springer eBooks · doi:10.1007/978-3-319-65058-6_13 · Cited by 31
Analysis of Ensemble Mean Forecasts: The Blessings of High Dimensionality
Christiansen, B. (2019) · Monthly Weather Review · doi:10.1175/mwr-d-18-0211.1 · Cited by 31
A Weather-Pattern-Based Evaluation of the Performance of CMIP5 Models over Mexico
Esteban, Y. D., de Raga, G. B., & Rodríguez, O. O. D. (2020) · Climate · doi:10.3390/cli8010005 · Cited by 31
“Note that the TME for mslp seems to be independent of the weather pattern, since the errors oscillate around 0.2 hPa for all regimes.”
BackgroundMethodInfluentialAssessment of the capability of CMIP6 global climate models to simulate Arctic cyclones
Song, J.-N., Fu, G., Xu, Y., Han, Z., Sun, Q., & Wang, H. (2021) · Advances in Climate Change Research · doi:10.1016/j.accre.2021.07.007 · Cited by 31
Wang, D., Liu, J., Wang, H. E., Shao, W., Mei, C., & Ding, X. (2022) · Journal of Water and Climate Change · doi:10.2166/wcc.2022.402 · Cited by 31
Process-Oriented Diagnostics: Principles, Practice, Community Development, and Common Standards
Neelin, J. D., Krasting, J. P., Radhakrishnan, A., Liptak, J., Jackson, T., Ming, Y., Dong, W., Gettelman, A., Coleman, D. R. B., Maloney, E. D., et al. (2023) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-21-0268.1 · Cited by 31
“Scalar evaluation metrics, commonly used to assess CMIP and development models (e.g., Gleckler et al. 2008; Reed et al. 2022), and the more complex process-oriented diagnostics can be viewed as opposite ends of a continuum of diagnostics.”
BackgroundMethodWeedon, G. P., Prudhomme, C., Crooks, S., Ellis, R. J., Folwell, S. S., & Best, M. (2014) · Journal of Hydrometeorology · doi:10.1175/jhm-d-14-0021.1 · Cited by 31
“…models, Taylor diagrams, based on standard deviation and correlation, are often used to combine multiple metrics visually, allowing one to assess whether a particular model outperforms others and/or whether model developments are leading to improved performance (Taylor 2001; Gleckler et al. 2008).”
MethodInterannual variability and expected regional climate change over North America
de Elía, R., Biner, S., & Frigon, A. (2013) · Climate Dynamics · doi:10.1007/s00382-013-1717-9 · Cited by 31
Effects of modeled tropical sea surface temperature variability on coral reef bleaching predictions
van Hooidonk, R. J., & Huber, M. S. (2011) · Coral Reefs · doi:10.1007/s00338-011-0825-4 · Cited by 31
“1 model is considered to have one of the better representations of the tropics (Gleckler et al. 2008).”
BackgroundInfluentialReliability and importance of structural diversity of climate model ensembles
Yokohata, T., Annan, J. D., Collins, M. D., Jackson, C. S., Shiogama, H., Watanabe, M., Emori, S., Yoshimori, M., Abe, M., Webb, M. J., et al. (2013) · Climate Dynamics · doi:10.1007/s00382-013-1733-9 · Cited by 31
Kim, J., Waliser, D. E., Mattmann, C. A., Mearns, L. O., Goodale, C. E., Hart, A. F., Crichton, D. J., McGinnis, S., Lee, H., Loikith, P. C., et al. (2013) · Journal of Climate · doi:10.1175/jcli-d-12-00452.1 · Cited by 31
“15) yields smaller RMSE than all RCMs included in the multimodel ensemble mean.”
BackgroundMethodInfluentialImprovements over three generations of climate model simulations for eastern India
Das, L., Annan, J., Hargreaves, J. C., & Emori, S. (2012) · Climate Research · doi:10.3354/cr01064 · Cited by 31
“Several papers have investigated the © Inter-Research 2012 · www.int-res.com*Email: daslalu@yahoo.co.in KEY WORDS: Seasonal cycle · Model improvement · Similarity statistics · Sub-regional scale · Topography · Model improvement index · Rank Resale or republication not permitted without written consent of the publisher…”
MethodMa, D., Qian, B., Gu, H., Sun, Z., & Xu, Y. (2021) · International Journal of Climatology · doi:10.1002/joc.7025 · Cited by 31
“…a multi-model ensemble often outperform the individual models within the ensemble with respect to the reproduction of the mean historical climate (Gleckler et al., 2008; Wang et al., 2019), largely resulting from the cancellation or compensation of errors in the individual models (Pierce et al.,…”
Evaluation of CMIP5 present climate simulations using the Köppen-Trewartha climate classification
Belda, M., Holtanová, E., Halenka, T., Kalvová, J., & Hlávka, Z. (2015) · Climate Research · doi:10.3354/cr01316 · Cited by 31
“This fact has already been shown in many studies, e.g. Gleckler et al. (2008), Pierce et al. (2009).”
MethodResultAhn, M., Ullrich, P. A., Gleckler, P. J., Lee, J., Ordoñez, A. C., & Pendergrass, A. G. (2023) · Geoscientific model development · doi:10.5194/gmd-16-3927-2023 · Cited by 30
Self-citationAddressing partial identification in climate modeling and policy analysis
Manski, C. F., Sanstad, A. H., & DeCanio, S. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2022886118 · Cited by 30
CMIP6 Model Evaluation for Mean and Extreme Precipitation Over India
Kushwaha, P., Pandey, V. K., Kumar, P., & Sardana, D. (2024) · Pure and Applied Geophysics · doi:10.1007/s00024-023-03409-5 · Cited by 30
“Gleckler et al. (2008) summarized the ranks for all models that depict the three skill parameters and the interannual variability skill score.”
Dioha, E. C., Chung, ., Ayugi, B. O., Babaousmail, H., & Sian, K. T. C. L. K. (2024) · Earth Systems and Environment · doi:10.1007/s41748-024-00397-x · Cited by 30
“On the other hand, the IVS of the simulations relative to the observations (Gleckler et al. 2008; Yang et al. 2021) is assessed using Eq.”
Ménard, C. B., Ikonen, J., Rautiainen, K., Aurela, M., Arslan, A. N., & Pulliainen, J. T. (2015) · Journal of Hydrometeorology · doi:10.1175/jhm-d-15-0013.1 · Cited by 30
“Moriasi et al. (2007), for hydrological models, and Gleckler et al. (2008), for climate models, have recommended the use of specific performance metrics to promote consistency when evaluating different models.”
BackgroundAssessing uncertainties in climate change impacts on runoff in Western Mediterranean basins
Ruelland, D., Hublart, P., & Tramblay, Y. (2015) · Proceedings of the International Association of Hydrological Sciences · doi:10.5194/piahs-371-75-2015 · Cited by 30
He, M., Yang, B. S., Валерьевич, Ш. В., Rossi, S., Bräuning, A., Ljungqvist, F. C., & Grießinger, J. (2017) · International Journal of Biometeorology · doi:10.1007/s00484-017-1472-4 · Cited by 30
“Since former studies indicated that the average of multiple models performs better than any single model when compared with observations (Gleckler et al. 2008; Reichler and Kim 2008), we averaged all related models to represent the Bpast^ and Bfuture^ climate under the three selected scenarios.”
MethodHillman, B., Marchand, R. T., & Ackerman, T. P. (2018) · Journal of Geophysical Research Atmospheres · doi:10.1029/2017jd027680 · Cited by 30
Transferability of climate simulation uncertainty to hydrological impacts
Wang, H., Chen, J., Cannon, A. J., Xu, C., & Chen, H. (2018) · Hydrology and earth system sciences · doi:10.5194/hess-22-3739-2018 · Cited by 30
“…who deal with the assessment of climate change impacts on hydrology, various selection methods have been proposed based on different criteria (Mendlik and Gobiet, 2016; Cannon, 2015; Gleckler et al., 2008; Lutz et al., 2016a; McSweeney et al., 2012; Warszawski et al., 2014; Perkins et al., 2007).”
BackgroundMethodInfluentialHiga, M., Nakao, K., Tsuyama, I., Nakazono, E., Yasuda, M., Matsui, T., & Tanaka, N. (2013) · Ecological Indicators · doi:10.1016/j.ecolind.2013.01.010 · Cited by 30
Song, X., Xu, M., Kang, S., Wang, R., & Wu, H. (2024) · International Journal of Climatology · doi:10.1002/joc.8622 · Cited by 29
“First, the IVS is a symmetric variable statistic that can measure interannual variability with equal values for each model that simulates observed variability (Gleckler et al., 2008).”
Understanding the Distribution of Multimodel Ensembles
Christiansen, B. (2020) · Journal of Climate · doi:10.1175/jcli-d-20-0186.1 · Cited by 29
Zhang, C., Golaz, J., Forsyth, R. M., Vo, T., Xie, S., Shaheen, Z., Potter, G. L., Asay‐Davis, X., Zender, C. S., Lin, W., et al. (2022) · Geoscientific model development · doi:10.5194/gmd-15-9031-2022 · Cited by 29
Testing climate models using an impact model: what are the advantages?
Stéfanon, M., Martin-StPaul, N. K., Leadley, P., Bastin, S., Dell’Aquila, A., Drobinski, P., & Gallardo, C. (2015) · Climatic Change · doi:10.1007/s10584-015-1412-4 · Cited by 29
“Other performance metrics exist and could provide additional insight into the evaluation of climatic variables (Gleckler et al. 2008).”
BackgroundHow well do coupled models replicate ocean energetics relevant to ENSO?
Brown, J. N., Fedorov, A. V., & Guilyardi, É. (2010) · Climate Dynamics · doi:10.1007/s00382-010-0926-8 · Cited by 29
“…University of Reading, Reading, UK Significant efforts are now directed towards defining process-based metrics and diagnostics that can help us understand why ENSO simulations are so different and suggest how we could improve the models (Gleckler et al. 2008; Sun et al. 2008; Lloyd et al. 2009).”
BackgroundThe cloud–aerosol–radiation (CAR) ensemble modeling system
Liang, X., & Zhang, F. (2013) · Atmospheric chemistry and physics · doi:10.5194/acp-13-8335-2013 · Cited by 29
On the projected increase of Sahel rainfall during the late rainy season
Monerie, P., Biasutti, M., & Roucou, P. (2016) · International Journal of Climatology · doi:10.1002/joc.4638 · Cited by 29
L’Hévéder, B., Speich, S., Ragueneau, O., Gohin, F., & Bryère, P. (2016) · International Journal of Climatology · doi:10.1002/joc.4882 · Cited by 29
“…number of indices and models analysed in this study, we have used a metric-based approach to assess model performance, based on the estimation of ‘model relative error’ of model climatologies (Gleckler et al., 2008) and adapted from Sillmann et al. (2013) application to climate extremes indices.”
MethodResultWater Security in an Uncertain Future: Contrasting Realities from an Availability-Demand Perspective
Sone, J. S., Araujo, T. F., Gesualdo, G. C., Ballarin, A. S., Carvalho, G. A., de Oliveira, P. T. S., & Wendland, E. C. (2022) · Water Resources Management · doi:10.1007/s11269-022-03160-x · Cited by 29
“A multimodel ensemble outperforms any individual model by reflecting uncertainties inherent to climate models (Gleckler et al. 2008).”
BackgroundPotential impact of future climate change on crop yield in northeastern China
Zhou, M., & Wang, H. (2015) · Advances in Atmospheric Sciences · doi:10.1007/s00376-014-4161-9 · Cited by 29
“The distance between the point and the reference point indicates the centered root-mean-square error (RMSE); that is, the closer a model is to the reference point, the better its performance (Gleckler et al., 2008; Jiang and Tian, 2013).”
BackgroundKhan, N., Nasara, M. A., Sa’adi, Z., Awhari, D. P., Asiri, M. I., Shahid, S., & Yaseen, Z. M. (2025) · Atmospheric Research · doi:10.1016/j.atmosres.2025.108300 · Cited by 28
Ahsan, S., Bhat, M. S., Alam, A., Farooq, H., & Sheikh, H. A. (2022) · Environmental Science and Pollution Research · doi:10.1007/s11356-022-24913-6 · Cited by 28
“In comparison to the outputs from separate models, the multi-model ensemble has demonstrated to be more effective in the representation and simulation of climatic variables (Gleckler et al., 2008; Knutti et al., 2010; Zhang & Huang, 2013).”
MethodAsokan, S. M., Rogberg, P., Bring, A., Jarsjö, J., & Destouni, G. (2015) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2015.11.017 · Cited by 28
Taniguchi, K. (2016) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2016.10.004 · Cited by 28
Guna, A., Zhang, J., Tong, S., Bao, Y., Han, A., & Li, K. (2019) · Water · doi:10.3390/w11102108 · Cited by 28
Short communication comments on ‘DISO: A rethink of Taylor diagram’
Xu, Z., & Han, Y. (2019) · International Journal of Climatology · doi:10.1002/joc.6359 · Cited by 28
“Model performance metrics that can quantitatively evaluate model overall performance in terms of multiple variables are desirable (Gleckler et al., 2008; Baker and Taylor, 2016).”
BackgroundThe Role of “Complex” Empiricism in the Debates About Satellite Data and Climate Models
Lloyd, E. A. (2018) · Springer eBooks · doi:10.1007/978-3-319-65058-6_6 · Cited by 28
Chang, H., Castro, C. L., Carrillo, C. M., & Domínguez, F. (2015) · Journal of Geophysical Research Atmospheres · doi:10.1002/2015jd023333 · Cited by 28
“…that are generally present in global climate change projection models: the widening of the tropical belt [Seidel et al., 2008; Lu et al., 2009] and a poleward shift of the westerlies [Archer and Caldeira, 2008; Ostermeier and Wallace, 2003; Thompson and Solomon, 2002; Gillett and Thompson, 2003].”
BackgroundGulakhmadov, A., Chen, X., Gulahmadov, N., Rizwan, M., Gulakhmadov, M., Nadeem, M. U., Rakhimova, M., & Liu, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-86366-4 · Cited by 27
Haywood, J., Boucher, O., Lennard, C., Storelvmo, T., Tilmes, S., & Visioni, D. (2025) · Frontiers in Climate · doi:10.3389/fclim.2025.1507479 · Cited by 27
Mumo, L., & Yu, J. (2019) · Atmospheric Research · doi:10.1016/j.atmosres.2019.104808 · Cited by 27
Evaluation and bias correction of global climate models in the CMIP5 over the Indian Ocean region
Mohan, S., & Bhaskaran, P. K. (2019) · Environmental Monitoring and Assessment · doi:10.1007/s10661-019-7700-0 · Cited by 27
“Reliability of future climate projections based on climate model experiments has a direct bearing on the present-day climate and that can be assessed by comparing with available observations (Pierce et al. 2006; Gleckler et al. 2008; Waliser et al. 2009; Su et al. 2013).”
BackgroundEbtehaj, I., & Bonakdari, H. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.128851 · Cited by 27
Identical hierarchy of physical drought types for climate change signals and uncertainty
Hosseinzadehtalaei, P., Schaeybroeck, B. V., Termonia, P., & Tabari, H. (2023) · Weather and Climate Extremes · doi:10.1016/j.wace.2023.100573 · Cited by 27
CMIP3 ensemble climate projections over the western tropical Pacific based on model skill
Perkins-Kirkpatrick, S. E., Irving, D., Brown, J. R., Power, S. B., Moise, A., Colman, R., & Smith, I. (2011) · Climate Research · doi:10.3354/cr01046 · Cited by 27
“While there is strong evidence that any given model does not perform consistently across multiple variables (e.g. Whetton et al. 2007, Gleckler et al. 2008), segregation of the models into ensembles is based on the overall evaluation scores presented in Irving et al. (2011; our Fig.”
BackgroundRuosteenoja, K., Jylhä, K., Räisänen, J., & Mäkelä, A. (2017) · Journal of Geophysical Research Atmospheres · doi:10.1002/2017jd026909 · Cited by 27
“In several responses from the modeling groups it was emphasized that RH at 2 m is only an output product of the run that is not fed back into the model algorithm [cf. Gleckler et al., 2008].”
Pareto‐Optimal Estimates of California Precipitation Change
Langenbrunner, B., & Neelin, J. D. (2017) · Geophysical Research Letters · doi:10.1002/2017gl075226 · Cited by 27
“A single best model or model subset is difficult to select across multiple measures (Gleckler et al., 2008; Pierce et al., 2009, 2013; Pincus et al., 2008; Reichler & Kim, 2008; Santer et al., 2009), as improvement in one metric is often accompanied by degradation in another, and these trade-offs…”
Background
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.