Performance metrics for climate models.
The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective measures of climate model performance are proposed and used to assess simulations of the 20th century, which are available from the Coupled Model Intercomparison Project (CMIP3) archive. The primary focus of this analysis is on the climatology of atmospheric fields. For each variable considered, the models are ranked according to a measure of relative error. Based on an average of the relative errors over all fields considered, some models appear to perform substantially better than others. Forming a single index of model performance, however, can be misleading in that it hides a more complex picture of the relative merits of different models. This is demonstrated by examining individual variables and showing that the relative ranking of models varies considerably from one variable to the next. A remarkable exception to this finding is that the so‐called “mean model” consistently outperforms all other models in nearly every respect. The usefulness, limitations and robustness of the metrics defined here are evaluated 1) by examining whether the information provided by each metric is correlated in any way with the others, and 2) by determining how sensitive the metrics are to such factors as observational uncertainty, spatial scale, and the domain considered (e.g., tropics versus extra‐tropics). An index that gauges the fidelity of model variability on interannual time‐scales is found to be only weakly correlated with an index of the mean climate performance. This illustrates the importance of evaluating a broad spectrum of climate processes and phenomena since accurate simulation of one aspect of climate does not guarantee accurate representation of other aspects. Once a broad suite of metrics has been developed to characterize model performance it may become possible to identify optimal subsets for various applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“not even the question of which model performs best during the control simulations can be readily answered but, rather, depends strongly on the skill metric, variable, and region considered (e.g., Gleckler et al. 2008).”
From Risks of Model Weighting in Multimodel Climate Projections (2010) · cites this paper as Gleckler et al. 2008
Citations
Cites
The NCEP/NCAR 40-Year Reanalysis Project
Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., et al. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0437:tnyrp>2.0.co;2 · Cited by 28,360
Summarizing multiple aspects of model performance in a single diagram
Taylor, K. E. (2001) · Journal of Geophysical Research: Atmospheres · doi:10.1029/2000jd900719 · Cited by 8,477
Self-citationAdler, R. F., Huffman, G. J., Chang, A., Ferraro, R., Xie, P.-P., Janowiak, J., Rudolf, B., Schneider, U., Curtis, S., Bolvin, D., et al. (2003) · Journal of Hydrometeorology · doi:10.1175/1525-7541(2003)004<1147:tvgpcp>2.0.co;2 · Cited by 3,808
Advances in Understanding Clouds from ISCCP
Rossow, W. B., & Schiffer, R. A. (1999) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1999)080<2261:aiucfi>2.0.co;2 · Cited by 2,256
Clouds and the Earth's Radiant Energy System (CERES): An Earth Observing System Experiment
Wielicki, B. A., Barkstrom, B. R., Harrison, E. F., Lee, R. B., Louis Smith, G., & Cooper, J. E. (1996) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1996)077<0853:catere>2.0.co;2 · Cited by 1,957
The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset
Huffman, G. J., Adler, R. F., Arkin, P., Chang, A., Ferraro, R., Gruber, A., Janowiak, J., McNab, A., Rudolf, B., & Schneider, U. (1997) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1997)078<0005:tgpcpg>2.0.co;2 · Cited by 1,723
Quantification of modelling uncertainties in a large ensemble of climate change simulations.
Murphy, J. M., Sexton, D. M. H., Barnett, D. N., Jones, G. S., Webb, M. J., Collins, M., & Stainforth, D. A. (2004) · Nature · doi:10.1038/nature02771 · Cited by 1,655
AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems
Aumann, H. H., Chahine, M. T., Gautier, C., Goldberg, M. D., Kalnay, E., McMillin, L. M., Revercomb, H., Rosenkranz, P. W., Smith, W. L., Staelin, D. H., et al. (2003) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2002.808356 · Cited by 1,542
Uncertainty in predictions of the climate response to rising levels of greenhouse gases
Stainforth, D. A., Aina, T., Christensen, C., Collins, M., Faull, N., Frame, D. J., Kettleborough, J. A., Knight, S., Martin, A., Murphy, J. M., et al. (2005) · Nature · doi:10.1038/nature03301 · Cited by 1,166
Research works printed but not linked: 12
- 10.1175/1520-0477(1984)065 · doi:10.1175/1520-0477(1984)065
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0477(1997)078 · doi:10.1175/1520-0477(1997)078
- 10.1175/1520-0442(1999)012 · doi:10.1175/1520-0442(1999)012
- 10.1175/1520-0477(1996)077 · doi:10.1175/1520-0477(1996)077
- 10.1175/1520-0477(2000)081 · doi:10.1175/1520-0477(2000)081
- 10.1175/1520-0477(1999)080 · doi:10.1175/1520-0477(1999)080
- 10.1175/1520-0442(1993)006 · doi:10.1175/1520-0442(1993)006
- Simmons A. J., 2000, The ERA‐40 project plan, ERA‐40 Project Report Series No. 1
- 10.1175/1520-0442(2004)017 · doi:10.1175/1520-0442(2004)017
- Taylor P.(Ed.) (2000) Final report of the Joint WCRP/SCOR Working Group on Air‐Sea Fluxes WCRP‐112.
- 10.1175/1520‐0477 · doi:10.1175/1520-0477
Cited by
Items 301–400 of 1,001
Zizinga, A., Mwanjalolo, J.-G. M., Tietjen, B., Bedadi, B., Pathak, H. K., Gabiri, G., & Beesigamukama, D. (2022) · Agricultural Systems · doi:10.1016/j.agsy.2022.103407 · Cited by 76
2016 International Land Model Benchmarking (ILAMB) Workshop Report
Hoffman, F. M., Koven, C. D., Keppel‐Aleks, G., Lawrence, D. M., Riley, W. J., Randerson, J. T., Ahlström, A., Abramowitz, G., Baldocchi, D., Best, M., et al. (2017) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/1330803 · Cited by 76
“The color scale portrays this as a relative error by normalizing the result by the median error of all model results (Gleckler et al., 2008).”
MethodSelf-citationDeidda, R., Marrocu, M., Caroletti, G. N., Pusceddu, G., Langousis, A., Lucarini, V., Puliga, M., & Speranza, A. (2013) · Hydrology and earth system sciences · doi:10.5194/hess-17-5041-2013 · Cited by 75
Assessing model performance of climate extremes in China: an intercomparison between CMIP5 and CMIP3
Chen, H., & Sun, J. (2015) · Climatic Change · doi:10.1007/s10584-014-1319-5 · Cited by 75
“To further understand the model evaluation, two additional metrics, i.e., an exploratory model climate performance index (MCPI) for climatology and an exploratory model variability index (MVI) for the variability of all fields proposed and constructed by Gleckler et al. (2008), are used.”
BackgroundMethodI—Elisabeth A. Lloyd: Varieties of Support and Confirmation of Climate Models
Lloyd, E. A. (2009) · Aristotelian Society Supplementary Volume · doi:10.1111/j.1467-8349.2009.00179.x · Cited by 75
Assessment of CMIP6 global climate models in reconstructing rainfall climatology of Bangladesh
Kamruzzaman, M., Shahid, S., Roy, D. K., Islam, A. R. M. T., Hwang, S., Cho, J., Zaman, M. A. U., Sultana, T., Rashid, T., & Akter, F. (2021) · International Journal of Climatology · doi:10.1002/joc.7452 · Cited by 74
“GCM's ability to reproduce the climate variable of interest is often recommended to increase confidence in projecting that variable (Gleckler et al., 2008; McMahon et al., 2015; Lutz et al., 2016; Sa'adi et al., 2017; Salman et al., 2018).”
BackgroundRao, K. K., Mandous, A. A., Ebri, M. A., Hameli, N. A., Rakib, M., & Kaabi, S. A. (2024) · Scientific Reports · doi:10.1038/s41598-023-49910-8 · Cited by 74
Tamoffo, A. T., Moufouma‐Okia, W., Dosio, A., James, R., Pokam, W. M., Vondou, D. A., Fotso‐Nguemo, T. C., Guenang, G. M., Tamo, P. H. K., Nikulin, G., et al. (2019) · Climate Dynamics · doi:10.1007/s00382-019-04751-y · Cited by 74
Distinctive South and East Asian monsoon circulation responses to global warming
Li, T., Wang, Y., Wang, B., Ting, M., Ding, Y., Sun, Y., He, C., & Yang, G. (2021) · 中国科学通报:英文版 · doi:10.1016/j.scib.2021.12.001 · Cited by 73
Future projections of extreme rainfall events in Indonesia
Kurniadi, A., Weller, E., Salmond, J., & Aldrian, E. (2023) · International Journal of Climatology · doi:10.1002/joc.8321 · Cited by 73
“The seasonal variations in extreme precipitation are analysed by utilising the accumulated precipitation over a season and generating a portrait plot (Gleckler et al., 2008; Kim et al., 2018).”
Barstad, I., Sorteberg, A., & Mesquita, M. D. S. (2012) · Renewable Energy · doi:10.1016/j.renene.2012.02.008 · Cited by 73
“g [4].”
Evaluation of CMIP3 and CMIP5 Models over the Australian Region to Inform Confidence in Projections
Moise, A. F., Wilson, L., Grose, M., Whetton, P. H., Watterson, I. G., Bhend, J., Bathols, J., Hanson, L., Erwin, T., Bedin, T., et al. (2015) · Australian Meteorological and Oceanographic Journal · doi:10.1071/es15004 · Cited by 73
The potential for structural errors in emergent constraints
Sanderson, B. M., Pendergrass, A. G., Koven, C. D., Brient, F., Booth, B. B. B., Fisher, R. A., & Knutti, R. (2021) · Earth System Dynamics · doi:10.5194/esd-12-899-2021 · Cited by 72
Climate change and the Portuguese precipitation: ENSEMBLES regional climate models results
Soares, P. M. M., Cardoso, R. M., Ferreira, J. J., & Miranda, P. (2014) · Climate Dynamics · doi:10.1007/s00382-014-2432-x · Cited by 72
“The simplest method is to equally average all available models, and rely in the finding that the ensemble mean performs better than individual ensemble models, regarding the mean value, but not the variances (Gleckler et al. 2008; Pincus et al. 2008; Reichler and Kim 2008; Pierce et al. 2009).”
MethodFeng, S., Ho, C.-H., Hu, Q., Oglesby, R. J., Jeong, S.-J., & Kim, B. (2011) · Climate Dynamics · doi:10.1007/s00382-011-1020-6 · Cited by 72
“These results support the previous finding that multiple model ensembles usually do a better job in simulating observed climate changes (Gleckler et al. 2008; Reichler and Kim 2008; Pierce et al. 2009).”
BackgroundResultClimate System Dynamics and Modelling
Goosse, H. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9781316018682 · Cited by 72
Ghorbani, M. A., Deo, R. C., Kim, S., Kashani, M. H., Karimi, V., & Izadkhah, M. (2020) · Soft Computing · doi:10.1007/s00500-019-04648-2 · Cited by 72
Do we (need to) care about canopy radiation schemes in DGVMs? Caveats and potential impacts
Loew, A., van Bodegom, P. M., Widlowski, J., Otto, J., Quaife, T. L., Pinty, B., & Raddatz, T. (2014) · Biogeosciences · doi:10.5194/bg-11-1873-2014 · Cited by 72
“…model benchmarking and evaluation studies detail particular deviations of model results compared to observations and assess (relative) model skill (Gleckler et al., 2008; Luo et al., 2012; Brovkin et al., 2013b; Hagemann et al., 2013), they typically do not provide an assessment of potential…”
ResultDecadal Climate Prediction: Opportunities and Challenges
J, W., J, W., J, W., J, W., J, W., J, W., J, W., J, W., J, W., J, W., et al. (2010) · doi:10.5270/oceanobs09.cwp.45 · Cited by 71
“climate application is problematic (Gleckler et al. 2008).”
BackgroundHydrological Modeling to Evaluate Climate Model Simulations and Their Bias Correction
Hakala, K., Addor, N., & Seibert, J. (2018) · Journal of Hydrometeorology · doi:10.1175/jhm-d-17-0189.1 · Cited by 71
Impact of global warming on heating and cooling degree days in major Indian cities
Ukey, R., & Aakash, C. (2021) · Energy and Buildings · doi:10.1016/j.enbuild.2021.111050 · Cited by 70
Verma, S., Kumar, K., Verma, M. K., Prasad, A. D., Mehta, D., & Rathnayake, U. (2023) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2023.101533 · Cited by 71
Dolan, A. M., Hunter, S. J., Hill, D. J., Haywood, A. M., Koenig, S. J., Otto‐Bliesner, B. L., Abe‐Ouchi, A., Bragg, F. J., Chan, W., Chandler, M. A., et al. (2015) · Climate of the past · doi:10.5194/cp-11-403-2015 · Cited by 71
Siew, J. H., Tangang, F., & Juneng, L. (2013) · International Journal of Climatology · doi:10.1002/joc.3880 · Cited by 70
“The ability of AOGCMs to simulate the annual cycles is a necessary condition for confidence in their prediction of long-term climatic changes (Wang et al., 2004; Gleckler et al., 2008; Bombardi and Carvalho, 2009; Kai et al., 2009; Rajeevan and Nanjundiah, 2009; Martin and Schumacher, 2011).”
BackgroundMethodAubry, T. J., Jellinek, M. A., Degruyter, W., Bonadonna, C., Radić, V., Clyne, M., & Quainoo, A. (2016) · Journal of Geophysical Research Atmospheres · doi:10.1002/2016jd025405 · Cited by 69
Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3
Lee, J., Gleckler, P. J., Ahn, M., Ordoñez, A. C., Ullrich, P. A., Sperber, K. R., Taylor, K. E., Planton, Y., Guilyardi, É., Durack, P. J., et al. (2024) · Geoscientific model development · doi:10.5194/gmd-17-3919-2024 · Cited by 72
“Thus, for each axis, the models in the lower half of the plot perform better than the CMIP5–CMIP6 multi-model median, while in the upper half, the opposite is true.”
BackgroundMethodSelf-citationWhat is the role of the observational dataset in the evaluation and scoring of climate models?
Gómez‐Navarro, J. J., Montávez, J. P., Jerez, S., Jiménez‐Guerrero, P., & Zorita, E. (2012) · Geophysical Research Letters · doi:10.1029/2012gl054206 · Cited by 64
“This approach is based in the assumption that the observational uncertainty is small compared with intermodel variability [Gleckler et al., 2008].”
BackgroundResultWeighting climate model ensembles for mean and variance estimates
Haughton, N., Abramowitz, G., Pitman, A. J., & Phipps, S. J. (2015) · Climate Dynamics · doi:10.1007/s00382-015-2531-3 · Cited by 70
“A complete description of the ensemble generation process, including parameter ranges and the sampling approach, is given in Haughton et al. (2014). The perturbed structure ensemble is intended to emulate an ensemble of single simulations from several climate models, each using comparable initial conditions and pertur…”
BackgroundMethodHistorical and Future Relations Between Large Storms and Droughts in California
Survey, U. S. G., & Dettinger, M. D. (2016) · San Francisco Estuary and Watershed Science · doi:10.15447/sfews.2016v14iss2art1 · Cited by 69
“…ESTUARY & WATERSHED SCIENCE 12 VOLUME 14, ISSUE 2, ARTICLE 1 terms of how well historical climatic metrics were simulated at global scales (Gleckler et al. 2008; Flato et al. 2013), over the southwestern U.S. (using the metrics of Rupp et al. 2013, as computed by Rupp for the southwest;…”
BackgroundMethodAssessing the extent of non-stationary biases in GCMs
Nahar, J., Johnson, F. M., & Sharma, A. (2017) · Journal of Hydrology · doi:10.1016/j.jhydrol.2017.03.045 · Cited by 69
Kaini, S., Nepal, S., Pradhananga, S., Gardner, T., & Sharma, A. K. (2019) · International Journal of Climatology · doi:10.1002/joc.6447 · Cited by 68
“Gleckler et al. (2008) have reported that all climate models/ensembles are not equally skillful to represent the annual cycle climatology and the variance of monthly anomalies.”
BackgroundThe impact of model fidelity on seasonal predictive skill
Kharin, V. V., & Scinocca, J. (2012) · Geophysical Research Letters · doi:10.1029/2012gl052815 · Cited by 68
“It is this assumption that underlies the substantial community activity surrounding the development and application of metrics, which seeks to more quantitatively characterize a model’s fidelity with respect to the observed record [Gleckler et al., 2008].”
BackgroundGlobal hydrology modelling and uncertainty: running multiple ensembles with a campus grid
Gosling, S. N., Bretherton, D. A., Haines, K., & Arnell, N. W. (2010) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2010.0164 · Cited by 68
“This is a reasonable assumption because of the challenge in defining appropriate measures of relative performance (Gleckler et al. 2008), but this does require further investigation.”
BackgroundConsistently Estimating Internal Climate Variability from Climate Model Simulations
Olonscheck, D., & Notz, D. (2017) · Journal of Climate · doi:10.1175/jcli-d-16-0428.1 · Cited by 68
“CMIP5 model plausibility subdivided for each individual simulation and each month is presented as a Unauthenticated | Downloaded 04/03/22 05:39 PM UTC portrait plot as introduced by Gleckler et al. (2008) (Fig.”
MethodFidelity of CMIP5 multi-model mean in assessing Indian monsoon simulations
Mishra, S. K., Sahany, S., Salunke, P., Kang, I., & Jain, S. (2018) · npj Climate and Atmospheric Science · doi:10.1038/s41612-018-0049-1 · Cited by 68
Assessment of the Extreme Precipitation by Satellite Estimates over South America
Palharini, R. S. A., Vila, D. A., Rodrigues, D. T., Pareja-Quispe, D., Palharini, R. C., de Siqueira, R. A., & Afonso, J. M. D. S. (2020) · Remote Sensing · doi:10.3390/rs12132085 · Cited by 68
Exploring Perturbed Physics Ensembles in a Regional Climate Model
Bellprat, O., Kotlarski, S., Lüthi, D., & Schär, C. (2012) · Journal of Climate · doi:10.1175/jcli-d-11-00275.1 · Cited by 68
Reliability of the global climate models during 1961–1999 in arid and semiarid regions of China
Yang, Y., Bai, L., Wang, B., Wu, J., & Fu, S. (2019) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2019.02.188 · Cited by 67
Landerer, F. W., Gleckler, P. J., & Lee, T. (2013) · Climate Dynamics · doi:10.1007/s00382-013-1939-x · Cited by 67
“As we did for the time-mean fields, we computed Taylor statistics of each CMIP5 model, but this time for the 12 monthly climatology anomaly SSH fields for each model (Gleckler et al. 2008).”
BackgroundMethodResultInfluentialSelf-citationDownscaling projections of Indian monsoon rainfall using a non‐homogeneous hidden Markov model
Greene, A. M., Robertson, A. W., Smyth, P., & Triglia, S. (2011) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.788 · Cited by 67
“Use of multimodel mean fields follows from work showing that such fields as a rule provide more consistently faithful representations of the observed climate than do individual models (Gleckler et al., 2008), and provides a natural way of estimating uncertainties arising from differences in GCM formulation.”
BackgroundA Weather-Type-Based Cross-Time-Scale Diagnostic Framework for Coupled Circulation Models
Muñoz, Á. G., Yang, X., Vecchi, G. A., Robertson, A. W., & Cooke, W. F. (2017) · Journal of Climate · doi:10.1175/jcli-d-17-0115.1 · Cited by 67
“For example, Gleckler et al. (2008) used different metrics to assess the climatological behavior of GCMs, Covey et al. (2016) focused on model evaluation of the rainfall diurnal cycle, and several studies have analyzed the present-time rainfall characteristics in state-of-theart climate change…”
MethodMulti-model ensembles for assessment of flood losses and associated uncertainty
Figueiredo, R., Schröter, K., Weiss-Motz, A., Martina, M. L. V., & Kreibich, H. (2018) · Natural hazards and earth system sciences · doi:10.5194/nhess-18-1297-2018 · Cited by 67
“Ensemble means have been shown to almost always outperform individual models (Georgakakos et al., 2004; Gleckler et al., 2008; Reichler and Kim, 2008), and the combination of the output of different models can be a pragmatic approach to estimate model uncertainty (Palmer et al., 2004; Weigel et…”
BackgroundAssessing water security in the São Paulo metropolitan region under projected climate change
Gesualdo, G. C., de Oliveira, P. T. S., Rodrigues, D. B. B., & Gupta, H. V. (2019) · Hydrology and earth system sciences · doi:10.5194/hess-23-4955-2019 · Cited by 67
“…instead of any single model projection, to reflect the range of uncertainties inherent to the current suite of GCMs, and also because reports have indicated that the ensembles, as a whole, provide superior performance to that of any individual model (Dhakal et al., 2018; Gleckler et al., 2008).”
Gain, A. K., Apel, H., Renaud, F. G., & Giupponi, C. (2013) · Climatic Change · doi:10.1007/s10584-013-0800-x · Cited by 66
“In climate predictions, a multi-model ensemble tends to give more reliable results than single model simulation (Gleckler et al. 2008; Knutti 2008).”
MethodSchwalm, C. R., Huntinzger, D. N., Michalak, A. M., Fisher, J. B., Kimball, J. S., Mueller, B., Zhang, K., & Zhang, Y. (2013) · Environmental Research Letters · doi:10.1088/1748-9326/8/2/024028 · Cited by 66
“More comprehensive benchmarking frameworks track model skill over successive versions of a given model (Gleckler et al 2008) and allow for a quantitative evaluation of model skill across multiple fields and models (Randerson et al 2009).”
BackgroundResultMonier, E., Scott, J. R., Sokolov, A., Forest, C. E., & Schlosser, C. A. (2013) · Geoscientific model development · doi:10.5194/gmd-6-2063-2013 · Cited by 67
Model performance metrics and process diagnostics for boreal summer intraseasonal variability
Neena, J. M., Waliser, D. E., & Jiang, X. (2016) · Climate Dynamics · doi:10.1007/s00382-016-3166-8 · Cited by 66
“Figure 9 presents the above discussed model evaluation metrics in the form of a “portrait diagram” (e.g. Gleckler et al. 2008), which readily summarizes the pattern correlation scores of the 27 models.”
BackgroundClimate Variability, Volcanic Forcing, and Last Millennium Hydroclimate Extremes
Stevenson, S. L., Overpeck, J. T., Fasullo, J., Coats, S., Parsons, L. A., Otto‐Bliesner, B. L., Ault, T. R., Loope, G. R., & Cole, J. E. (2018) · Journal of Climate · doi:10.1175/jcli-d-17-0407.1 · Cited by 66
“…a useful framework for diagnosing the mechanisms responsible for megaevent generation, despite some caveats related to the role of structural biases and issues with the amplitude of low-frequency variability (Gleckler et al. 2008; Ault et al. 2013; Bellenger et al. 2014; Coats et al. 2015a).”
BackgroundScherrer, S. C. (2011) · International Journal of Climatology · doi:10.1002/joc.2170 · Cited by 65
“To avoid these cancellation effects the variability index (VI ) as introduced by Gleckler et al. (2008) is used to analyse the performance for each model, variable and grid-point: V Ix,yvar = ( sx,yvar s x,y var,ref − 1 sx,yvar / s x,y var,ref )2 = ( sx,yvar s x,y var,ref − s x,y var,ref sx,yvar )2…”
BackgroundMethodWill population exposure to heat extremes intensify over Southeast Asia in a warmer world?
Sun, X., Ge, F., Fan, Y., Zhu, S., & Chen, Q. (2022) · Environmental Research Letters · doi:10.1088/1748-9326/ac48b6 · Cited by 65
Okkan, U., & İnan, G. (2014) · International Journal of Climatology · doi:10.1002/joc.4206 · Cited by 65
“In the literature, there exist important studies focusing on multiple GCMs and the differences and uncertainties between the forecasts of different GCMs (Gleckler et al., 2008; Knutti, 2008; Reichler and Kim, 2008).”
BackgroundThe ratio of land to ocean temperature change under global warming
Boer, G. J. (2011) · Climate Dynamics · doi:10.1007/s00382-011-1112-3 · Cited by 65
“Lambert and Boer 2001; Boer and Lambert 2008; Gleckler et al. 2008 ), and are also”
BackgroundEvaluating global climate models for the Pacific island region
Irving, D., Perkins-Kirkpatrick, S. E., Brown, J. R., Gupta, A. S., Moise, A., Murphy, B., Muir, L., Colman, R., Power, S. B., Delage, F. P. D., et al. (2011) · Climate Research · doi:10.3354/cr01028 · Cited by 65
“In fact, there are relatively few studies in the peer reviewed literature that have attempted to combine metrics of model performance pertaining to such a wide variety of climate processes and phenomena (e.g. Gleckler et al. 2008, Pierce et al. 2009, Santer et al. 2009).”
BackgroundMethodResultInfluentialZiehn, T., Lenton, A. A., Law, R. M., Matear, R. J., & Chamberlain, M. A. (2017) · Geoscientific model development · doi:10.5194/gmd-10-2591-2017 · Cited by 65
Downscaling of monthly precipitation using CMIP5 climate models operated under RCPs
Okkan, U., & Kırdemir, U. (2016) · Meteorological Applications · doi:10.1002/met.1575 · Cited by 64
“(Gleckler et al., 2008). However, some biases may be observed in ensemble simulations (Knutti et al., 2010). In this section, following the ensemble process based on the BMA scheme given in the previous section, a bias correction technique was used so as to lessen possible biases in ensemble forecasts. It is possible…”
MethodInfluentialDoes the weighting of climate simulations result in a better quantification of hydrological impacts?
Wang, H., Chen, J., Xu, C., Chen, H., Guo, S., Xie, P., & Li, X. (2019) · Hydrology and earth system sciences · doi:10.5194/hess-23-4033-2019 · Cited by 64
“…the equal average of an MME often outperforms any individual model in regards to the reproduction of the mean state of observed historical climate (Gleckler et al., 2008; Reichler and Kim, 2008), whether the equal weighting is a better strategy for Published by Copernicus Publications on behalf…”
Han, Y., Zhang, M., Xu, Z., & Guo, W. (2021) · Climate Dynamics · doi:10.1007/s00382-021-05986-4 · Cited by 63
“The MVIE method is based on a vector field evaluation diagram, which is a generalization of the Taylor diagram (Taylor, 2001; Gleckler et al., 2008; Xu et al., 2016; 2017).”
The role of ‘complex’ empiricism in the debates about satellite data and climate models
Lloyd, E. A. (2012) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2012.02.001 · Cited by 63
Wang, X., Zhang, B., Xu, X., Tian, J., & He, C. (2019) · Journal of Hydrology · doi:10.1016/j.jhydrol.2019.124246 · Cited by 63
U.K. Community Earth System Modeling for CMIP6
Senior, C. A., Jones, C., Wood, R., Sellar, A., Belcher, S. E., Tank, A. K., Sutton, R. T., Walton, J. P. R. B., Lawrence, B., Andrews, T., et al. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms002004 · Cited by 63
“5 of 11 of relative root‐mean‐square (RMS) error (Gleckler et al., 2008) to illustrate that whenmeasured over a range of atmospheric variables for the period 1997–2010, UKESM1 and HadGEM3‐GC3.”
BackgroundLi, L., Li, W., & Deng, Y. (2013) · Journal of Geophysical Research Atmospheres · doi:10.1002/jgrd.50136 · Cited by 62
“The MME emphasizes precipitation changes due to climate forcing and deemphasizes differences in models’ dynamic cores and parameterization schemes [Gleckler et al., 2008].”
BackgroundChange in the precipitation intensity of the East Asian summer monsoon projected by CMIP3 models
Kusunoki, S., & Arakawa, O. (2011) · Climate Dynamics · doi:10.1007/s00382-011-1234-7 · Cited by 62
Assessment of indices of temperature extremes simulated by multiple CMIP5 models over China
Dong, S., Xu, Y., Zhou, B., & Shi, Y. (2015) · Advances in Atmospheric Sciences · doi:10.1007/s00376-015-4152-5 · Cited by 62
“Each field is normalized by the corresponding standard deviation of the reference data (hereafter referred to as NSD) (Gleckler et al., 2008), which allows the ensemble mean in the different subregions (distinguished by number) to be shown in each panel.”
BackgroundMethodResultInfluentialImpact of climate change on precipitation patterns: a comparative approach
Djebou, D. C. S., & Singh, V. P. (2016) · International Journal of Climatology · doi:10.1002/joc.4578 · Cited by 63
“This is beneficial because multimodel generally provides better climate estimates than a single model (Phillips and Gleckler, 2006; Gleckler et al., 2008).”
BackgroundComparison of Monthly Temperature Extremes Simulated by CMIP3 and CMIP5 Models
Yao, Y., Luo, Y. M., Huang, J., & Zhao, Z. (2013) · Journal of Climate · doi:10.1175/jcli-d-12-00560.1 · Cited by 61
“To evaluate the simulations of the extreme monthly-mean temperatures in different regions, the relative error defined by Gleckler et al. (2008) is applied as a metric in this study.”
MethodA Framework for Evaluating Climate Model Performance Metrics
Baker, N. C., & Taylor, P. C. (2016) · Journal of Climate · doi:10.1175/jcli-d-15-0114.1 · Cited by 61
“Gleckler et al. (2008) created a set of performancemetrics using the root-meansquared differences between simulated and observed climate.”
MethodFossey, M., & Rousseau, A. N. (2016) · Journal of Hydrology · doi:10.1016/j.jhydrol.2016.08.032 · Cited by 61
Evaluation and application of Bayesian multi-model estimation in temperature simulations
Miao, C., Duan, Q., Sun, Q., & Li, J. (2013) · Progress in Physical Geography Earth and Environment · doi:10.1177/0309133313494961 · Cited by 60
“…and that the ensembles may even outperform the best single participating model (Buser et al., 2009; Duan and Phillips, 2010; Fischer et al., 2012; Gleckler et al., 2008; IPCC, 2001; Lambert and Boer, 2001; Phillips and Gleckler, 2006; Pincus et al., 2008; Reichler and Kim, 2008; Weigel et al.,…”
BackgroundGreenland’s contribution to global sea-level rise by the end of the 21st century
Graversen, R. G., Drijfhout, S., Hazeleger, W., van de Wal, R. S. W., Bintanja, R., & Helsen, M. M. (2010) · Climate Dynamics · doi:10.1007/s00382-010-0918-8 · Cited by 60
“They represent the models with the highest performance score for the Northern Hemisphere extratropics in an evaluation by Gleckler et al. (2008) .”
MethodChen, J., St-Denis, B. G., Brissette, F. P., & Lucas‐Picher, P. (2016) · Journal of Hydrometeorology · doi:10.1175/jhm-d-15-0099.1 · Cited by 60
“The performance of these four bias correction methods is expressed using the portrait diagram in Gleckler et al. (2008), in which different colors indicate the remaining biases of precipitation, temperature, and streamflow in terms of the natural variability units.”
Ahmadalipour, A., Moradkhani, H., & Rana, A. (2017) · Climate Dynamics · doi:10.1007/s00382-017-3639-4 · Cited by 60
“In general, models have proved to simulate mean state and variability of many variables reasonably well (Gleckler et al. 2008; Rupp et al. 2013; Ahmadalipour et al. 2015).”
BackgroundCook, L. M., Anderson, C., & Samaras, C. (2017) · Journal of Infrastructure Systems · doi:10.1061/(asce)is.1943-555x.0000382 · Cited by 60
Shamshirband, S., Esmaeilbeiki, F., Zarehaghi, D., Neyshabouri, M., Samadianfard, S., Ghorbani, M. A., Mosavi, A. H., Nabipour, N., & Chau, K. (2020) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2020.1788644 · Cited by 60
“In the Taylor diagram, several aspects can be reviewed instantly (Gleckler et al., 2008; Taylor, 2001).”
BackgroundChen, R., Duan, K., Shang, W., Shi, P., Meng, Y., & Zhang, Z. (2022) · Atmospheric Research · doi:10.1016/j.atmosres.2022.106306 · Cited by 60
Evaluation of CMIP5 Models in the Context of Dynamical Downscaling over Europe
Jury, M., Prein, A. F., Truhetz, H., & Gobiet, A. (2015) · Journal of Climate · doi:10.1175/jcli-d-14-00430.1 · Cited by 59
“Several methods to analyze model performance are described in literature (e.g., Pierce et al. 2009; Reichler and Kim 2008; Gleckler et al. 2008).”
MethodBrands, S., Herrera, S., San-Martín, D., & Gutiérrez, J. M. (2011) · Climate Research · doi:10.3354/cr00995 · Cited by 59
Ensemble Averaging and the Curse of Dimensionality
Christiansen, B. (2017) · Journal of Climate · doi:10.1175/jcli-d-17-0197.1 · Cited by 59
Precipitation Bias Correction: A Novel Semi‐parametric Quantile Mapping Method
Rajulapati, C. R., & Papalexiou, S. M. (2023) · Earth and Space Science · doi:10.1029/2023ea002823 · Cited by 64
“Climate models may have systematic biases with respect to observed hydrological variables such as precipitation (Flato et al., 2013; Gleckler et al., 2008).”
BackgroundConsistent large‐scale temperature responses in warm and cold climates
Izumi, K., Bartlein, P. J., & Harrison, S. P. (2013) · Geophysical Research Letters · doi:10.1002/grl.50350 · Cited by 58
“…model and the observations is consistent with the idea that the mean model is usually closer to observations than most individual climate models [see, e.g., Gleckler et al., 2008] and provides support for the continued use of multimodel ensembles for paleoclimate analyses and future projections.”
ResultJohnson, T. E., Butcher, J., Parker, A. J., & Weaver, C. (2011) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)wr.1943-5452.0000175 · Cited by 58
Implications of future bioclimatic shifts on Portuguese forests
Costa, R., Fraga, H., Fernandes, P. M., & Santos, J. A. (2016) · Regional Environmental Change · doi:10.1007/s10113-016-0980-9 · Cited by 58
“Multi-model ensemble means are consistently more skilful than single-model experiments in reproducing observed fields and are a common procedure in climate research (Gleckler et al. 2008).”
BackgroundUsing climate models to estimate the quality of global observational data sets
Massonnet, F., Bellprat, O., Guemas, V., & Doblas-Reyes, F. J. (2016) · Science · doi:10.1126/science.aaf6369 · Cited by 55
Modelling the effect of climate on maple syrup production in Québec, Canada
Duchesne, L., Houle, D., Côté, M., & Logan, T. (2009) · Forest Ecology and Management · doi:10.1016/j.foreco.2009.09.035 · Cited by 57
The upper end of climate model temperature projections is inconsistent with past warming
Stott, P. A., Good, P., Jones, G. S., Gillett, N. P., & Hawkins, E. (2013) · Environmental Research Letters · doi:10.1088/1748-9326/8/1/014024 · Cited by 57
“By averaging over many models the risk of model pattern error is reduced, as has also been demonstrated for modelled simulation of observed climatology [25].”
Climate model simulation of the South Indian Ocean Convergence Zone: mean state and variability
Lazenby, M. J., Todd, M. C., & Wang, Y. (2016) · Climate Research · doi:10.3354/cr01382 · Cited by 57
Future changes in extreme precipitation indices over Korea
Kim, G., Cha, D., Park, C., Lee, G., Jin, C., Lee, D., Suh, M., Ahn, J., Min, S., Hong, S., et al. (2018) · International Journal of Climatology · doi:10.1002/joc.5414 · Cited by 57
“Seasonal variations in extreme precipitation are then analysed using accumulated precipitation per period, and a portrait plot (Gleckler et al., 2008).”
Assessing Mountains as Natural Reservoirs With a Multimetric Framework
Rhoades, A. M., Jones, A. D., & Ullrich, P. A. (2018) · Earth s Future · doi:10.1002/2017ef000789 · Cited by 57
“To date, several studies have used various metrics and methodologies to evaluate climate models at global scales (CCTAG, 2015; Gleckler et al., 2008; Hawkins & Sutton, 2009; Pierce et al., 2009), regional scales (CCTAG, 2015; Pierce et al.”
BackgroundAssessment of climate change impacts on water quality parameters of Lake Burullus, Egypt
Shalby, A., Elshemy, M., & Zeidan, B. A. (2019) · Environmental Science and Pollution Research · doi:10.1007/s11356-019-06105-x · Cited by 57
Unravelling the influence of subjectivity on ranking of CMIP6 based climate models: A case study
Anil, S., Manikanta, V., & Pallakury, A. R. (2021) · International Journal of Climatology · doi:10.1002/joc.7164 · Cited by 59
Douglas, D. C. (2010) · Antarctica A Keystone in a Changing World · doi:10.3133/ofr20101176 · Cited by 57
Metrics and Diagnostics for Precipitation-Related Processes in Climate Model Short-Range Hindcasts
Ma, H., Xie, S., Boyle, J. S., Klein, S. A., & Zhang, Y. (2012) · Journal of Climate · doi:10.1175/jcli-d-12-00235.1 · Cited by 56
“…12:36 AM UTC To better assess and demonstrate performance skill of these models in simulating tropical precipitation (208S– 208N, 08–3608) and to better understand the model biases, several standard metrics described in Gleckler et al. (2008) are implemented to evaluate the CAPT and AMIP runs.”
BackgroundMethodInfluentialChen, J., Brissette, F. P., & Lucas‐Picher, P. (2016) · Climate Dynamics · doi:10.1007/s00382-016-3030-x · Cited by 55
“For example, Gleckler et al. (2008) defined a large suite of metrics to characterize model performance and highlighted the importance of evaluating a broad spectrum of climate processes and phenomena.”
BackgroundProjected changes in mean and extreme climates over Hindu Kush Himalayan region by 21 CMIP5 models
Wu, J., Xu, Y., & Gao, X. (2017) · Advances in Climate Change Research · doi:10.1016/j.accre.2017.03.001 · Cited by 55
Tropical semi-arid regions expanding over temperate latitudes under climate change
Rajaud, A., & de Noblet‐Ducoudré, N. (2017) · Climatic Change · doi:10.1007/s10584-017-2052-7 · Cited by 55
He, C., & Zhou, W. (2020) · Journal of Climate · doi:10.1175/jcli-d-20-0304.1 · Cited by 55
Pradhanang, S. M., Mukundan, R., Schneiderman, E. M., Zion, M. S., Anandhi, A., Pierson, D. C., Frei, A., Easton, Z. M., Fuka, D. R., & Steenhuis, T. S. (2013) · JAWRA Journal of the American Water Resources Association · doi:10.1111/jawr.12086 · Cited by 54
“The skill of a multimodel ensemble is known to consistently outperform individual models for detection and attribution studies (Brekke et al., 2008; Gleckler et al., 2008; Pierce et al., 2009).”
BackgroundHerrera-Estrada, J. E., & Sheffield, J. (2017) · Journal of Climate · doi:10.1175/jcli-d-16-0491.1 · Cited by 54
Bukovsky, M., Carrillo, C. M., Gochis, D., Hammerling, D., McCrary, R., & Mearns, L. O. (2015) · Journal of Climate · doi:10.1175/jcli-d-14-00695.1 · Cited by 54
“…to evaluate GCM and RCM simulations in order to either eliminate ensemble members (of too poor quality) or differentially weight them for the sake of coming up with more robust estimates of future climate on regional scales (e.g., Gleckler et al. 2008; Knutti et al. 2010; Flato et al. 2013).”
BackgroundGood practice for the usage of climate model simulation results - a discussion paper
Kreienkamp, F., Huebener, H., Linke, C., & Spekat, A. (2012) · ENVIRONMENTAL SYSTEMS RESEARCH · doi:10.1186/2193-2697-1-9 · Cited by 53
“It has been empirically indicated that a joint application of several models is able to represent reality better than each individual model, see (Zhang et al. 2007), Reichler and Kim (2008) or Gleckler et al. (2008)g.”
BackgroundInvestigating the impact of climate change on New York City’s primary water supply
Matonse, A. H., Pierson, D. C., Frei, A., Zion, M. S., Anandhi, A., Schneiderman, E. M., & Wright, B. (2012) · Climatic Change · doi:10.1007/s10584-012-0515-4 · Cited by 53
“planning and management in many regions throughout the world (Bates et al. 2008; Hunt and Watkiss 2011; Vicuna and Dracup 2007; Gleckler et al. 2008) including the New York City (Rosenzweig and Solecki 2001).”
Background
Metadata
- Authors
- Gleckler, P. J., Taylor, K. E., & Doutriaux, C.
- Affiliations
- Lawrence Livermore National Laboratory (US)
- Venue
- Journal of Geophysical Research: Atmospheres
- Publisher
- American Geophysical Union (AGU)
- Published
- 2008-03-20
- Volume
- 113
- Issue
- D6
- ISSN
- 0148-0227 (ISSN-L)
- OpenAlex
- W1996478169
- Cited by
- 1,328
- References
- 31
- Citations by year
- 2026: 64 · 2025: 73 · 2024: 85 · 2023: 88 · 2022: 69 · 2021: 116 · 2020: 103 · 2019: 83 · 2018: 65 · 2017: 76 · 2016: 63 · 2015: 77 · 2014: 65 · 2013: 87 · 2012: 73
- FWCI
- 41.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Geophysical Research Atmospheres (publishedVersion)
- Topics
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate Change and Geoengineering
- Keywords
- climate model evaluation, performance indicators, CMIP3, relative error, atmospheric climatology, model ranking, interannual variability, observational uncertainty, spatial scale, 20th century climate simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.