Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size

Richardson, David S.

doi:10.1002/qj.49712757715

Cited by 213Meteorological Phenomena and SimulationsForecasting Techniques and ApplicationsClimate variability and models

Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size

Richardson, David S.·2001·Quarterly Journal of the Royal Meteorological Society

DOI 10.1002/qj.49712757715

Open this paper in CoChat — free account

Abstract

Ensemble forecasts provide probabilistic predictions for the future state of the atmosphere. Usually the probability of a given event E is determined from the fraction of ensemble members which predict the event. Hence there is a degree of sampling error inherent in the predictions. In this paper a theoretical study is made of the effect of ensemble size on forecast performance as measured by a reliability diagram and Brier (skill) score, and on users by using a simple cost‐loss decision model. The relationship between skill and value, and a generalized skill score, dependent on the distribution of users, are discussed. The Brier skill score is reduced from its potential level for all finite‐sized ensembles. The impact is most significant for small ensembles, especially when the variance of forecast probabilities is also small. The Brier score for a set of deterministic forecasts is a measure of potential predictability, assuming the forecasts are representative selections from a reliable ensemble prediction system (EPS). There is a consistent effect of finite ensemble size on the reliability diagram. Even if the underlying distribution is perfectly reliable, sampling this using only a small number of ensemble members introduces considerable unreliability. There is a consistent over‐forecasting which appears as a clockwise tilt of the reliability diagram. It is important to be aware of the expected effect of ensemble size to avoid misinterpreting results. An ensemble of ten o…

Integrity signals

Registry

Crossref · 10.1002/qj.49712757715

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

For a perfectly reliable forecast system, RPSS depends on ensemble size N according to the relationship given in the paper.
“Generalizing the results of Richardson (2001) and Tippett et al. (2007), the RPSS of a perfectly reliable forecast system depends on ensemble size N according to (see appendix B for details):”
From Skill of Multimodel ENSO Probability Forecasts (2008) · cites this paper as Richardson (2001)
Richardson (2001) showed that a reliable forecast system's Brier score depends on ensemble size N according to the given formula.
“Richardson (2001) showed that the Brier score (BS) of a reliable forecast system depends on ensemble size N according to ͗BS͑N͒͘ ϭ ͩ 1 ϩ 1 N ͪ͗ BS͑ϱ͒͘, ͑B1͒”
From Skill of Multimodel ENSO Probability Forecasts (2008) · cites this paper as Richardson (2001)
Machine learning models have significantly higher computational efficiency, facilitating the generation of a large number of ensemble members which are crucial for prediction skill and reliability.
“Machine learning models have the advantages of significantly higher computational efficiency, facilitating the generation of a large number of ensemble members which are crucial for prediction skill and reliability24.”
From A machine learning model that outperforms conventional global subseasonal forecast models (2024) · cites this paper as [24]

Statistical Methods in the Atmospheric Sciences
Ramsey, P. H., & Wilks, D. S. (1996) · Technometrics · doi:10.2307/1271312 · Cited by 7,681 · see citing works
Global and Regional Scale Precipitation Patterns Associated with the El Niño/Southern Oscillation
Ropelewski, C. F., & Halpert, M. S. (1987) · Monthly Weather Review · doi:10.1175/1520-0493(1987)115 · Cited by 3,662 · see citing works
Skill and relative economic value of the ECMWF ensemble prediction system
Richardson, D. S. (2000) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.49712656313 · Cited by 491 · see citing works
Self-citation
Preface
Wilks, D. S. (1995) · International geophysics/International geophysics series · doi:10.1016/s0074-6142(06)80036-7 · Cited by 319 · see citing works
The Finley Affair: A Signal Event in the History of Forecast Verification
Murphy, A. H. (1996) · Weather and Forecasting · doi:10.1175/1520-0434(1996)011 · Cited by 246 · see citing works
The Value of Climatological, Categorical and Probabilistic Forecasts in the Cost-Loss Ratio Situation
Murphy, A. H. (1977) · Monthly Weather Review · doi:10.1175/1520-0493(1977)105 · Cited by 229 · see citing works
On the Relationship between the Accuracy and Value of Forecasts in the Cost–Loss Ratio Situation
Murphy, A. H., & Ehrendorfer, M. (1987) · Weather and Forecasting · doi:10.1175/1520-0434(1987)002 · Cited by 69 · see citing works
A Note on the Utility of Probabilistic Predictions and the Probability Score in the Cost-Loss Ratio Decision Situation
Murphy, A. H. (1966) · Journal of applied meteorology · doi:10.1175/1520-0450(1966)005 · Cited by 51 · see citing works
Untitled work
(1995) · doi:10.1175/1520-0493(1995)123

Flagged first, then by citations

Sorted by citations

Forecast verification : a practitioner's guide in atmospheric science
Jolliffe, I., & Stephenson, D. (2011) · Wiley eBooks · doi:10.1002/9781119960003 · Cited by 1,457 · see citing works
Ensemble flood forecasting: A review
Cloke, H., & Pappenberger, F. (2009) · Journal of Hydrology · doi:10.1016/j.jhydrol.2009.06.005 · Cited by 1,146 · see citing works
Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation
Gneiting, T., Raftery, A. E., Westveld, A. H., & Goldman, T. (2005) · Monthly Weather Review · doi:10.1175/mwr2904.1 · Cited by 1,142 · see citing works
The North American Multimodel Ensemble: Phase-1 Seasonal-to-Interannual Prediction; Phase-2 toward Developing Intraseasonal Prediction
Kirtman, B. P., Min, D., Infanti, J. M., Kinter, J. L., Paolino, D. A., Zhang, Q., van den Dool, H., Saha, S., Mendez, M. P., Becker, E. J., et al. (2013) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-12-00050.1 · Cited by 1,032 · see citing works
Ensemble forecasting
Leutbecher, M., & Palmer, T. N. (2007) · Journal of Computational Physics · doi:10.1016/j.jcp.2007.02.014 · Cited by 920 · see citing works
Forecast Verification
Wilks, D. S. (2011) · International Geophysics · doi:10.1016/b978-0-12-385022-5.00008-7 · Cited by 727 · see citing works
The ROC Curve and the Area under It as Performance Measures
Marzban, C. (2004) · Weather and Forecasting · doi:10.1175/825.1 · Cited by 337 · see citing works
Can multi‐model combination really enhance the prediction skill of probabilistic ensemble forecasts?
Weigel, A. P., Liniger, M. A., & Appenzeller, C. (2008) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.210 · Cited by 329 · see citing works
Economic Value of Weather and Climate Forecasts
Katz, R. W., & Lazo, J. K. (2012) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780195398649.013.0021 · Cited by 304 · see citing works
The economic value of ensemble forecasts as a tool for risk assessment: From days to decades
Palmer, T. N. (2002) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/0035900021643593 · Cited by 277 · see citing works
Measuring forecast skill: is it real skill or is it the varying climatology?
Hamill, T., & Juras, J. (2006) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/qj.06.25 · Cited by 266 · see citing works
The Discrete Brier and Ranked Probability Skill Scores
Weigel, A. P., Liniger, M. A., & Appenzeller, C. (2007) · Monthly Weather Review · doi:10.1175/mwr3280.1 · Cited by 254 · see citing works
RECENT CHANGES IN FROST DAYS AND THE FROST-FREE SEASON IN THE UNITED STATES
Easterling, D. R. (2002) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477-83.9.1327 · Cited by 216 · see citing works
Self-citation
An Overview of the 2010 Hazardous Weather Testbed Experimental Forecast Program Spring Experiment
Clark, A. J., Weiss, S. J., Kain, J. S., Jirak, I. L., Coniglio, M. C., Melick, C. J., Siewert, C., Sobash, R. A., Marsh, P. T., Dean, A. R., et al. (2011) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-11-00040.1 · Cited by 243 · see citing works
A Comparison of Precipitation Forecast Skill between Small Convection-Allowing and Large Convection-Parameterizing Ensembles
Clark, A. J., Gallus, W. A., Xue, M., & Kong, F. (2009) · Weather and Forecasting · doi:10.1175/2009waf2222222.1 · Cited by 236 · see citing works
The rationale behind the success of multi-model ensembles in seasonal forecasting - II. Calibration and combination
Doblas-Reyes, F. J., Hagedorn, R., & Palmer, T. N. (2005) · Tellus A: Dynamic Meteorology and Oceanography · doi:10.3402/tellusa.v57i3.14658 · Cited by 340 · see citing works
On the effect of ensemble size on the discrete and continuous ranked probability scores
Ferro, C. A. T., Richardson, D., & Weigel, A. (2008) · Meteorological Applications · doi:10.1002/met.45 · Cited by 169 · see citing works
Self-citation
The Development of the NCEP Global Ensemble Forecast System Version 12
Zhou, X., Zhu, Y., Hou, D., Fu, B., Li, W., Guan, H., Sinsky, E., Kolczynski, W. C., Xue, X., Luo, Y., et al. (2022) · Weather and Forecasting · doi:10.1175/waf-d-21-0112.1 · Cited by 164 · see citing works
Skill and relative economic value of medium-range hydrological ensemble predictions
Roulin, E. (2006) · Hydrology and earth system sciences · doi:10.5194/hess-11-725-2007 · Cited by 156 · see citing works
Fog Prediction from a Multimodel Mesoscale Ensemble Prediction System
Zhou, B., & Du, J. (2009) · Weather and Forecasting · doi:10.1175/2009waf2222289.1 · Cited by 158 · see citing works
The development of seasonal climate forecasting for agricultural producers
Klemm, T., & McPherson, R. A. (2016) · Agricultural and Forest Meteorology · doi:10.1016/j.agrformet.2016.09.005 · Cited by 154 · see citing works
Probabilistic Precipitation Forecast Skill as a Function of Ensemble Size and Spatial Scale in a Convection-Allowing Ensemble
Clark, A. J., Kain, J. S., Stensrud, D. J., Xue, M., Kong, F., Coniglio, M. C., Thomas, K. W., Wang, Y., Brewster, K. A., Gao, J., et al. (2011) · Monthly Weather Review · doi:10.1175/2010mwr3624.1 · Cited by 140 · see citing works
From Climate-Change Spaghetti to Climate-Change Distributions for 21st Century California
Survey, U. S. G., & Dettinger, M. D. (2005) · San Francisco Estuary and Watershed Science · doi:10.15447/sfews.2005v3iss1art6 · Cited by 139 · see citing works
Ensemble Bayesian model averaging using Markov Chain Monte Carlo sampling
Vrugt, J. A., Diks, C. G. H., & Clark, M. (2008) · Environmental Fluid Mechanics · doi:10.1007/s10652-008-9106-3 · Cited by 137 · see citing works
Addressing model uncertainty in seasonal and annual dynamical ensemble forecasts
Doblas-Reyes, F. J., Weisheimer, A., Déqué, M., Keenlyside, N. S., McVean, M., Murphy, J. M., Rogel, P., Smith, D. M., & Palmer, T. N. (2009) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.464 · Cited by 134 · see citing works
Ensemble size: How suboptimal is less than infinity?
Leutbecher, M. (2019) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3387 · Cited by 126 · see citing works
Verification of an Ensemble Prediction System against Observations
Candille, G., Cote, C., Houtekamer, P., & Pellerin, G. (2007) · Monthly Weather Review · doi:10.1175/mwr3414.1 · Cited by 117 · see citing works
Comparison of ensemble‐MOS methods in the Lorenz '96 setting
Wilks, D. S. (2006) · Meteorological Applications · doi:10.1017/s1350482706002192 · Cited by 142 · see citing works
Improvement of ensemble reliability with a new dressing kernel
Wang, X., & Bishop, C. (2005) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/qj.04.120 · Cited by 115 · see citing works
A machine learning model that outperforms conventional global subseasonal forecast models
Chen, L., Zhong, X., Li, H., Wu, J., Lu, B., Chen, D., Xie, S.-P., Wu, L., Chao, Q., Lin, C., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50714-1 · Cited by 105 · see citing works
Comparing Probabilistic Forecasting Systems with the Brier Score
Ferro, C. (2007) · Weather and Forecasting · doi:10.1175/waf1034.1 · Cited by 105 · see citing works
Extended range prediction of active-break spells of Indian summer monsoon rainfall using an ensemble prediction system in NCEP Climate Forecast System
Abhilash, S. R., Sahai, A. K., Pattnaik, S., Goswami, B. N., & Kumar, A. (2013) · International Journal of Climatology · doi:10.1002/joc.3668 · Cited by 100 · see citing works
Subseasonal Tropical Cyclone Genesis Prediction and MJO in the S2S Dataset
Lee, C., Camargo, S. J., Vitart, F., Sobel, A. H., & Tippett, M. K. (2018) · Weather and Forecasting · doi:10.1175/waf-d-17-0165.1 · Cited by 98 · see citing works
Trends in the predictive performance of raw ensemble weather forecasts
Hemri, S., Scheuerer, M., Pappenberger, F., Bogner, K., & Haiden, T. (2014) · Geophysical Research Letters · doi:10.1002/2014gl062472 · Cited by 98 · see citing works
A Laboratory Study of the Benefits of Including Uncertainty Information in Weather Forecasts
Roulston, M. S., Bolton, G. E., Kleit, A. N., & Sears-Collins, A. L. (2006) · Weather and Forecasting · doi:10.1175/waf887.1 · Cited by 98 · see citing works
Promoting the use of probabilistic weather forecasts through a dialogue between scientists, developers and end‐users
Fundel, V. J., Fleischhut, N., Herzog, S. M., Göber, M., & Hagedorn, R. (2019) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3482 · Cited by 95 · see citing works
Forecast of Low Visibility and Fog from NCEP: Current Status and Efforts
Zhou, B., Du, J., Gültepe, I., & DiMego, G. (2011) · Pure and Applied Geophysics · doi:10.1007/s00024-011-0327-x · Cited by 92 · see citing works
Perturbation of convection-permitting NWP forecasts for flash-flood ensemble forecasting
Vincendon, B., Ducrocq, V., Nuissier, O., & Vié, B. (2011) · Natural hazards and earth system sciences · doi:10.5194/nhess-11-1529-2011 · Cited by 74 · see citing works
Spatial and Interannual Variability of the Reliability of Ensemble-Based Probabilistic Forecasts: Consequences for Calibration
Atger, F. (2003) · Monthly Weather Review · doi:10.1175//1520-0493(2003)131 · Cited by 85 · see citing works
The skill of multi-model seasonal forecasts of the wintertime North Atlantic Oscillation
Doblas-Reyes, F. J., Pavan, V., & Stephenson, D. B. (2003) · Climate Dynamics · doi:10.1007/s00382-003-0350-4 · Cited by 86 · see citing works
How confident are predictability estimates of the winter North Atlantic Oscillation?
Weisheimer, A., Decremer, D., MacLeod, D. A., O’Reilly, C., Stockdale, T., Johnson, S. J., & Palmer, T. N. (2018) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3446 · Cited by 81 · see citing works
Smoothing forecast ensembles with fitted probability distributions
Wilks, D. S. (2002) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/qj.01.215 · Cited by 81 · see citing works
Benefits of increased resolution in the ECMWF ensemble system and comparison with poor‐man's ensembles
Buizza, R., Richardson, D. S., & Palmer, T. N. (2003) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/qj.02.92 · Cited by 79 · see citing works
Self-citation
Forecast Verification
Wilks, D. S. (2019) · Statistical Methods in the Atmospheric Sciences · doi:10.1016/b978-0-12-815823-4.00009-2 · Cited by 130 · see citing works
The Impact of Horizontal Resolution and Ensemble Size on Probabilistic Forecasts of Precipitation by the ECMWF Ensemble Prediction System
Mullen, S., & Buizza, R. (2002) · Weather and Forecasting · doi:10.1175/1520-0434(2002)017 · Cited by 87 · see citing works
Analog Probabilistic Precipitation Forecasts Using GEFS Reforecasts and Climatology-Calibrated Precipitation Analyses*
Hamill, T. M., Scheuerer, M., & Bates, G. T. (2015) · Monthly Weather Review · doi:10.1175/mwr-d-15-0004.1 · Cited by 77 · see citing works
The impact of horizontal resolution and ensemble size for convective‐scale probabilistic forecasts
Raynaud, L., & Bouttier, F. (2017) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3159 · Cited by 77 · see citing works
The Multiensemble Approach: The NAEFS Example
Candille, G. (2008) · Monthly Weather Review · doi:10.1175/2008mwr2682.1 · Cited by 71 · see citing works
Estimation of Seasonal Precipitation Tercile-Based Categorical Probabilities from Ensembles
Tippett, M. K., Barnston, A. G., & Robertson, A. W. (2007) · Journal of Climate · doi:10.1175/jcli4108.1 · Cited by 69 · see citing works
Seasonal ensemble forecasts: Are recalibrated single models better than multimodels?
Weigel, A. P., Liniger, M. A., & Appenzeller, C. (2009) · Monthly Weather Review · doi:10.1175/2008mwr2773.1 · Cited by 66 · see citing works
Skill of Multimodel ENSO Probability Forecasts
Tippett, M. K., & Barnston, A. G. (2008) · Monthly Weather Review · doi:10.1175/2008mwr2431.1 · Cited by 64 · see citing works
A forecast quality assessment of an end-to-end probabilistic multi-model seasonal forecast system using a malaria model
Morse, A. P., Doblas-Reyes, F. J., Hoshen, M. B., Hagedorn, R., & Palmer, T. N. (2005) · Tellus A Dynamic Meteorology and Oceanography · doi:10.3402/tellusa.v57i3.14668 · Cited by 64 · see citing works
Reliability in ensemble data assimilation
Rodwell, M. J., Lang, S. T. K., Ingleby, N. B., Bormann, N., Hólm, E. V., Rabier, F., Richardson, D. S., & Yamaguchi, M. (2015) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.2663 · Cited by 63 · see citing works
Self-citation
Probabilistic Seasonal Forecasts in the North American Multimodel Ensemble: A Baseline Skill Assessment
Becker, E., & van den Dool, H. M. (2016) · Journal of Climate · doi:10.1175/jcli-d-14-00862.1 · Cited by 56 · see citing works
Verification of inflow into hydropower reservoirs using ensemble forecasts of the TIGGE database for large scale basins in Brazil
Fan, F. M., Schwanenberg, D., Collischonn, W., & Weerts, A. (2015) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2015.05.012 · Cited by 61 · see citing works
Seasonal streamflow forecasts for Europe – Part 2: Sources of skill
Greuell, W., Franssen, W., & Hutjes, R. (2019) · Hydrology and earth system sciences · doi:10.5194/hess-23-371-2019 · Cited by 60 · see citing works
Convection-Allowing and Convection-Parameterizing Ensemble Forecasts of a Mesoscale Convective Vortex and Associated Severe Weather Environment
Clark, A. J., Gallus, W. A., Xue, M., & Kong, F. (2010) · Weather and Forecasting · doi:10.1175/2010waf2222390.1 · Cited by 59 · see citing works
Probabilistic Verification of Monthly Temperature Forecasts
Weigel, A. P., Baggenstos, D., Liniger, M. A., Vitart, F., & Appenzeller, C. (2008) · Monthly Weather Review · doi:10.1175/2008mwr2551.1 · Cited by 59 · see citing works
Bayesian model averaging using particle filtering and Gaussian mixture modeling: Theory, concepts, and simulation experiments
Rings, J., Vrugt, J. A., Schoups, G., Huisman, J. A., & Vereecken, H. (2012) · Water Resources Research · doi:10.1029/2011wr011607 · Cited by 59 · see citing works
The NCEP–FNMOC Combined Wave Ensemble Product: Expanding Benefits of Interagency Probabilistic Forecasts to the Oceanic Environment
Alves, J.-H., Wittmann, P. A., Sestak, M., Schauer, J., Stripling, S., Bernier, N. B., McLean, J., Chao, Y. Y., Chawla, A., Tolman, H. L., et al. (2013) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-12-00032.1 · Cited by 58 · see citing works
The Effect of Probabilistic Information on Threshold Forecasts
Joslyn, S. L., Pak, K., Jones, D. A., Pyles, J. A., & Hunt, E. B. (2007) · Weather and Forecasting · doi:10.1175/waf1020.1 · Cited by 54 · see citing works
GROUND WATER/SURFACE WATER RESPONSES TO GLOBAL CLIMATE SIMULATIONS, SANTA CLARA-CALLEGUAS BASIN, VENTURA, CALIFORNIA
Hanson, R. T., & Dettinger, M. D. (2005) · JAWRA Journal of the American Water Resources Association · doi:10.1111/j.1752-1688.2005.tb03752.x · Cited by 52 · see citing works
Performance targets and the Brier score
Roulston, M. (2007) · Meteorological Applications · doi:10.1002/met.21 · Cited by 56 · see citing works
Contributions of Mixed Physics versus Perturbed Initial/Lateral Boundary Conditions to Ensemble-Based Precipitation Forecast Skill
Clark, A. J., Gallus, W. A., & Chen, T.-C. (2008) · Monthly Weather Review · doi:10.1175/2007mwr2029.1 · Cited by 48 · see citing works
How does the quantification of uncertainties affect the quality and value of flood early warning systems?
Thiboult, A., Anctil, F., & Ramos, M. (2017) · Journal of Hydrology · doi:10.1016/j.jhydrol.2017.05.014 · Cited by 47 · see citing works
Using Varied Microphysics to Account for Uncertainty in Warm-Season QPF in a Convection-Allowing Ensemble
Duda, J. D., Wang, X., Kong, F., & Xue, M. (2014) · Monthly Weather Review · doi:10.1175/mwr-d-13-00297.1 · Cited by 46 · see citing works
High-resolution operational monsoon forecasts: an objective assessment
Sahai, A. K., Abhilash, S. R., Chattopadhyay, R., Borah, N., Joseph, S., Sharmila, S., & Rajeevan, M. N. (2014) · Climate Dynamics · doi:10.1007/s00382-014-2210-9 · Cited by 46 · see citing works
A laboratory‐based study of understanding of uncertainty in 5‐day site‐specific temperature forecasts
Roulston, M. S., & Kaplan, T. R. (2008) · Meteorological Applications · doi:10.1002/met.113 · Cited by 46 · see citing works
A Hybrid NWP–Analog Ensemble
Eckel, F. A., & Monache, L. D. (2015) · Monthly Weather Review · doi:10.1175/mwr-d-15-0096.1 · Cited by 44 · see citing works
An effective configuration of ensemble size and horizontal resolution for the NCEP GEFS
Ma, J., Zhu, Y., Wobus, R. L., & Wang, P. (2012) · Advances in Atmospheric Sciences · doi:10.1007/s00376-012-1249-y · Cited by 42 · see citing works
The Effect of Decision Analysis on Power System Resilience and Economic Value During a Severe Weather Event
Wu, Y., Chen, Y., Chang, H.-L., & Hong, J. (2022) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2022.3145753 · Cited by 41 · see citing works
An alternate approach to ensemble ENSO forecast spread: Application to the 2014 forecast
Larson, S. M., & Kirtman, B. P. (2015) · Geophysical Research Letters · doi:10.1002/2015gl066173 · Cited by 40 · see citing works
Real-time assimilation of streamflow observations into a hydrological routing model: effects of model structures and updating methods
Mazzoleni, M., Noh, S. J., Lee, H., Liu, Y., Seo, D., Amaranto, A., Alfonso, L., & SOLOMATINE, D. P. (2018) · Hydrological Sciences Journal · doi:10.1080/02626667.2018.1430898 · Cited by 40 · see citing works
Quality and Value of Seasonal Precipitation Forecasts Issued by the West African Regional Climate Outlook Forum
Bliefernicht, J., Waongo, M., Salack, S., Seidel, J., Laux, P., & Kunstmann, H. G. (2018) · Journal of Applied Meteorology and Climatology · doi:10.1175/jamc-d-18-0066.1 · Cited by 39 · see citing works
Ensemble Methods for Meteorological Predictions
Du, J., Berner, J., Buizza, R., Charron, M., Houtekamer, P. L., Hou, D., Jankov, I., Mu, M., Wang, X., Wei, M., et al. (2018) · Springer eBooks · doi:10.1007/978-3-642-40457-3_13-1 · Cited by 39 · see citing works
On the Reliability of the Rank Histogram
Wilks, D. S. (2010) · Monthly Weather Review · doi:10.1175/2010mwr3446.1 · Cited by 39 · see citing works
What determines perceived value of seasonal climate forecasts? A theoretical analysis
Millner, A., & Washington, R. (2010) · Global Environmental Change · doi:10.1016/j.gloenvcha.2010.08.001 · Cited by 38 · see citing works
Evaluation of Short-Range Quantitative Precipitation Forecasts from a Time-Lagged Multimodel Ensemble
Yuan, H., Lu, C., McGinley, J. A., Schultz, P. J., Jamison, B. D., Wharton, L. S., & Anderson, C. J. (2008) · Weather and Forecasting · doi:10.1175/2008waf2007053.1 · Cited by 36 · see citing works
Short-Range Ensemble Forecasts of Precipitation Type
Wandishin, M. S., Baldwin, M. E., Mullen, S. L., & Cortinas, J. (2005) · Weather and Forecasting · doi:10.1175/waf871.1 · Cited by 36 · see citing works
Generalization of the discrete Brier and ranked probability skill scores for weighted multimodel ensemble forecasts.
Weigel, A. P., Liniger, M. A., & Appenzeller, C. (2007) · Monthly Weather Review · doi:10.1175/mwr3428.1 · Cited by 35 · see citing works
Model complexity versus ensemble size: allocating resources for climate prediction
Ferro, C. A. T., Jupp, T. E., Lambert, F. H., Huntingford, C., & Cox, P. M. (2012) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2011.0307 · Cited by 34 · see citing works
ENSO-conditioned weather resampling method for seasonal ensemble streamflow prediction
Beckers, J. V. L., Weerts, A., Tijdeman, E., & Welles, E. (2016) · Hydrology and earth system sciences · doi:10.5194/hess-20-3277-2016 · Cited by 33 · see citing works
Improving Week-2 Forecasts with Multimodel Reforecast Ensembles
Whitaker, J. S., Wei, X., & Vitart, F. (2006) · Monthly Weather Review · doi:10.1175/mwr3175.1 · Cited by 33 · see citing works
Lagged Ensembles, Forecast Configuration, and Seasonal Predictions
Chen, M., Wang, W., & Kumar, A. (2013) · Monthly Weather Review · doi:10.1175/mwr-d-12-00184.1 · Cited by 32 · see citing works
Bayesian Approach to Decision Making Using Ensemble Weather Forecasts
Katz, R. W., & Ehrendorfer, M. (2006) · Weather and Forecasting · doi:10.1175/waf913.1 · Cited by 32 · see citing works
Hydrometeorological Short-Range Ensemble Forecasts in Complex Terrain. Part II: Economic Evaluation
McCollor, D., & Stull, R. B. (2008) · Weather and Forecasting · doi:10.1175/2007waf2007064.1 · Cited by 32 · see citing works
The fractions skill score for ensemble forecast verification
Necker, T., Wolfgruber, L., Kugler, L., Weißmann, M., Dorninger, M., & Serafin, S. (2024) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.4824 · Cited by 30 · see citing works
Enforcing calibration in ensemble postprocessing
Wilks, D. S. (2017) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3185 · Cited by 30 · see citing works
Statistical Forecasting
Wilks, D. S. (2011) · International geophysics/International geophysics series · doi:10.1016/b978-0-12-385022-5.00007-5 · Cited by 29 · see citing works
Evaluating decadal predictions of northern hemispheric cyclone frequencies
Kruschke, T., Rust, H. W., Kadow, C., Leckebusch, G. C., & Ulbrich, U. (2014) · Tellus A Dynamic Meteorology and Oceanography · doi:10.3402/tellusa.v66.22830 · Cited by 28 · see citing works
Predictive verification for the design of partially exchangeable multi-model ensembles
Bouallègue, Z. B., Ferro, C. A. T., Leutbecher, M., & Richardson, D. S. (2019) · Tellus A Dynamic Meteorology and Oceanography · doi:10.1080/16000870.2019.1697165 · Cited by 26 · see citing works
Self-citation
Assessing the Impact of L-Band Observations on Drought and Flood Risk Estimation: A Decision-Theoretic Approach in an OSSE Environment
Kumar, S. V., Harrison, K. W., Peters‐Lidard, C. D., Santanello, J. A., & Kirschbaum, D. B. (2014) · Journal of Hydrometeorology · doi:10.1175/jhm-d-13-0204.1 · Cited by 26 · see citing works
An assessment of real‐time extended range forecast of 2013 Indian summer monsoon
Borah, N., Sahai, A. K., Abhilash, S. R., Chattopadhyay, R., Joseph, S., Sharmila, S., & Kumar, A. (2014) · International Journal of Climatology · doi:10.1002/joc.4178 · Cited by 26 · see citing works
Ensemble Methods for Meteorological Predictions
Du, J., Berner, J., Buizza, R., Charron, M., Houtekamer, P. L., Hou, D., Jankov, I., Mu, M., Wang, X., Wei, M., et al. (2019) · Springer eBooks · doi:10.1007/978-3-642-39925-1_13 · Cited by 25 · see citing works
Huge ensembles – Part 2: Properties of a huge ensemble of hindcasts generated with spherical Fourier neural operators
Mahesh, A., Collins, W. D., Bonev, B., Brenowitz, N., Cohen, Y., Harrington, P., Kashinath, K., Kurth, T., North, J. S., O'Brien, T. A., et al. (2025) · Geoscientific model development · doi:10.5194/gmd-18-5605-2025 · Cited by 24 · see citing works
Predictability and economic value
Richardson, D. S. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511617652.026 · Cited by 24 · see citing works
Self-citation
Practical Aspects of Statistical Postprocessing
Hamill, T. M. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-812372-0.00007-8 · Cited by 24 · see citing works
The ensemble‐adjusted Ignorance Score for forecasts issued as normal distributions
Siegert, S., Ferro, C. A. T., Stephenson, D. B., & Leutbecher, M. (2018) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3447 · Cited by 24 · see citing works
Calibrated Probabilistic Forecasts of Arctic Sea Ice Concentration
Dirkson, A., Merryfield, W. J., & Monahan, A. H. (2018) · Journal of Climate · doi:10.1175/jcli-d-18-0224.1 · Cited by 24 · see citing works
The Development of Seasonal and Inter-Annual Climate Forecasting
Harrison, M. (2005) · Climatic Change · doi:10.1007/s10584-005-5947-7 · Cited by 24 · see citing works

Atlantic Hurricane Season of 1997
Rappaport, E. N. (1999) · Monthly Weather Review · doi:10.1175/1520-0493(1999)127 · Cited by 271 · see citing works
Comparing Probabilistic Forecasting Systems with the Brier Score
Ferro, C. (2007) · Weather and Forecasting · doi:10.1175/waf1034.1 · Cited by 105 · see citing works
Relative impact of model quality and ensemble deficiencies on the performance of ensemble based probabilistic forecasts evaluated through the Brier score
Atger, F. (2004) · Nonlinear processes in geophysics · doi:10.5194/npg-11-399-2004 · Cited by 11 · see citing works
Effect of ensemble size on reliability and Brier score
Richardson, D. S. (2000) · European Centre for Medium-Range Weather Forecasts · doi:10.21957/pqpqimjz8
Self-citation
Ensemble size: How suboptimal is less than infinity?
Leutbecher, M. (2019) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.3387 · Cited by 126 · see citing works

Authors: Richardson, D. S.
Affiliations: European Centre for Medium-Range Weather Forecasts (GB)
Corresponding author: Richardson, David S.
ORCID: Richardson, David S.
Venue: Quarterly Journal of the Royal Meteorological Society
Publisher: Wiley
Published: 2001-10
Volume: 127
Issue: 577
Pages: 2473-2489
ISSN: 0035-9009 (ISSN-L)
1477-870X
DOI: 10.1002/qj.49712757715
OpenAlex: W2089463042
Cited by: 212
References: 9
Citations by year: 2026: 8 · 2025: 9 · 2024: 10 · 2023: 8 · 2022: 11 · 2021: 7 · 2020: 6 · 2019: 13 · 2018: 14 · 2017: 6 · 2016: 8 · 2015: 8 · 2014: 12 · 2013: 7 · 2012: 6
FWCI: 3.05
Type: journal article
Language: en
Open access: closed
Licence: onlinelibrary.wiley.com/termsAndConditions#vor (vor, 1905 d)
Locations: Quarterly Journal of the Royal Meteorological Society (publishedVersion)
Topics: Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
Keywords: ensemble prediction systems, ensemble size, Brier skill score, reliability diagrams, forecast reliability, sampling error, forecast value, probabilistic forecasting, predictability, extreme events, user sensitivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks