The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size
Abstract
Ensemble forecasts provide probabilistic predictions for the future state of the atmosphere. Usually the probability of a given event E is determined from the fraction of ensemble members which predict the event. Hence there is a degree of sampling error inherent in the predictions. In this paper a theoretical study is made of the effect of ensemble size on forecast performance as measured by a reliability diagram and Brier (skill) score, and on users by using a simple cost‐loss decision model. The relationship between skill and value, and a generalized skill score, dependent on the distribution of users, are discussed. The Brier skill score is reduced from its potential level for all finite‐sized ensembles. The impact is most significant for small ensembles, especially when the variance of forecast probabilities is also small. The Brier score for a set of deterministic forecasts is a measure of potential predictability, assuming the forecasts are representative selections from a reliable ensemble prediction system (EPS). There is a consistent effect of finite ensemble size on the reliability diagram. Even if the underlying distribution is perfectly reliable, sampling this using only a small number of ensemble members introduces considerable unreliability. There is a consistent over‐forecasting which appears as a clockwise tilt of the reliability diagram. It is important to be aware of the expected effect of ensemble size to avoid misinterpreting results. An ensemble of ten o…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Generalizing the results of Richardson (2001) and Tippett et al. (2007), the RPSS of a perfectly reliable forecast system depends on ensemble size N according to (see appendix B for details):”
From Skill of Multimodel ENSO Probability Forecasts (2008) · cites this paper as Richardson (2001)
In the citing paper
“Richardson (2001) showed that the Brier score (BS) of a reliable forecast system depends on ensemble size N according to ͗BS͑N͒͘ ϭ ͩ 1 ϩ 1 N ͪ͗ BS͑ϱ͒͘, ͑B1͒”
From Skill of Multimodel ENSO Probability Forecasts (2008) · cites this paper as Richardson (2001)
In the citing paper
“Machine learning models have the advantages of significantly higher computational efficiency, facilitating the generation of a large number of ensemble members which are crucial for prediction skill and reliability24.”
From A machine learning model that outperforms conventional global subseasonal forecast models (2024) · cites this paper as [24]
Citations
Cites
Statistical Methods in the Atmospheric Sciences
Ramsey, P. H., & Wilks, D. S. (1996) · Technometrics · doi:10.2307/1271312 · Cited by 7,681
Skill and relative economic value of the ECMWF ensemble prediction system
Richardson, D. S. (2000) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.49712656313 · Cited by 491
Self-citationWilks, D. S. (1995) · International geophysics/International geophysics series · doi:10.1016/s0074-6142(06)80036-7 · Cited by 319
On the Evolution of the Southern Oscillation
Trenberth, K. E., & Shea, D. J. (1987) · Monthly Weather Review · doi:10.1175/1520-0493(1987)115 · Cited by 316
The Finley Affair: A Signal Event in the History of Forecast Verification
Murphy, A. H. (1996) · Weather and Forecasting · doi:10.1175/1520-0434(1996)011 · Cited by 246
Some Aspects of Circulation and Climate over the Eastern Equatorial Atlantic
HASTENRATH, S. L., & Lamb, P. (1977) · Monthly Weather Review · doi:10.1175/1520-0493(1977)105 · Cited by 82
On the Relationship between the Accuracy and Value of Forecasts in the Cost–Loss Ratio Situation
Murphy, A. H., & Ehrendorfer, M. (1987) · Weather and Forecasting · doi:10.1175/1520-0434(1987)002 · Cited by 69
Murphy, A. H. (1966) · Journal of applied meteorology · doi:10.1175/1520-0450(1966)005 · Cited by 52
Research works printed but not linked: 1
- Talagrand O. Vautard R.andStrauss B.1997‘Evaluation of probabilistic prediction systems’. Pp.157–157in Proceedings of the ECMWF workshop on predictability 20–22 October 1997 ECMWF Shinfield Park Reading UK
Cited by
Sorted by citations
Items 201–213 of 213
Korhonen, N. (2024) · Natural hazards and earth system sciences · doi:10.5194/nhess-2024-75-ac2
Leutbecher, M., & Baran, S. (2024) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.4898
The Economic Value of Forecasts in Reducing Extreme Total Losses
Stephenson, D. B. (2025) · Meteorological Applications · doi:10.1002/met.70117
Forecast system design, configuration, and complexity
Takaya, Y., & Vitart, F. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-31538-1.00019-1
Predicting East Africa’s rainfall extremes with calibrated, hybrid physical and AI systemsPreprint
Nath, S., Koros, D., Cooper, F., MacLeod, D., Kimani, H., Mwai, Z., Maswi, C., Chanzu, B., Teshome, A., Tamene, B., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-8534913/v1
Weather intelligence – transforming economies
Rogers, D. P., Thorpe, A. J., Kappes, M., & Bogdanova, A.-M. (2026) · Advances in science and research · doi:10.5194/asr-22-119-2026
(2026) · Elsevier eBooks · doi:10.1016/b978-0-443-49002-6.00012-7
Jewson, S. P. (2026) · Frontiers in Communication · doi:10.3389/fcomm.2026.1762033
Yang, D., Wu, Y. N., Yang, X. Y., Tang, Y., Yan, X., Wang, Q. L., Song, X., & yangdj77, Y. (2026) · Journal of Geophysical Research Atmospheres · doi:10.1029/2026jd046318
Preserving forecast accuracy with reduced ensemble size: A framework for optimal ensemble selection
Teja, K. N., & Umamahesh, N. V. (2026) · Journal of Hydrology · doi:10.1016/j.jhydrol.2026.136113
Predicting East Africa’s rainfall extremes with calibrated AI post-processing systems
Nath, S., Koros, D., Cooper, F., MacLeod, D., Kimani, H., Mwai, Z., Maswi, C., Chanzu, B., Teshome, A., Tamene, B., et al. (2026) · npj natural hazards. · doi:10.1038/s44304-026-00283-8
Rajagopalan (2002) · doi:10.1175/1520-0493(2002)130
Metadata
- Authors
- Richardson, D. S.
- Affiliations
- European Centre for Medium-Range Weather Forecasts (GB)
- Corresponding author
- Richardson, David S.
- ORCID
- Richardson, David S.
- Venue
- Quarterly Journal of the Royal Meteorological Society
- Publisher
- Wiley
- Published
- 2001-10
- Volume
- 127
- Issue
- 577
- Pages
- 2473-2489
- ISSN
- 0035-9009 (ISSN-L)
- 1477-870X
- OpenAlex
- W2089463042
- Cited by
- 212
- References
- 9
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 10 · 2023: 8 · 2022: 11 · 2021: 7 · 2020: 6 · 2019: 13 · 2018: 14 · 2017: 6 · 2016: 8 · 2015: 8 · 2014: 12 · 2013: 7 · 2012: 6
- FWCI
- 3.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Quarterly Journal of the Royal Meteorological Society (publishedVersion)
- Topics
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Environmental Science › Global and Planetary Change › Climate variability and models
- Keywords
- ensemble prediction systems, ensemble size, Brier skill score, reliability diagrams, forecast reliability, sampling error, forecast value, probabilistic forecasting, predictability, extreme events, user sensitivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
