Increasing the Reliability of Reliability Diagrams
Abstract
The reliability diagram is a common diagnostic graph used to summarize and evaluate probabilistic forecasts. Its strengths lie in the ease with which it is produced and the transparency of its definition. While visually appealing, major long-noted shortcomings lie in the difficulty of interpreting the graph visually; for the most part, ambiguities arise from variations in the distributions of forecast probabilities and from various binning procedures. A resampling method for assigning consistency bars to the observed frequencies is introduced that allows for immediate visual evaluation as to just how likely the observed relative frequencies are under the assumption that the predicted probabilities are reliable. Further, an alternative presentation of the same information on probability paper eases quantitative evaluation and comparison. Both presentations can easily be employed for any method of binning.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The decomposition of proper scoring rule into calibration loss and refinement loss is the sum of expected loss between S and C and expected loss between C and Y.
“The decomposition of proper scoring rule into calibration loss and refinement loss is then the sum of expected loss between S and C and expected loss between C and Y (Bröcker & Smith, 2007;Kull & Flach, 2015):”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as Bröcker & Smith, 2007
Making the assumption of fixed bins, score averages and proportion of positives is unrealistic unless the score distribution is uniform.
“However, making the assumption of fixed bins, score averages and proportion of positives is unrealistic; unless the score distribution is uniform (see Bröcker and Smith (2007) for a detailed discussion).”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as Bröcker and Smith (2007)
Positioning markers at the centre of bins can be misleading, as even perfectly calibrated scores may result in a visually non-calibrated reliability diagram with a deviation of up to half the bin width from the diagonal.
“However, positioning the markers at the centre of the bins can be misleading, as even perfectly calibrated scores may result in a visually non-calibrated reliability diagram with a deviation of up to half the bin width from the diagonal (Bröcker & Smith, 2007)”
From Classifier calibration: a survey on how to assess and improve predicted class probabilities (2023) · cites this paper as Bröcker & Smith, 2007
Cites
Reliability of Subjective Probability Forecasts of Precipitation and Temperature
Murphy, A. H., & Winkler, R. L. (1977) · Applied Statistics · doi:10.2307/2346866 · Cited by 386 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A tutorial on calibration measurements and calibration models for clinical prediction models
Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz228 · Cited by 486 · see citing works
Confidence, uncertainty and decision-support relevance in climate predictions.
Stainforth, D. A., Allen, M. R., Tredger, E. R., & Smith, L. A. (2007) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2007.2074 · Cited by 464 · see citing works
Self-citationClassifier calibration: a survey on how to assess and improve predicted class probabilities
Silva Filho, T., Song, H., Perello-Nieto, M., Santos-Rodriguez, R., Kull, M., & Flach, P. (2023) · Machine Learning · doi:10.1007/s10994-023-06336-7 · Cited by 228 · see citing works
Stable reliability diagrams for probabilistic classifiers
Dimitriadis, T., Gneiting, T., & Jordan, A. I. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016191118 · Cited by 84 · see citing works
Predicting optimum crop designs using crop models and seasonal climate forecasts
Rodriguez, D., de Voil, P., Hudson, D., Brown, J. N., Hayman, P., Marrou, H., & Meinke, H. (2018) · Scientific Reports · doi:10.1038/s41598-018-20628-2 · Cited by 82 · see citing works
Progress and opportunities in advancing near-term forecasting of freshwater quality
Lofton, M. E., Howard, D. W., Thomas, R. Q., & Carey, C. C. (2022) · doi:10.1002/essoar.10512426.1 · Cited by 38 · see citing works
- Authors
- Bröcker, J., & Smith, L. A.
- Affiliations
- London School of Economics and Political Science (GB)
- University of Oxford (GB)
- Corresponding author
- Bröcker, Jochen
- Venue
- Weather and Forecasting
- Publisher
- American Meteorological Society
- Published
- 2007-06-01
- Volume
- 22
- Issue
- 3
- Pages
- 651-661
- ISSN
- 0882-8156 (ISSN-L)
- 1520-0434
- OpenAlex
- W2146193817
- Cited by
- 316
- References
- 1
- Citations by year
- 2026: 20 · 2025: 22 · 2024: 31 · 2023: 27 · 2022: 23 · 2021: 32 · 2020: 34 · 2019: 23 · 2018: 16 · 2017: 10 · 2016: 15 · 2015: 10 · 2014: 14 · 2013: 6 · 2012: 8
- FWCI
- 3.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Weather and Forecasting (publishedVersion)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (acceptedVersion)
- Topics
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Precipitation Measurement and Analysis
- Keywords
- reliability diagrams, probabilistic forecasting, forecast verification, resampling, probability paper, visual interpretation, forecast reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
