The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Measuring forecast skill: is it real skill or is it the varying climatology?
Abstract
It is common practice to summarize the skill of weather forecasts from an accumulation of samples spanning many locations and dates. In calculating many of these scores, there is an implicit assumption that the climatological frequency of event occurrence is approximately invariant over all samples. If the event frequency actually varies among the samples, the metrics may report a skill that is different from that expected. Many common deterministic verification metrics, such as threat scores, are prone to mis‐reporting skill, and probabilistic forecast metrics such as the Brier skill score and relative operating characteristic skill score can also be affected. Three examples are provided that demonstrate unexpected skill, two from synthetic data and one with actual forecast data. In the first example, positive skill was reported in a situation where metrics were calculated from a composite of forecasts that were comprised of random draws from the climatology of two distinct locations. As the difference in climatological event frequency between the two locations was increased, the reported skill also increased. A second example demonstrates that when the climatological event frequency varies among samples, the metrics may excessively weight samples with the greatest observational uncertainty. A final example demonstrates unexpectedly large skill in the equitable threat score of deterministic precipitation forecasts. Guidelines are suggested for how to adjust skill computations to minimize these effects. Copyright © 2006 Royal Meteorological Society
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Statistical Methods in the Atmospheric Sciences
Ramsey, P. H. (1996) · Technometrics · doi:10.1080/00401706.1996.10484553 · Cited by 7,601
Agresti, A. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/0471249688 · Cited by 6,746
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Cochran (1954) also is unambiguous with regards to inferences from contingency tables: “One method that is sometimes used is to combine all the data into a single 2x2 table … this procedure is legitimate only if the probability p of an occurrence (on the null hypothesis) can be assumed to be the…”
MethodNorth American Regional Reanalysis
Mesinger, F., DiMego, G., Kalnay, E., Mitchell, K., Shafran, P. C., Ebisuzaki, W., Jović, D., Woollen, J., Rogers, E., Berbery, E. H., et al. (2006) · Bulletin of the American Meteorological Society · doi:10.1175/bams-87-3-343 · Cited by 3,378
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301
Forecast Verification: A Practitioner's Guide in Atmospheric Science
(2011) · Wiley eBooks · doi:10.1002/9781119960003 · Cited by 1,267
Mason, S. J., & Graham, N. E. (2002) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/003590002320603584 · Cited by 868
The Critical Success Index as an Indicator of Warning Skill
Schaefer, J. T. (1990) · Weather and Forecasting · doi:10.1175/1520-0434(1990)005<0570:tcsiaa>2.0.co;2 · Cited by 867
A Scoring System for Probability Forecasts of Ranked Categories
Epstein, E. S. (1969) · Journal of Applied Meteorology · doi:10.1175/1520-0450(1969)008<0985:assfpf>2.0.co;2 · Cited by 563
Skill and relative economic value of the ECMWF ensemble prediction system
Richardson, D. S. (2000) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.49712656313 · Cited by 477
Stensrud, D. J., Bao, J.-W., & Warner, T. T. (2000) · Monthly Weather Review · doi:10.1175/1520-0493(2000)128<2077:uicamp>2.0.co;2 · Cited by 428
Hypothesis Tests for Evaluating Numerical Precipitation Forecasts
Hamill, T. M. (1999) · Weather and Forecasting · doi:10.1175/1520-0434(1999)014<0155:htfenp>2.0.co;2 · Cited by 403
Self-citationAbility of a Poor Man's Ensemble to Predict the Probability and Distribution of Precipitation
Ebert, E. E. (2001) · Monthly Weather Review · doi:10.1175/1520-0493(2001)129<2461:aoapms>2.0.co;2 · Cited by 386
The ROC Curve and the Area under It as Performance Measures
Marzban, C. (2004) · Weather and Forecasting · doi:10.1175/825.1 · Cited by 335
Conditional Probabilities, Relative Operating Characteristics, and Relative Operating Levels
Mason, S. J., & Graham, N. E. (1999) · Weather and Forecasting · doi:10.1175/1520-0434(1999)014<0713:cproca>2.0.co;2 · Cited by 331
Hamill, T. M., & Whitaker, J. S. (2006) · Monthly Weather Review · doi:10.1175/mwr3237.1 · Cited by 313
Self-citationAccadia, C., Mariani, S., Casaioli, M., Lavagnini, A., & Speranza, A. (2003) · Weather and Forecasting · doi:10.1175/1520-0434(2003)018<0918:sopfss>2.0.co;2 · Cited by 259
The Economic Value Of Ensemble-Based Weather Forecasts
Zhu, Y., Toth, Z., Wobus, R., Richardson, D., & Mylne, K. (2002) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(2002)083<0073:tevoeb>2.3.co;2 · Cited by 247
Reforecasts: An Important Dataset for Improving Weather Predictions
Hamill, T. M., Whitaker, J. S., & Mullen, S. L. (2006) · Bulletin of the American Meteorological Society · doi:10.1175/bams-87-1-33 · Cited by 246
Self-citationRogers, E., Black, T. L., Deaven, D. G., DiMego, G. J., Zhao, Q., Baldwin, M. E., Junker, N. W., & Lin, Y. (1996) · Weather and Forecasting · doi:10.1175/1520-0434(1996)011<0391:cttoea>2.0.co;2 · Cited by 214
Equitable Skill Scores for Categorical Forecasts
Gandin, L. S., & Murphy, A. H. (1992) · Monthly Weather Review · Cited by 213
Richardson, D. S. (2001) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.49712757715 · Cited by 212
Impact of Ensemble Size on Ensemble Prediction
Buizza, R., & Palmer, T. N. (1998) · Monthly Weather Review · doi:10.1175/1520-0493(1998)126<2503:ioesoe>2.0.co;2 · Cited by 203
Probabilistic Predictions of Precipitation Using the ECMWF Ensemble Prediction System
Buizza, R., Hollingsworth, A., Lalaurette, F., & Ghelli, A. (1999) · Weather and Forecasting · doi:10.1175/1520-0434(1999)014<0168:ppoput>2.0.co;2 · Cited by 193
A probability and decision‐model analysis of PROVOST seasonal multi‐model ensemble integrations
Palmer, T. N., Branković, Č., & Richardson, D. S. (2000) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.49712656703 · Cited by 189
A Note on the Ranked Probability Score
Murphy, A. H. (1971) · Journal of applied meteorology · doi:10.1175/1520-0450(1971)010<0155:anotrp>2.0.co;2 · Cited by 186
The Application of Signal Detection Theory to Weather Forecasting Behavior
Harvey, L. O., Hammond, K. R., Lusk, C. M., & Mross, E. F. (1992) · Monthly Weather Review · doi:10.1175/1520-0493(1992)120<0863:taosdt>2.0.co;2 · Cited by 160
A skill score based on economic value for probability forecasts
Wilks, D. (2001) · Meteorological Applications · doi:10.1017/s1350482701002092 · Cited by 155
On Using “Climatology” as a Reference Strategy in the Brier and Ranked Probability Skill Scores
Mason, S. J. (2004) · Monthly Weather Review · doi:10.1175/1520-0493(2004)132<1891:oucaar>2.0.co;2 · Cited by 146
Hamill, T. M., Snyder, C., & Morss, R. E. (2000) · Monthly Weather Review · doi:10.1175/1520-0493(2000)128<1835:acopff>2.0.co;2 · Cited by 143
Self-citationImpact of model resolution and ensemble size on the performance of an Ensemble Prediction System
Buizza, R., Petroliagis, T. I., Palmer, T. N., Barkmeijer, J., Hamrud, M., Hollingsworth, A., Simmons, A. J., & Wedi, N. (1998) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.49712455008 · Cited by 132
Rogers, E., Deaven, D. G., & DiMego, G. (1995) · Weather and Forecasting · Cited by 92
Current status and future developments of the ECMWF Ensemble Prediction System
Buizza, R., Barkmeijer, J., Palmer, T. N., & Richardson, D. (2000) · Meteorological Applications · doi:10.1017/s1350482700001456 · Cited by 90
Atger, F. (2003) · Monthly Weather Review · doi:10.1175//1520-0493(2003)131<1509:saivot>2.0.co;2 · Cited by 85
Buizza, R., Richardson, D. S., & Palmer, T. N. (2003) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/qj.02.92 · Cited by 79
Mullen, S. L., & Buizza, R. (2002) · Weather and Forecasting · Cited by 78
Stefanova, L., & Krishnamurti, T. N. (2002) · Journal of Climate · doi:10.1175/1520-0442(2002)015<0537:ioscfu>2.0.co;2 · Cited by 65
Accuracy and Potential Economic Value of Categorical and Probabilistic Forecasts of Discrete Events
Buizza, R. (2001) · Monthly Weather Review · Cited by 40
Comments on “Probabilistic Predictions of Precipitation Using the ECMWF Ensemble Prediction System”
Wilson, L. J. (2000) · Weather and Forecasting · Cited by 37
Fairplay in the verification of operational quantitative precipitation forecasts
Göber, M., Wilson, C. A., Milton, S. F., & Stephenson, D. B. (2004) · Journal of Hydrology · doi:10.1016/j.jhydrol.2003.11.016 · Cited by 35
Glahn, B. (2004) · Weather and Forecasting · Cited by 18
Comments on “Probabilistic Predictions of Precipitation Using the ECMWF Ensemble Prediction System”
Juras, J. (2000) · Weather and Forecasting · Cited by 15
Self-citationVerification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 2×
“The long-used Brier score (Brier 1950) is a measure of the mean-square error of probability forecasts for a dichotomous (two-category) event, such as the occurrence/nonoccurrence of precipitation.”
MethodVERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
(2003) · doi:10.1175/1520-0442(2003)016
Cited by
Wilks, D. S. (2011) · International Geophysics · doi:10.1016/b978-0-12-385022-5.00008-7 · Cited by 727
Hamill, T. M., & Whitaker, J. S. (2006) · Monthly Weather Review · doi:10.1175/mwr3237.1 · Cited by 313
Self-citationForecast verification: Current status and future directions
Casati, B., Wilson, L. J., Stephenson, D. B., Nurmi, P., Ghelli, A., Pocernich, M., Damrath, U., Ebert, E. E., Brown, B. G., & Mason, S. (2008) · Meteorological Applications · doi:10.1002/met.52 · Cited by 306
van Straaten, C., Whan, K., Coumou, D., van den Hurk, B., & Schmeits, M. (2022) · Monthly Weather Review · doi:10.1175/mwr-d-21-0201.1 · Cited by 50
Metadata
- Authors
- Hamill, T. M., & Juras, J.
- Venue
- Quarterly Journal of the Royal Meteorological Society
- Publisher
- Wiley
- Published
- 2006-10
- Volume
- 132
- Issue
- 621C
- Pages
- 2905-2923
- ISSN
- 0035-9009 (ISSN-L)
- 1477-870X
- OpenAlex
- W2170766202
- Cited by
- 264
- References
- 47
- FWCI
- 9.15
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.