The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Measuring forecast skill: is it real skill or is it the varying climatology?
Abstract
It is common practice to summarize the skill of weather forecasts from an accumulation of samples spanning many locations and dates. In calculating many of these scores, there is an implicit assumption that the climatological frequency of event occurrence is approximately invariant over all samples. If the event frequency actually varies among the samples, the metrics may report a skill that is different from that expected. Many common deterministic verification metrics, such as threat scores, are prone to mis‐reporting skill, and probabilistic forecast metrics such as the Brier skill score and relative operating characteristic skill score can also be affected. Three examples are provided that demonstrate unexpected skill, two from synthetic data and one with actual forecast data. In the first example, positive skill was reported in a situation where metrics were calculated from a composite of forecasts that were comprised of random draws from the climatology of two distinct locations. As the difference in climatological event frequency between the two locations was increased, the reported skill also increased. A second example demonstrates that when the climatological event frequency varies among samples, the metrics may excessively weight samples with the greatest observational uncertainty. A final example demonstrates unexpectedly large skill in the equitable threat score of deterministic precipitation forecasts. Guidelines are suggested for how to adjust skill computations to minimize these effects. Copyright © 2006 Royal Meteorological Society
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease
(1959) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/22.4.719 · Cited by 14,689
Statistical Methods in the Atmospheric Sciences
Ramsey, P. H. (1996) · Technometrics · doi:10.1080/00401706.1996.10484553 · Cited by 7,601
Agresti, A. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/0471249688 · Cited by 6,746
Some Methods for Strengthening the Common χ 2 Tests
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001616 · Cited by 3,718 · cited 1×
“Cochran (1954) also is unambiguous with regards to inferences from contingency tables: “One method that is sometimes used is to combine all the data into a single 2x2 table … this procedure is legitimate only if the probability p of an occurrence (on the null hypothesis) can be assumed to be the…”
MethodNorth American Regional Reanalysis
Mesinger, F., DiMego, G., Kalnay, E., Mitchell, K., Shafran, P. C., Ebisuzaki, W., Jović, D., Woollen, J., Rogers, E., Berbery, E. H., et al. (2006) · Bulletin of the American Meteorological Society · doi:10.1175/bams-87-3-343 · Cited by 3,378
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301
Forecast Verification: A Practitioner's Guide in Atmospheric Science
(2011) · Wiley eBooks · doi:10.1002/9781119960003 · Cited by 1,267
Mason, S. J., & Graham, N. E. (2002) · Quarterly Journal of the Royal Meteorological Society · doi:10.1256/003590002320603584 · Cited by 868
The Critical Success Index as an Indicator of Warning Skill
Schaefer, J. T. (1990) · Weather and Forecasting · doi:10.1175/1520-0434(1990)005<0570:tcsiaa>2.0.co;2 · Cited by 867
Cited by
Wilks, D. S. (2011) · International Geophysics · doi:10.1016/b978-0-12-385022-5.00008-7 · Cited by 727
The MOGREPS short‐range ensemble prediction system
Bowler, N. E., Arribas, A., Mylne, K. R., Robertson, K. B., & Beare, S. E. (2008) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.234 · Cited by 468
Can multi‐model combination really enhance the prediction skill of probabilistic ensemble forecasts?
Weigel, A. P., Liniger, M. A., & Appenzeller, C. (2008) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.210 · Cited by 330
Hamill, T. M., & Whitaker, J. S. (2006) · Monthly Weather Review · doi:10.1175/mwr3237.1 · Cited by 313
Self-citationForecast verification: Current status and future directions
Casati, B., Wilson, L. J., Stephenson, D. B., Nurmi, P., Ghelli, A., Pocernich, M., Damrath, U., Ebert, E. E., Brown, B. G., & Mason, S. (2008) · Meteorological Applications · doi:10.1002/met.52 · Cited by 306
Comparison of Ensemble-MOS Methods Using GFS Reforecasts
Wilks, D. S., & Hamill, T. M. (2007) · Monthly Weather Review · doi:10.1175/mwr3402.1 · Cited by 258
Self-citationvan Straaten, C., Whan, K., Coumou, D., van den Hurk, B., & Schmeits, M. (2022) · Monthly Weather Review · doi:10.1175/mwr-d-21-0201.1 · Cited by 50
“Subseasonal windows of predictability, being sourced from the features and interactions that we wish to research, would exist on top of this climate change signal (Hamill and Juras 2006).”
Background
Metadata
- Authors
- Hamill, T. M., & Juras, J.
- Venue
- Quarterly Journal of the Royal Meteorological Society
- Publisher
- Wiley
- Published
- 2006-10
- Volume
- 132
- Issue
- 621C
- Pages
- 2905-2923
- ISSN
- 0035-9009 (ISSN-L)
- 1477-870X
- OpenAlex
- W2170766202
- Cited by
- 264
- References
- 47
- FWCI
- 9.15
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.