The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Evaluating epidemic forecasts in an interval format
Open accessAbstract
For practical reasons, many forecasts of case, hospitalization, and death counts in the context of the current Coronavirus Disease 2019 (COVID-19) pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub(https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Self-citationDiebold, F., & Mariano, R. (1994) · National Bureau of Economic Research · doi:10.3386/t0169 · Cited by 5,185
Diebold, F. X., & Mariano, R. S. (1995) · Journal of Business & Economic Statistics · doi:10.1080/07350015.1995.10524599 · Cited by 5,130
Probabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00587.x · Cited by 1,783
Self-citationMaking and Evaluating Point Forecasts
Gneiting, T. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.r10138 · Cited by 1,154
Self-citationDawid, A. P. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981683 · Cited by 1,085
Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond
Hong, T., Pinson, P., Fan, S., Zareipour, H., Troccoli, A., & Hyndman, R. J. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2016.02.001 · Cited by 1,044
The M4 Competition: 100,000 time series and 61 forecasting methods
Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.04.014 · Cited by 969
Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules
Gneiting, T., & Ranjan, R. (2011) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2010.08110 · Cited by 507
Self-citationPredictive Model Assessment for Count Data
Czado, C., Gneiting, T., & Held, L. (2009) · Biometrics · doi:10.1111/j.1541-0420.2009.01191.x · Cited by 464
Self-citationVerification tools for probabilistic forecasts of continuous hydrological variables
Laio, F., & Tamea, S. (2007) · Hydrology and Earth System Sciences · doi:10.5194/hess-11-1267-2007 · Cited by 358
Reich, N. G., Brooks, L. C., Fox, S. J., Kandula, S., McGowan, C. J., Moore, E., Osthus, D., Ray, E. L., Tushar, A., Yamana, T. K., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1812594116 · Cited by 305
Self-citationGneiting, T., Stanberry, L. I., Grimit, E. P., Held, L., & Johnson, N. A. (2008) · TEST · doi:10.1007/s11749-008-0114-x · Cited by 265
Self-citationEvaluating the quality of scenarios of short-term wind power generation
Pinson, P., & Girard, R. (2012) · Applied Energy · doi:10.1016/j.apenergy.2011.11.004 · Cited by 262
A Decision-Theoretic Approach to Interval Estimation
Winkler, R. L. (1972) · Journal of the American Statistical Association · doi:10.1080/01621459.1972.10481224 · Cited by 249
Scoring rules and the evaluation of probabilities
Winkler, R. L., Muñoz, J., Cervera, J. L., Bernardo, J. M., Blattenberger, G., Kadane, J. B., Lindley, D. V., Murphy, A. H., Oliver, R. M., & Ríos-Insua, D. (1996) · Test · doi:10.1007/bf02562681 · Cited by 227
An open challenge to advance probabilistic forecasting for dengue epidemics
Johansson, M. A., Apfeldorf, K. M., Dobson, S., Devita, J., Buczak, A. L., Baugher, B., Moniz, L. J., Bagley, T., Babin, S. M., Guven, E., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909865116 · Cited by 218
Self-citationEvaluating predictive count data distributions in retail sales forecasting
Kolassa, S. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.12.004 · Cited by 178
Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies
Piray, P., Dezfouli, A., Heskes, T., Frank, M. J., & Daw, N. D. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007043 · Cited by 173
Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.
Ray, E. L., Wattanachit, N., Niemi, J., Kanji, A. H., House, K., Cramer, E. Y., Bracher, J., Zheng, A., Yamana, T. K., Xiong, X., et al. (2020) · medRxiv · doi:10.1101/2020.08.19.20177493 · Cited by 161
Self-citationA novel sub-epidemic modeling framework for short-term forecasting epidemic waves
Chowell, G., Tariq, A., & Hyman, J. M. (2019) · BMC Medicine · doi:10.1186/s12916-019-1406-6 · Cited by 159
Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016
McGowan, C. J., Biggerstaff, M., Johansson, M. A., Apfeldorf, K. M., Ben-Nun, M., Brooks, L., Convertino, M., Erraguntla, M., Farrow, D., Freeze, J., et al. (2019) · Scientific Reports · doi:10.1038/s41598-018-36361-9 · Cited by 142
Self-citationFunk, S., Camacho, A., Kucharski, A. J., Lowe, R., Eggo, R. M., & Edmunds, W. J. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006785 · Cited by 113
Decomposition and graphical portrayal of the quantile score
Wahl, S., & Friederichs, P. (2013) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.2284 · Cited by 99
Why the “best” point forecast depends on the error or accuracy measure
Kolassa, S. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.02.017 · Cited by 99
Generation and evaluation of space–time trajectories of photovoltaic power
Golestaneh, F., Gooi, H. B., & Pinson, P. (2016) · Applied Energy · doi:10.1016/j.apenergy.2016.05.025 · Cited by 85
Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture
Held, L., Meyer, S., & Bracher, J. (2017) · Statistics in Medicine · doi:10.1002/sim.7363 · Cited by 82
Self-citationInfectious disease prediction with kernel conditional density estimation
Ray, E. L., Sakrejda, K., Lauer, S. A., Johansson, M. A., & Reich, N. G. (2017) · Statistics in Medicine · doi:10.1002/sim.7488 · Cited by 60
Self-citationA Bayesian Approach to Calibration
DUNSMORE, I. R. (1968) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1968.tb00740.x · Cited by 57
On the Comparison of Interval Forecasts
Askanazi, R., Diebold, F. X., Schorfheide, F., & Shin, M. (2018) · Journal of Time Series Analysis · doi:10.1111/jtsa.12426 · Cited by 53
Fixed-time descriptive statistics underestimate extremes of epidemic curve ensembles
Juul, J. L., Græsbøll, K., Christiansen, L. E., & Lehmann, S. (2020) · Nature Physics · doi:10.1038/s41567-020-01121-y · Cited by 29
On the multibin logarithmic score used in the FluSight competitions
Bracher, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912147116 · Cited by 15
Self-citationReply to Bracher: Scoring probabilistic forecasts to maximize public health interpretability
Reich, N. G., Osthus, D., Ray, E. L., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912694116 · Cited by 12
Self-citation
Cited by
Cramer, E. Y., Ray, E. L., Lopez, V. K., Bracher, J., Brennen, A., Castro Rivadeneira, A. J., Gerding, A., Gneiting, T., House, K. H., Huang, Y., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2113561119 · Cited by 309
Self-citationForecast combinations: An over 50-year review
Wang, X., Hyndman, R. J., Li, F., & Kang, Y. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2022.11.005 · Cited by 280
Metadata
- Authors
- Bracher, J., Ray, E. L., Gneiting, T., & Reich, N. G.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-02-12
- Volume
- 17
- Issue
- 2
- Pages
- e1008618
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W3031381838
- Cited by
- 285
- References
- 33
- FWCI
- 15.75
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.