The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States
Open accessAbstract
Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub ( https://covid19forecasthub.org/ ) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An interactive web-based dashboard to track COVID-19 in real time
Dong, E., Du, H., & Gardner, L. (2020) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(20)30120-1 · Cited by 10,048
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Self-citationSimple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Bates, J. M., & Granger, C. W. J. (1969) · Journal of the Operational Research Society · doi:10.1057/jors.1969.103 · Cited by 3,201
Ensemble based systems in decision making
Polikar, R. (2006) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2006.1688199 · Cited by 2,938
Leutbecher, M., & Palmer, T. N. (2008) · Journal of Computational Physics · doi:10.1016/j.jcp.2007.02.014 · Cited by 886
Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble
Krishnamurti, T. N., Kishtawal, C. M., LaRow, T. E., Bachiochi, D. R., Zhang, Z., Williford, C. E., Gadgil, S., & Surendran, S. (1999) · Science · doi:10.1126/science.285.5433.1548 · Cited by 816
Weather Forecasting with Ensemble Methods
Gneiting, T., & Raftery, A. E. (2005) · Science · doi:10.1126/science.1115255 · Cited by 636
Self-citationBates, J. M., & Granger, C. W. J. (1969) · OR · doi:10.2307/3008764 · Cited by 502
Reich, N. G., Brooks, L. C., Fox, S. J., Kandula, S., McGowan, C. J., Moore, E., Osthus, D., Ray, E. L., Tushar, A., Yamana, T. K., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1812594116 · Cited by 305
Self-citationEvaluating epidemic forecasts in an interval format
Bracher, J., Ray, E. L., Gneiting, T., & Reich, N. G. (2021) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1008618 · Cited by 285
Self-citationThe RAPIDD Ebola forecasting challenge: synthesis and lessons learnt
Viboud, C., Sun, K., Gaffey, R., Ajelli, M., Fumanelli, L., Merler, S., Zhang, Q., Chowell, G., Simonsen, L., & Vespignani, A. (2018) · Epidemics · doi:10.1016/j.epidem.2017.08.002 · Cited by 251
Forecaster’s Dilemma: Extreme Events and Forecast Evaluation
Lerch, S., Thorarinsdottir, T. L., Ravazzolo, F., & Gneiting, T. (2017) · Statistical Science · doi:10.1214/16-sts588 · Cited by 234
Self-citationAn open challenge to advance probabilistic forecasting for dengue epidemics
Johansson, M. A., Apfeldorf, K. M., Dobson, S., Devita, J., Buczak, A. L., Baugher, B., Moniz, L. J., Bagley, T., Babin, S. M., Guven, E., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909865116 · Cited by 218
Self-citationRenda, M. E., & Straccia, U. (2003) · ACM Symposium on Applied Computing · doi:10.1145/952532.952698 · Cited by 195
Lutz, C. S., Huynh, M. P., Schroeder, M., Anyatonwu, S. C., Dahlgren, F. S., Danyluk, G. M., Fernandez, D., Greene, S. K., Kipshidze, N. N., Liu, L., et al. (2019) · BMC Public Health · doi:10.1186/s12889-019-7966-8 · Cited by 194
Self-citationAccuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.
Reich, N. G., McGowan, C. J., Yamana, T. K., Tushar, A., Ray, E. L., Osthus, D., Kandula, S., Brooks, L. C., Crawford-Crudell, W., Gibson, G. C., et al. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007486 · Cited by 188
Self-citationOutbreak analytics: a developing data science for informing the response to emerging pathogens
Polonsky, J. A., Baidjoe, A., Kamvar, Z. N., Cori, A., Durski, K., Edmunds, W. J., Eggo, R. M., Funk, S., Kaiser, L., Keating, P., et al. (2019) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2018.0276 · Cited by 174
Self-citationImpact of Rank-Based Normalizing Transformations on the Accuracy of Test Scores
Soloman, S. R., & Sawilowsky, S. S. (2009) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1257034080 · Cited by 161
Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016
McGowan, C. J., Biggerstaff, M., Johansson, M. A., Apfeldorf, K. M., Ben-Nun, M., Brooks, L., Convertino, M., Erraguntla, M., Farrow, D., Freeze, J., et al. (2019) · Scientific Reports · doi:10.1038/s41598-018-36361-9 · Cited by 157
Self-citationMoran, K. R., Fairchild, G. C., Generous, N., Hickmann, K. S., Osthus, D., Priedhorsky, R., Hyman, J. M., & Valle, S. Y. D. (2016) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiw375 · Cited by 112
Self-citationBracher, J., Wolffram, D., Deuschel, J., Görgen, K., Ketterer, J. L., Ullrich, A., Abbott, S., Barbarossa, M. V., Bertsimas, D., Bhatia, S., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-25207-0 · Cited by 99
Self-citationPollett, S., Johansson, M. A., Reich, N. G., Brett‐Major, D. M., Valle, S. Y. D., Venkatramanan, S., Lowe, R., Porco, T. C., Berry, I. M., Deshpande, A., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003793 · Cited by 91
Self-citationCan auxiliary indicators improve COVID-19 forecasting and hotspot prediction?
McDonald, D. J., Bien, J., Green, A., Hu, A. J., DeFries, N., Hyun, S., Oliveira, N. L., Sharpnack, J., Tang, J., Tibshirani, R. J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2111453118 · Cited by 58
Self-citationThe Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research
Reich, N. G., Cornell, M., Ray, E. L., House, K., & Le, K. (2021) · Scientific Data · doi:10.1038/s41597-021-00839-5 · Cited by 12
Self-citation
Metadata
- Authors
- Cramer, E. Y., Ray, E. L., Lopez, V. K., Bracher, J., Brennen, A., Castro Rivadeneira, A. J., Gerding, A., Gneiting, T., House, K. H., Huang, Y., Jayawardena, D., Kanji, A. H., Khandelwal, A., Le, K., Mühlemann, A., Niemi, J., Shah, A., Stark, A., Wang, Y., Wattanachit, N., Zorn, M. W., Gu, Y., Jain, S., Bannur, N., Deva, A., Kulkarni, M., Merugu, S., Raval, A., Shingi, S., Tiwari, A., White, J., Abernethy, N. F., Woody, S., Dahan, M., Fox, S., Gaither, K., Lachmann, M., Meyers, L. A., Scott, J. G., Tec, M., Srivastava, A., George, G. E., Cegan, J. C., Dettwiller, I. D., England, W. P., Farthing, M. W., Hunter, R. H., Lafferty, B., Linkov, I., Mayo, M. L., Parno, M. D., Rowland, M. A., Trump, B. D., Zhang-James, Y., Chen, S., Faraone, S. V., Hess, J., Morley, C. P., Salekin, A., Wang, D., Corsetti, S. M., Baer, T. M., Eisenberg, M. C., Falb, K., Huang, Y., Martin, E. T., McCauley, E., Myers, R. L., Schwarz, T., Sheldon, D., Gibson, G. C., Yu, R., Gao, L., Ma, Y., Wu, D., Yan, X., Jin, X., Wang, Y.-X., Chen, Y., Guo, L., Zhao, Y., Gu, Q., Chen, J., Wang, L., Xu, P., Zhang, W., Zou, D., Biegel, H., Lega, J., McConnell, S., Nagraj, V. P., Guertin, S. L., Hulme-Lowe, C., Turner, S. D., Shi, Y., Ban, X., Walraven, R., Hong, Q.-J., Kong, S., van de Walle, A., Turtle, J. A., Ben-Nun, M., Riley, S., Riley, P., Koyluoglu, U., DesRoches, D., Forli, P., Hamory, B., Kyriakides, C., Leis, H., Milliken, J., Moloney, M., Morgan, J., Nirgudkar, N., Ozcan, G., Piwonka, N., Ravi, M., Schrader, C., Shakhnovich, E., Siegel, D., Spatz, R., Stiefeling, C., Wilkinson, B., Wong, A., Cavany, S., España, G., Moore, S., Oidtman, R., Perkins, A., Kraus, D., Kraus, A., Gao, Z., Bian, J., Cao, W., Lavista Ferres, J., Li, C., Liu, T.-Y., Xie, X., Zhang, S., Zheng, S., Vespignani, A., Chinazzi, M., Davis, J. T., Mu, K., Pastore y Piontti, A., Xiong, X., Zheng, A., Baek, J., Farias, V., Georgescu, A., Levi, R., Sinha, D., Wilde, J., Perakis, G., Bennouna, M. A., Nze-Ndong, D., Singhvi, D., Spantidakis, I., Thayaparan, L., Tsiourvas, A., Sarker, A., Jadbabaie, A., Shah, D., Della Penna, N., Celi, L. A., Sundar, S., Wolfinger, R., Osthus, D., Castro, L., Fairchild, G., Michaud, I., Karlen, D., Kinsey, M., Mullany, L. C., Rainwater-Lovett, K., Shin, L., Tallaksen, K., Wilson, S., Lee, E. C., Dent, J., Grantz, K. H., Hill, A. L., Kaminsky, J., Kaminsky, K., Keegan, L. T., Lauer, S. A., Lemaitre, J. C., Lessler, J., Meredith, H. R., Perez-Saez, J., Shah, S., Smith, C. P., Truelove, S. A., Wills, J., Marshall, M., Gardner, L., Nixon, K., Burant, J. C., Wang, L., Gao, L., Gu, Z., Kim, M., Li, X., Wang, G., Wang, Y., Yu, S., Reiner, R. C., Barber, R., Gakidou, E., Hay, S. I., Lim, S., Murray, C., Pigott, D., Gurung, H. L., Baccam, P., Stage, S. A., Suchoski, B. T., Prakash, B. A., Adhikari, B., Cui, J., Rodríguez, A., Tabassum, A., Xie, J., Keskinocak, P., Asplund, J., Baxter, A., Oruc, B. E., Serban, N., Arik, S. O., Dusenberry, M., Epshteyn, A., Kanal, E., Le, L. T., Li, C.-L., Pfister, T., Sava, D., Sinha, R., Tsai, T., Yoder, N., Yoon, J., Zhang, L., Abbott, S., Bosse, N. I., Funk, S., Hellewell, J., Meakin, S. R., Sherratt, K., Zhou, M., Kalantari, R., Yamana, T. K., Pei, S., Shaman, J., Li, M. L., Bertsimas, D., Skali Lami, O., Soni, S., Tazi Bouardi, H., Ayer, T., Adee, M., Chhatwal, J., Dalgic, O. O., Ladd, M. A., Linas, B. P., Mueller, P., Xiao, J., Wang, Y., Wang, Q., Xie, S., Zeng, D., Green, A., Bien, J., Brooks, L., Hu, A. J., Jahja, M., McDonald, D., Narasimhan, B., Politsch, C., Rajanala, S., Rumack, A., Simon, N., Tibshirani, R. J., Tibshirani, R., Ventura, V., Wasserman, L., O’Dea, E. B., Drake, J. M., Pagano, R., Tran, Q. T., Ho, L. S. T., Huynh, H., Walker, J. W., Slayton, R. B., Johansson, M. A., Biggerstaff, M., & Reich, N. G.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2022-04-08
- Volume
- 119
- Issue
- 15
- Pages
- e2113561119-e2113561119
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W4223487063
- Cited by
- 309
- References
- 25
- FWCI
- 34.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.