The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Making and Evaluating Point Forecasts
Abstract
Typically, point forecasting methods are compared and assessed by means of an error measure or scoring function, with the absolute error and the squared error being key examples. The individual scores are averaged over forecast cases, to result in a summary measure of the predictive performance, such as the mean absolute error or the mean squared error. I demonstrate that this common practice can lead to grossly misguided inferences, unless the scoring function and the forecasting task are carefully matched. Effective point forecasting requires that the scoring function be specified ex ante, or that the forecaster receives a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. If the scoring function is specified ex ante, the forecaster can issue the optimal point forecast, namely, the Bayes rule. If the forecaster receives a directive in the form of a functional, it is critical that the scoring function be consistent for it, in the sense that the expected score is minimized when following the directive. A functional is elicitable if there exists a scoring function that is strictly consistent for it. Expectations, ratios of expectations and quantiles are elicitable. For example, a scoring function is consistent for the mean functional if and only if it is a Bregman function. It is consistent for a quantile if and only if it is generalized piecewise linear. Similar characterizations apply to ratios of expectations and to expectiles. Weighted scoring functions are consistent for functionals that adapt to the weighting in peculiar ways. Not all functionals are elicitable; for instance, conditional value-at-risk is not, despite its popularity in quantitative finance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Generalized autoregressive conditional heteroskedasticity
Bollerslev, T. (1986) · Journal of Econometrics · doi:10.1016/0304-4076(86)90063-1 · Cited by 22,254
Engle, R. F. (1982) · Econometrica · doi:10.2307/1912773 · Cited by 20,795
Koenker, R., & Bassett, G. (1978) · Econometrica · doi:10.2307/1913643 · Cited by 13,316
Ridge regression: Biased estimation for nonorthogonal problems
Hoerl, A. E., & Kennard, R. W. (1970) · Technometrics · doi:10.1080/00401706.1970.10488634 · Cited by 9,648
Artzner, P., Delbaen, F., Eber, J., & Heath, D. (1999) · Mathematical Finance · doi:10.1111/1467-9965.00068 · Cited by 8,806
Robust estimation of a location parameter
Huber, P. J. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703732 · Cited by 7,228
Optimization of conditional value-at-risk
Rockafellar, R. T., & Uryasev, S. (2000) · The Journal of Risk · doi:10.21314/jor.2000.038 · Cited by 6,725
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Self-citationAnother look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
Diebold, F., & Mariano, R. (1994) · National Bureau of Economic Research · doi:10.3386/t0169 · Cited by 5,185
Ziegel, E. R., Lehmann, E. L., & Casella, G. (1999) · Technometrics · doi:10.2307/1270597 · Cited by 4,203
Conditional value-at-risk for general loss distributions
Rockafellar, R. T., & Uryasev, S. (2002) · Journal of Banking & Finance · doi:10.1016/s0378-4266(02)00271-6 · Cited by 3,820
Park, T., & Casella, G. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000337 · Cited by 2,934
Diebold, F. X., & Mariano, R. S. (1995) · Journal of Business and Economic Statistics · doi:10.2307/1392185 · Cited by 2,340
Estimation with Quadratic Loss
James, W., & Stein, C. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_30 · Cited by 2,252
Applied Statistical Decision Theory
Lindley, D. V., Raiffa, H., & Schlaifer, R. (1962) · Econometrica · doi:10.2307/1909900 · Cited by 2,172
Quasi-likelihood functions, generalized linear models, and the Gauss—Newton method
WEDDERBURN, R. W. M. (1974) · Biometrika · doi:10.1093/biomet/61.3.439 · Cited by 2,122 · cited 1×
“Technically, our discussion connects to M-estimation (Huber 1964; Huber and Ronchetti 2009) and quasi-likelihood (Wedderburn 1974; Dawid and Lauritzen 2006).”
BackgroundRidge Regression: Biased Estimation for Nonorthogonal Problems
Hoerl, A. E., & Kennard, R. W. (1970) · Technometrics · doi:10.2307/1267351 · Cited by 1,955
The M3-Competition: results, conclusions and implications
Makridakis, S., & Hibon, M. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00057-1 · Cited by 1,695
(1993) · Choice Reviews Online · doi:10.5860/choice.31-0986 · Cited by 1,540
Mathematical Statistics A Decision Theoretic Approach
Rogers, G. S. (1968) · Technometrics · doi:10.1080/00401706.1968.10490555 · Cited by 1,488
The accuracy of extrapolation (time series) methods: Results of a forecasting competition
Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., & Winkler, R. (1982) · Journal of Forecasting · doi:10.1002/for.3980010202 · Cited by 1,462
Evaluating Density Forecasts with Applications to Financial Risk Management
Diebold, F. X., Gunther, T. A., & Tay, A. S. (1998) · International Economic Review · doi:10.2307/2527342 · Cited by 1,452
Asymptotic Inference about Predictive Ability
West, K. D. (1996) · Econometrica · doi:10.2307/2171956 · Cited by 1,401
Error measures for generalizing about forecasting methods: Empirical comparisons
Armstrong, J. S., & Collopy, F. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90008-w · Cited by 1,396
Duffie, D., & Pan, J. (1997) · The Journal of Derivatives · doi:10.3905/jod.1997.407971 · Cited by 1,296
Volatility forecast comparison using imperfect volatility proxies
Patton, A. J. (2011) · Journal of Econometrics · doi:10.1016/j.jeconom.2010.03.034 · Cited by 1,264
(2001) · Lecture Notes in Mathematics · doi:10.1007/b76887 · Cited by 1,235
Elicitation of Personal Probabilities and Expectations
Savage, L. J. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482346 · Cited by 1,226
Dawid, A. P. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981683 · Cited by 1,085 · cited 2×
“Ideally, forecasts ought to be probabilistic, taking the form of predictive distributions over future quantities and events (Dawid 1984; Diebold, Gunther, and Tay 1998; Granger and Pesaran 2000a, 2000b; Geweke and Whiteman 2006; Gneiting 2008a).”
BackgroundAsymmetric Least Squares Estimation and Testing
Newey, W. K., & Powell, J. L. (1987) · Econometrica · doi:10.2307/1911031 · Cited by 1,048
Bayesian Estimation and Prediction Using Asymmetric Loss Functions
Zellner, A. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478289 · Cited by 913
A General Framework for Forecast Verification
Murphy, A. H., & Winkler, R. L. (1987) · Monthly Weather Review · doi:10.1175/1520-0493(1987)115<1330:agfffv>2.0.co;2 · Cited by 667
The Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604
Accuracy of Forecasting: An Empirical Investigation
Makridakis, S., Hibon, M., & Moser, C. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345077 · Cited by 514
On a Geometric Notion of Quantiles for Multivariate Data
Chaudhuri, P. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476954 · Cited by 414
Economic and statistical measures of forecast accuracy
Granger, C. W. J., & Pesaran, M. H. (2000) · Journal of Forecasting · doi:10.1002/1099-131x(200012)19:7<537::aid-for769>3.0.co;2-g · Cited by 399
A Survey of Multidimensional Medians
Small, C. G. (1990) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1403809 · Cited by 393
Prediction with a Generalized Cost of Error Function
Granger, C. W. J. (1969) · Journal of the Operational Research Society · doi:10.1057/jors.1969.52 · Cited by 324
Buja, A., Stuetzle, W., & Shen, Y. (2005) · Cited by 308
Forecasting and operational research: a review
Fildes, R., Nikolopoulos, K., Crone, S. F., & Syntetos, A. A. (2008) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2602597 · Cited by 287 · cited 1×
“has changed since. Armstrong (2001) called for further research, while Moskaitis and Hansen (2006) asked “Deterministic forecasting and verification: A busted system?” Similarly, the recent review by Fildes et al. (2008, p. 1158) states that “Defining the basic requirements of a good error measure is still a controversi…”
BackgroundInfluentialThe tourism forecasting competition
Athanasopoulos, G., Hyndman, R. J., Song, H., & Wu, D. C. (2011) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2010.04.009 · Cited by 281
Evaluation and Combination of Conditional Quantile Forecasts
Giacomini, R., & Komunjer, I. (2005) · Journal of Business & Economic Statistics · doi:10.1198/073500105000000018 · Cited by 278
Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L., & Johnson, N. A. (2008) · TEST · doi:10.1007/s11749-008-0114-x · Cited by 265
Self-citationThe evaluation of extrapolative forecasting methods
Fildes, R. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90009-x · Cited by 265
The M2-competition: A real-time judgmentally based forecasting study
Makridakis, S., Chatfield, C., Hibon, M., Lawrence, M., Mills, T., Ord, K., & Simmons, L. F. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90044-n · Cited by 263
On the Optimality of Conditional Expectation as a Bregman Predictor
Banerjee, A., Guo, X., & Wang, H. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.850145 · Cited by 240
Evaluating Volatility and Correlation Forecasts
Patton, A. J., & Sheppard, K. (2009) · Handbook of Financial Time Series · doi:10.1007/978-3-540-71297-8_36 · Cited by 232
Scoring rules and the evaluation of probabilities
Winkler, R. L., Muñoz, J., Cervera, J. L., Bernardo, J. M., Blattenberger, G., Kadane, J. B., Lindley, D. V., Murphy, A. H., Oliver, R. M., & Ríos-Insua, D. (1996) · Test · doi:10.1007/bf02562681 · Cited by 227
Fildes, R., & Goodwin, P. (2007) · Interfaces · doi:10.1287/inte.1070.0309 · Cited by 226
Eliciting properties of probability distributions
Lambert, N. S., Pennock, D. M., & Shoham, Y. (2008) · ACM Conference on Electronic Commerce (EC) · doi:10.1145/1386790.1386813 · Cited by 219
The geometry of proper scoring rules
Dawid, A. P. (2006) · Annals of the Institute of Statistical Mathematics · doi:10.1007/s10463-006-0099-8 · Cited by 218
Evaluating predictive performance of value-at-risk models in emerging markets: a reality check
Bao, Y., Lee, T.-H., & Saltoglu, B. (2006) · Journal of Forecasting · doi:10.1002/for.977 · Cited by 218
Quantiles as optimal point forecasts
Gneiting, T. (2011) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2009.12.015 · Cited by 215
Self-citationMEASURES OF THE VALUE OF INFORMATION
McCarthy, J. (1956) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.42.9.654 · Cited by 198
Evaluating Forecasting Methods
Armstrong, J. S. (2001) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-0-306-47630-3_20 · Cited by 183
A General Method for Comparing Probability Assessors
Schervish, M. J. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347398 · Cited by 181
Powering Up With Space-Time Wind Forecasting
Hering, A. S., & Genton, M. G. (2010) · Journal of the American Statistical Association · doi:10.1198/jasa.2009.ap08117 · Cited by 171
Further results on forecasting and model selection under asymmetric loss
Christoffersen, P. F., & Diebold, F. X. (1996) · Journal of Applied Econometrics · doi:10.1002/(sici)1099-1255(199609)11:5<561::aid-jae406>3.0.co;2-s · Cited by 170
Coherent dispersion criteria for optimal experimental design
Dawid, A. P., & Sebastiani, P. (1999) · The Annals of Statistics · doi:10.1214/aos/1018031101 · Cited by 162
The evolution of sales forecasting management: a 20-year longitudinal study of forecasting practices
McCarthy, T. M., Davis, D. F., Golicic, S. L., & Mentzer, J. T. (2006) · Journal of Forecasting · doi:10.1002/for.989 · Cited by 155
Quasi-maximum likelihood estimation for conditional quantiles
Komunjer, I. (2004) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.08.010 · Cited by 153
A General Concept of Unbiasedness
Lehmann, E. L. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729549 · Cited by 147
Carbone, R., & Armstrong, J. S. (1982) · Journal of Forecasting · doi:10.1002/for.3980010207 · Cited by 138
Editorial: Probabilistic Forecasting
Gneiting, T. (2008) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2007.00522.x · Cited by 137
Self-citationDecision‐Based Methods for Forecast Evaluation
Pesaran, M. H., & Skouras, S. (2004) · Wiley eBooks · doi:10.1002/9780470996430.ch11 · Cited by 116
Identification and estimation.
Bush, R. R., & Mosteller, F. C. (2014) · John Wiley & Sons, Inc. eBooks · doi:10.1037/14496-009 · Cited by 103
Evaluating Quantile Assessments
Jose, V. R. R., & Winkler, R. L. (2009) · Operations Research · doi:10.1287/opre.1080.0665 · Cited by 86
Eliciting production possibilities from a well-informed manager
THOMSON, W. J. (1979) · Journal of Economic Theory · doi:10.1016/0022-0531(79)90042-5 · Cited by 75
Accuracy of point predictions in survival analysis
Henderson, R., Jones, M., & Stare, J. (2001) · Statistics in Medicine · doi:10.1002/sim.913 · Cited by 67
Evaluating value-at-risk measures in the presence of long memory conditional volatility
Caporin, M. (2008) · The Journal of Risk · doi:10.21314/jor.2008.172 · Cited by 65
Error measures and the choice of a forecast method
Ahlburg, D. A. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90010-7 · Cited by 65
Building cost functions minimizing to some summary statistics
Saerens, M. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.883416 · Cited by 57
Identification and estimation of statistical functionals using incomplete data
Horowitz, J. L., & Manski, C. F. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2005.02.007 · Cited by 55
A commentary on error measures
Chatfield, C. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90011-w · Cited by 40
The impenetrable hedge: a note on propriety, equitability and consistency
Jolliffe, I. T. (2008) · Meteorological Applications · doi:10.1002/met.60 · Cited by 35
Hedging and Skill Scores for Probability Forecasts
Murphy, A. H. (1973) · Journal of Applied Meteorology · doi:10.1175/1520-0450(1973)012<0215:hassfp>2.0.co;2 · Cited by 33
Penalized Bregman divergence for large-dimensional regression and classification
Zhang, C., Jiang, Y., & Chai, Y. (2010) · Biometrika · doi:10.1093/biomet/asq033 · Cited by 32
A Bayesian Model for Sales Forecasting at Sun Microsystems
Yelland, P. M., Kim, S., & Stratulate, R. (2010) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.1090.0477 · Cited by 30
Unbiasedness as the Dual of Being Bayes
Noorbaloochi, S., & Meeden, G. D. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10478020 · Cited by 25
Predicting Inflation: Professional Experts Versus No-Change ForecastsPreprint
Gneiting, T., & Thorarinsdottir, T. L. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.2318 · Cited by 16
Self-citation
Cited by
Understanding predictive information criteria for Bayesian models
Gelman, A., Hwang, J., & Vehtari, A. (2014) · Statistics and Computing · doi:10.1007/s11222-013-9416-2 · Cited by 2,104
“We use the squared error as an example scoring function for point prediction, because the squared error and its derivatives seem to be the most common scoring functions in predictive literature (Gneiting, 2011).”
BackgroundMethodInfluentialForecasting: theory and practice
Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Ben Taieb, S., Bergmeir, C., Bessa, R. J., Bijak, J., Boylan, J. E., et al. (2022) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.11.001 · Cited by 921
A guide to Bayesian model selection for ecologists
Hooten, M. B., & Hobbs, N. T. (2015) · Ecological Monographs · doi:10.1890/14-0661.1 · Cited by 891
Gneiting, T., & Katzfuss, M. (2014) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-062713-085831 · Cited by 888
Self-citationRiebler, A., & Held, L. (2017) · Biometrical Journal · doi:10.1002/bimj.201500263 · Cited by 624
Metadata
- Authors
- Gneiting, T.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2011-06
- Volume
- 106
- Issue
- 494
- Pages
- 746-762
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2075965721
- Cited by
- 1,154
- References
- 81
- FWCI
- 49.42
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.