The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Evaluating time series forecasting models: an empirical study on performance estimation methods
Open accessAbstract
Performance estimation aims at estimating the loss that a predictive model will incur on unseen data. This process is a fundamental stage in any machine learning project. In this paper we study the application of these methods to time series forecasting tasks. For independent and identically distributed data the most common approach is cross-validation. However, the dependency among observations in time series raises some caveats about the most appropriate way to estimate performance in this type of data. Currently, there is no consensual approach. We contribute to the literature by presenting an extensive empirical study which compares different performance estimation methods for time series forecasting tasks. These methods include variants of cross-validation, out-of-sample (holdout), and prequential approaches. Two case studies are analysed: One with 174 real-world time series and another with three synthetic time series. Results show noticeable differences in the performance estimation methods in the two scenarios. In particular, empirical experiments suggest that blocked cross-validation can be applied to stationary time series. However, when the time series are non-stationary, the most accurate estimates are produced by out-of-sample methods, particularly the holdout approach repeated in multiple testing periods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,410
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
(2005) · Statistics for Biology and Health · doi:10.1007/0-387-27255-0_9 · Cited by 20,133
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927
Testing the null hypothesis of stationarity against the alternative of a unit root
Kwiatkowski, D., Phillips, P. C. B., Schmidt, P., & Shin, Y. (1992) · Journal of Econometrics · doi:10.1016/0304-4076(92)90104-y · Cited by 12,633
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
Detecting strange attractors in turbulence
Takens, F. (1981) · Lecture Notes in Mathematics · doi:10.1007/bfb0091924 · Cited by 10,459
Time Series: Theory and Methods
Ziegel, E. R., Brockwell, P., & Davis, R. (1992) · Technometrics · doi:10.2307/1270070 · Cited by 5,012
Forecasting principles and practice
Hyndman, R. J., & Athanasopoulos, G. (2018) · Cited by 4,405
Time Series: Theory and Methods
Brockwell, P. J., & Davis, R. A. (1991) · Springer Series in Statistics · doi:10.1007/978-1-4419-0320-4 · Cited by 4,122
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · cited 1×
“For large scale data sets the recommendation by Dietterich [13], and usually adopted in practice, is to apply a simple out-of-sample estimation procedure (Holdout).”
MethodDetermining embedding dimension for phase-space reconstruction using a geometrical construction
Kennel, M. B., Brown, R., & Abarbanel, H. D. I. (1992) · Physical Review A · doi:10.1103/physreva.45.3403 · Cited by 3,776
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522
The Predictive Sample Reuse Method with Applications
Geisser, S. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479865 · Cited by 2,748 · cited 2×
“g a performance estimation method often depends on the data one is modelling. For example, when one can assume independence and an identical distribution (i.i.d.) among observations, cross-validation [17] is typically the most appropriate method. This is mainly due to its ecient use of data [1]. However, there are som…”
BackgroundMethodInfluentialKwiatkowski, D., Phillips, P. C. B., & Schmidt, P. S. (1991) · RePEc: Research Papers in Economics · Cited by 1,617
The accuracy of extrapolation (time series) methods: Results of a forecasting competition
Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., & Winkler, R. (1982) · Journal of Forecasting · doi:10.1002/for.3980010202 · Cited by 1,462
On the use of cross-validation for time series predictor evaluation
Bergmeir, C., & Benítez, J. M. (2012) · Information Sciences · doi:10.1016/j.ins.2011.12.028 · Cited by 1,292
Dawid, A. P. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981683 · Cited by 1,085
Out-of-sample tests of forecasting accuracy: an analysis and review
Tashman, L. J. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00065-0 · Cited by 1,021
McCullagh, P., & Nelder, J. A. (2019) · doi:10.1201/9780203753736 · Cited by 1,013
Combining Instance-Based and Model-Based Learning
Quinlan, J. R. (1993) · Elsevier eBooks · doi:10.1016/b978-1-55860-307-3.50037-x · Cited by 773
A note on the validity of cross-validation for evaluating autoregressive time series prediction
Bergmeir, C., Hyndman, R. J., & Koo, B. (2018) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2017.11.003 · Cited by 743
Characteristic-Based Clustering for Time Series Data
Wang, X., Smith, K., & Hyndman, R. (2006) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-005-0039-x · Cited by 679
Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysisPreprint
Benavoli, A., Corani, G., Demšar, J., & Zaffalon, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04316 · Cited by 631
On evaluating stream learning algorithms
Gama, J., Sebastião, R., & Rodrigues, P. P. (2013) · Machine Learning · doi:10.1007/s10994-012-5320-9 · Cited by 532
Regression and Time Series Model Selection
McQuarrie, A. D. R., & Tsai, C.-L. (1998) · WORLD SCIENTIFIC eBooks · doi:10.1142/3573 · Cited by 513
Cross-validation and multinomial prediction
STONE, M. (1974) · Biometrika · doi:10.1093/biomet/61.3.509 · Cited by 488
Wald, A. (1947) · CERN Document Server (European Organization for Nuclear Research) · Cited by 438
DATA STREAM MINING A Practical Approach
Bifet, A., & Kirkby, R. (2009) · Cited by 268
Kernel Regression Estimation Using Repeated Measurements Data
Hart, J. D., & Wehrly, T. E. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478377 · Cited by 241
Consistent cross-validatory model-selection for dependent data: hv-block cross-validation
Racine, J. (2000) · Journal of Econometrics · doi:10.1016/s0304-4076(00)00030-0 · Cited by 205
Cubist: Rule- And Instance-Based Regression Modeling
Kühn, M., & Quinlan, R. (2011) · doi:10.32614/cran.package.cubist · Cited by 173
ranger: A Fast Implementation of Random Forests
Wright, M. N. (2015) · doi:10.32614/cran.package.ranger · Cited by 145
On the usefulness of cross-validation for directional forecast evaluation
Bergmeir, C., Costantini, M., & Benítez, J. M. (2014) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2014.02.001 · Cited by 83
Evaluation of Aggregate and Individual Forecast Method Selection Rules
Fildes, R. A. (1989) · Management Science · doi:10.1287/mnsc.35.9.1056 · Cited by 67
Arbitrated Ensemble for Time Series Forecasting
Cerqueira, V., Torgo, L., Pinto, F., & Soares, C. (2017) · ECML/PKDD · doi:10.1007/978-3-319-71246-8_29 · Cited by 67
Self-citationArbitrage of forecasting experts
Cerqueira, V., Torgo, L., Pinto, F., & Soares, C. (2018) · Machine Learning · doi:10.1007/s10994-018-05774-y · Cited by 63
Self-citationOn Cross-Validation for Predictor Evaluation in Time Series
Snijders, T. A. B. (1988) · Lecture notes in economics and mathematical systems · doi:10.1007/978-3-642-61564-1_4 · Cited by 57
A Comparative Study of Performance Estimation Methods for Time Series Forecasting
Cerqueira, V., Torgo, L., Smailović, J., & Mozetič, I. (2017) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2017.7 · Cited by 51
Self-citationYearly and seasonal models for electricity load forecasting
Koprinska, I., Rana, M., & Agelidis, V. G. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033398 · Cited by 39
How to evaluate sentiment classifiers for Twitter time-ordered data?
Mozetič, I., Torgo, L., Cerqueira, V., & Smailović, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194317 · Cited by 37
Self-citationPrequential and Cross-Validated Regression Estimation
Modha, D. S., & Masry, E. (1998) · Machine Learning · doi:10.1023/a:1007577530334 · Cited by 24
Evaluation Procedures for Forecasting with Spatio-Temporal Data
Oliveira, M., Torgo, L., & Costa, V. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-10925-7_43 · Cited by 17
Self-citation
Cited by
Forecast evaluation for data scientists: common pitfalls and best practices
Hewamalage, H., Ackermann, K., & Bergmeir, C. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00894-5 · Cited by 263
Metadata
- Authors
- Cerqueira, V., Torgo, L., & Mozetič, I.
- Venue
- Machine Learning
- Published
- 2020-10-13
- Volume
- 109
- Issue
- 11
- Pages
- 1997-2028
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W2947014533
- Cited by
- 485
- References
- 47
- FWCI
- 20.71
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.