The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
M5 accuracy competition: Results, findings, and conclusions
Open accessAbstract
In this study, we present the results of the M5 “Accuracy” competition, which was the first of two parallel challenges in the latest M competition with the aim of advancing the theory and practice of forecasting. The main objective in the M5 “Accuracy” competition was to accurately predict 42,840 time series representing the hierarchical unit sales for the largest retail company in the world by revenue, Walmart. The competition required the submission of 30,490 point forecasts for the lowest cross-sectional aggregation level of the data, which could then be summed up accordingly to estimate forecasts for the remaining upward levels. We provide details of the implementation of the M5 “Accuracy” challenge, as well as the results and best performing methods, and summarize the major findings and conclusions. Finally, we discuss the implications of these findings and suggest directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Another look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
Forecasting with artificial neural networks:
Zhang, G., Eddy Patuwo, B., & Y. Hu, M. (1998) · International Journal of Forecasting · doi:10.1016/s0169-2070(97)00044-7 · Cited by 4,229
A scaled conjugate gradient algorithm for fast supervised learning
Møller, M. F. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80056-5 · Cited by 4,052
DeepAR: Probabilistic forecasting with autoregressive recurrent networks
Salinas, D., Flunkert, V., Gasthaus, J., & Januschowski, T. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.07.001 · Cited by 2,661
The M3-Competition: results, conclusions and implications
Makridakis, S., & Hibon, M. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00057-1 · Cited by 1,695
Self-citationStatistical and machine learning forecasting methods: Concerns and ways forward
Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018) · PLOS ONE · doi:10.1371/journal.pone.0194889 · Cited by 1,528
Self-citationThe accuracy of extrapolation (time series) methods: Results of a forecasting competition
Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., & Winkler, R. (1982) · Journal of Forecasting · doi:10.1002/for.3980010202 · Cited by 1,462
Self-citationExponential smoothing: The state of the art
Gardner, E. S. (1985) · Journal of Forecasting · doi:10.1002/for.3980040103 · Cited by 1,458
A state space framework for automatic forecasting using exponential smoothing methods
Hyndman, R. J., Koehler, A. B., Snyder, R. D., & Grose, S. (2002) · International Journal of Forecasting · doi:10.1016/s0169-2070(01)00110-8 · Cited by 1,066
Out-of-sample tests of forecasting accuracy: an analysis and review
Tashman, L. J. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00065-0 · Cited by 1,021
Automatic time series forecasting: the forecast package for R
Hyndman, R. J., & Khandakar, Y. (2017) · RePEc: Research Papers in Economics · doi:10.26180/5090230 · Cited by 978
The M4 Competition: 100,000 time series and 61 forecasting methods
Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.04.014 · Cited by 969
Self-citationAccuracy measures: theoretical and practical concerns
Makridakis, S. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90079-3 · Cited by 869
Self-citationforecast: Forecasting Functions for Time Series and Linear Models
Hyndman, R., Athanasopoulos, G., Bergmeir, C., Caceres, G., Chhay, L., Kuroptev, K., Mücke, M., O'Hara-Wild, M., Petropoulos, F., Razbash, S., et al. (2009) · CRAN: Contributed Packages · doi:10.32614/cran.package.forecast · Cited by 791
A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting
Smyl, S. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.03.017 · Cited by 761
Forecasting and Stock Control for Intermittent Demands
Croston, J. D. (1972) · Journal of the Operational Research Society · doi:10.1057/jors.1972.50 · Cited by 753
A note on the validity of cross-validation for evaluating autoregressive time series prediction
Bergmeir, C., Hyndman, R. J., & Koo, B. (2018) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2017.11.003 · Cited by 743
Judgmental forecasting: A review of progress over the last 25years
Lawrence, M., Goodwin, P., O'Connor, M., & Önkal, D. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.007 · Cited by 534
The theta model: a decomposition approach to forecasting
Assimakopoulos, V., & Nikolopoulos, K. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00066-2 · Cited by 503
Self-citationThe accuracy of intermittent demand estimates
Syntetos, A. A., & Boylan, J. E. (2005) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2004.10.001 · Cited by 489
Optimal combination forecasts for hierarchical time series
Hyndman, R. J., Ahmed, R. A., Athanasopoulos, G., & Shang, H. L. (2011) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2011.03.006 · Cited by 467
Fildes, R., Goodwin, P., Lawrence, M., & Nikolopoulos, K. (2009) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2008.11.010 · Cited by 446
On the categorization of demand patterns
Syntetos, A. A., Boylan, J. E., & Croston, J. D. (2005) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601841 · Cited by 425
randomForest: Breiman and Cutlers Random Forests for Classification and Regression
Breiman, L., Cutler, A., Liaw, A., & Wiener, M. (2002) · CRAN: Contributed Packages · doi:10.32614/cran.package.randomforest · Cited by 358
On the asymmetry of the symmetric MAPE
Goodwin, P., & Lawton, R. (1999) · International Journal of Forecasting · doi:10.1016/s0169-2070(99)00007-2 · Cited by 328
FFORMA: Feature-based forecast model averaging
Montero-Manso, P., Athanasopoulos, G., Hyndman, R. J., & Talagala, T. S. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.02.011 · Cited by 311
Neural network ensemble operators for time series forecasting
Kourentzes, N., Barrow, D. K., & Crone, S. F. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.12.011 · Cited by 292
Kaggle forecasting competitions: An overlooked learning opportunity
Bojer, C. S., & Meldgaard, J. P. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.07.007 · Cited by 264
The M2-competition: A real-time judgmentally based forecasting study
Makridakis, S., Chatfield, C., Hibon, M., Lawrence, M., Mills, T., Ord, K., & Simmons, L. F. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90044-n · Cited by 263
Self-citationThe M5 competition: Background, organization, and implementation
Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.07.007 · Cited by 258
Self-citationIntermittent demand: Linking forecasting to inventory obsolescence
Teunter, R. H., Syntetos, A. A., & Zied Babai, M. (2011) · European Journal of Operational Research · doi:10.1016/j.ejor.2011.05.018 · Cited by 252
The forecast combination puzzle: A simple theoretical explanation
Claeskens, G., Magnus, J. R., Vasnev, A. L., & Wang, W. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.12.005 · Cited by 244
Ghobbar, A. A., & Friend, C. H. (2003) · Computers & Operations Research · doi:10.1016/s0305-0548(02)00125-9 · Cited by 235
‘Horses for Courses’ in demand forecasting
Petropoulos, F., Makridakis, S., Assimakopoulos, V., & Nikolopoulos, K. (2014) · European Journal of Operational Research · doi:10.1016/j.ejor.2014.02.036 · Cited by 201
Self-citationThe M3 competition: Statistical tests of the results
Koning, A., Franses, P. H., Hibon, M., & Stekler, H. O. (2004) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2004.10.003 · Cited by 197
Nikolopoulos, K. I., Syntetos, A. A., Boylan, J. E., Petropoulos, F., & Assimakopoulos, V. (2010) · Journal of the Operational Research Society · doi:10.1057/jors.2010.32 · Cited by 194
Self-citationMeasuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts
Davydenko, A., & Fildes, R. (2013) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2012.09.002 · Cited by 193
Improving forecasting by estimating time series structural components across multiple frequencies
Kourentzes, N., Petropoulos, F., & Trapero, J. R. (2014) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2013.09.006 · Cited by 182
Evaluating predictive count data distributions in retail sales forecasting
Kolassa, S. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.12.004 · Cited by 178
Ma, S., Fildes, R., & Huang, T. (2016) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.08.029 · Cited by 174
Forecast combinations for intermittent demand
Petropoulos, F., & Kourentzes, N. (2014) · Journal of the Operational Research Society · doi:10.1057/jors.2014.62 · Cited by 170
The M5 uncertainty competition: Results, findings and conclusions
Makridakis, S., Spiliotis, E., Assimakopoulos, V., Chen, Z., Gaba, A., Tsetlin, I., & Winkler, R. L. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.10.009 · Cited by 164
Self-citationComparison of statistical and machine learning methods for daily SKU demand forecasting
Spiliotis, E., Makridakis, S., Semenoglou, A.-A., & Assimakopoulos, V. (2020) · Operational Research · doi:10.1007/s12351-020-00605-2 · Cited by 155
Self-citationOn the selection of forecasting accuracy measures
Koutsandreas, D., Spiliotis, E., Petropoulos, F., & Assimakopoulos, V. (2021) · Journal of the Operational Research Society · doi:10.1080/01605682.2021.1892464 · Cited by 138
Self-citationA brief history of forecasting competitions
Hyndman, R. J. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.03.015 · Cited by 131
Judging the judges through accuracy-implication metrics: The case of inventory forecasting
Syntetos, A. A., Nikolopoulos, K. I., & Boylan, J. E. (2009) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2009.05.016 · Cited by 117
Why the “best” point forecast depends on the error or accuracy measure
Kolassa, S. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.02.017 · Cited by 99
Investigating the accuracy of cross-learning time series forecasting methods
Semenoglou, A.-A., Spiliotis, E., Makridakis, S., & Assimakopoulos, V. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.11.009 · Cited by 91
Self-citationAre forecasting competitions data representative of the reality?
Spiliotis, E., Kouloumos, A., Assimakopoulos, V., & Makridakis, S. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2018.12.007 · Cited by 67
Self-citationThe value added by machine learning approaches in forecasting
Gilliland, M. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.04.016 · Cited by 59
Forecasting for big data: Does suboptimality matter?
Nikolopoulos, K. I., & Petropoulos, F. (2017) · Computers & Operations Research · doi:10.1016/j.cor.2017.05.007 · Cited by 50
Objectivity, reproducibility and replicability in forecasting research
Makridakis, S., Assimakopoulos, V., & Spiliotis, E. (2018) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2018.05.001 · Cited by 42
Self-citationImproving the forecasting performance of temporal hierarchies
Spiliotis, E., Petropoulos, F., & Assimakopoulos, V. (2019) · PLoS ONE · doi:10.1371/journal.pone.0223422 · Cited by 42
Self-citationExploring the representativeness of the M5 competition data
Theodorou, E., Wang, S., Kang, Y., et al. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.07.006 · Cited by 24
Cited by
Forecast combinations: An over 50-year review
Wang, X., Hyndman, R. J., Li, F., & Kang, Y. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2022.11.005 · Cited by 280
TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
Ekambaram, V., Jati, A., Nguyen, N., Sinthong, P., & Kalagnanam, J. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599533 · Cited by 278
Forecast evaluation for data scientists: common pitfalls and best practices
Hewamalage, H., Ackermann, K., & Bergmeir, C. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00894-5 · Cited by 263
Uncertainty estimation with deep learning for rainfall–runoff modeling
Klotz, D., Kratzert, F., Gauch, M., Keefe Sampson, A., Brandstetter, J., Klambauer, G., Hochreiter, S., & Nearing, G. (2022) · Hydrology and Earth System Sciences · doi:10.5194/hess-26-1673-2022 · Cited by 256
Metadata
- Authors
- Makridakis, S., Spiliotis, E., & Assimakopoulos, V.
- Venue
- International Journal of Forecasting
- Publisher
- Elsevier BV
- Published
- 2022-10
- Volume
- 38
- Issue
- 4
- Pages
- 1346-1364
- ISSN
- 0169-2070 (ISSN-L)
- OpenAlex
- W4206173445
- Cited by
- 554
- References
- 56
- FWCI
- 54.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.