The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
A better measure of relative prediction accuracy for model selection and model estimation
Open accessAbstract
Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of prediction accuracy in businesses and organizations. It is, however, biased: when used to select among competing prediction methods it systematically selects those whose predictions are too low. This has not been widely discussed and so is not generally known among practitioners. We explain why this happens. We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio, that is, log (prediction/actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection. Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Error measures for generalizing about forecasting methods: Empirical comparisons
Armstrong, J. S., & Collopy, F. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90008-w · Cited by 1,396
Making and Evaluating Point Forecasts
Gneiting, T. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.r10138 · Cited by 1,154
Accuracy measures: theoretical and practical concerns
Makridakis, S. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90079-3 · Cited by 869
A simulation study of the model evaluation criterion mmre
Foss, T., Stensrud, E., Kitchenham, B., & Myrtveit, I. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1245300 · Cited by 480
How Should Relative Changes be Measured?
Törnqvist, L., Vartia, P., & Vartia, Y. O. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479385 · Cited by 466
What accuracy statistics really measure
Kitchenham, B. A., Pickard, L. M., MacDonell, S. G., & Shepperd, M. J. (2001) · IEE Proceedings - Software · doi:10.1049/ip-sen:20010506 · Cited by 437
Evaluating prediction systems in software project estimation
Shepperd, M., & MacDonell, S. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2011.12.008 · Cited by 405
Reliability and validity in comparative studies of software prediction models
Myrtveit, I., Stensrud, E., & Shepperd, M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.58 · Cited by 259
Robust regression for developing software estimation models
Miyazaki, Y., Terakado, M., Ozaki, K., & Nozaki, H. (1994) · Journal of Systems and Software · doi:10.1016/0164-1212(94)90110-4 · Cited by 178
Method to estimate parameter values in software prediction models
Miyazaki, Y., Takanou, A., Nozaki, H., et al. (1991) · Information and Software Technology · doi:10.1016/0950-5849(91)90139-3 · Cited by 62
Cited by
Predictive model assessment in PLS-SEM: guidelines for using PLSpredict
Shmueli, G., Sarstedt, M., Hair, J. F., Cheah, J.-H., Ting, H., Vaithilingam, S., & Ringle, C. M. (2019) · European Journal of Marketing · doi:10.1108/ejm-02-2019-0189 · Cited by 4,618
A new metric of absolute percentage error for intermittent demand forecasts
Kim, S., & Kim, H. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.12.003 · Cited by 1,146
Machine learning methods for wind turbine condition monitoring: A review
Stetco, A., Dinmohammadi, F., Zhao, X., Robu, V., Flynn, D., Barnes, M., Keane, J., & Nenadic, G. (2019) · Renewable Energy · doi:10.1016/j.renene.2018.10.047 · Cited by 831
Botchkarev, A. (2019) · Interdisciplinary Journal of Information, Knowledge, and Management · doi:10.28945/4184 · Cited by 616
Rana, N. P., Chatterjee, S., Dwivedi, Y. K., & Akter, S. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1955628 · Cited by 404
Metadata
- Authors
- Tofallis, C.
- Venue
- Journal of the Operational Research Society
- Publisher
- Informa UK Limited
- Published
- 2015-08
- Volume
- 66
- Issue
- 8
- Pages
- 1352-1362
- ISSN
- 0160-5682 (ISSN-L)
- 1476-9360
- OpenAlex
- W1988608305
- Cited by
- 439
- References
- 11
- FWCI
- 17.66
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A better measure of relative prediction accuracy for model selection and model estimation. https://app.cochat.ai/papers/10.1057/jors.2014.103
- The paper
- Tofallis, C. (2015). A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society. https://doi.org/10.1057/jors.2014.103
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.