The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
A New Typology Design of Performance Metrics to Measure Errors in Machine Learning Regression Algorithms
Open accessAbstract
Aim/Purpose: The aim of this study was to analyze various performance metrics and approaches to their classification. The main goal of the study was to develop a new typology that will help to advance knowledge of metrics and facilitate their use in machine learning regression algorithms Background: Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. A performance metric can be defined as a logical and mathematical construct designed to measure how close are the actual results from what has been expected or predicted. A vast variety of performance metrics have been described in academic literature. The most commonly mentioned metrics in research studies are Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), etc. Knowledge about metrics properties needs to be systematized to simplify the design and use of the metrics. Methodology: A qualitative study was conducted to achieve the objectives of identifying related peer-reviewed research studies, literature reviews, critical thinking and inductive reasoning. Contribution: The main contribution of this paper is in ordering knowledge of performance metrics and enhancing understanding of their structure and properties by proposing a new typology, generic primary metrics mathematical formula and a visualization chart Findings: Based on the analysis of the structure of numerous performance metrics, we proposed a framework of metrics which includes four (4) categories: primary metrics, extended metrics, composite metrics, and hybrid sets of metrics. The paper identified three (3) key components (dimensions) that determine the structure and properties of primary metrics: method of determining point distance, method of normalization, method of aggregation of point distances over a data set. For each component, implementation options have been identified. The suggested new typology has been shown to cover a total of over 40 commonly used primary metrics Recommendations for Practitioners: Presented findings can be used to facilitate teaching performance metrics to university students and expedite metrics selection and implementation processes for practitioners Recommendation for Researchers: By using the proposed typology, researchers can streamline development of new metrics with predetermined properties Impact on Society: The outcomes of this study could be used for improving evaluation results in machine learning regression, forecasting and prognostics with direct or indirect positive impacts on innovation and productivity in a societal sense Future Research: Future research is needed to examine the properties of the extended metrics, composite metrics, and hybrid sets of metrics. Empirical study of the metrics is needed using R Studio or Azure Machine Learning Studio, to find associations between the properties of primary metrics and their “numerical” behavior in a wide spectrum of data characteristics and business or research requirements
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193
Chai, T., & Draxler, R. R. (2014) · Geoscientific Model Development · doi:10.5194/gmd-7-1247-2014 · Cited by 7,178 · cited 4×
“Chai and Draxler (2014) clarified that “as every statistical measure condenses a large number of data into a single value, it only provides one projection of the model errors emphasizing a certain aspect of the error characteristics of the model performance.””
BackgroundInfluentialWillmott, C., & Matsuura, K. (2005) · Climate Research · doi:10.3354/cr030079 · Cited by 6,512 · cited 3×
“Suggested in this paper generic formula for primary performance metrics is more comprehensive than used by Willmott & Matsuura (2005), as their definition can be applied only to metrics with mean-averaging type of error aggregation.”
BackgroundMethodAnother look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
Analysis of Ecological Communities
Anderson, M. J. (2003) · Journal of Experimental Marine Biology and Ecology · doi:10.1016/s0022-0981(03)00091-1 · Cited by 5,086
Statistics for the evaluation and comparison of models
Willmott, C. J., Ackleson, S. G., Davis, R. E., Feddema, J. J., Klink, K. M., Legates, D. R., O'Donnell, J., & Rowe, C. M. (1985) · Journal of Geophysical Research: Oceans · doi:10.1029/jc090ic05p08995 · Cited by 2,026
Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions
Cha, S.-H. (2007) · Cited by 1,644
25 years of time series forecasting
De Gooijer, J. G., & Hyndman, R. J. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.01.001 · Cited by 1,587
Statistical and machine learning forecasting methods: Concerns and ways forward
Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018) · PLOS ONE · doi:10.1371/journal.pone.0194889 · Cited by 1,528
Error measures for generalizing about forecasting methods: Empirical comparisons
Armstrong, J. S., & Collopy, F. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90008-w · Cited by 1,396
Making and Evaluating Point Forecasts
Gneiting, T. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.r10138 · Cited by 1,154
A new metric of absolute percentage error for intermittent demand forecasts
Kim, S., & Kim, H. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.12.003 · Cited by 1,146
Accuracy measures: theoretical and practical concerns
Makridakis, S. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90079-3 · Cited by 869
Experience with Forecasting Univariate Time Series and the Combination of Forecasts
Newbold, P., & Granger, C. W. J. (1974) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344546 · Cited by 848
Effects of Distance Measure Choice on K-Nearest Neighbor Classifier Performance: A Review
Abu Alfeilat, H. A., Hassanat, A. B. A., Lasassmeh, O., Tarawneh, A. S., Alhasanat, M. B., Eyal Salman, H. S., & Prasath, V. B. S. (2019) · Big Data · doi:10.1089/big.2018.0175 · Cited by 667
Metrics for evaluating performance of prognostic techniques
Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M. (2008) · 2008 International Conference on Prognostics and Health Management · doi:10.1109/phm.2008.4711436 · Cited by 518
The accuracy of intermittent demand estimates
Syntetos, A. A., & Boylan, J. E. (2005) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2004.10.001 · Cited by 489
A simulation study of the model evaluation criterion mmre
Foss, T., Stensrud, E., Kitchenham, B., & Myrtveit, I. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1245300 · Cited by 480
How Should Relative Changes be Measured?
Törnqvist, L., Vartia, P., & Vartia, Y. O. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479385 · Cited by 466
A better measure of relative prediction accuracy for model selection and model estimation
Tofallis, C. (2015) · Journal of the Operational Research Society · doi:10.1057/jors.2014.103 · Cited by 439
What accuracy statistics really measure
Kitchenham, B. A., Pickard, L. M., MacDonell, S. G., & Shepperd, M. J. (2001) · IEE Proceedings - Software · doi:10.1049/ip-sen:20010506 · Cited by 437
On the asymmetry of the symmetric MAPE
Goodwin, P., & Lawton, R. (1999) · International Journal of Forecasting · doi:10.1016/s0169-2070(99)00007-2 · Cited by 328
Measures of Model Performance Based On the Log Accuracy Ratio
Morley, S. K., Brito, T. V., & Welling, D. T. (2018) · Space Weather · doi:10.1002/2017sw001669 · Cited by 313
New unbiased symmetric metrics for evaluation of air quality models
Yu, S., Eder, B., Dennis, R., Chu, S., & Schwartz, S. E. (2006) · Atmospheric Science Letters · doi:10.1002/asl.125 · Cited by 290
The Internet Encyclopedia of Philosophy
(2009) · Reference Reviews · doi:10.1108/09504120910958313 · Cited by 284
Another Look at Forecast Accuracy Metrics for Intermittent Demand
Hyndman, R. J. (2006) · Cited by 270
Distances in evidence theory: Comprehensive survey and generalizations
Jousselme, A.-L., & Maupin, P. (2012) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2011.07.006 · Cited by 266
The evaluation of extrapolative forecasting methods
Fildes, R. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90009-x · Cited by 265
A new accuracy measure based on bounded relative error for time series forecasting
Chen, C., Twycross, J., & Garibaldi, J. M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0174202 · Cited by 264
A suite of metrics for assessing the performance of solar power forecasting
Zhang, J., Florita, A. R., Hodge, B. S., Lu, S., Hamann, H. F., Banunarayanan, V., & Brockway, A. M. (2014) · Solar Energy · doi:10.1016/j.solener.2014.10.016 · Cited by 254
Fildes, R., & Goodwin, P. (2007) · Interfaces · doi:10.1287/inte.1070.0309 · Cited by 226
A Survey of Distance and Similarity Measures Used Within Network Intrusion Anomaly Detection
Weller-Fahy, D. J., Borghetti, B. J., & Sodemann, A. A. (2015) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2336610 · Cited by 216
Ambiguities inherent in sums-of-squares-based error statistics
Willmott, C. J., Matsuura, K., & Robeson, S. M. (2009) · Atmospheric Environment · doi:10.1016/j.atmosenv.2008.10.005 · Cited by 212
Erratum: A better measure of relative prediction accuracy for model selection and model estimation
Tofallis, C. (2015) · Journal of the Operational Research Society · doi:10.1057/jors.2014.124 · Cited by 208
Assessing the accuracy of predictive models for numerical data: Not r nor r2, why not? Then what?
Li, J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0183250 · Cited by 193
Botchkarev, A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3177507 · Cited by 166
Self-citationThe evolution of sales forecasting management: a 20-year longitudinal study of forecasting practices
McCarthy, T. M., Davis, D. F., Golicic, S. L., & Mentzer, J. T. (2006) · Journal of Forecasting · doi:10.1002/for.989 · Cited by 155
A Taxonomy of Similarity Mechanisms for Case-Based Reasoning
Cunningham, P. (2008) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.227 · Cited by 150
Heyer, H. (1982) · Springer series in statistics · doi:10.1007/978-1-4613-8218-8_7 · Cited by 147
Carbone, R., & Armstrong, J. S. (1982) · Journal of Forecasting · doi:10.1002/for.3980010207 · Cited by 138
Improving Energy Use Forecast for Campus Micro-grids Using Indirect Indicators
Aman, S., Simmhan, Y. L., & Prasanna, V. K. (2011) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2011.95 · Cited by 70
Mean-based error measures for intermittent demand forecasting
Prestwich, S., Rossi, R., Tarim, Ş. A., & Brahim Hnich, B. (2014) · International Journal of Production Research · doi:10.1080/00207543.2014.917771 · Cited by 61
Holistic Measures for Evaluating Prediction Models in Smart Grids
Aman, S., Simmhan, Y. L., & Prasanna, V. K. (2014) · NOT FOUND REPOSITORY (Indian Institute of Science Bangalore) · Cited by 38
Forecasting Multivariate Time Series with the Theta Method
Thomakos, D. D., & Nikolopoulos, K. I. (2015) · Journal of Forecasting · doi:10.1002/for.2334 · Cited by 32
Hernández–Rivera, E., Coleman, S. P., & Tschopp, M. A. (2016) · ACS Combinatorial Science · doi:10.1021/acscombsci.6b00142 · Cited by 28
Martín, F., Moreno, L. E., Garrido, S., & Blanco, D. (2015) · Sensors · doi:10.3390/s150923431 · Cited by 11
Cited by
Chicco, D., Warrens, M. J., & Jurman, G. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.623 · Cited by 5,380
Naser, M. Z., & Alavi, A. H. (2023) · Architecture, Structures and Construction · doi:10.1007/s44150-021-00015-8 · Cited by 464
A Deep Learning Framework for Optimization of MISO Downlink Beamforming
Xia, W., Zheng, G., Zhu, Y., Zhang, J., Wang, J., & Petropulu, A. P. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2019.2960361 · Cited by 345
“On the contrary, the MSE loss function is highly sensitive to outliers in the dataset because the MSE function tries to adjust the model according to these outlier values, at the expense of other samples [50].”
BackgroundTapeh, A. T. G., & Naser, M. Z. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-022-09793-w · Cited by 320
Chaves, D., Fidalgo, E., Alegre, E., Alaiz-Rodríguez, R., Jáñez-Martino, F., & Azzopardi, G. (2020) · Sensors · doi:10.3390/s20164491 · Cited by 286
Metadata
- Authors
- Botchkarev, A.
- Venue
- Interdisciplinary Journal of Information, Knowledge, and Management
- Publisher
- Informing Science Institute
- Published
- 2019
- Volume
- 14
- Pages
- 045-076
- ISSN
- 1555-1229 (ISSN-L)
- 1555-1237
- OpenAlex
- W3105640742
- Cited by
- 616
- References
- 46
- FWCI
- 59.98
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.