A Review on Evaluation Metrics for Data Classification Evaluations
Open accessAbstract
Evaluation metric plays a critical role in achieving the optimal classifier during the classification training.Thus, a selection of suitable evaluation metric is an important key for discriminating and obtaining the optimal classifier.This paper systematically reviewed the related evaluation metrics that are specifically designed as a discriminator for optimizing generative classifier.Generally, many generative classifiers employ accuracy as a measure to discriminate the optimal solution during the classification training.However, the accuracy has several weaknesses which are less distinctiveness, less discriminability, less informativeness and bias to majority class data.This paper also briefly discusses other metrics that are specifically designed for discriminating the optimal solution.The shortcomings of these alternative metrics are also discussed.Finally, this paper suggests five important aspects that must be taken into consideration in constructing a new discriminator metric.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evaluation metrics adopted within DL tasks play a crucial role in achieving the optimized classifier.
“Evaluation metrics adopted within DL tasks play a crucial role in achieving the optimized classifier [335].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [335]
The Introduction and §2 say evaluation metrics help select an optimal solution during classifier training. The paper focuses on data classification, particularly prototype-selection classifiers; these passages do not establish the claim specifically for deep-learning (DL) tasks. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515 · see citing works
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
Information Theory, Inference, and Learning Algorithms
(2004) · Kybernetes · doi:10.1108/03684920410534506 · Cited by 4,937 · see citing works
An Introduction to Data Mining
Stephens, C. R., & Sukumar, R. (2006) · Sage eBooks · doi:10.4135/9781412973380.n22 · Cited by 4,614 · see citing works
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148 · see citing works
Cost curves: An improved method for visualizing classifier performance
Drummond, C., & Holte, R. C. (2006) · Machine Learning · doi:10.1007/s10994-006-8199-5 · Cited by 329 · see citing works
Evaluation Measures of the Classification Performance of Imbalanced Data Sets
Gu, Q., Zhu, L., & Cai, Z. (2009) · Communications in computer and information science · doi:10.1007/978-3-642-04962-0_53 · Cited by 263 · see citing works
The geometry of ROC space: understanding machine learning metrics through ROC isometrics
Flach, P. A. (2003) · Cited by 254 · see citing works
Nearest prototype classification: clustering, genetic algorithms, or random search?
Kuncheva, L. I., & Bezdek, J. C. (1998) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/5326.661099 · Cited by 215 · see citing works
ROC curve, lift chart and calibration plot
Vuk, M., & Curk, T. (2006) · Advances in Methodology and Statistics · doi:10.51936/noqf3710 · Cited by 178 · see citing works
Wilson, S. W. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44640-0_11 · Cited by 175 · see citing works
Assessment Metrics for Imbalanced Learning
Japkowicz, N. (2013) · doi:10.1002/9781118646106.ch8 · Cited by 161 · see citing works
Rosset, S. (2004) · doi:10.1145/1015330.1015400 · Cited by 109 · see citing works
On evaluating performance of classifiers for rare classes
Joshi, M. V. (2003) · Proceedings · doi:10.1109/icdm.2002.1184018 · Cited by 79 · see citing works
A cooperative coevolutionary algorithm for instance selection for instance-based learning
García‐Pedrajas, N., Romero-del-Castillo, J. A., & Ortíz-Boyer, D. (2009) · Machine Learning · doi:10.1007/s10994-009-5161-3 · Cited by 75 · see citing works
Optimized Precision - A New Measure for Classifier Performance Evaluation
Ranawana, R., & Palade, V. (2006) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2006.1688586 · Cited by 70 · see citing works
Optimal reference subset selection for nearest neighbor classification by tabu search
Zhang, H., & Sun, G. (2002) · Pattern Recognition · doi:10.1016/s0031-3203(01)00137-6 · Cited by 53 · see citing works
Constructing new and better evaluation measures for machine learning
Huang, J., & Ling, C. X. (2007) · Cited by 37 · see citing works
García, V., Mollineda, R. A., & Sánchez, J. S. (2014) · Knowledge-Based Systems · doi:10.1016/j.knosys.2014.01.021 · Cited by 30 · see citing works
Rare Class Mining: Progress and Prospect
Han, S., Yuan, B., & Liu, W. (2009) · Chinese Conference on Pattern Recognition · doi:10.1109/ccpr.2009.5344137 · Cited by 25 · see citing works
A hybrid evaluation metric for optimizing classifier
Hossin, M. E., Sulaiman, M. N., Mustapha, A., Mustapha, N., & Rahmat, W. (2011) · Conference on Data Mining and Optimization (DMO) · doi:10.1109/dmo.2011.5976522 · Cited by 24 · see citing works
Generic Methods for Multi-criteria Evaluation
Lavesson, N., & Davidsson, P. (2008) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972788.49 · Cited by 19 · see citing works
B-ROC curves for the assessment of classifiers over imbalanced data sets
Cárdenas, Á. A., & Baras, J. S. (2006) · Cited by 11 · see citing works
A Novel Performance Metric for Building an Optimized Classifier
Hossin (2011) · Journal of Computer Science · doi:10.3844/jcssp.2011.582.590 · Cited by 7 · see citing works
Hossin, M. E., Sulaiman, M. N., Mustapha, A., et al. (2011) · Journal of Artificial Intelligence · doi:10.3923/jai.2011.187.196 · Cited by 5 · see citing works
Precision and Recall in Rough Support Vector Machines
Lingras, P. J., & Butz, C. J. (2007) · IEEE International Conference on Granular Computing (GrC) · doi:10.1109/grc.2007.4403181 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- R. Caruana and A. Niculescu-Mizil, "Data mining in metric space: an empirical analysis of supervised learning performance criteria", in Proc. of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '04), New York, NY, USA, ACM 2004, pp. 69-78.
- N.V. Chawla, N. Japkowicz and A. Kolcz, "Editorial: Special issue on learning from imbalanced data sets", SIGKDD Explorations, 6 (2004) 1-6.
- S. Garcia and F. Herrera, "Evolutionary training set selection to optimize C4.5 in imbalance problems", in Proc. of 8th Int. Conference on Hybrid Intelligent Systems (HIS 2008), Washington, DC, USA, IEEE Computer Society, 2008, pp.567-572.
- Q. Gu, L. Zhu and Z. Cai, "Evaluation Measures of the Classification Performance of Imbalanced Datasets", in Z. Cai et al. (Eds.) ISICA 2009, CCIS 51. Berlin, Heidelberg: Springer-Verlag, 2009, pp. 461-471.
- T. Kohonen, Self-Organizing Maps, 3rd ed., Berlin Heidelberg: Springer-Verlag, 2001.
- R. Prati, G. Batista, and M. Monard, "A survery on graphical methods for classification predictive performance evaluation", IEEE Trans. Knowl. Data Eng. 23(2011) 1601-1618.
- A. Rakotomamonyj, "Optimizing area under ROC with SVMs", in J. Hernandez-Orallo, C. Ferri, N. Lachiche and P. A. Flach (Eds.) 1st Int. Workshop on ROC Analysis in Artificial Intelligence (ROCAI 2004), Valencia, Spain, 2004, pp. 71-80.
- D.B. Skalak, "Prototype and feature selection by sampling and random mutation hill climbing algorithm", in W. W. Cohen and H. Hirsh (Eds.) Proc. of the 11th Int. Conference on Machine Learning (ICML 1994), New Brunswick, NJ, USA: Morgan Kaufmann, 1994, pp.293-301.
Cited by
Sorted by citations
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
Attack and anomaly detection in IoT sensors in IoT sites using machine learning approaches
Hasan, M., Islam, M. M., Zarif, M. I. I., & Hashem, M. M. A. (2019) · Internet of Things · doi:10.1016/j.iot.2019.100059 · Cited by 841 · see citing works
Facial Expression Recognition: A Survey
Huang, Y., Chen, F., Lv, S., & Wang, X. (2019) · Symmetry · doi:10.3390/sym11101189 · Cited by 195 · see citing works
Huggins, C. J., Escudero, J., Parra, M. A., Scally, B., Anghinah, R., Vitória Lacerda De Araújo, A., Basile, L. F., & Abasolo, D. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac05d8 · Cited by 105 · see citing works
Systematic review of machine learning in PTSD studies for automated diagnosis evaluation
Wu, Y., Mao, K., Dennett, L., Zhang, Y., & Chen, J. (2023) · npj Mental Health Research · doi:10.1038/s44184-023-00035-w · Cited by 30 · see citing works
One-Class Anomaly Detection for Industrial Applications: A Comparative Survey and Experimental Study
Paolini, D., Dini, P., Soldaini, E., & Saponara, S. (2025) · Computers · doi:10.3390/computers14070281 · Cited by 21 · see citing works
An ensemble model for predicting dispositions of emergency department patients
Kuo, K.-M., Lin, Y.-L., Chang, C. S., & Kuo, T. J. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02503-5 · Cited by 10 · see citing works
Ismailov, M., Tsikerdekis, M., & Zeadally, S. (2020) · Future Internet · doi:10.3390/fi12090148 · Cited by 6 · see citing works
Anomaly Detection Method Considering PLC Control Logic Structure for ICS Cyber Threat Detection
Lee, J. H., Ji, I. H., Jeon, S. H., & Seo, J. T. (2025) · Applied Sciences · doi:10.3390/app15073507 · Cited by 5 · see citing works
- Authors
- M, H., & M.N, S.
- Venue
- International Journal of Data Mining & Knowledge Management Process
- Published
- 2015-03-31
- Volume
- 5
- Issue
- 2
- Pages
- 01-11
- ISSN
- 2230-9608 (ISSN-L)
- 2231-007X
- OpenAlex
- W2330219538
- Cited by
- 2,693
- References
- 27
- Citations by year
- 2026: 123 · 2025: 459 · 2024: 539 · 2023: 487 · 2022: 436 · 2021: 312 · 2020: 198 · 2019: 98 · 2018: 32 · 2017: 7 · 2016: 1 · 2015: 1
- FWCI
- 13.38
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Journal of Data Mining & Knowledge Management Process (publishedVersion)
- Funders
- Universiti Malaysia Sarawak
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- classification evaluation metrics, generative classifiers, classifier optimization, majority class bias, metric design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
