The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
The impact of class imbalance in classification performance metrics based on the binary confusion matrix
Open accessAbstract
A major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain other well-known performance metrics. In this paper, our approach goes beyond simply studying case studies and develops a systematic analysis of this impact by simulating the results obtained using binary classifiers. A set of functions and numerical indicators are attained which enables the comparison of the behaviour of several performance metrics based on the binary confusion matrix when they are faced with imbalanced datasets. Throughout the paper, a new way to measure the imbalance is defined which surpasses the Imbalance Ratio used in previous studies. From the simulation results, several clusters of performance metrics have been identified that involve the use of Geometric Mean or Bookmaker Informedness as the best null-biased metrics if their focus on classification successes (dismissing the errors) presents no limitation for the specific application where they are used. However, if classification errors must also be considered, then the Matthews Correlation Coefficient arises as the best choice. Finally, a set of null-biased multi-perspective Class Balance Metrics is proposed which extends the concept of Class Balance Accuracy to other performance metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007735 · Cited by 4,203
A Review on Evaluation Metrics for Data Classification Evaluations
M, H., & M.N, S. (2015) · International Journal of Data Mining & Knowledge Management Process · doi:10.5121/ijdkp.2015.5201 · Cited by 2,693
Learning from imbalanced data: open challenges and future directions
Krawczyk, B. (2016) · Progress in Artificial Intelligence · doi:10.1007/s13748-016-0094-0 · Cited by 2,459
Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation
Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11941439_114 · Cited by 1,974
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816
Selecting and interpreting measures of thematic classification accuracy
Stehman, S. V. (1997) · Remote Sensing of Environment · doi:10.1016/s0034-4257(97)00083-7 · Cited by 1,729
International Encyclopedia of Statistical Science
(2011) · Springer eBooks · doi:10.1007/978-3-642-04898-2 · Cited by 1,653
López, V., Fernández, A., García, S., Palade, V., & Herrera, F. (2013) · Information Sciences · doi:10.1016/j.ins.2013.07.007 · Cited by 1,590
Data Mining and Knowledge Discovery Handbook
(2005) · Springer eBooks · doi:10.1007/b107408 · Cited by 1,548
Cost-sensitive boosting for classification of imbalanced data
Sun, Y., Kamel, M. S., Wong, A. K. C., & Wang, Y. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2007.04.009 · Cited by 1,423
Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric
Boughorbel, S., Jarray, F., & El-Anbari, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177678 · Cited by 1,222
Jolliffe, I. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat06472 · Cited by 1,221
Data Mining for Imbalanced Datasets: An Overview
Chawla, N. V. (2009) · Data Mining and Knowledge Discovery Handbook · doi:10.1007/978-0-387-09823-4_45 · Cited by 1,082
An experimental comparison of performance measures for classification
Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.010 · Cited by 889
Ten quick tips for machine learning in computational biology
Chicco, D. (2017) · BioData Mining · doi:10.1186/s13040-017-0155-3 · Cited by 878
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & De La Torre, F. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.47 · Cited by 790
Weighted extreme learning machine for imbalance learning
Zong, W., Huang, G.-B., & Chen, Y. (2013) · Neurocomputing · doi:10.1016/j.neucom.2012.08.010 · Cited by 743
Comparing two K-category assignments by a K-category correlation coefficient
Gorodkin, J. (2004) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2004.09.006 · Cited by 545
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406
Amin, A., Anwar, S., Adnan, A., Nawaz, M., Howard, N., Qadir, J., Hawalah, A., & Hussain, A. (2016) · IEEE Access · doi:10.1109/access.2016.2619719 · Cited by 310
Data mining in metric space: an empirical analysis of supervised learning performance criteria
Caruana, R., & Niculescu-Mizil, A. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014063 · Cited by 303
Addressing imbalance in multilabel classification: Measures and random resampling algorithms
Charte, F., Rivera, A. J., del Jesus, M. J., & Herrera, F. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.08.091 · Cited by 288
Evaluation Measures of the Classification Performance of Imbalanced Data Sets
Gu, Q., Zhu, L., & Cai, Z. (2009) · Communications in Computer and Information Science · doi:10.1007/978-3-642-04962-0_53 · Cited by 254
The geometry of ROC space: understanding machine learning metrics through ROC isometrics
Flach, P. A. (2003) · Cited by 251
An insight into imbalanced Big Data classification: outcomes and challenges
Fernández, A., del Río, S., Chawla, N. V., & Herrera, F. (2017) · Complex & Intelligent Systems · doi:10.1007/s40747-017-0037-9 · Cited by 230
Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions
García, V., Mollineda, R. A., & Sánchez, J. S. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02172-5_57 · Cited by 214
Yuan, X., Xie, L., & Abouelenien, M. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.12.017 · Cited by 185
Classifying imbalanced data sets using similarity based hierarchical decomposition
Beyan, C., & Fisher, R. B. (2014) · Pattern Recognition · doi:10.1016/j.patcog.2014.10.032 · Cited by 177
EVALUATION OF CLASSIFIERS FOR AN UNEVEN CLASS DISTRIBUTION PROBLEM
Daskalaki, S., Kopanas, I., & Avouris, N. (2006) · Applied Artificial Intelligence · doi:10.1080/08839510500313653 · Cited by 169
A balanced approach to the multi-class imbalance problem
Mosley, L. (2013) · Iowa State University Digital Repository (Iowa State University) · doi:10.31274/etd-180810-3375 · Cited by 145
Pattern Recognition, 2008. ICPR 2008. 19th International Conference on
Belle, V., Deselaers, T., & Schiffer, S. (2008) · Cited by 145
Generic performance measure for multiclass-classifiers
Kautz, T., Eskofier, B. M., & Pasluosta, C. F. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.03.008 · Cited by 95
Relevance-Based Evaluation Metrics for Multi-class Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-57454-7_54 · Cited by 75
Improving SVM Classification on Imbalanced Datasets by Introducing a New Bias
Núñez, H., Abril, L. G., & Ángulo, C. (2017) · Journal of Classification · doi:10.1007/s00357-017-9242-x · Cited by 41
Enhancing Learning using Feature and Example selection
Raman, B., & Ioerger, T. R. (2003) · Cited by 36
Cited by
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Khan, A. A., Chaudhari, O., & Chandra, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122778 · Cited by 530
Cross validation for model selection: A review with examples from ecology
Yates, L. A., Aandahl, Z., Richards, S. A., & Brook, B. W. (2023) · Ecological Monographs · doi:10.1002/ecm.1557 · Cited by 459
Pick and Choose: A GNN-based Imbalanced Learning Approach for Fraud Detection
Liu, Y., Ao, X., Qin, Z., Chi, J., Feng, J., Yang, H., & He, Q. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449989 · Cited by 446
Metadata
- Authors
- Luque, A., Carrasco, A., Martín, A., & de las Heras, A.
- Venue
- Pattern Recognition
- Publisher
- Elsevier BV
- Published
- 2019-07
- Volume
- 91
- Pages
- 216-231
- ISSN
- 0031-3203 (ISSN-L)
- OpenAlex
- W2918408501
- Cited by
- 1,128
- References
- 42
- FWCI
- 52.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.