The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.
Why Cohen’s Kappa should be avoided as performance measure in classification
Open accessAbstract
We show that Cohen's Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Cohen’s Kappa is an useful metric for imbalanced datasets; however, the interpretation of its value is problematic [84].”
From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [84]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,543
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,346
Measures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 12,187
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
(2005) · PubMed · Cited by 7,011
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 2×
“Matthews Correlation Coefficient MCC was first introduced in the binary case by B.W. Matthews [4] to assess the performance of protein secondary structure prediction, as the ϕ -coefficient, which is the measure of association obtained by discretization of the Pearson’s correlation coefficient for…”
MethodALLOUCHE, O., TSOAR, A., & KADMON, R. (2006) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2006.01214.x · Cited by 5,172
Flagged first, then by citations
2 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- M. Kuhn, 2008, Caret package, Journal of statistical software, 28, 1
- X. Zhao, 2013, Communication Yearbook, 419
Cited by
Sorted by citations
Items 301–324 of 324
Hahn, L., Glöckner, A., Gollwitzer, M., Hellmann, J. H., Lange, J., Schindler, S., & Sassenberg, K. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357568 · Cited by 2
de la Cruz, S. A., Arenas-Hernández, P. E., Fernández-Herrera, M. F., Quiñones-Díaz, R. A., Llaca-Díaz, J. M., Díaz‐Chuc, E. A., Robles-Espino, D. G., & Miguel-Garay, E. A. S. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14186676 · Cited by 1
Almehrizi, R. S. (2025) · Educational and Psychological Measurement · doi:10.1177/00131644251380540
A framework for binary classification evaluation metrics
Shirdel, M., Mauro, M. D., & Liotta, A. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123540
Analyzing Noise Data From a Noise Audit: A Guide for Industrial‐Organizational Practitioners
Nordmo, M., & Nordmo, M. (2025) · International Journal of Selection and Assessment · doi:10.1111/ijsa.70034
Assessment of transformer-based AI in clinical oncology
Seong, L. W., Lió, P., Taib, N. A. M., Ganggayah, M. D., & Dhillon, S. K. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03306-y · Cited by 1
Diogo, V. S., Ferreira, H. A., & Prata, D. P. (2022) · Figshare · doi:10.6084/m9.figshare.20430647
Additional file 1 of Interrater reliability estimators tested against true interrater reliabilities
Zhao, X., Feng, G. C., Ao, S. H., & Liu, P. L. (2022) · Figshare · doi:10.6084/m9.figshare.20727709
Harahap, Z. A., Bengen, D. G., Nurjaya, I. W., Agus, S. B., & Naulita, Y. (2026) · Regional Studies in Marine Science · doi:10.1016/j.rsma.2026.104748
Fault Diagnosis for IP-Based Networks Using Incremental Learning Algorithms and Data Stream Methods
Vargas-Arcila, A. M., Rodriguez-Vivas, A., Corrales, J. C., Sanchis, A., & Gallón, Á. R. (2026) · Technologies · doi:10.3390/technologies14020132
Kratzer, L., Knoblauch, H., Powers, A. D., Katrinli, Ş., Michopoulos, V., Fani, N., Gillespie, C. F., Jovanović, T., Ressler, K. J., Smith, A. D., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-42512-0
Loggenberg, K., Strever, A. E., & Münch, Z. (2026) · Precision Agriculture · doi:10.1007/s11119-026-10333-w
Karasiak, N. (2020) · theses.fr (ABES) · doi:10.70675/2bd6267dze870z47f5z84dcz624eefe9d4af
Principles Over Preferences: Quality-Aware Metric Selection for Machine Learning Systems
Grillmeyer, D., Jahn, V., Hadry, M., Lesch, V., & Kounev, S. (2026) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3777911.3800639
Scam detection between individuals with and without prior victimization
Rong, Z., & Khan, M. M. H. (2026) · Computers & Security · doi:10.1016/j.cose.2026.104961
Slope stability prediction using machine learning models
Geem, Z. W., Bekdaş, G., Ni̇gdeli, S. M., Aydın, Y., Işıkdağ, Ü., & Kim, T.-H. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-41481-7.00006-7
Yusufoğlu, E., Özçelik, S. T. A., Atıla, O., Guldemir, N. H., & Şengür, A. (2026) · Tomography · doi:10.3390/tomography12060076
Leal, M. A., Tovar, D., Hernández, M. Á. D. P., Bonilla, M. A., Sánchez, J., Goyanes, G., Molina, A., Leal, M., Manjarrés, I., Ríos, I., et al. (2026) · Acta Astronautica · doi:10.1016/j.actaastro.2026.05.065
Maw, A., Lupi, A. L., Johnson-Koenke, R. A., Coats, H., Zhou, L., Li, B. H., Mitchell, J., Kotz, A., Plasek, J. M., & Goss, F. R. (2026) · JMIR Formative Research · doi:10.2196/89837
Gunasekaran, H., Rajkumar, S., & Benjamin, L. K. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1880070
Lambert, P. J., Pitz, M. W., Singh, H., & Decker, K. (2026) · Discover Data · doi:10.1007/s44248-026-00119-w
Mostafa, W., Aldosari, A. A., Kheir, A. M. S., Abdelzaher, M. R., Chen, S., & El-Khalafy, M. M. (2026) · Frontiers in Environmental Science · doi:10.3389/fenvs.2026.1900117
Metadata
- Authors
- Delgado, R., & Tibau, X.-A.
- Affiliations
- Universitat Autònoma de Barcelona (ES)
- Corresponding author
- Delgado, Rosario
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-09-26
- Volume
- 14
- Issue
- 9
- Pages
- e0222916
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6762152
- OpenAlex
- W2975256032
- Cited by
- 324
- References
- 53
- Citations by year
- 2026: 22 · 2025: 47 · 2024: 53 · 2023: 60 · 2022: 69 · 2021: 50 · 2020: 22 · 2019: 1
- FWCI
- 16.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- Cohen's kappa, Matthews correlation coefficient, multi-class classification, class imbalance, confusion matrix, entropy (information theory), classifier comparison
- MeSH terms
- Supervised Machine Learning, Data Analysis, Classification, Models, Theoretical, Reproducibility of Results, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
