CoChat
Sign Up
Cited by 324Imbalanced Data Classification TechniquesMachine Learning and Data ClassificationReliability and Agreement in Measurement

The DOI matches its Crossref record; 51 of the 53 references it cites were found in a registry or an index.

Why Cohen’s Kappa should be avoided as performance measure in classification

Open access
Delgado, Rosario, Tibau, Xavier-Andoni·2019·PLOS ONE
DOI 10.1371/journal.pone.0222916PMC PMC6762152
Open this paper in CoChat — free account

Abstract

We show that Cohen's Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0222916

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Agreement between MCC and Kappa for C0. Unbalanced case with underrepresentation of the negative class, which is perfectly classified. (a) With a = d = 1, as function of b: positive class mainly misclassified. (b) With b = d = 1 as function of a: positive class mainly well classified.
Fig 1 Agreement between MCC and Kappa for C0. Unbalanced case with underrepresentation of the negative class, which is perfectly classified. (a) With a = d = 1, as function of b: positive class mainly misclassified. (b) With b = d = 1 as function of a: positive class mainly well classified.CC BY · PubMed Central
Fig 2 Disagreement between MCC and Kappa for C1,a^(a,b) with a = 0.2, as function of b ≥ 0. If b > 1, the negative class is underrepresented and quite misclassified, and the positive class is mainly misclassified. (a) A zoom of the detail for b ≤ 2. (b) For b ≤ 30.
Fig 2 Disagreement between MCC and Kappa for C1,a^(a,b) with a = 0.2, as function of b ≥ 0. If b > 1, the negative class is underrepresented and quite misclassified, and the positive class is mainly misclassified. (a) A zoom of the detail for b ≤ 2. (b) For b ≤ 30.CC BY · PubMed Central
Fig 3 Disagreement between MCC and Kappa for C1,0^(a,b) with a = 1, as function of b ≥ 0. If b > 1, the negative class is underrepresented and systematically misclassified, and the positive class is also mainly misclassified. (a) A zoom of the detail for b ≤ 2. (b) For b ≤ 30.
Fig 3 Disagreement between MCC and Kappa for C1,0^(a,b) with a = 1, as function of b ≥ 0. If b > 1, the negative class is underrepresented and systematically misclassified, and the positive class is also mainly misclassified. (a) A zoom of the detail for b ≤ 2. (b) For b ≤ 30.CC BY · PubMed Central
Fig 4 Disagreement between MCC and Kappa for C1,1^(a,b) with a = 0.2, as function of b ≥ 0. The negative class is classified at random. If b > 1 the positive class is mainly misclassified, and the negative class is underrepresented. (a) A zoom of the detail for b ≤ 2. (b) For b ≤ 30.
Fig 4 Disagreement between MCC and Kappa for C1,1^(a,b) with a = 0.2, as function of b ≥ 0. The negative class is classified at random. If b > 1 the positive class is mainly misclassified, and the negative class is underrepresented. (a) A zoom of the detail for b ≤ 2. (b) For b ≤ 30.CC BY · PubMed Central
Fig 5 Disagreement between MCC and Kappa for ZA, for different values of N. (a) N = 2, a zoom of the detail for A ≤ 5. (b) N = 2, A ≤ 100. (c) N = 5, A ≤ 500. (d) N = 10, A ≤ 1000.
Fig 5 Disagreement between MCC and Kappa for ZA, for different values of N. (a) N = 2, a zoom of the detail for A ≤ 5. (b) N = 2, A ≤ 100. (c) N = 5, A ≤ 500. (d) N = 10, A ≤ 1000.CC BY · PubMed Central
Fig 6 Experimental agreement between MCC and Kappa for M1(A). Increasing asymmetry but constant entropy.
Fig 6 Experimental agreement between MCC and Kappa for M1(A). Increasing asymmetry but constant entropy.CC BY · PubMed Central
Fig 7 Experimental disagreement between MCC and Kappa for M2(A). Decreasing to zero entropy, which implies increasing asymmetry.
Fig 7 Experimental disagreement between MCC and Kappa for M2(A). Decreasing to zero entropy, which implies increasing asymmetry.CC BY · PubMed Central
Fig 8 Experimental agreement between MCC and Kappa for M3(A). Decreasing entropy to a positive limit and constant asymmetry.
Fig 8 Experimental agreement between MCC and Kappa for M3(A). Decreasing entropy to a positive limit and constant asymmetry.CC BY · PubMed Central
Fig 9 Experimental disagreement between MCC and Kappa for M4(A). Entropy decreases to zero, which implies that asymmetry increases, for A increasing from 50 to 100, and from A decreasing from 50 to 0, by symmetry.
Fig 9 Experimental disagreement between MCC and Kappa for M4(A). Entropy decreases to zero, which implies that asymmetry increases, for A increasing from 50 to 100, and from A decreasing from 50 to 0, by symmetry.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. M. Kuhn, 2008, Caret package, Journal of statistical software, 28, 1
  2. X. Zhao, 2013, Communication Yearbook, 419

Cited by

Sorted by citations

See all 324

Related

See all

Metadata

Authors
Delgado, R., & Tibau, X.-A.
Affiliations
Universitat Autònoma de Barcelona (ES)
Corresponding author
Delgado, Rosario
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2019-09-26
Volume
14
Issue
9
Pages
e0222916
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0222916
PMC
PMC6762152
OpenAlex
W2975256032
Cited by
324
References
53
Citations by year
2026: 22 · 2025: 47 · 2024: 53 · 2023: 60 · 2022: 69 · 2021: 50 · 2020: 22 · 2019: 1
FWCI
16.20
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
Cohen's kappa, Matthews correlation coefficient, multi-class classification, class imbalance, confusion matrix, entropy (information theory), classifier comparison
MeSH terms
Supervised Machine Learning, Data Analysis, Classification, Models, Theoretical, Reproducibility of Results, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review