The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment
Open accessAbstract
Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F1score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen's Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen's Kappa and the Brier score to evaluate binary classifications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 5×
“Finally, κ can be undefined in some cases, but these cases can be handled with mathematical operations similar to the ones needed when MCC is undefined [3].”
BackgroundMethodResultInfluentialSelf-citationMathematical Methods of Statistics.
Geary, R. C., & Cramer, H. (1947) · The Economic Journal · doi:10.2307/2226151 · Cited by 5,927
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 3×
“The Matthews correlation coefficient (MCC) [1] is a case of the Cramér’s V [19] applied to a 2 × 2 traditional confusion matrix, having true positives (TP), true negatives (TN), false negatives (FN), and false positives (FP) (Equation 1).”
BackgroundMethodThe kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Taha, A. A., & Hanbury, A. (2015) · BMC Medical Imaging · doi:10.1186/s12880-015-0068-x · Cited by 2,920
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179 · cited 3×
“Nevertheless, multiple researchers have proposed alternative metrics for 2 × 2 confusion tables [54], [55], [58].”
BackgroundAssessing the accuracy of prediction algorithms for classification: an overview
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., & Nielsen, H. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.5.412 · Cited by 2,111
Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
Pontius, R. G., & Millones, M. (2011) · International Journal of Remote Sensing · doi:10.1080/01431161.2011.552923 · Cited by 2,046
High agreement but low kappa: II. Resolving the paradoxes
Cicchetti, D. V., & Feinstein, A. R. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90159-m · Cited by 1,838 · cited 4×
“Nevertheless, multiple researchers have proposed alternative metrics for 2 × 2 confusion tables [54], [55], [58].”
BackgroundMethodInfluentialA Method for Comparing Two Hierarchical Clusterings
Fowlkes, E. B., & Mallows, C. L. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10478008 · Cited by 1,614
Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric
Boughorbel, S., Jarray, F., & El-Anbari, M. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177678 · Cited by 1,222 · cited 1×
“Recently, Boughorbel and colleagues [13] described an enhanced classifier based on the Matthews correlation coefficient, while Zhu [14] investigated the behavior of MCC on several imbalanced cases.”
BackgroundSEQC/MAQC-III Consortium (2014) · Nature Biotechnology · doi:10.1038/nbt.2957 · Cited by 983 · cited 1×
“The Matthews correlation coefficient gained popularity when the US Food and Drug Administration (FDA) agency employed it as the main evaluation metric in the MicroArray / Sequencing Quality Control (MAQC/SEQC) comprehensive analyses in 2010 and 2014 [11], [12].”
MethodBeyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
Superforecasting: the art and science of prediction
(2016) · Choice Reviews Online · doi:10.5860/choice.196400 · Cited by 919
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Self-citationMAQC Consortium (2010) · Nature Biotechnology · doi:10.1038/nbt.1665 · Cited by 889
Ten quick tips for machine learning in computational biology
Chicco, D. (2017) · BioData Mining · doi:10.1186/s13040-017-0155-3 · Cited by 878
Self-citationProperties of the Hubert-Arable Adjusted Rand Index.
Steinley, D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.3.386 · Cited by 754
Comparing two K-category assignments by a K-category correlation coefficient
Gorodkin, J. (2004) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2004.09.006 · Cited by 545
Gerds, T. A., & Schumacher, M. (2006) · Biometrical Journal · doi:10.1002/bimj.200610301 · Cited by 519
Use of Brier score to assess binary predictions
Rufibach, K. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.009 · Cited by 454
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406
Self-citationProbabilistic prediction in patient management and clinical trials
Spiegelhalter, D. J. (1986) · Statistics in Medicine · doi:10.1002/sim.4780050506 · Cited by 358
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324
Cohen's kappa coefficient as a performance measure for feature selection
Vieira, S. M., Kaymak, U., & Sousa, J. M. C. (2010) · International Conference on Fuzzy Systems · doi:10.1109/fuzzy.2010.5584447 · Cited by 300
Meta-analysis of Cohen’s kappa
Sun, S. (2011) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-011-0077-3 · Cited by 282
Five Ways to Look at CohenâÂÂs Kappa
Warrens, M. J. (2015) · Journal of Psychology & Psychotherapy · doi:10.4172/2161-0487.1000197 · Cited by 280
Self-citationSample-size calculations for Cohen's kappa.
Cantor, A. B. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.150 · Cited by 240 · cited 1×
“Tables for sample size determination for a variety of common study designs involving Cohen’s Kappa can be found in a study of Cantor [45], and standard errors for Cohen’s Kappa can be found in works of Garner [46] and Shan and Wang [47].”
MethodA Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters
Berry, K. J., & Mielke, P. W. (1988) · Educational and Psychological Measurement · doi:10.1177/0013164488484007 · Cited by 239
About the relationship between ROC curves and Cohen's kappa
Ben-David, A. (2008) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2007.09.009 · Cited by 210
On the performance of Matthews correlation coefficient (MCC) for imbalanced dataset
Zhu, Q. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.03.030 · Cited by 202
Lai, Y.-H., Chen, W.-N., Hsu, T.-C., Lin, C., Tsao, Y., & Wu, S. (2020) · Scientific Reports · doi:10.1038/s41598-020-61588-w · Cited by 191
The dependence of Cohen's kappa on the prevalence does not matter
Vach, W. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.021 · Cited by 182
Hernández‐Orallo, J., Flach, P. A., & Ferri, C. (2012) · Bristol Research (University of Bristol) · Cited by 181
The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models
Assel, M. J., Sjoberg, D. D., & Vickers, A. J. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0020-3 · Cited by 171
On the Equivalence of Cohen’s Kappa and the Hubert-Arabie Adjusted Rand Index
Warrens, M. J. (2008) · Journal of Classification · doi:10.1007/s00357-008-9023-7 · Cited by 160
Self-citationKvålseth, T. O. (1989) · Psychological Reports · doi:10.2466/pr0.1989.65.1.223 · Cited by 138
Aickin, M. (1990) · Biometrics · doi:10.2307/2531434 · Cited by 122
A New Metric for Evaluating Semantic Segmentation: Leveraging Global and Contour Accuracy
Fernández-Moral, E., Martins, R., Wolf, D. F., & Rives, P. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500497 · Cited by 121
Interrater agreement statistics with skewed data: Evaluation of alternatives to Cohen’s kappa.
Xu, S., & Lorber, M. F. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0037489 · Cited by 120
Chicco, D., Старовойтов, В. В., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3068614 · Cited by 112
Self-citationComparing Probabilistic Forecasting Systems with the Brier Score
Ferro, C. A. T. (2007) · Weather and Forecasting · doi:10.1175/waf1034.1 · Cited by 109
Sampling Uncertainty and Confidence Intervals for the Brier Score and Brier Skill Score
Bradley, A., Schwartz, S., & Hashino, T. (2008) · Weather and Forecasting · doi:10.1175/2007waf2007049.1 · Cited by 101
Two Extra Components in the Brier Score Decomposition
Stephenson, D. B., Coelho, C. A. S., & Jolliffe, I. T. (2008) · Weather and Forecasting · doi:10.1175/2007waf2006116.1 · Cited by 98
Sampling distributions of the Brier score and Brier skill score under serial dependence
Wilks, D. S. (2010) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.709 · Cited by 87
The MCC-F1 curve: a performance evaluation technique for binary classificationPreprint
Cao, C., Chicco, D., & Hoffman, M. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11278 · Cited by 78
Self-citationSeparating the Brier Score into Calibration and Refinement Components: A Graphical Exposition
Blattenberger, G., & Lad, F. (1985) · The American Statistician · doi:10.1080/00031305.1985.10479382 · Cited by 75
On Generalizations Of The G Index And The Phi Coefficient To Nominal Scales
Janson, S., & Vegelius, J. (1979) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr1402_9 · Cited by 71
Cohen’s kappa is a weighted average
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.06.002 · Cited by 70
Self-citationA Formal Proof of a Paradox Associated with Cohen’s Kappa
Warrens, M. J. (2010) · Journal of Classification · doi:10.1007/s00357-010-9060-x · Cited by 67
Self-citationConditional inequalities between Cohen’s kappa and weighted kappas
Warrens, M. J. (2012) · Statistical Methodology · doi:10.1016/j.stamet.2012.05.004 · Cited by 62
Self-citationA critical analysis of metrics used for measuring progress in artificial intelligencePreprint
Blagec, K., Dorffner, G., Moradi, M., & Samwald, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02577 · Cited by 60
Performance targets and the Brier score
Roulston, M. (2007) · Meteorological Applications · doi:10.1002/met.21 · Cited by 56
New Interpretations of Cohen’s Kappa
Warrens, M. J. (2014) · Journal of Mathematics · doi:10.1155/2014/203907 · Cited by 49
Self-citationPrevalence threshold (ϕe) and the geometry of screening curves
Balayla, J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240215 · Cited by 45
Feature Selection via Cramer's V-Test Discretization for Remote-Sensing Image Classification
Wu, B., Zhang, L., & Zhao, Y. (2013) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2013.2263510 · Cited by 39
The standard error of Cohen's Kappa
Garner, J. B. (1991) · Statistics in Medicine · doi:10.1002/sim.4780100512 · Cited by 36
Casati, B., & Wilson, L. J. (2007) · Monthly Weather Review · doi:10.1175/mwr3442.1 · Cited by 34
Cohen’s kappa can always be increased and decreased by combining categories
Warrens, M. J. (2010) · Statistical Methodology · doi:10.1016/j.stamet.2010.05.003 · Cited by 32
Self-citationUse of the Bradley-Terry model to quantify association in remotely sensed images
Stein, A., Aryal, J., & Gort, G. (2005) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2005.843569 · Cited by 32
Steinijans, V. W., Diletti, E., Bömches, B., Greis, C., & Solleder, P. (1997) · PubMed · Cited by 31
Cohen's kappa for capturing discrimination
Chang, C. (2014) · International Health · doi:10.1093/inthealth/ihu010 · Cited by 31
Weighted kappa is higher than Cohen’s kappa for tridiagonal agreement tables
Warrens, M. J. (2010) · Statistical Methodology · doi:10.1016/j.stamet.2010.09.004 · Cited by 28
Self-citationA New Decomposition of the Brier Score: Formulation and Interpretation
Murphy, A. H. (1986) · Monthly Weather Review · doi:10.1175/1520-0493(1986)114<2671:andotb>2.0.co;2 · Cited by 28
Simon, P. (2006) · Educational and Psychological Measurement · doi:10.1177/0013164405285548 · Cited by 20
Assessing Binary Diagnoses of Bio-behavioral Disorders
Cicchetti, D. V., Klin, A., & Volkmar, F. R. (2016) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000598 · Cited by 18
DREAMTools: a Python package for scoring collaborative challenges
Cokelaer, T., Bansal, M., Bare, C., Bilal, E., Bot, B. M., Neto, E. C., Eduati, F., de la Fuente, A., Gönen, M., Hill, S. M., et al. (2016) · F1000Research · doi:10.12688/f1000research.7118.2 · Cited by 17
DREAMTools: a Python package for scoring collaborative challenges
Cokelaer, T., Bansal, M., Bare, C., Bilal, E., Bot, B. M., Neto, E. C., Eduati, F., Gönen, M., Hill, S. M., Hoff, B., et al. (2015) · F1000Research · doi:10.12688/f1000research.7118.1 · Cited by 17
Decomposition of the Brier score for weighted forecast‐verification pairs
Young, R. M. B. (2010) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.641 · Cited by 15
Variance estimation for Brier Score decomposition
Siegert, S. (2013) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.2228 · Cited by 14
Power Weighted Versions of Bennett, Alpert, and Goldstein’s S
Warrens, M. J. (2014) · Journal of Mathematics · doi:10.1155/2014/231909 · Cited by 13
Self-citationGenetic rule extraction optimizing brier score
Johansson, U., König, R., & Niklasson, L. (2010) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1830483.1830668 · Cited by 13
Fixed-effects modeling of Cohen’s weighted kappa for bivariate multinomial data
Yang, J., & Chinchilli, V. M. (2010) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2010.08.014 · Cited by 12
On Marginal Dependencies of the 2 × 2 Kappa
Warrens, M. J. (2014) · Advances in Statistics · doi:10.1155/2014/759527 · Cited by 11
Self-citationWeighted Brier score decompositions for topically heterogenous forecasting tournaments
Merkle, E. C., & Hartman, R. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500007099 · Cited by 11
Fixed-Effects Modeling of Cohen's Kappa for Bivariate Multinomial Data
Yang, J., & Chinchilli, V. M. (2009) · Communication in Statistics- Theory and Methods · doi:10.1080/03610920802618426 · Cited by 8
Weighted Brier score decompositions for topically heterogenous forecasting tournamentsPreprint
Merkle, E. C. (2018) · doi:10.31219/osf.io/p6wk5 · Cited by 6
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 2×
“The Brier score [74] is a strictly proper scoring function that is equivalent to the mean squared error : (worst value = 1; best value = 0) where N is the number of samples in the dataset, x i is the predicted value for the i th element and y i is the actual value of the i th element.”
BackgroundMethodVERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
Cited by
Chicco, D., Warrens, M. J., & Jurman, G. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.623 · Cited by 5,380
Self-citationAlzubaidi, L., Bai, J., Al-Sabaawi, A., Santamaría, J., Albahri, A. S., Al-dabbagh, B. S. N., Fadhel, M. A., Manoufali, M., Zhang, J., Al-Timemy, A. H., et al. (2023) · Journal of Big Data · doi:10.1186/s40537-023-00727-2 · Cited by 907
Chicco, D., & Jurman, G. (2023) · BioData Mining · doi:10.1186/s13040-023-00322-4 · Cited by 597
Self-citationCross validation for model selection: A review with examples from ecology
Yates, L. A., Aandahl, Z., Richards, S. A., & Brook, B. W. (2023) · Ecological Monographs · doi:10.1002/ecm.1557 · Cited by 459
Augmenting assessment with AI coding of online student discourse: A question of reliability
Misiejuk, K., Kaliisa, R., & Scianna, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100216 · Cited by 33
Metadata
- Authors
- Chicco, D., Warrens, M. J., & Jurman, G.
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 78368-78381
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3172444956
- Cited by
- 444
- References
- 83
- FWCI
- 23.29
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.