CoChat
Sign Up
Cited by 6,143Imbalanced Data Classification TechniquesMachine Learning and Data ClassificationStatistical Methods in Epidemiology

The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation

Open access
Chicco, Davide, Jurman, Giuseppe·2020·BMC Genomics
DOI 10.1186/s12864-019-6413-7PMC PMC6941312
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: To evaluate binary classifications and their confusion matrices, scientific researchers can employ several statistical rates, accordingly to the goal of the experiment they are investigating. Despite being a crucial issue in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F 1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on imbalanced datasets. Results: The Matthews correlation coefficient (MCC), instead, is a more reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset. Conclusions: In this article, we show how MCC produces a more informative and truthful score in evaluating binary classifications than accuracy and F 1 score, by first explaining the mathematical properties, and then the asset of MCC in six synthetic use cases and in a real genomics scenario. We believe that the Matthews correlation coefficient should be preferred to accuracy and F 1 score in evaluating binary classification tasks by all scientific communities.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper argues that the Matthews correlation coefficient (MCC) is a more reliable metric than accuracy and F1 score for evaluating binary classifications, especially on imbalanced datasets, because MCC requires good performance on both positive and negative classes. It supports this through mathematical analysis, six synthetic use cases, and a real colon cancer gene expression experiment.

Key findings

  • In six synthetic use cases, MCC was the only metric that correctly indicated the prediction problem in all six. For Use case A1 (91 positives, 9 negatives; TP=90, FN=1, TN=0, FP=9), accuracy=0.90 and F1=0.95, but MCC=−0.03. For Use case C2 (11 positives, 89 negatives; TP=2, FN=9, TN=88, FP=1), accuracy=0.90, F1=0.29, and MCC=+0.31.
  • Correlation analysis over all possible confusion matrices showed MCC and accuracy were strongly correlated, while F1 correlated less than 0.8 with the other two measures. For N=1000, PCC(MCC,F1)=0.775982, PCC(MCC,accuracy)=0.921030, and PCC(accuracy,F1)=0.778652.
  • In the colon cancer gene expression dataset (2,000 gene probesets, 62 patients: 22 healthy controls and 40 colon cancer; 35.48% negatives, 64.52% positives), rankings differed by metric. MCC ranked gradient boosting top (MCC=+0.55, TP rate=0.85, TN rate=0.69), while F1 and accuracy ranked k-NN top (F1=0.87, accuracy=0.80, TP rate=0.92, TN rate=0.52). Radial SVM had F1=0.75 and accuracy=0.67 but TN rate=0.40 and MCC=+0.29.
  • MCC produces a high score only if the classifier correctly predicts the majority of positive and negative instances; F1 is independent of true negatives and is not symmetric for class swapping, while accuracy can be overoptimistic on imbalanced datasets.

Methods

  • The paper defines accuracy, MCC, and F1 score mathematically, explores their relationships using Pearson correlation coefficient over all possible confusion matrices for N≥10 up to 1000, and plots all 21,084,251 possible confusion matrices for a 500-sample dataset on the MCC/F1 plane.
  • Six synthetic use cases vary class balance and imbalance with specified confusion matrices to compare metric behavior.
  • The real genomics experiment uses colon cancer gene expression data from Alon et al. via the plsgenomics R package, applying gradient boosting, decision tree, k-nearest neighbors, linear SVM, and radial SVM. Training/test splits were 80/20 or 60/20/20 with validation for hyperparameter grid search; experiments were repeated ten times and averaged for MCC, F1, accuracy, TP rate, and TN rate.

Limitations

  • The paper states that additional comparisons with other rates such as Cohen’s Kappa, Cramér’s V, and K measure would have provided further information, but the authors focused on accuracy and F1 because they are more commonly used in biomedical machine learning studies.
  • The paper notes that MCC can be undefined when a whole row or column of the confusion matrix is zero, and handles these cases via an approximation using an arbitrarily small value ε; it also cites situations where MCC can display large fluctuations due to imbalanced outcomes.
  • Evident from the text: the real-world validation uses a single colon cancer gene expression dataset, the six use cases are synthetic, and Table 5 excludes standard deviations to keep the table simple.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12864-019-6413-7

Cites

101 of 101 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 27 cited sources checked · 0 supported · 0 flagged

  • Findingreasoning

    MCC is undefined when a whole row or column of the confusion matrix M is zero.

    “A potential problem with MCC lies in the fact that MCC is undefined when a whole row or column of M is zero, as it happens in the previously cited case of the trivial majority classifier.”
    Quote in the paperSubstantial claim
  • Findingnone given

    F1 score is independent from the number of samples correctly classified as negative.

    “Second, F1 score is independent from the number of samples correctly classified as negative.”
    Quote in the paperSubstantial claim
  • Findingown data

    For the radial SVM, the F1 score was 0.75.

    “she/he would notice good results (F1 score = 0.75 and accuracy = 0.67)”
    Quote in the paperNot a substantial claim
  • Findingown data

    In Use case A1, the machine learning classifier was unable to correctly predict negative data instances, producing few true negatives.

    “For the Use case A1 (negatively imbalanced dataset), the machine learning classifier was unable to correctly predict negative data instances, and it therefore produced confusion matrices featuring few true negatives (TN).”
    Quote in the paperSubstantial claim
  • Findingown data

    The top classifier changes when the ranking is based on MCC, F1 score, or accuracy.

    “First, the top classifier changes when we consider the ranking based on MCC, F1 score, or accuracy.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 15
  • The quote only partly states this claim. 5
  • May combine two claims. 36

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Findingown data

    In Use case C2, with confusion matrix TP=2, FN=9, TN=88, FP=1, accuracy=0.90.

    “Here, accuracy showed a high value: 0.90 in the [0, 1] interval.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Use case A2, with confusion matrix TP=5, FN=70, TN=19, FP=6, accuracy=0.24, F1=0.12, and MCC=-0.24, all three rates showed a low score.

    “In this case, all three statistical rates showed a low score which emphasized the deficiency in the prediction process (accuracy = 0.24, F1 score = 0.12, and MCC = −0.24).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    MCC ranges in [-1,+1], with -1 and +1 reached in perfect misclassification and perfect classification respectively, while MCC=0 is the expected value for the coin tossing classifier.

    “It ranges in the interval [−1,+1], with extreme values –1 and +1 reached in case of perfect misclassification and perfect classification, respectively, while MCC=0 is the expected value for the coin tossing classifier.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    F1 ranges in [0,1], with minimum reached for TP=0 and maximum for FN=FP=0.

    “F1 ranges in [0,1], where the minimum is reached for TP=0, that is, when all the positive samples are misclassified, and the maximum for FN=FP=0, that is for perfect classification.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In Use case B2, with confusion matrix TP=10, FN=40, TN=46, FP=4, accuracy=0.56, F1=0.31, and MCC=+0.17, all three scores were average or low and correctly informed about non-optimal performance.

    “Like for the Use case A2, accuracy, F1 and MCC show average or low result scores (accuracy = 0.56, F1 score = 0.31, and MCC = +0.17), correctly informing you about the non-optimal performance of the prediction method (Additional file 3a).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    If M has only one non-zero entry, MCC=1 when all samples are correctly classified and MCC=-1 when all samples are incorrectly classified.

    “If M has only one non-zero entry, this means that all samples in the dataset belong to one class, and they are either all correctly (for TP≠0 or TN≠0) or incorrectly (for FP≠0 or FN≠0) classified. In this situations, MCC=1 for the former case and MCC=−1 for the latter case.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    For a highly unbalanced dataset, the trivial majority classifier yields accuracy approximately equal to the no-information error rate, so accuracy gives an unreliable estimation of the goodness of the classifier.

    “If the dataset is highly unbalanced, ni≈1, and thus the accuracy measure gives an unreliable estimation of the goodness of the classifier.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In Use case B1, with balanced 50/50 dataset and confusion matrix TP=47, FN=3, TN=5, FP=45, F1=0.66, MCC=+0.07, and accuracy=0.52.

    “Checking only F1, one would read a good value (0.66 in the [0, 1] interval), and would be overall satisfied about the prediction (Additional file 2a).”
    • Also stated in: § Use cases · hedged
      “The Matthews correlation coefficient, instead, by showing a score close to random guessing (+0.07 in the [–1, +1] interval) would be able to inform that the machine learning method has been on the wrong track. Also, it is worth noticing that accuracy would provide with an informative result in this case (0.52 in the [0, 1] interval).”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In Use case A1, with 9 negatives and 91 positives and confusion matrix TP=90, FN=1, TN=0, FP=9, accuracy=0.90 and F1=0.95, while MCC=-0.03.

    “However, accuracy and of F1 showed high values in this case: accuracy = 0.90 and F1 score = 0.95, both close to the best possible value 1.00 in the [0, 1] interval (Fig. 2a).”
    • Also stated in: § Use cases · hedged
      “Instead, if one decided to take advantage of the Matthews correlation coefficient in the Use case A1, he/she would notice the resulting MCC = –0.03 (Fig. 2a).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Use case C1, with 90 negatives and 10 positives and confusion matrix TP=9, FN=1, TN=1, FP=89, F1=0.17, accuracy=0.10, and MCC=-0.19, all three scores were low and reflected mediocre prediction quality.

    “Similar to the Use case A2 and B2, all three statistical scores generated low results that reflect the mediocre quality of the prediction: F1 score = 0.17 and accuracy = 0.10 in the [0, 1] interval, and MCC = −0.19 in the [–1, +1] interval (Additional file 4a).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    For a given MCC=μ, F1 can range in [0,μ+1] if μ≤0 and in [μ,1] if μ>0, so the width of the range is 1−|μ|.

    “On the other hand, for a given value MCC=μ, the F1 score can range in [0,μ+1] if μ≤0 and in [μ,1] if μ>0, so that the width of the range is 1−|μ|, that is, it depends on the MCC value μ.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    MCC and F1 are reasonably concordant, but the scatterplot cloud is wide, implying that for each F1 value there is a corresponding range of MCC values and vice versa, with different width.

    “The two measures are reasonably concordant, but the scatterplot cloud is wide, implying that for each value of F1 score there is a corresponding range of values of MCC and vice versa, although with different width.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Accuracy, F1, and MCC show reliable concordant scores for predictions that correctly classify both positives and negatives, and for predictions that incorrectly classify both; however, they show discordant behaviors when the prediction performs well with only one of the two binary classes.

    “Overall, accuracy, F1, and MCC show reliable concordant scores for predictions that correctly classify both positives and negatives (having therefore many TP and TN), and for predictions that incorrectly classify both positives and negatives (having therefore few TP and TN); however, these measures show discordant behaviors when the prediction performs well just with one of the two binary classes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    F1 varies for class swapping, while MCC is invariant if the positive class is renamed negative and vice versa.

    “First, F1 varies for class swapping, while MCC is invariant if the positive class is renamed negative and vice versa.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The correlation between these measures grows with N, but the increments are limited.

    “Interestingly, the correlation grows with N, but the increments are limited.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    With epsilon substitution, MCC is defined for all confusion matrices M; MCC=0 for the trivial majority classifier and 0 is also the expected value for the coin tossing classifier.

    “With these positions MCC is now defined for all confusion matrices M. As a consequences, MCC=0 for the trivial majority classifier, and 0 is also the expected value for the coin tossing classifier.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For N≥10, considering all possible confusion matrices for a dataset with N samples, MCC and accuracy are strongly correlated, while the Pearson coefficient is less than 0.8 for the correlation of F1 with the other two measures.

    “MCC and accuracy resulted strongly correlated, while the Pearson coefficient is less than 0.8 for the correlation of F1 with the other two measures (Table 3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    For any value F1=φ, MCC varies approximately between [φ−1,φ], so the width of the variability range is 1, independent from φ.

    “In fact, for any value F1=ϕ, the MCC varies approximately between [ϕ−1,ϕ], so that the width of the variability range is 1, independent from the value of ϕ.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Micro/macro averaged F1 is invariant for class swapping and its behaviour is more similar to MCC.

    “The micro/macro averaged F1 is invariant for class swapping and its behaviour is more similar to MCC.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In Use case C2, accuracy was 0.90 on the [0, 1] interval.

    “Here, accuracy showed a high value: 0.90 in the [0, 1] interval.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Linear SVM is ranked fourth in the MCC ranking and in the accuracy ranking, but ranked second in the F1 score ranking.

    “linear SVM is ranked forth in the MCC ranking and in the accuracy ranking, but ranked second in the F1 score ranking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    k-NN, the top performing method in both the F1 score ranking and the accuracy ranking, obtained an excellent score for recall (TP rate = 0.92) but just sufficient specificity (TN rate = 0.52).

    “k-NN, that is the top performing method both in the F1 score ranking and in the accuracy ranking, instead, obtained an excellent score for recall (TP rate = 0.92) but just sufficient on the specificity (TN rate = 0.52).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    ROC AUC presents several flaws and is sensitive to class imbalance.

    “However, ROC AUC presents several flaws [92], and it is sensitive to class imbalance [93].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [92] Lobo JM, Jiménez-Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Glob Ecol Biogeogr. 2008; 17(2):145–51. source

      Not checked in the free run.

    • cites [93] Hanczar B, Hua J, Sima C, Weinstein J, Bittner M, Dougherty ER. Small-sample precision of ROC-related estimates. Bioinformatics. 2010; 26(6):822–30. source

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    In Use case B1, F1 hides poor performance on the negative subset, MCC close to random guessing informs that the method was on the wrong track, and accuracy provides an informative result.

    “Once again, this score would hide the truth: the classification algorithm has performed poorly on the negative subset.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The effectiveness of MCC has been shown in other scientific fields.

    “The effectiveness of MCC has been shown in other scientific fields as well [43, 44].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [43] Liu Y, Cheng J, Yan C, Wu X, Chen F. Research on the Matthews correlation coefficients metrics of personalized recommendation algorithm evaluation. Int J Hybrid Informa Technol. 2015; 8(1):163–72. source

      Not checked in the free run.

    • cites [44] Naulaerts S, Dang CC, Ballester PJ. Precision and recall oncology: combining multiple gene mutations for improved identification of drug-sensitive tumours. Oncotarget. 2017; 8(57):97025. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several scientists consider the PR curve more informative than the ROC curve, especially on imbalanced biological and medical datasets.

    “Even if less common than the ROC curve, several scientists consider the PR curve more informative than the ROC curve, especially on imbalanced biological and medical datasets [48, 97, 98].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [48] Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE. 2015; 10(3):0118432. source

      Not checked in the free run.

    • cites [97] Chicco D. Ten quick tips for machine learning in computational biology. BioData Min. 2017; 10(35):1–17.

      Not checked in the free run.

    • cites [98] Ozenne B, Subtil F, Maucort-Boulch D. The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J Clin Epidemiol. 2015; 68(8):855–9. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    MCC was re-proposed by Baldi and colleagues in 2000 as a standard performance metric for machine learning with a natural extension to the multiclass case.

    “MCC was re-proposed by Baldi and colleagues [39] in 2000 as a standard performance metric for machine learning with a natural extension to the multiclass case [40].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [39] Baldi P, Brunak S, Chauvin Y, Andersen CA, Nielsen H. Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics. 2000; 16(5):412–24. source

      Not checked in the free run.

    • cites [40] Gorodkin J. Comparing two K-category assignments by a K-category correlation coefficient. Comput Biol Chem. 2004; 28(5–6):367–74. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Whalen and colleagues showed TargetFinder results measured only by F1 score, making it impossible to detect the actual true positive rate and true negative rate of their tests.

    “For example, when Whalen and colleagues released TargetFinder, a tool to predict enhancer-promoters interactions in genomics, they showed its results measured only by F1 score [13], making it impossible to detect the actual true positive rate and true negative rate of their tests [69].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [13] Whalen S, Truty RM, Pollard KS. Enhancer–promoter interactions are encoded by complex genomic signatures on looping chromatin. Nat Genet. 2016; 48(5):488. source

      Not checked in the free run.

    • cites [69] Xi W, Beer MA. Local epigenomic state cannot discriminate interacting and non-interacting enhancer–promoter pairs with high accuracy. PLoS Comput Biol. 2018; 14(12):1006625. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There are extreme situations where MCC cannot be defined or displays large fluctuations due to imbalanced outcomes in classification.

    “Although being widely acknowledged as a reliable metric, there are situations - albeit extreme - where either MCC cannot be defined or it displays large fluctuations [45], due to imbalanced outcomes in the classification.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Brown JB. Classifiers and their metrics quantified. Mol Inform. 2018; 37:1700127. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MCC is a special case of the phi coefficient.

    “An effective solution overcoming the class imbalance issue comes from the Matthews correlation coefficient (MCC), a special case of the ϕ phi coefficient [36].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] Guilford JP. Psychometric Methods. New York City: McGraw-Hill; 1954.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The FDA employed MCC as the main evaluation measure in the MAQC/SEQC projects.

    “for instance, the Food and Drug Administration (FDA) agency of the USA employed the MCC as the main evaluation measure in the MicroArray II / Sequencing Quality Control (MAQC/SEQC) projects [41, 42].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [41] The MicroArray Quality Control (MAQC) Consortium. The MAQC-II Project: a comprehensive study of common practices for the development and validation of microarray-based predictive models. Nat Biotechnol. 2010; 28(8):827–38. source

      Not checked in the free run.

    • cites [42] The SEQC/MAQC-III Consortium. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequence Quality Control consortium. Nat Biotechnol. 2014; 32:903–14. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The micro/macro averaged F1 procedure is biased and is not accepted as a standard practice by the community.

    “However, this procedure is biased [65], and it is still far from being accepted as a standard practice by the community.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [65] Van Asch V. Macro-and micro-averaged evaluation measures. Technical report. 2013:1–27.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    In Use case A1, the classifier predicted positive instances well but failed to identify healthy controls, so the overall performance should be judged poor.

    “In this case, the algorithm showed its ability to predict the positive data instances (90 sick patients out of 91 were correctly predicted), but it also displayed its lack of talent in identifying healthy controls (only 1 healthy individual out of 9 was correctly recognized) (Fig. 2b). Therefore, the overall performance should be judged poor.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    In Use case C2, the method correctly identified multiple negative instances but was unable to correctly predict most sick patients.

    “Similar to the Use case A1 and B1, the method was able to correctly identify multiple negative data instances (88 healthy patients out of 89), but unable to correctly predict most of sick patients (only 2 true positives out of 11 possible elements).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MCC was originally developed by Matthews in 1975 for comparison of chemical structures.

    “Originally developed by Matthews in 1975 for comparison of chemical structures [38]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] Matthews BW. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct. 1975; 405(2):442–51. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Cohen's kappa has issues in the learning context causing unreliable results, including high sensitivity to the distribution of marginal totals.

    “Despite its popularity, in the learning context there are a number of issues causing the kappa measure to produce unreliable results (for instance, its high sensitivity to the distribution of the marginal totals [73–75])”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [73] Flight L, Julious SA. The disagreeable behaviour of the kappa statistic. Pharm Stat. 2015; 14:74–8. source

      Not checked in the free run.

    • cites [74] Powers DMW. The problem with Kappa. In: Proceedings of EACL 2012 – the 13th Conference of the European Chapter of the Association for Computational Linguistics. Avignon: ACL: 2012. p. 345–55.

      Not checked in the free run.

    • cites [75] Delgado R, Tibau X-A. Why Cohen’s Kappa should be avoided as performance measure in classification. PloS ONE. 2019; 14(9):0222916.

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    When a prediction displays many true positives but few true negatives, or many true negatives but few true positives, F1 and accuracy can provide misleading information, while MCC always generates results that reflect the overall prediction issues.

    “In fact, when a prediction displays many true positives but few true negatives (or many true negatives but few true positives) we will show that F1 and accuracy can provide misleading information, while MCC always generates results that reflect the overall prediction issues.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data

    These results show that, while accuracy and F1 score often generate high scores that do not inform the user about ongoing prediction issues, MCC is a robust, useful, reliable, truthful statistical measure able to correctly reflect the deficiency of any prediction in any dataset.

    “These results show that, while accuracy and F1 score often generate high scores that do not inform the user about ongoing prediction issues, the MCC is a robust, useful, reliable, truthful statistical measure able to correctly reflect the deficiency of any prediction in any dataset.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    In Use case C2, accuracy generated overoptimistic, misleading, and inflated high results.

    “Here accuracy again generated overoptimistic, misleading, and inflated high results”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The F1 score ranking and the accuracy ranking hide the important flaw of the top classifier: k-NN was unable to correctly predict a high percentage of patients.

    “The F1 score ranking and the accuracy ranking, in conclusion, are hiding this important flaw of the top classifier: k-NN was unable top correctly predict a high percentage of patients.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    In Use case A1, accuracy and F1 generated overoptimistic and inflated results.

    “There, accuracy and F1 generated overoptimistic and inflated results”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Accuracy and F1 score can generate misleading results on imbalanced datasets, because they fail to consider the ratio between positive and negative elements; they generate reliable results only when applied to balanced datasets.

    “Accuracy and F1 score, although popular, can generate misleading results on imbalanced datasets, because they fail to consider the ratio between positive and negative elements.”
    • Also stated in: § Conclusions · plain
      “F1 and accuracy, instead, generate reliable results only when applied to balanced datasets, and produce misleading results when applied to imbalanced cases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    For the radial SVM, the low MCC value (+0.29) informs readers about the poor general outcome, while accuracy and F1 score show misleading values.

    “With its low value (MCC = +0.29), the MCC informs the readers about the poor general outcome of the radial SVM, while the accuracy and F1 score show misleading values.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The colon cancer gene expression dataset contains 2,000 gene probesets for 62 patients, of which 22 are healthy controls and 40 have colon cancer (35.48% negatives and 64.52% positives).

    “The dataset contains 2,000 gene probsets for 62 patients, of which 22 are healthy controls and 40 have colon cancer (35.48% negatives and 64.52% positives) [106].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [106] Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ. Data pertaining to the article ‘Broad patterns of gene expression revealed by clustering of tumor and normal colon tissues probed by oligonucleotide arrays’. 2000. http://genomics-pubs.princeton.edu/oncology/affydata/index.html. Accessed 10 Dec 2019.

      Not checked in the free run.

44 of 44

  • The paper relies on this without support.

    Accuracy and F1 score are the most common metrics used for binary classification in machine learning.

    “We decided to focus on accuracy and F1 score because they are the most common metrics used for binary classification in machine learning.”

  • The paper relies on this without support.

    The synthetic use cases are realistic representations of binary classification scenarios.

    “here we describe some synthetic, realistic scenarios where MCC results are more informative and truthful than the other two measures analyzed.”

  • The paper relies on this without support.

    Accuracy and F1 score are more commonly used in machine learning studies related to biomedical applications than other rates.

    “because accuracy and F1 score are more commonly used in machine learning studies related to biomedical applications.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • C1's accuracy of 0.10 is described as inflated

    Major

    The recap calls C1's accuracy overoptimistic and inflated, but the results explicitly give it as 0.10 and describe all three scores as low.

  • C2's prediction failure is assigned to the wrong class

    Minor

    The recap says C2 failed to predict negatives and had few true negatives. Its earlier account says it correctly identified 88 of 89 negatives but only 2 of 11 positives.

  • A2's low accuracy is described as excessively high

    Minor

    The recap says A2 accuracy was excessively high, contradicting the reported value of 0.24 and the earlier description of all three scores as low.

  • The best k-NN accuracy differs between text and table

    Minor

    The results text gives k-NN accuracy as 0.81, while every k-NN entry in Table 5 gives 0.80.

  • A1 reports one correctly identified healthy control despite TN=0

    Minor

    The A1 confusion matrix has zero true negatives, so no healthy controls were correctly recognized, not one.

  • The recap reverses C2's class imbalance

    Minor

    The recap calls C2 positively imbalanced, but Table 4 lists 11 positives and 89 negatives and labels it negatively imbalanced.

  • The recap reverses A1's class imbalance

    Minor

    The recap calls A1 negatively imbalanced, but its dataset has 91 positives and 9 negatives.

  • The recap reverses C1's class imbalance

    Minor

    The recap calls C1 positively imbalanced, whereas the dataset has 10 positives and 90 negatives.

  • A2 uses 70 instead of 75 as the number of positives

    Minor

    A2 has 5 true positives and 70 false negatives, making 75 sick patients in total, not 70.

  • A1 has two different reported MCC values

    Minor

    The text reports MCC as −0.03, while the figure description identifies the A1 point at −0.04.

10 of 10

Cites

Flagged first, then by citations

3 of the 104 references couldn’t be matched to a known paper.

Research works printed but not linked: 7

  1. Akosa JS. Predictive accuracy: a misleading performance measure for highly imbalanced data. In: Proceedings of the SAS Global Forum 2017 Conference. Cary, North Carolina: SAS Institute Inc.: 2017. p. 942–2017.
  2. Guilford JP. Psychometric Methods. New York City: McGraw-Hill; 1954.
  3. van Rijsbergen CJ, Joost C. Information Retrieval. New York City: Butterworths; 1979.
  4. Sasaki Y. The truth of the F-measure. Teach Tutor Mater. 2007; 1(5):1–5.
  5. Dubey A, Tarar S. Evaluation of approximate rank-order clustering using Matthews correlation coefficient. Int J Eng Adv Technol. 2018; 8(2):106–13.
  6. Suresh Babu N. Various performance measures in binary classification – An overview of ROC study. Int J Innov Sci Eng Technol. 2015; 2(9):596–605.
  7. Ferri C, Hernández-Orallo J, Flach PA. A coherent interpretation of AUC as a measure of aggregated classification performance. In: Proceedings of ICML 2011 – the 28th International Conference on Machine Learning. Norristown: Omnipress: 2011. p. 657–64.

Cited by

Sorted by citations

Related

Authors
Chicco, D., & Jurman, G.
Affiliations
  • Krembil Research Institute (CA)
  • Fondazione Bruno Kessler (IT)
Corresponding author
Chicco, Davide
ORCID
Venue
BMC Genomics
Publisher
Springer Science and Business Media LLC
Published
2020-01-02
Volume
21
Issue
1
Pages
6-6
ISSN
1471-2164 (ISSN-L)
DOI
10.1186/s12864-019-6413-7
PMC
PMC6941312
OpenAlex
W2999309192
Semantic Scholar
44858b662822d291a5e1e9e3501ad19a9f1572c2
Cited by
6,143
References
104
Citations by year
2026: 1118 · 2025: 1152 · 2024: 1060 · 2023: 1032 · 2022: 883 · 2021: 682 · 2020: 211 · 2019: 1 · 2016: 1
FWCI
276.55
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Genomics (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • BOA (University of Milano-Bicocca) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Iris (University of Trento) (submittedVersion)
Funders
University of Toronto
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
Matthews correlation coefficient, F1 score, accuracy, confusion matrix, class imbalance, classification metrics
MeSH terms
Machine Learning, Correlation of Data, Algorithms, Data Interpretation, Statistical, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review