The DOI matches its Crossref record; 101 of the 104 references it cites were found in a registry or an index; 2 of the 27 sources its claims cite were checked for support.
The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation
Open accessAbstract
Background: To evaluate binary classifications and their confusion matrices, scientific researchers can employ several statistical rates, accordingly to the goal of the experiment they are investigating. Despite being a crucial issue in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F 1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on imbalanced datasets. Results: The Matthews correlation coefficient (MCC), instead, is a more reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset. Conclusions: In this article, we show how MCC produces a more informative and truthful score in evaluating binary classifications than accuracy and F 1 score, by first explaining the mathematical properties, and then the asset of MCC in six synthetic use cases and in a real genomics scenario. We believe that the Matthews correlation coefficient should be preferred to accuracy and F 1 score in evaluating binary classification tasks by all scientific communities.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper argues that the Matthews correlation coefficient (MCC) is a more reliable metric than accuracy and F1 score for evaluating binary classifications, especially on imbalanced datasets, because MCC requires good performance on both positive and negative classes. It supports this through mathematical analysis, six synthetic use cases, and a real colon cancer gene expression experiment.
Key findings
- In six synthetic use cases, MCC was the only metric that correctly indicated the prediction problem in all six. For Use case A1 (91 positives, 9 negatives; TP=90, FN=1, TN=0, FP=9), accuracy=0.90 and F1=0.95, but MCC=−0.03. For Use case C2 (11 positives, 89 negatives; TP=2, FN=9, TN=88, FP=1), accuracy=0.90, F1=0.29, and MCC=+0.31.
- Correlation analysis over all possible confusion matrices showed MCC and accuracy were strongly correlated, while F1 correlated less than 0.8 with the other two measures. For N=1000, PCC(MCC,F1)=0.775982, PCC(MCC,accuracy)=0.921030, and PCC(accuracy,F1)=0.778652.
- In the colon cancer gene expression dataset (2,000 gene probesets, 62 patients: 22 healthy controls and 40 colon cancer; 35.48% negatives, 64.52% positives), rankings differed by metric. MCC ranked gradient boosting top (MCC=+0.55, TP rate=0.85, TN rate=0.69), while F1 and accuracy ranked k-NN top (F1=0.87, accuracy=0.80, TP rate=0.92, TN rate=0.52). Radial SVM had F1=0.75 and accuracy=0.67 but TN rate=0.40 and MCC=+0.29.
- MCC produces a high score only if the classifier correctly predicts the majority of positive and negative instances; F1 is independent of true negatives and is not symmetric for class swapping, while accuracy can be overoptimistic on imbalanced datasets.
Methods
- The paper defines accuracy, MCC, and F1 score mathematically, explores their relationships using Pearson correlation coefficient over all possible confusion matrices for N≥10 up to 1000, and plots all 21,084,251 possible confusion matrices for a 500-sample dataset on the MCC/F1 plane.
- Six synthetic use cases vary class balance and imbalance with specified confusion matrices to compare metric behavior.
- The real genomics experiment uses colon cancer gene expression data from Alon et al. via the plsgenomics R package, applying gradient boosting, decision tree, k-nearest neighbors, linear SVM, and radial SVM. Training/test splits were 80/20 or 60/20/20 with validation for hyperparameter grid search; experiments were repeated ten times and averaged for MCC, F1, accuracy, TP rate, and TN rate.
Limitations
- The paper states that additional comparisons with other rates such as Cohen’s Kappa, Cramér’s V, and K measure would have provided further information, but the authors focused on accuracy and F1 because they are more commonly used in biomedical machine learning studies.
- The paper notes that MCC can be undefined when a whole row or column of the confusion matrix is zero, and handles these cases via an approximation using an arbitrarily small value ε; it also cites situations where MCC can display large fluctuations due to imbalanced outcomes.
- Evident from the text: the real-world validation uses a single colon cancer gene expression dataset, the six use cases are synthetic, and Table 5 excludes standard deviations to keep the table simple.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 2 Use case A1 — Positively imbalanced dataset. a Barplot representing accuracy, F1, and normalized Matthews correlation coefficient (normMCC = (MCC + 1) / 2), all in the [0, 1] interval, where 0 is the worst possible score and 1 is the best possible score, applied to the Use case A1 positively imbalanced dataset. b Pie chart representing the amounts of true positives (TP), false negatives (FN), true negatives (TN), and false positives (FP). c Pie chart representing the dataset balance, as the amounts of positive data instances and negative data instances](/paper-figures/988371a13eb649a5bff767f781052a4f/the-advantages-of-the-matthews-correlation-coefficient-mcc-over-f1-score-and-figure-2.jpg)
Verification
Claims the paper makes
29 Own interpretation · 28 Own result · 14 Not checked yet · 13 No basis given · 1 Partly supported · accuracy not yet measured
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A potential problem with MCC lies in the fact that MCC is undefined when a whole row or column of M is zero, as it happens in the previously cited case of the trivial majority classifier.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Notation and mathematical foundations
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, F1 score is independent from the number of samples correctly classified as negative.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“she/he would notice good results (F1 score = 0.75 and accuracy = 0.67)”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the Use case A1 (negatively imbalanced dataset), the machine learning classifier was unable to correctly predict negative data instances, and it therefore produced confusion matrices featuring few true negatives (TN).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, the top classifier changes when we consider the ranking based on MCC, F1 score, or accuracy.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MCC = +0.31 in the [–1, +1] interval”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the Use case B1 (balanced dataset), the machine learning method was unable to correctly predict negative data instances, and therefore produced a confusion matrix featuring few true negatives (TN).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, in the last Use case C2 (positively imbalanced dataset), the prediction technique failed in predicting negative elements, and therefore its confusion matrix showed a low percentage of true negatives.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“F1 score = 0.29 in the [0, 1] interval”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the radial SVM has performed poorly on the true negatives (TN rate = 0.40), by correctly predicting less than half patients.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 15
- The quote only partly states this claim. 5
- It may state two separate claims as one. 36
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The effectiveness of MCC has been shown in other scientific fields as well [43, 44].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Partly supportedcites [43] Liu Y, Cheng J, Yan C, Wu X, Chen F. Research on the Matthews correlation coefficients metrics of personalized recommendation algorithm evaluation. Int J Hybrid Informa Technol. 2015; 8(1):163–72. sourceExperimental — accuracy not yet measured
The Introduction and the recommendation-algorithm results report that the authors tested MCCP, an MCC-based metric, and found it applicable to evaluating recommendation systems. This supports an example of effectiveness in another field, but the paper does not establish MCC’s effectiveness across multiple other scientific fields. (LLM, conf=0.82)
- Could not verifycites [44] Naulaerts S, Dang CC, Ballester PJ. Precision and recall oncology: combining multiple gene mutations for improved identification of drug-sensitive tumours. Oncotarget. 2017; 8(57):97025. sourceExperimental — accuracy not yet measured
The Materials and Methods section defines MCC as a measure of classifier performance, and the Results section uses it to assess predictions of drug sensitivity. Neither section reports that MCC has been shown effective in other scientific fields; searches for that claim found only its use in this oncology study. Page numbers were not available in our copy. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, accuracy showed a high value: 0.90 in the [0, 1] interval.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, all three statistical rates showed a low score which emphasized the deficiency in the prediction process (accuracy = 0.24, F1 score = 0.12, and MCC = −0.24).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It ranges in the interval [−1,+1], with extreme values –1 and +1 reached in case of perfect misclassification and perfect classification, respectively, while MCC=0 is the expected value for the coin tossing classifier.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Notation and mathematical foundations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“F1 ranges in [0,1], where the minimum is reached for TP=0, that is, when all the positive samples are misclassified, and the maximum for FN=FP=0, that is for perfect classification.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Notation and mathematical foundations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Like for the Use case A2, accuracy, F1 and MCC show average or low result scores (accuracy = 0.56, F1 score = 0.31, and MCC = +0.17), correctly informing you about the non-optimal performance of the prediction method (Additional file 3a).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If M has only one non-zero entry, this means that all samples in the dataset belong to one class, and they are either all correctly (for TP≠0 or TN≠0) or incorrectly (for FP≠0 or FN≠0) classified. In this situations, MCC=1 for the former case and MCC=−1 for the latter case.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Notation and mathematical foundations
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“If the dataset is highly unbalanced, ni≈1, and thus the accuracy measure gives an unreliable estimation of the goodness of the classifier.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Notation and mathematical foundations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Checking only F1, one would read a good value (0.66 in the [0, 1] interval), and would be overall satisfied about the prediction (Additional file 2a).”
- Also stated in: § Use cases · hedged
“The Matthews correlation coefficient, instead, by showing a score close to random guessing (+0.07 in the [–1, +1] interval) would be able to inform that the machine learning method has been on the wrong track. Also, it is worth noticing that accuracy would provide with an informative result in this case (0.52 in the [0, 1] interval).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Use cases · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, accuracy and of F1 showed high values in this case: accuracy = 0.90 and F1 score = 0.95, both close to the best possible value 1.00 in the [0, 1] interval (Fig. 2a).”
- Also stated in: § Use cases · hedged
“Instead, if one decided to take advantage of the Matthews correlation coefficient in the Use case A1, he/she would notice the resulting MCC = –0.03 (Fig. 2a).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Use cases · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to the Use case A2 and B2, all three statistical scores generated low results that reflect the mediocre quality of the prediction: F1 score = 0.17 and accuracy = 0.10 in the [0, 1] interval, and MCC = −0.19 in the [–1, +1] interval (Additional file 4a).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the other hand, for a given value MCC=μ, the F1 score can range in [0,μ+1] if μ≤0 and in [μ,1] if μ>0, so that the width of the range is 1−|μ|, that is, it depends on the MCC value μ.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The two measures are reasonably concordant, but the scatterplot cloud is wide, implying that for each value of F1 score there is a corresponding range of values of MCC and vice versa, although with different width.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Overall, accuracy, F1, and MCC show reliable concordant scores for predictions that correctly classify both positives and negatives (having therefore many TP and TN), and for predictions that incorrectly classify both positives and negatives (having therefore few TP and TN); however, these measures show discordant behaviors when the prediction performs well just with one of the two binary classes.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, F1 varies for class swapping, while MCC is invariant if the positive class is renamed negative and vice versa.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, the correlation grows with N, but the increments are limited.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“With these positions MCC is now defined for all confusion matrices M. As a consequences, MCC=0 for the trivial majority classifier, and 0 is also the expected value for the coin tossing classifier.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Notation and mathematical foundations
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MCC and accuracy resulted strongly correlated, while the Pearson coefficient is less than 0.8 for the correlation of F1 with the other two measures (Table 3).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In fact, for any value F1=ϕ, the MCC varies approximately between [ϕ−1,ϕ], so that the width of the variability range is 1, independent from the value of ϕ.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The micro/macro averaged F1 is invariant for class swapping and its behaviour is more similar to MCC.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, accuracy showed a high value: 0.90 in the [0, 1] interval.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“linear SVM is ranked forth in the MCC ranking and in the accuracy ranking, but ranked second in the F1 score ranking.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“k-NN, that is the top performing method both in the F1 score ranking and in the accuracy ranking, instead, obtained an excellent score for recall (TP rate = 0.92) but just sufficient on the specificity (TN rate = 0.52).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“However, ROC AUC presents several flaws [92], and it is sensitive to class imbalance [93].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [92] Lobo JM, Jiménez-Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Glob Ecol Biogeogr. 2008; 17(2):145–51. source
- Not checked yetcites [93] Hanczar B, Hua J, Sima C, Weinstein J, Bittner M, Dougherty ER. Small-sample precision of ROC-related estimates. Bioinformatics. 2010; 26(6):822–30. source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Once again, this score would hide the truth: the classification algorithm has performed poorly on the negative subset.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Even if less common than the ROC curve, several scientists consider the PR curve more informative than the ROC curve, especially on imbalanced biological and medical datasets [48, 97, 98].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [48] Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE. 2015; 10(3):0118432. source
- Not checked yetcites [97] Chicco D. Ten quick tips for machine learning in computational biology. BioData Min. 2017; 10(35):1–17.
- Not checked yetcites [98] Ozenne B, Subtil F, Maucort-Boulch D. The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J Clin Epidemiol. 2015; 68(8):855–9. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“MCC was re-proposed by Baldi and colleagues [39] in 2000 as a standard performance metric for machine learning with a natural extension to the multiclass case [40].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [39] Baldi P, Brunak S, Chauvin Y, Andersen CA, Nielsen H. Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics. 2000; 16(5):412–24. source
- Not checked yetcites [40] Gorodkin J. Comparing two K-category assignments by a K-category correlation coefficient. Comput Biol Chem. 2004; 28(5–6):367–74. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“For example, when Whalen and colleagues released TargetFinder, a tool to predict enhancer-promoters interactions in genomics, they showed its results measured only by F1 score [13], making it impossible to detect the actual true positive rate and true negative rate of their tests [69].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [13] Whalen S, Truty RM, Pollard KS. Enhancer–promoter interactions are encoded by complex genomic signatures on looping chromatin. Nat Genet. 2016; 48(5):488. source
- Not checked yetcites [69] Xi W, Beer MA. Local epigenomic state cannot discriminate interacting and non-interacting enhancer–promoter pairs with high accuracy. PLoS Comput Biol. 2018; 14(12):1006625. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Although being widely acknowledged as a reliable metric, there are situations - albeit extreme - where either MCC cannot be defined or it displays large fluctuations [45], due to imbalanced outcomes in the classification.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [45] Brown JB. Classifiers and their metrics quantified. Mol Inform. 2018; 37:1700127. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“An effective solution overcoming the class imbalance issue comes from the Matthews correlation coefficient (MCC), a special case of the ϕ phi coefficient [36].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [36] Guilford JP. Psychometric Methods. New York City: McGraw-Hill; 1954.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“for instance, the Food and Drug Administration (FDA) agency of the USA employed the MCC as the main evaluation measure in the MicroArray II / Sequencing Quality Control (MAQC/SEQC) projects [41, 42].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [41] The MicroArray Quality Control (MAQC) Consortium. The MAQC-II Project: a comprehensive study of common practices for the development and validation of microarray-based predictive models. Nat Biotechnol. 2010; 28(8):827–38. source
- Not checked yetcites [42] The SEQC/MAQC-III Consortium. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequence Quality Control consortium. Nat Biotechnol. 2014; 32:903–14. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“However, this procedure is biased [65], and it is still far from being accepted as a standard practice by the community.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [65] Van Asch V. Macro-and micro-averaged evaluation measures. Technical report. 2013:1–27.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, the algorithm showed its ability to predict the positive data instances (90 sick patients out of 91 were correctly predicted), but it also displayed its lack of talent in identifying healthy controls (only 1 healthy individual out of 9 was correctly recognized) (Fig. 2b). Therefore, the overall performance should be judged poor.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to the Use case A1 and B1, the method was able to correctly identify multiple negative data instances (88 healthy patients out of 89), but unable to correctly predict most of sick patients (only 2 true positives out of 11 possible elements).”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plainn = 88- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Originally developed by Matthews in 1975 for comparison of chemical structures [38]”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [38] Matthews BW. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct. 1975; 405(2):442–51. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Despite its popularity, in the learning context there are a number of issues causing the kappa measure to produce unreliable results (for instance, its high sensitivity to the distribution of the marginal totals [73–75])”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Background
What the sources say
- Not checked yetcites [73] Flight L, Julious SA. The disagreeable behaviour of the kappa statistic. Pharm Stat. 2015; 14:74–8. source
- Not checked yetcites [74] Powers DMW. The problem with Kappa. In: Proceedings of EACL 2012 – the 13th Conference of the European Chapter of the Association for Computational Linguistics. Avignon: ACL: 2012. p. 345–55.
- Not checked yetcites [75] Delgado R, Tibau X-A. Why Cohen’s Kappa should be avoided as performance measure in classification. PloS ONE. 2019; 14(9):0222916.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, when a prediction displays many true positives but few true negatives (or many true negatives but few true positives) we will show that F1 and accuracy can provide misleading information, while MCC always generates results that reflect the overall prediction issues.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results show that, while accuracy and F1 score often generate high scores that do not inform the user about ongoing prediction issues, the MCC is a robust, useful, reliable, truthful statistical measure able to correctly reflect the deficiency of any prediction in any dataset.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
boosteddirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here accuracy again generated overoptimistic, misleading, and inflated high results”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The F1 score ranking and the accuracy ranking, in conclusion, are hiding this important flaw of the top classifier: k-NN was unable top correctly predict a high percentage of patients.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There, accuracy and F1 generated overoptimistic and inflated results”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Accuracy and F1 score, although popular, can generate misleading results on imbalanced datasets, because they fail to consider the ratio between positive and negative elements.”
- Also stated in: § Conclusions · plain
“F1 and accuracy, instead, generate reliable results only when applied to balanced datasets, and produce misleading results when applied to imbalanced cases.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Conclusions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Conclusions · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With its low value (MCC = +0.29), the MCC informs the readers about the poor general outcome of the radial SVM, while the accuracy and F1 score show misleading values.”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The dataset contains 2,000 gene probsets for 62 patients, of which 22 are healthy controls and 40 have colon cancer (35.48% negatives and 64.52% positives) [106].”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
What the sources say
- Not checked yetcites [106] Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ. Data pertaining to the article ‘Broad patterns of gene expression revealed by clustering of tumor and normal colon tissues probed by oligonucleotide arrays’. 2000. http://genomics-pubs.princeton.edu/oncology/affydata/index.html. Accessed 10 Dec 2019.
plainn = 62- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Accuracy and F1 score are the most common metrics used for binary classification in machine learning.
“We decided to focus on accuracy and F1 score because they are the most common metrics used for binary classification in machine learning.”
The paper relies on this without support.
The synthetic use cases are realistic representations of binary classification scenarios.
“here we describe some synthetic, realistic scenarios where MCC results are more informative and truthful than the other two measures analyzed.”
The paper relies on this without support.
Accuracy and F1 score are more commonly used in machine learning studies related to biomedical applications than other rates.
“because accuracy and F1 score are more commonly used in machine learning studies related to biomedical applications.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“(3) the absolute value of F1 changes by simply switching the outcome labels (ie, when 1 becomes 0 and 0 becomes 1).”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [77]
The cited source says what the claim attributes to it.
In the citing paper
“MCC is a more reliable metric, whereby a high score is assigned only if the prediction was correct for the four base categories (TP, FP, TN and FP), taking into account the size of P and N [85].”
From A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance (2022) · cites this paper as [85]
In the citing paper
“since it has been shown to present a balanced score when dealing with imbalanced data [11, 6] as is our case with fewer manipulated images.”
From BioFors: A Large Biomedical Image Forensics Dataset (2021) · cites this paper as [11]
In the citing paper
“(2) F1 has no intuitive interpretation”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [77]
In the citing paper
“(1) F1 ignores true negatives”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [77]
In the citing paper
“F1 and MCC were introduced to address the challenges caused by class imbalance, where classification accuracy can be inflated by classifying all patients as low risk when the event is rare.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“Here, accuracy again generated an overoptimistic inflated score”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“all three statistical scores generated low results that reflect the mediocre quality of the prediction: F1 score = 0.17 and accuracy = 0.10 in the [0, 1] interval”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“the prediction technique failed in predicting negative elements, and therefore its confusion matrix showed a low percentage of true negatives”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“the method was able to correctly identify multiple negative data instances (88 healthy patients out of 89), but unable to correctly predict most of sick patients (only 2 true positives out of 11 possible elements)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“Even if accuracy showed an excessively high result score”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“all three statistical rates showed a low score which emphasized the deficiency in the prediction process (accuracy = 0.24, F1 score = 0.12, and MCC = −0.24)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
“the best classifier resulted being k-NN (F1 score = 0.87 and accuracy = 0.81)”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Genomics scenario: colon cancer gene expression
“k-nearest neighbors | +0.48 | 0.87 | 0.80 | 0.92 | 0.52”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“only 1 healthy individual out of 9 was correctly recognized”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“TP=90, FN=1, TN=0, FP=9”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“Finally, in the last Use case C2 (positively imbalanced dataset)”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“Use case C2 Negatively imbalanced dataset | 11 | 89 | 2 | 9 | 88 | 1”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“For the Use case A1 (negatively imbalanced dataset)”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“made of 9 healthy individuals (negatives =9%) and 91 sick patients (positives =91%)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“Also in the Use case C1 (positively imbalanced dataset)”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“90 healthy controls (negatives =90%) and 10 sick patients (positives =10%)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“only 5 sick patients out of 70”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“TP = 5, FN = 70, TN = 19, FP = 6”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Results and discussion > Use cases
“the resulting MCC = –0.03”
§ The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation > Methods > Relationship between measures
“In red, the (−0.04, 0.95) point corresponding to use case A1”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Measures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 12,187
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 12,051
Research works printed but not linked: 7
- Akosa JS. Predictive accuracy: a misleading performance measure for highly imbalanced data. In: Proceedings of the SAS Global Forum 2017 Conference. Cary, North Carolina: SAS Institute Inc.: 2017. p. 942–2017.
- Guilford JP. Psychometric Methods. New York City: McGraw-Hill; 1954.
- van Rijsbergen CJ, Joost C. Information Retrieval. New York City: Butterworths; 1979.
- Sasaki Y. The truth of the F-measure. Teach Tutor Mater. 2007; 1(5):1–5.
- Dubey A, Tarar S. Evaluation of approximate rank-order clustering using Matthews correlation coefficient. Int J Eng Adv Technol. 2018; 8(2):106–13.
- Suresh Babu N. Various performance measures in binary classification – An overview of ROC study. Int J Innov Sci Eng Technol. 2015; 2(9):596–605.
- Ferri C, Hernández-Orallo J, Flach PA. A coherent interpretation of AUC as a measure of aggregated classification performance. In: Proceedings of ICML 2011 – the 28th International Conference on Machine Learning. Norristown: Omnipress: 2011. p. 657–64.
Cited by
Sorted by citations
Chicco, D., Warrens, M. J., & Jurman, G. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.623 · Cited by 5,380
“…of determination more reliable and trustworthy: similarly to the Matthews correlation coef fi cient (MCC) ( Matthews, 1975 ) in binary classi fi cation ( Chicco & Jurman, 2020 ; Chicco, Tötsch & Jurman, 2021 ; Tötsch & Hoffmann, 2021 ; Chicco, Starovoitov & Jurman, 2021 ; Chicco, Warrens &…”
Self-citationXiong, G., Wu, Z., Yi, J., Fu, L., Yang, Z., Hsieh, C., Yin, M., Zeng, X., Wu, C., Lü, A., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab255 · Cited by 3,023
Fu, L., Shi, S., Yi, J., Wang, N., He, Y., Wu, Z., Peng, J., Deng, Y., Wang, W., Wu, C., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae236 · Cited by 1,396
“…the performance of the models, R-square (R 2 ), root mean squared error (RMSE), and mean absolute error (MAE) were applied for regression tasks, while the area under the ROC curve (AUC), accuracy (ACC), and Matthews correlation coefficient (MCC) ( 39 ) were applied for classification tasks.”
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Self-citationDeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 736
“A threshold for each output label is computed by maximizing Matthew’s Correlation Coefficient (MCC) (30) on the training data.”
InfluentialChicco, D., & Jurman, G. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1023-5 · Cited by 664
Self-citationChicco, D., & Jurman, G. (2023) · BioData Mining · doi:10.1186/s13040-023-00322-4 · Cited by 737
“In some previous studies, we argued that the MCC is more informative than confusion entropy [15], F 1 score [6], accuracy [6], balanced accuracy [4], bookmaker informed-ness [4], markedness [4], diagnostic odds ratio [16], Cohen’s Kappa [17], and Brier score [17].”
BackgroundMethodInfluentialSelf-citationEnd-to-end privacy preserving deep learning on multi-institutional medical imaging
Kaissis, G., Ziller, A., Passerat-Palmbach, J., Ryffel, T., Usynin, D., Trask, A., Lima, I., Mancuso, J., Jungmann, F., Steinborn, M.-M., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00337-8 · Cited by 513
Masud, M., Sikder, N., Nahid, A.-A., Bairagi, A. K., & AlZain, M. A. (2021) · Sensors · doi:10.3390/s21030748 · Cited by 489
“Their proposed method does not require the segmentation of nuclei and can provide a maximum F-measure of 80.2% while classifying them.”
BackgroundInfluentialBoosting methods for multi-class imbalanced data classification: an experimental review
Tanha, J., Abdi, Y., Samadi, N., Razzaghi-Asl, N., & Asadpour, M. (2020) · Journal Of Big Data · doi:10.1186/s40537-020-00349-y · Cited by 497
“To have an intuitive view of the performance of the classifier based on these metrics, F-measure [72] can be used.”
MethodChicco, D., Warrens, M. J., & Jurman, G. (2021) · IEEE Access · doi:10.1109/access.2021.3084050 · Cited by 503
“Finally, κ can be undefined in some cases, but these cases can be handled with mathematical operations similar to the ones needed when MCC is undefined [3].”
BackgroundMethodResultInfluentialSelf-citationBentivoglio, R., Isufi, E., Jonkman, S. N., & Taormina, R. (2022) · Hydrology and earth system sciences · doi:10.5194/hess-26-4345-2022 · Cited by 529
Marcos-Zambrano, L. J., Karađuzović-Hadžiabdić, K., Lončar-Turukalo, T., Przymus, P., Trajkovik, V., Aasmets, O., Berland, M., Gruca, A., Telalović, J. H., Hron, K., et al. (2021) · Frontiers in Microbiology · doi:10.3389/fmicb.2021.634511 · Cited by 447
“MCC was used since it is a suitable performance measure to evaluate model performance even in the case of imbalanced classes (Chicco and Jurman, 2020).”
MethodAn open-source, high-performance tool for automated sleep staging
Vallat, R., & Walker, M. P. (2021) · eLife · doi:10.7554/elife.70092 · Cited by 366
“The latter is thought to be the most robust and informative score for classification as it naturally takes into account imbalance between sleep stages ( Chicco and Jurman, 2020 ).”
BackgroundWang, Y., Wang, H., & Peng, Z. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114770 · Cited by 352
“Although Matthews correlation coefficient is identified to perform better for binary classification, it is not suitable for our multi-class classification tasks(Chicco and Jurman, 2020).”
MethodFletcher, R. R., Nakeshimana, A., & Olubeko, O. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.561802 · Cited by 305
“More recently, other measures, such as the Matthews correlation coef fi cient (MCC) (Chicco and Jurman, 2020), are also being used as a preferred optimization criteria for evaluating binary classi fi ers when a class imbalance exists.”
Patent Quality: Towards a Systematic Framework for Analysis and Measurement
Higham, K., de Rassenfosse, G., & Jaffe, A. B. (2021) · Research Policy · doi:10.1016/j.respol.2021.104215 · Cited by 294
Machine learning algorithms for social media analysis: A survey
T.K., B., Annavarapu, C. S. R., & Bablani, A. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100395 · Cited by 276
Deep learning as a tool for ecology and evolution
Borowiec, M. L., Dikow, R. B., Frandsen, P. B., McKeeken, A. J., Valentini, G., & White, A. E. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13901 · Cited by 295
“TensorFlow (Abadi et al., 2016) is another library with a Python or R interface and currently the most popular deep learning framework.”
BackgroundOn the equivalency of factor and network loadings
Christensen, A. P., & Golino, H. F. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01500-6 · Cited by 264
Imani, M., Beikmohammadi, A., & Arabnia, H. R. (2025) · Technologies · doi:10.3390/technologies13030088 · Cited by 361
“This characteristic makes MCC especially valuable for evaluating imbalanced datasets, where metrics like accuracy often fail to provide reliable insights [69,70].”
AI-Based Intrusion Detection Systems for In-Vehicle Networks: A Survey
Rajapaksha, S., Kalutarage, H., Al-Kadri, M. O., Petrovski, A., Madzudzo, G., & Cheah, M. (2022) · ACM Computing Surveys · doi:10.1145/3570954 · Cited by 278
“Matthews correlation coeicient (MCC) [26] is considered a more reliable metric for binary classiication problems such as anomaly detection.”
MethodIleberi, E., Sun, Y., & Wang, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3134330 · Cited by 293
“In this research, we consider the Matthews correlation coefficient ( MCC ) [31], [32], the AUC [33], and the Confusion Matrix (CM) as additional performance indicators.”
MethodSixteen facial expressions occur in similar contexts worldwide
Cowen, A., Keltner, D. J., Schroff, F., Jou, B., Adam, H., & Prasad, G. (2020) · Nature · doi:10.1038/s41586-020-3037-7 · Cited by 247
A review of the F-measure: its history, properties, criticism, and alternatives
Christen, P., Hand, D. J., & Kirielle, N. (2024) · ACM Computing Surveys · doi:10.1145/3606367 · Cited by 236
“Various reviews which look at different aspects of classification performance measures have been written, including [11, 15, 19, 23, 27, 50, 57], and others.”
BackgroundF*: an interpretable transformation of the F-measure
Hand, D. J., Christen, P., & Kirielle, N. (2021) · Machine Learning · doi:10.1007/s10994-021-05964-1 · Cited by 224
“These simple proportions can be combined to yield familiar performance measures, including the misclassification rate, the kappa statistic, the Youden index, the Matthews coefficient, and the F-measure or F1-score (Chicco and Jurman 2020; Hand 2012).”
BackgroundMethodInfluentialDeepLoc 2.1: multi-label membrane protein type prediction using protein language models
Ødum, M. T., Teufel, F., Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae237 · Cited by 230
“As opposed to other accuracy metrics, the threshold optimization using the MCC-score better accommodates label imbalance ( 22 ).”
Louk, M. H. L., & Tama, B. A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119030 · Cited by 227
Autopilot control unmanned aerial vehicle system for sewage defect detection using deep learning
Pandey, B. K., Pandey, D., & Sahani, K. (2024) · Engineering Reports · doi:10.1002/eng2.12852 · Cited by 220
“56 In addition, the outcomes of the suggested technique, including precision, specificity, recall, sensitivity, accuracy, and F1-score, 1,57 have been determined with various UAV-collected input images, as shown in Table 2.”
Deep Learning in Virtual Screening: Recent Applications and Developments
Kimber, T. B., Chen, Y., & Volkamer, A. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22094435 · Cited by 215
Cost-sensitive learning for imbalanced medical data: a review
Araf, I., Idri, A., & Chairi, I. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10652-8 · Cited by 276
Wang, X., Wang, H., Bhandari, B., & Cheng, L. (2023) · International Journal of Precision Engineering and Manufacturing-Green Technology · doi:10.1007/s40684-023-00537-0 · Cited by 248
“Some research suggests that the G-mean score provides a more balanced interpretation of classification performance, while others claim that the Matthew correlation coefficient (MCC) is more suitable [156].”
The receiver operating characteristic curve accurately assesses imbalanced datasets
Richardson, E., Trevizani, R., Greenbaum, J., Carter, H., Nielsen, M., & Peters, B. (2024) · Patterns · doi:10.1016/j.patter.2024.100994 · Cited by 280
Deep learning ensemble 2D CNN approach towards the detection of lung cancer
Shah, A. A., Malik, H. A. M., Muhammad, A., Alourani, A., & Butt, Z. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-29656-z · Cited by 211
“After the prediction was made, the next step was to check the accuracy of the predictions, and for this purpose, we made a confusion matrix 38,39 .”
MethodYounes, R., Caviglia, G. P., Govaere, O., Rosso, C., Armandi, A., Sanavia, T., Pennisi, G., Liguori, A., Francione, P., Gallego-Durán, R., et al. (2021) · Journal of Hepatology · doi:10.1016/j.jhep.2021.05.008 · Cited by 196
Selvaraj, C., Chandra, I., & Singh, S. K. (2021) · Molecular Diversity · doi:10.1007/s11030-021-10326-z · Cited by 220
Electricity Consumer Characteristics Identification: A Federated Learning Approach
Wang, Y., Bennani, I. L., Liu, X., Sun, M., & Zhou, Y. (2021) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2021.3066577 · Cited by 192
“In addition, a high score can only be achieved if the classifier has been able to correctly predict the majority of positive data instances and the majority of negative data instances while reducing FP and FN [37].”
BackgroundA general model to predict small molecule substrates of enzymes based on machine and deep learning
Kroll, A., Ranjan, S., Engqvist, M. K. M., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38347-2 · Cited by 193
“As the test datasets used by Yang et al. 15 are imbalanced, with a proportion of 18 – 31% of positive data points, the MCC is a more meaningful score compared to the accuracy 52 ; we hence conclude that ESP outperforms GT-Predict.”
MethodAn update on global mining land use
MAUS, V. W., Giljum, S., da Silva, D. M., Gutschlhofer, J., da Rosa, R. P., Luckeneder, S., Gass, S. L. B., Lieber, M., & McCallum, I. (2022) · Scientific Data · doi:10.1038/s41597-022-01547-4 · Cited by 196
“The Kappa index was 0.77 and Matthews correlation coefficient (MCC) 0.78 (Kappa and MCC range from − 1 to 1 40 ).”
InfluentialA survey on deep learning for skin lesion segmentation
Mirikharaji, Z., Abhishek, K., Bissoto, A. E., Barata, C., Avila, S., Junior, E. A. D. V., Celebi, M. E., & Hamarneh, G. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102863 · Cited by 220
“, 2020; Zhu, 2020), this measure appears to be suitable for class-imbalanced domains (Boughorbel et al., 2017; Chicco and Jurman, 2020; Luque et al., 2020).”
BackgroundMethodKumar, V., Azamathulla, H. M., Sharma, K. V., Mehta, D., & Tota‐Maharaj, K. (2023) · Sustainability · doi:10.3390/su151310543 · Cited by 217
“The accuracy of predictions may be evaluated using measures including accuracy, precision, recall, and F1 score [162].”
MethodFoody, G. M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0291908 · Cited by 248
“Recent literature has also promoted the use of the MCC in accuracy assessment in all subjects and highlighted merits relative to other popular and widely used measures such as recall, precision and F1 [11, 15].”
BackgroundMethodInfluentialThe False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184
“While the so-called Matthews correlation coefficient is better than the F1 score if researchers use only one specific threshold, PR curves are recommended [34, 35] if different threshold levels exist as in the case with Botometer.”
BackgroundAlphaFold2: A Role for Disordered Protein/Region Prediction?
Wilson, C. J., Choy, W.-Y., & Karttunen, M. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23094591 · Cited by 173
“Thresholds for n were chosen based on the Matthews correlation coefficient (MCC), which has been documented to be an excellent metric for assessing the accuracy of binary classifiers [33] and was the approach used at CAID [29].”
MethodLandscape of mobile genetic elements and their antibiotic resistance cargo in prokaryotic genomes
Khedkar, S., Smyshlyaev, G., Letunić, I., Maistrenko, O. M., Coelho, L. P., Orakov, A. N., Forslund, S. K., Hildebrand, F., Lütge, M., Schmidt, T., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac163 · Cited by 195
“Median of gathering thresholds of folds with Matthews Correlation Coefficient (27) MCC > 0.”
BackgroundInfluentialOn the platform but will they buy? Predicting customers' purchase behavior using deep learning
Chaudhuri, N., Gupta, G., Vallurupalli, V., & Bose, I. (2021) · Decision Support Systems · doi:10.1016/j.dss.2021.113622 · Cited by 208
Znet: Deep Learning Approach for 2D MRI Brain Tumor Segmentation
Ottom, M. A., Rahman, H. A., & Dinov, I. D. (2022) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2022.3176737 · Cited by 184
“9 for evaluating the proposed approach [39], (5)–(9), as shown at the bottom of the page.”
BackgroundA general Monte Carlo method for sample size analysis in the context of network models.
Constantin, M. A., Schuurman, N. K., & Vermunt, J. K. (2023) · Psychological Methods · doi:10.1037/met0000555 · Cited by 217
Li, Y., Cao, J., Xu, Y. H., Zhu, L., & Dong, Z. Y. (2023) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2023.113913 · Cited by 191
Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features
Tursunov, A., Mustaqeem, M., & Kwon, S. (2020) · Sensors · doi:10.3390/s20185212 · Cited by 174
“The F1measure is defined as the harmonic mean of the precision rate and the recall rate [51].”
BackgroundCrack Detection in Concrete Structures Using Deep Learning
Golding, V. P., Gharineiat, Z., Munawar, H. S., & Ullah, F. (2022) · Sustainability · doi:10.3390/su14138117 · Cited by 178
Financial Sentiment Analysis: Techniques and Applications
Du, K., Xing, F., Mao, R., & Cambria, E. (2024) · ACM Computing Surveys · doi:10.1145/3649451 · Cited by 215
Melaiu, O., Chierici, M., Lucarini, V., Jurman, G., Conti, L. A., Vito, R. D., Boldrini, R., Cifaldi, L., Castellano, A., Furlanello, C., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19781-y · Cited by 164
Self-citationShao, Z., Ahmad, M. N., & Javed, A. (2024) · Remote Sensing · doi:10.3390/rs16040665 · Cited by 199
Wei, L., He, W., Malik, A., Su, R., Cui, L., & Manavalan, B. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa275 · Cited by 162
A review of deep learning applications in human genomics using next-generation sequencing data
Alharbi, W., & Rashid, M. (2022) · Human Genomics · doi:10.1186/s40246-022-00396-x · Cited by 179
“A sufficient number of solutions usually applied in this case such as transfer learning [33] and Matthews correlation coefficient (MCC) [34].”
BackgroundKumarasan, Y., Rajandas, H., Parimannan, S., Ravichandran, M., Marimuthu, K., Petersen, B., Clokie, M. R. J., Millard, A. D., & Sicheritz‐Pontén, T. (2022) · Viruses · doi:10.3390/v14020342 · Cited by 161
“The accuracy, F1 score, and area under the Curve (AUC) scores were calculated for each of the predictors, but the performance of each model was evaluated based on MCC scores, as these provide a much more reliable scoring metric for data with imbalanced classes [44].”
MethodA deep learning approach for intrusion detection in Internet of Things using focal loss function
Dina, A. S., Siddique, A. B., & Manivannan, D. (2023) · Internet of Things · doi:10.1016/j.iot.2023.100699 · Cited by 173
Olsen, C. R., Mentz, R. J., Anstrom, K. J., Page, D. I., & Patel, P. A. (2020) · American Heart Journal · doi:10.1016/j.ahj.2020.07.009 · Cited by 164
A review of deep learning-based detection methods for COVID-19
Subramanian, N., Elharrouss, O., Al-Maadeed, S. A., & Chowdhury, M. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105233 · Cited by 159
“The Matthews Correlation Coefficient (MCC) [107] is the extra metric that is used in Ref.”
MethodChatzimparmpas, A., Martins, R. M., Kucher, K., & Kerren, A. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030352 · Cited by 164
“However, comparison and selection between multiple performance indicators is not trivial, even for widely used metrics [13, 50]; alternative ones such as Matthews correlation coefficient (MCC) might be more informative for different problems [9].”
BackgroundSanavia, T., Birolo, G., Montanucci, L., Turina, P., Capriotti, E., & Fariselli, P. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.07.011 · Cited by 157
“Given this unbalance, the most recommended measures of accuracy in the context of classification are the MCC and the harmonic mean between precision and recall (F1) [64].”
General Performance Score for classification problems
de Diego, I. M., Redondo, A. R., Fernández, R. R., Navarro, J., & Moguerza, J. M. (2022) · Applied Intelligence · doi:10.1007/s10489-021-03041-7 · Cited by 175
“Balanced Accuracy majority class [4].”
A Survey for Cervical Cytopathology Image Analysis Using Deep Learning
Rahaman, M. M., Li, C., Wu, X., Yao, Y., Hu, Z., Jiang, T., Li, X., & Qi, S. (2020) · IEEE Access · doi:10.1109/access.2020.2983186 · Cited by 156
“A coefficient of + 1 signifies a perfect classification, 0 is the expected value for arbitrary prediction, and − 1 indicates perfect misclassification [100], [101].”
BackgroundOpitz, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00675 · Cited by 244
“Now it suffices to see that x prevalence ( x ) = x bias ( x ) and HM ( a, a ) = a .”
BackgroundNguyễn, T. T., Pham, T. D., Nguyen, C. T., Delfos, J., Archibald, R., Dang, K. B., Hoang, N. B., Guo, W., & Ngo, H. H. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.150187 · Cited by 153
IoMT-Enabled Real-Time Blood Glucose Prediction With Deep Learning and Edge Computing
Zhu, T., Kuang, L., Daniels, J., Herrero, P., Li, K., & Georgiou, P. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3143375 · Cited by 164
“…in binary classifications since high MCC scores can be obtained only if the classifier performs well in all the categories of the confusion matrix [65], which can be denoted as where TP stands for the number of true positives, i.e., the hypoglycemic events that are correctly detected by the…”
BackgroundComprehensive Survey of Recent Drug Discovery Using Deep Learning
Kim, J., Park, S., Min, D., & Kim, W. (2021) · International Journal of Molecular Sciences · doi:10.3390/ijms22189983 · Cited by 147
“Chicco and Jurman [178] showed that MCC is more informative in evaluating binary classifications than accuracy and F1 score.”
MethodLee, M. C. H., Braet, J., & Springael, J. (2024) · Applied Sciences · doi:10.3390/app14219863 · Cited by 213
Houssein, E. H., Abohashima, Z., Elhoseny, M., & Mohamed, W. M. (2022) · Journal of Computational Design and Engineering · doi:10.1093/jcde/qwac003 · Cited by 154
A survey on intrusion detection system in IoT networks
Rahman, M. M., Shakil, S. A., & Mustakim, M. R. (2024) · Cyber Security and Applications · doi:10.1016/j.csa.2024.100082 · Cited by 140
Massively parallel functional testing of MSH2 missense variants conferring Lynch syndrome risk
Jia, X., Burugula, B. B., Chen, V., Lemons, R. M., Jayakody, S., Maksutova, M., & Kitzman, J. O. (2020) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2020.12.003 · Cited by 141
Two-stage convolutional neural network for road crack detection and segmentation
Nguyen, N. H. T., Perry, S. W., Bone, D. J., Le, H. T., & Thủy, N. T. T. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115718 · Cited by 145
Xu, D., Quan, W., Zhou, H., Sun, D., Baker, J. S., & Gu, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-07054-1 · Cited by 146
“Matthew’s correlation coefficient (MCC) is a contingency matrix method 34 .”
BackgroundMethodInfluentialMachine learning approaches in microbiome research: challenges and best practices
Παπουτσόγλου, Γ., Tarazona, S., Lopes, M. B., Klammsteiner, T., Ibrahimi, E., Eckenberger, J., Novielli, P., Tonda, A. P., Mihajlović, A., Shigdel, R., et al. (2023) · Frontiers in Microbiology · doi:10.3389/fmicb.2023.1261889 · Cited by 153
“This is especially useful when the class distribution is imbalanced or when the costs associated with different types of errors vary (Chicco and Jurman, 2020).”
BackgroundThe Role of Near-Infrared Spectroscopy in Food Quality Assurance: A Review of the Past Two Decades
Fodor, M., Matkovits, A., Benes, E., & Jókai, Z. (2024) · Foods · doi:10.3390/foods13213501 · Cited by 145
Arreche, O., Guntur, T. R., & Abdallah, M. (2024) · Applied Sciences · doi:10.3390/app14104170 · Cited by 189
Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review
Giuste, F., Shi, W., Zhu, Y., Naren, T., Isgut, M., Sha, Y., Li, T., Gupte, M., & Wang, M. D. (2022) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3185953 · Cited by 145
“More robust metrics include Area Under the Receiver Operating Curve (AUROC) and Matthews Correlation Coefficient (MCC), the latter being appropriate even in highly imbalanced binary classification tasks [111].”
MethodOlczak, J., Pavlopoulos, J., Prijs, J., IJpma, F. F. A., Doornberg, J. N., Lundström, C. F., Hedlund, J., & Gordon, M. (2021) · Acta Orthopaedica · doi:10.1080/17453674.2021.1918389 · Cited by 145
“MCC or precision-recall analysis with AUPR and F1-score are more suitable, as AUC could overestimate the model’s performance (Chicco and Jurman 2020).”
MethodChang, T., Cao, Y., Sfreddo, H. J., Dhruba, S. R., Lee, S., Valero, C., Yoo, S., Chowell, D., Morris, L. G. T., & Ruppin, E. (2024) · Nature Cancer · doi:10.1038/s43018-024-00772-7 · Cited by 149
Classification of Heart Sounds Using Convolutional Neural Network
Li, F., Tang, H. X., Shang, S., Mathiak, K., & Cong, F. (2020) · Applied Sciences · doi:10.3390/app10113956 · Cited by 140
“The sensitivity, specificity, MCC [42] and mean accuracy (Macc) for each fold of the cross-validation were used to evaluate the performance of the proposed model.”
MethodPratama, A. A., Bolduc, B., Zayed, A. A., Zhong, Z.-P., Guo, J., Vik, D. R., Gazitúa, M. C., Wainaina, J. M., Roux, S., & Sullivan, M. B. (2021) · PeerJ · doi:10.7717/peerj.11447 · Cited by 122
E-XAI: Evaluating Black-Box Explainable AI Frameworks for Network Intrusion Detection
Arreche, O., Guntur, T. R., Roberts, J. W., & Abdallah, M. (2024) · IEEE Access · doi:10.1109/access.2024.3365140 · Cited by 153
“These are accuracy (Acc), precision (Prec), recall (Rec), F1-score (F1), Matthews correlation coefficient (Mcc) [74], and balanced accuracy (Bacc), and the AucRoc (area under the ROC curve).”
BackgroundColorectal Histology Tumor Detection Using Ensemble Deep Neural Network
Ghosh, S., Bandyopadhyay, A., Sahay, S., Ghosh, R., Kundu, I., & Santosh, K. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104202 · Cited by 129
Ågren, A. M., Larson, J., Paul, S. S., Laudon, H., & Lidberg, W. (2021) · Geoderma · doi:10.1016/j.geoderma.2021.115280 · Cited by 127
Murphy, A. E., & Skene, N. (2022) · Nature Communications · doi:10.1038/s41467-022-35519-4 · Cited by 121
A review of model evaluation metrics for machine learning in genetics and genomics
Miller, C., Portlock, T., Nyaga, D. M., & O'Sullivan, J. M. (2024) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2024.1457619 · Cited by 187
Real World Object Detection Dataset for Quadcopter Unmanned Aerial Vehicle Detection
Pawelczyk, M. L., & Wojtyra, M. (2020) · IEEE Access · doi:10.1109/access.2020.3026192 · Cited by 121
“Inthe case of unbalanced classes, another useful metric is the Matthews Correlation Coefficient (MCC), which is specifically designed for binary classification [61].”
MethodAgricultural Drought Prediction Based on Conditional Distributions of Vine Copulas
Wu, H., Su, X., Singh, V. P., Feng, K., & Niu, J. (2021) · Water Resources Research · doi:10.1029/2021wr029562 · Cited by 127
“…both lower than −0.5 in the sample of SSI observations lower than −0.5; and F1S is the harmonic mean value between precision and recall, which is the most popular metric used in binary classification evaluation and helps comprehensively measure the accuracy of a model (Chicco & Jurman, 2020).”
InfluentialElgendi, M., Nasir, M. U., Tang, Q., Smith, D., Grenier, J., Batte, C., Spieler, B. M., Leslie, W. D., Menon, C., Fletcher, R., et al. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.629134 · Cited by 131
“The accuracy is considered as the most popular adopted metrics in binary classification tasks; however, for imbalanced datasets, such as the case in our work, the MCC is recommended (41).”
BackgroundSharma, R., Shrivastava, S., Singh, S. K., Kumar, A., Saxena, S., & Singh, R. K. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab065 · Cited by 122
Machine and deep learning for longitudinal biomedical data: a review of methods and applications
Cascarano, A., Mur-Petit, J., Hernández-González, J., Camacho, M., Eadie, N. D. T., Gkontra, P., Chadeau‐Hyam, M., Vitria, J. M., & Lekadir, K. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10561-w · Cited by 147
A Deep Learning Approach for Human Activities Recognition From Multimodal Sensing Devices
Ihianle, I. K., Nwajana, A. O., Ebenuwa, S. H., Otuka, R. I., Owa, K., & Orisatoki, M. O. (2020) · IEEE Access · doi:10.1109/access.2020.3027979 · Cited by 138
Diabetic foot ulcer detection using deep learning approaches
Thotad, P. N., Bharamagoudar, G. R., & Anami, B. S. (2022) · Sensors International · doi:10.1016/j.sintl.2022.100210 · Cited by 126
A guide to formulating fairness in an optimization model
Chen, V. X., & Hooker, J. (2023) · Annals of Operations Research · doi:10.1007/s10479-023-05264-y · Cited by 139
“Beyond these, the Matthews correlation coefficient (Matthews 1975, Chicco and Jurman 2020) is often regarded as the most comprehensive measure of classification accuracy, but it corresponds to a complicated, nonconvex SWF that could be quite difficult to optimize.”
MethodAn attention-based CNN-BiLSTM model for depression detection on social media text
Thekkekara, J. P., Yongchareon, S., & Liesaputra, V. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123834 · Cited by 141
State-of-the-art retinal vessel segmentation with minimalistic models
Galdrán, A., Anjos, A., Dolz, J., Chakor, H., Lombaert, H., & Ayed, I. B. (2022) · Scientific Reports · doi:10.1038/s41598-022-09675-y · Cited by 124
Vision-Transformer-Based Transfer Learning for Mammogram Classification
Ayana, G., Dese, K., Dereje, Y., Kebede, Y., Barki, H., Amdissa, D., Ibrahim, N. H., Mulugeta, F., Habtamu, B., & Choe, S. (2023) · Diagnostics · doi:10.3390/diagnostics13020178 · Cited by 132
Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening
Ünlü, O., Shin, J., Mailly, C. J., Oates, M., Tucci, M. R., Varugheese, M., Wagholikar, K. B., Wang, F., Scirica, B. M., Blood, A. J., et al. (2024) · NEJM AI · doi:10.1056/aioa2400181 · Cited by 128
Liu, J., Kantarcı, B., & Adams, C. M. (2020) · ACM Workshop on Wireless Security and Machine Learning (WiseML) · doi:10.1145/3395352.3402621 · Cited by 119
“, and is a useful ML algo-rithm metric especially for imbalanced datasets, which results in more meaningful results than F score [5] in such settings as it considers the proportion of the classes inside the confusion matrix.”
Background
Metadata
- Authors
- Chicco, D., & Jurman, G.
- Affiliations
- Krembil Research Institute (CA)
- Fondazione Bruno Kessler (IT)
- Corresponding author
- Chicco, Davide
- Venue
- BMC Genomics
- Published
- 2020-01-02
- Volume
- 21
- Issue
- 1
- Pages
- 6-6
- ISSN
- 1471-2164 (ISSN-L)
- PMC
- PMC6941312
- OpenAlex
- W2999309192
- Semantic Scholar
- 44858b662822d291a5e1e9e3501ad19a9f1572c2
- Cited by
- 6,143
- References
- 104
- Citations by year
- 2026: 1118 · 2025: 1152 · 2024: 1060 · 2023: 1032 · 2022: 883 · 2021: 682 · 2020: 211
- FWCI
- 276.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Genomics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Iris (University of Trento) (submittedVersion)
- Funders
- University of Toronto
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- Matthews correlation coefficient, F1 score, accuracy, confusion matrix, class imbalance, classification metrics
- MeSH terms
- Machine Learning, Correlation of Data, Algorithms, Data Interpretation, Statistical, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
