The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification
Open accessAbstract
Binary classification is a common task for which machine learning and computational statistics are used, and the area under the receiver operating characteristic curve (ROC AUC) has become the common standard metric to evaluate binary classifications in most scientific fields. The ROC curve has true positive rate (also called sensitivity or recall) on the y axis and false positive rate on the x axis, and the ROC AUC can range from 0 (worst result) to 1 (perfect result). The ROC AUC, however, has several flaws and drawbacks. This score is generated including predictions that obtained insufficient sensitivity and specificity, and moreover it does not say anything about positive predictive value (also known as precision) nor negative predictive value (NPV) obtained by the classifier, therefore potentially generating inflated overoptimistic results. Since it is common to include ROC AUC alone without precision and negative predictive value, a researcher might erroneously conclude that their classification was successful. Furthermore, a given point in the ROC space does not identify a single confusion matrix nor a group of matrices sharing the same MCC value. Indeed, a given (sensitivity, specificity) pair can cover a broad MCC range, which casts doubts on the reliability of ROC AUC as a performance measure. In contrast, the Matthews correlation coefficient (MCC) generates a high score in its $$[-1; +1]$$ [ - 1 ; + 1 ] interval only if the classifier scored a high value for all the four basic rates of the confusion matrix: sensitivity, specificity, precision, and negative predictive value. A high MCC (for example, MCC $$=$$ = 0.9), moreover, always corresponds to a high ROC AUC, and not vice versa. In this short study, we explain why the Matthews correlation coefficient should replace the ROC AUC as standard statistic in all the scientific studies involving a binary classification, in all scientific fields.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · cited 1×
“The area under the ROC curve (ROC AUC, also known as c statistic ) is one of the most common statistics used in scientific research to assess binary classifications, and can range from 0 (worst result) to 1 (perfect result) [30].”
BackgroundThe use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
Zweig, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,569
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 5×
“In some previous studies, we argued that the MCC is more informative than confusion entropy [15], F 1 score [6], accuracy [6], balanced accuracy [4], bookmaker informed-ness [4], markedness [4], diagnostic odds ratio [16], Cohen’s Kappa [17], and Brier score [17].”
BackgroundMethodInfluentialSelf-citationBasic principles of ROC analysis
Metz, C. E. (1978) · Seminars in Nuclear Medicine · doi:10.1016/s0001-2998(78)80014-2 · Cited by 6,026
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 2×
“…the φ coefficient [7] for 2 × 2 confusion matrices: a + 1 value corresponds to perfect classification; a value close to 0 corresponds to a prediction made by chance; and − 1 corresponds to a perfectly opposite prediction, where all the negative samples were predicted as positive and vice versa [8].”
BackgroundAUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Taha, A. A., & Hanbury, A. (2015) · BMC Medical Imaging · doi:10.1186/s12880-015-0068-x · Cited by 2,920
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
Cook, N. R. (2007) · Circulation · doi:10.1161/circulationaha.106.672402 · Cited by 2,112
Cited by
Amorim, A. M. B., Piochi, L. F., Gaspar, A. T., Preto, A. J., Rosário-Ferreira, N., & Moreira, I. S. (2024) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.3c00352 · Cited by 132
Screening articles for systematic reviews with ChatGPT
Syriani, E., David, I., & Kumar, G. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101287 · Cited by 82
Ahluwalia, M., Abdalla, M., Sanayei, J., Seyyed-Kalantari, L., Hussain, M., Ali, A., & Fine, B. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220270 · Cited by 26
Richter-Laskowska, M., Sobotnicka, E., & Bednorz, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-90460-y · Cited by 16
Ismail, M., Alrabaee, S., Choo, K.-K. R., Ali, L., & Harous, S. (2024) · Mobile Networks and Applications · doi:10.1007/s11036-024-02367-z · Cited by 12
Metadata
- Authors
- Chicco, D., & Jurman, G.
- Venue
- BioData Mining
- Published
- 2023-02-17
- Volume
- 16
- Issue
- 1
- Pages
- 4-4
- ISSN
- 1756-0381 (ISSN-L)
- OpenAlex
- W4321214126
- Cited by
- 597
- References
- 59
- FWCI
- 52.45
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. https://app.cochat.ai/papers/10.1186/s13040-023-00322-4
- The paper
- Chicco, D., & Jurman, G. (2023). The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining. https://doi.org/10.1186/s13040-023-00322-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.