Comparing two K-category assignments by a K-category correlation coefficient
Abstract
Predicted assignments of biological sequences are often evaluated by Matthews correlation coefficient. However, Matthews correlation coefficient applies only to cases where the assignments belong to two categories, and cases with more than two categories are often artificially forced into two categories by considering what belongs and what does not belong to one of the categories, leading to the loss of information. Here, an extended correlation coefficient that applies to K-categories is proposed, and this measure is shown to be highly applicable for evaluating prediction of RNA secondary structure in cases where some predicted pairs go into the category "unknown" due to lack of reliability in predicted pairs or unpaired residues. Hence, predicting base pairs of RNA secondary structure can be a three-category problem. The measure is further shown to be well in agreement with existing performance measures used for ranking protein secondary structure predictions. Server and software is available at http://rk.kvl.dk/.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The phi coefficient has no canonical 3×3 analog; the multi-class MCC extension of Gorodkin (2004) does not reduce to either Pearson on a chosen encoding or to Kendall's τb.
“The phi coefficient has no canonical 3 × 3 analog; the multi-class MCC extension of Gorodkin (2004) does not reduce to either Pearson on a chosen encoding or to Kendall's τ b .”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Gorodkin (2004)
MCC was re-proposed by Baldi and colleagues in 2000 as a standard performance metric for machine learning with a natural extension to the multiclass case.
“MCC was re-proposed by Baldi and colleagues [39] in 2000 as a standard performance metric for machine learning with a natural extension to the multiclass case [40].”
From The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020) · cites this paper as [40]
Cites
Kabsch, W., & Sander, C. (1983) · Biopolymers · doi:10.1002/bip.360221211 · Cited by 15,320 · see citing works
The protein data bank: A computer-based archival file for macromolecular structures
Bernstein, F. C., Koetzle, T. F., Williams, G. J. B., Meyer, E. F., Brice, M. D., Rodgers, J. R., Kennard, O., Shimanouchi, T., & Tasumi, M. (1977) · Journal of Molecular Biology · doi:10.1016/s0022-2836(77)80200-3 · Cited by 8,582 · see citing works
Protein secondary structure prediction based on position-specific scoring matrices.
Jones, D. C. (1999) · Journal of Molecular Biology · doi:10.1006/jmbi.1999.3091 · Cited by 5,741 · see citing works
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · see citing works
Identification of prokaryotic and eukaryotic signal peptides and prediction of their cleavage sites
Nielsen, H., Engelbrecht, J., Brunak, S., & von HEIJNE, G. (1997) · Protein Engineering Design and Selection · doi:10.1093/protein/10.1.1 · Cited by 5,324 · see citing works
Prediction of Protein Secondary Structure at Better than 70% Accuracy
Rost, B., & Sander, C. (1993) · Journal of Molecular Biology · doi:10.1006/jmbi.1993.1413 · Cited by 2,927 · see citing works
A hidden Markov model for predicting transmembrane helices in protein sequences.
Sonnhammer, E. L. L., von HEIJNE, G., & Krogh, A. (1998) · PubMed · Cited by 2,483 · see citing works
Assessing the accuracy of prediction algorithms for classification: an overview
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., & Nielsen, H. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.5.412 · Cited by 2,111 · see citing works
[31] PHD: Predicting one-dimensional protein structure by profile-based neural networks
Rost, B. (1996) · Methods in enzymology on CD-ROM/Methods in enzymology · doi:10.1016/s0076-6879(96)66033-9 · Cited by 1,282 · see citing works
Hidden Markov models for detecting remote protein homologies.
Karplus, K. J., Barrett, C., & Hughey, R. (1998) · Bioinformatics · doi:10.1093/bioinformatics/14.10.846 · Cited by 1,108 · see citing works
Human Retroviruses and Aids - 1999
KUIKEN, C. L., & Foley, B. T. (1999) · doi:10.2172/1186019 · Cited by 952 · see citing works
Multi-class protein fold recognition using support vector machines and neural networks
Ding, C. H. Q., & Dubchak, I. L. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.4.349 · Cited by 845 · see citing works
Cuff, J. A., & Barton, G. J. (2000) · Proteins Structure Function and Bioinformatics · doi:10.1002/1097-0134(20000815)40:3 · Cited by 768 · see citing works
Pollastri, G., Przybylski, D., Rost, B., & Baldi, P. (2002) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.10082 · Cited by 736 · see citing works
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
Cuff, J. A., & Barton, G. J. (1999) · Proteins Structure Function and Bioinformatics · doi:10.1002/(sici)1097-0134(19990301)34:4 · Cited by 660 · see citing works
Le Traitement des Variables Vectorielles
Escoufier, Y. (1973) · Biometrics · doi:10.2307/2529140 · Cited by 625 · see citing works
Finding the genes in genomic DNA
Burge, C. B., & Karlin, S. (1998) · Current Opinion in Structural Biology · doi:10.1016/s0959-440x(98)80069-9 · Cited by 586 · see citing works
Hansen, J. E., Lund, O., Tolstrup, N., Gooley, A. A., Williams, K. L., & Brunak, S. (1998) · Glycoconjugate Journal · doi:10.1023/a:1006960004440 · Cited by 508 · see citing works
Dynalign: an algorithm for finding the secondary structure common to two RNA sequences
Mathews, D. H., & Turner, D. H. (2002) · Journal of Molecular Biology · doi:10.1006/jmbi.2001.5351 · Cited by 374 · see citing works
Cascaded multiple classifiers for secondary structure prediction
Ouali, M., & King, R. D. (2000) · Protein Science · doi:10.1110/ps.9.6.1162 · Cited by 340 · see citing works
Redefining the goals of protein secondary structure prediction
Rost, B., Sander, C., & Schneider, R. (1994) · Journal of Molecular Biology · doi:10.1016/s0022-2836(05)80007-5 · Cited by 333 · see citing works
Benson, D. A. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg057 · Cited by 312 · see citing works
Dowell, R. D., & Eddy, S. R. (2004) · BMC Bioinformatics · doi:10.1186/1471-2105-5-71 · Cited by 280 · see citing works
Alignments grow, secondary structure prediction improves
Przybylski, D., & Rost, B. (2001) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.10029 · Cited by 206 · see citing works
Prediction of RNA Secondary Structure by Energy Minimization
Zuker, M. (1994) · Humana Press eBooks · doi:10.1385/0-89603-276-0:267 · Cited by 201 · see citing works
Using reliability information to annotate RNA secondary structures
Zuker, M., & Jacobson, A. B. (1998) · RNA · doi:10.1017/s1355838298980116 · Cited by 174 · see citing works
EVA: evaluation of protein structure prediction servers
Koh, I. Y. Y. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg619 · Cited by 168 · see citing works
Discovering common stem-loop motifs in unaligned RNA sequences
Gorodkin, J. (2001) · Nucleic Acids Research · doi:10.1093/nar/29.10.2135 · Cited by 138 · see citing works
Self-citationRNA Interactions in the 5′ Region of the HIV-1 Genome
Damgaard, C. K., Andersen, E. S., Knudsen, B., Gorodkin, J., & Kjems, J. (2003) · Journal of Molecular Biology · doi:10.1016/j.jmb.2003.12.010 · Cited by 123 · see citing works
Self-citationRefined approximations to permutation tests for multivariate inference
Kazi-Aoual, F., Hitier, S., Sabatier, R., & Lebreton, J. (1995) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(94)00064-2 · Cited by 86 · see citing works
Combining many multiple alignments in one improved alignment.
Bucka-Lassen, K., Caprani, O., & Hein, J. (1999) · Bioinformatics · doi:10.1093/bioinformatics/15.2.122 · Cited by 42 · see citing works
Flexible structural comparison allowing hinge-bending, swiveling motions
Verbitsky, G., Nussinov, R., & Wolfson, H. J. (1999) · Proteins Structure Function and Bioinformatics · doi:10.1002/(sici)1097-0134(19990201)34:2 · Cited by 38 · see citing works
MatrixPlot: visualizing sequence constraints
Gorodkin, J., Stærfeldt, H., Lund, O., & Brunak, S. (1999) · Computer applications in the biosciences · doi:10.1093/bioinformatics/15.9.769 · Cited by 35 · see citing works
Self-citationSemi-automated update and cleanup of structural RNA alignment databases
Gorodkin, J., Zwieb, C. W., & Knudsen, B. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.7.642 · Cited by 16 · see citing works
Self-citationEvolutionary rate variation and RNA secondary structure prediction
Knudsen, B., Andersen, E. S., Damgaard, C. K., et al. (2004) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2004.04.001 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606 · see citing works
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406 · see citing works
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324 · see citing works
A mountable toilet system for personalized health monitoring via the analysis of excreta
Park, S.-M., Won, D. D., Lee, B. J., Escobedo, D., Esteva, A., Aalipour, A., Ge, T. J., Kim, J. H., Suh, S., Choi, E. H., et al. (2020) · Nature Biomedical Engineering · doi:10.1038/s41551-020-0534-9 · Cited by 175 · see citing works
Marchese Robinson, R. L., Palczewska, A., Palczewski, J., & Kidley, N. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00753 · Cited by 167 · see citing works
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130 · see citing works
Laengsri, V., Shoombuatong, W., Adirojananon, W., Nantasenamat, C., Prachayasittikul, V., & Nuchnoi, P. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0929-2 · Cited by 66 · see citing works
Automated Identification of Eviction Status From Electronic Health Record Notes
Yao, Z., Tsai, J., Liu, W., Levy, D. A., Druhl, E., Reisman, J. I., & Yu, H. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad081 · Cited by 15 · see citing works
- Authors
- Gorodkin, J.
- Corresponding author
- Gorodkin, J.
- ORCID
- Gorodkin, J.
- Venue
- Computational Biology and Chemistry
- Publisher
- Elsevier BV
- Published
- 2004-12
- Volume
- 28
- Issue
- 5-6
- Pages
- 367-374
- ISSN
- 1476-9271 (ISSN-L)
- OpenAlex
- W2059185913
- Cited by
- 545
- References
- 35
- Citations by year
- 2026: 63 · 2025: 48 · 2024: 49 · 2023: 50 · 2022: 60 · 2021: 84 · 2020: 56 · 2019: 38 · 2018: 22 · 2017: 18 · 2016: 7 · 2015: 10 · 2014: 11 · 2013: 8 · 2012: 5
- FWCI
- 0.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Computational Biology and Chemistry (publishedVersion)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
- Funders
- Teknologi og Produktion, Det Frie Forskningsråd
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Keywords
- Matthews correlation coefficient, multi-class classification, forecast evaluation, RNA secondary structure prediction, secondary structure prediction, performance indicators
- MeSH terms
- Molecular Sequence Data, Proteins, RNA, Databases, Factual, Sequence Alignment, Protein Structure, Secondary, Computational Biology, Base Pairing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
