The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis
Open accessAbstract
BACKGROUND: Prediction methods are increasingly used in biosciences to forecast diverse features and characteristics. Binary two-state classifiers are the most common applications. They are usually based on machine learning approaches. For the end user it is often problematic to evaluate the true performance and applicability of computational tools as some knowledge about computer science and statistics would be needed. RESULTS: Instructions are given on how to interpret and compare method evaluation results. For systematic method performance analysis is needed established benchmark datasets which contain cases with known outcome, and suitable evaluation measures. The criteria for benchmark datasets are discussed along with their implementation in VariBench, benchmark database for variations. There is no single measure that alone could describe all the aspects of method performance. Predictions of genetic variation effects on DNA, RNA and protein level are important as information about variants can be produced much faster than their disease relevance can be experimentally verified. Therefore numerous prediction tools have been developed, however, systematic analyses of their performance and comparison have just started to emerge. CONCLUSIONS: The end users of prediction tools should be able to understand how evaluation is done and how to interpret the results. Six main performance evaluation measures are introduced. These include sensitivity, specificity, positive predictive value, negative predictive value, accuracy and Matthews correlation coefficient. Together with receiver operating characteristics (ROC) analysis they provide a good picture about the performance of methods and allow their objective and quantitative comparison. A checklist of items to look at is provided. Comparisons of methods for missense variant tolerance, protein stability changes due to amino acid substitutions, and effects of variations on mRNA splicing are presented.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MUSCLE: multiple sequence alignment with high accuracy and high throughput
Edgar, R. C. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh340 · Cited by 44,662
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
CATH – a hierarchic classification of protein domain structures
Orengo, C., Michie, A., Jones, S., Jones, D., Swindells, M., & Thornton, J. (1997) · Structure · doi:10.1016/s0969-2126(97)00260-8 · Cited by 2,628
Assessing the accuracy of prediction algorithms for classification: an overview
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., & Nielsen, H. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.5.412 · Cited by 2,111
SCOP: a Structural Classification of Proteins database
Hubbard, T. J. P., Murzin, A. G., Brenner, S. E., & Chothia, C. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.1.236 · Cited by 940
Critical assessment of methods of protein structure prediction (CASP) — round x
Moult, J., Fidelis, K., Kryshtafovych, A., Schwede, T., & Tramontano, A. (2014) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.24452 · Cited by 652
SCOP: a Structural Classification of Proteins database
Lo Conte, L. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.257 · Cited by 581
Performance of mutation pathogenicity prediction methods on missense variants
Thusberg, J., Olatubosun, A., & Vihinen, M. (2011) · Human Mutation · doi:10.1002/humu.21445 · Cited by 537
Self-citationHOMSTRAD: A database of protein structure alignments for homologous families
Mizuguchi, K., Deane, C. M., Blundell, T. L., & Overington, J. P. (1998) · Protein Science · doi:10.1002/pro.5560071126 · Cited by 501
Cited by
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143
Integrated omics: tools, advances and future approaches
Misra, B. B., Langefeld, C., Olivier, M., & Cox, L. A. (2019) · Journal of Molecular Endocrinology · doi:10.1530/jme-18-0055 · Cited by 686
Grimm, D. G., Azencott, C., Aicheler, F., Gieraths, U., MacArthur, D. G., Samocha, K. E., Cooper, D. N., Stenson, P. D., Daly, M. J., Smoller, J. W., et al. (2015) · Human Mutation · doi:10.1002/humu.22768 · Cited by 356
Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis
Kolbinger, F. R., Veldhuizen, G. P., Zhu, J., Truhn, D., & Kather, J. N. (2024) · Communications Medicine · doi:10.1038/s43856-024-00492-0 · Cited by 160
Towards precision medicine: advances in computational approaches for the analysis of human variants
Peterson, T. A., Doughty, E., & Kann, M. G. (2013) · Journal of Molecular Biology · doi:10.1016/j.jmb.2013.08.008 · Cited by 123
Metadata
- Authors
- Vihinen, M.
- Venue
- BMC Genomics
- Published
- 2012
- Volume
- 13
- Issue
- Suppl 4
- Pages
- S2
- ISSN
- 1471-2164 (ISSN-L)
- OpenAlex
- W2146841526
- Cited by
- 301
- References
- 33
- Type
- conference abstract
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis. https://app.cochat.ai/papers/10.1186/1471-2164-13-s4-s2
- The paper
- Vihinen, M. (2012). How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis. BMC Genomics. https://doi.org/10.1186/1471-2164-13-s4-s2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.