The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Performance measures in evaluating machine learning based bioinformatics predictors for classifications
Open accessAbstract
Background: Many existing bioinformatics predictors are based on machine learning technology. When applying these predictors in practical studies, their predictive performances should be well understood. Different performance measures are applied in various studies as well as different evaluation methods. Even for the same performance measure, different terms, nomenclatures or notations may appear in different context. Results: We carried out a review on the most commonly used performance measures and the evaluation methods for bioinformatics predictors. Conclusions: It is important in bioinformatics to correctly understand and interpret the performance, as it is the key to rigorously compare performances of different predictors and to choose the right predictor.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 1×
“The Matthew’s Correlation Coefficient was first proposed in 1975 [52].”
BackgroundSaito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
ML-KNN: A lazy learning approach to multi-label learning
Zhang, M.-L., & Zhou, Z.-H. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.12.019 · Cited by 3,552
Multi-Label Classification: An Overview
Tsoumakas, G., & Katakis, I. (2007) · International Journal of Data Warehousing and Mining · doi:10.4018/jdwm.2007070101 · Cited by 2,508
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1×
“A safe evaluation protocol involving the feature selection is to leave the testing sample out before the feature selection process [60].”
BackgroundMachine learning applications in genetics and genomics
Libbrecht, M. W., & Noble, W. S. (2015) · Nature Reviews Genetics · doi:10.1038/nrg3920 · Cited by 2,084
The Balanced Accuracy and Its Posterior Distribution
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.764 · Cited by 1,816 · cited 1×
“The balanced accuracy solves most of the bias problems in an imbalanced dataset test [45,49].”
BackgroundTsoumakas, G., Katakis, I., & Vlahavas, I. (2009) · Data Mining and Knowledge Discovery Handbook · doi:10.1007/978-0-387-09823-4_34 · Cited by 1,410
Some remarks on protein attribute prediction and pseudo amino acid composition
Chou, K.-C. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2010.12.024 · Cited by 1,225 · cited 1×
“The jackknife test has been widely applied in evaluating many bioinformatics predictors [4,17,38,39], as it can easily generate fixed values of performance measures, which makes it easy to compare different predictors [40,41].”
MethodChou, K.-C., & Shen, H.-B. (2008) · Nature Protocols · doi:10.1038/nprot.2007.494 · Cited by 1,223
Data mining and knowledge discovery handbook
(2011) · Choice Reviews Online · doi:10.5860/choice.48-5729 · Cited by 1,156
Prediction of Protein Structural Classes
Chou, K.-C., & Zhang, C.-T. (1995) · Critical Reviews in Biochemistry and Molecular Biology · doi:10.3109/10409239509083488 · Cited by 1,058
Network‐based global inference of human disease genes
Wu, X., Jiang, R., Zhang, M. Q., & Li, S. (2008) · Molecular Systems Biology · doi:10.1038/msb.2008.27 · Cited by 660
(1998) · Machine Learning · doi:10.1023/a:1017181826899 · Cited by 629
The Precision Medicine Initiative: A New National Effort
Ashley, E. A. (2015) · JAMA · doi:10.1001/jama.2015.3595 · Cited by 560
GPS-SUMO: a tool for the prediction of sumoylation sites and SUMO-interaction motifs
Zhao, Q., Xie, Y., Zheng, Y., Jiang, S., Liu, W., Mu, W., Liu, Z., Zhao, Y., Xue, Y., & Ren, J. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku383 · Cited by 522
Impacts of Bioinformatics to Medicinal Chemistry
Chou, K.-C. (2015) · Medicinal Chemistry · doi:10.2174/1573406411666141229162834 · Cited by 518
Lin, H., Deng, E.-Z., Ding, H., Chen, W., & Chou, K.-C. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku1019 · Cited by 508 · cited 1×
“The jackknife test has been widely applied in evaluating many bioinformatics predictors [4,17,38,39], as it can easily generate fixed values of performance measures, which makes it easy to compare different predictors [40,41].”
MethodPseudo Amino Acid Composition and its Applications in Bioinformatics, Proteomics and System Biology
Chou, K.-C. (2009) · Current Proteomics · doi:10.2174/157016409789973707 · Cited by 478
Some remarks on predicting multi-label attributes in molecular biosystems
Chou, K.-C. (2013) · Molecular BioSystems · doi:10.1039/c3mb25555g · Cited by 413
Andreassen, O. A., Thompson, W. K., Schork, A. J., Ripke, S., Mattingsdal, M., Kelsoe, J. R., Kendler, K. S., O'Donovan, M. C., Rujescu, D., Werge, T., et al. (2013) · PLoS Genetics · doi:10.1371/journal.pgen.1003455 · Cited by 407
Guo, S.-H., Deng, E.-Z., Xu, L.-Q., Ding, H., Lin, H., Chen, W., & Chou, K.-C. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu083 · Cited by 375
Normalization, testing, and false discovery rate estimation for RNA-sequencing data
Li, J., Witten, D. M., Johnstone, I. M., & Tibshirani, R. (2012) · Biostatistics · doi:10.1093/biostatistics/kxr031 · Cited by 369
iRNA-Methyl: Identifying N6-methyladenosine sites using pseudo nucleotide composition
Chen, W., Feng, P., Ding, H., Lin, H., & Chou, K.-C. (2015) · Analytical Biochemistry · doi:10.1016/j.ab.2015.08.021 · Cited by 349
Chou, K.-C., Wu, Z.-C., & Xiao, X. (2011) · Molecular BioSystems · doi:10.1039/c1mb05420a · Cited by 334
Application of high-dimensional feature selection: evaluation for genomic prediction in man
Bermingham, M. L., Pong-Wong, R., Spiliopoulou, A., Hayward, C., Rudan, I., Campbell, H., Wright, A. F., Wilson, J. F., Agakov, F., Navarro, P., et al. (2015) · Scientific Reports · doi:10.1038/srep10312 · Cited by 322
A random forest approach to the detection of epistatic interactions in case-control studies
Jiang, R., Tang, W., Wu, X., & Fu, W. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-s1-s65 · Cited by 306
The promise of single-cell sequencing
Eberwine, J., Sul, J.-Y., Bartfai, T., & Kim, J. (2014) · Nature Methods · doi:10.1038/nmeth.2769 · Cited by 304
Model Assessment and Selection
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7_7 · Cited by 265
Modeling gene expression using chromatin features in various cellular contexts
Dong, X., Greven, M. C., Kundaje, A., Djebali, S., Brown, J. B., Cheng, C., Gingeras, T. R., Gerstein, M., Guigó, R., Birney, E., et al. (2012) · Genome Biology · doi:10.1186/gb-2012-13-9-r53 · Cited by 256
Using subsite coupling to predict signal peptides
Chou, K.-C. (2001) · Protein Engineering, Design and Selection · doi:10.1093/protein/14.2.75 · Cited by 255
Hum-PLoc: A novel ensemble classifier for predicting human protein subcellular localization
Chou, K.-C., & Shen, H.-B. (2006) · Biochemical and Biophysical Research Communications · doi:10.1016/j.bbrc.2006.06.059 · Cited by 248
Xue, Y., Liu, Z., Cao, J., Ma, Q., Gao, X., Wang, Q., Jin, C., Zhou, Y., Wen, L., & Ren, J. (2011) · Protein Engineering Design and Selection · doi:10.1093/protein/gzq094 · Cited by 246
iNitro-Tyr: Prediction of Nitrotyrosine Sites in Proteins with General Pseudo Amino Acid Composition
Xu, Y., Wen, X., Wen, L.-S., Wu, L.-Y., Deng, N.-Y., & Chou, K.-C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105018 · Cited by 232
Too Big to Ignore: The Business Case for Big Data
Simon, P. (2013) · Cited by 220
Adaptive least squares support vector machines filter for hand tremor canceling in microsurgery
Liu, Z., Wu, Q., Zhang, Y., & Philip Chen, C. L. (2011) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-011-0012-5 · Cited by 214
Predicting drug-target interactions using restricted Boltzmann machines
Wang, Y., & Zeng, J. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt234 · Cited by 211
The Correlation Between Histone Modifications and Gene Expression
Dong, X., & Weng, Z. (2013) · Epigenomics · doi:10.2217/epi.13.13 · Cited by 194
PREP-Mt: predictive RNA editor for plant mitochondrial genes
Mower, J. P. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-96 · Cited by 180
Du, P., & Li, Y. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-518 · Cited by 163
Self-citationCascleave: towards more accurate prediction of caspase substrate cleavage sites
Song, J., Tan, H., Shen, H., Mahmood, K., Boyd, S. E., Webb, G. I., Akutsu, T., & Whisstock, J. C. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq043 · Cited by 156
Predicting chromatin organization using histone marks
Huang, J., Marco, E., Pinello, L., & Yuan, G. (2015) · Genome biology · doi:10.1186/s13059-015-0740-z · Cited by 122
Epistatic Module Detection for Case-Control Studies: A Bayesian Model with a Gibbs Sampling Strategy
Tang, W., Wu, X., Jiang, R., & Li, Y. (2009) · PLoS Genetics · doi:10.1371/journal.pgen.1000464 · Cited by 120
Kong, L., Zhang, L., & Lv, J. (2013) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2013.11.021 · Cited by 92
Shao, J., Xu, D., Hu, L., Kwan, Y. W., Wang, Y., Kong, X., & Ngai, S. M. (2012) · Molecular BioSystems · doi:10.1039/c2mb25251a · Cited by 72
Proteome-wide discovery of mislocated proteins in cancer
Lee, K., Byun, K., Hong, W.-P., Chuang, H., Pack, C., Bayarsaikhan, E., Paek, S. H., Kim, H., Shin, H. Y., Ideker, T., et al. (2013) · Genome Research · doi:10.1101/gr.155499.113 · Cited by 66
The False Discovery Rate: A Key Concept in Large-Scale Genetic Studies
Chen, J. J., Robeson, P. K., & Schell, M. J. (2010) · Cancer Control · doi:10.1177/107327481001700108 · Cited by 61
Li, T., Du, P.-F., & Xu, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0015411 · Cited by 52
Self-citationCombining multiple approaches for gene microarray classification
Nanni, L., Brahnam, S., & Lumini, A. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts108 · Cited by 49
Recent progress in predicting protein sub-subcellular locations
Du, P.-F., Li, T., & Wang, X. (2011) · Expert Review of Proteomics · doi:10.1586/epr.11.20 · Cited by 48
Self-citationPredicting multisite protein subcellular locations: progress and challenges
Du, P.-F., & Xu, C. (2013) · Expert Review of Proteomics · doi:10.1586/epr.13.16 · Cited by 45
Self-citationJiao, Y.-S., & Du, P.-F. (2015) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2015.11.009 · Cited by 39
Self-citationFan, Y., Zhang, Y., & Shen, H. (2012) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24217 · Cited by 31
Li, X., Liu, T., Tao, P., Wang, C., & Chen, L. (2015) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2015.08.012 · Cited by 23
Cheng, C., Shou, C., Yip, K. Y.-L., & Gerstein, M. (2011) · Genome biology · doi:10.1186/gb-2011-12-11-r111 · Cited by 18
Dayarian, A., Romero, R. J., Wang, Z., Biehl, M. L., Bilal, E., Hormoz, S., Meyer, P., Norel, R., Rhrissorrakrai, K., Bhanot, G., et al. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu490 · Cited by 16
Cited by
Applications of machine learning in drug discovery and development
Vamathevan, J., Clark, D., Czodrowski, P., Dunham, I., Ferran, E., Lee, G., Li, B., Madabhushi, A., Shah, P., Spitzer, M., et al. (2019) · Nature Reviews Drug Discovery · doi:10.1038/s41573-019-0024-5 · Cited by 3,228
Diagnosis of Diabetes Mellitus Using Gradient Boosting Machine (LightGBM)
Rufo, D. D., Debelee, T. G., Ibenthal, A., & Negera, W. G. (2021) · Diagnostics · doi:10.3390/diagnostics11091714 · Cited by 317
ICface: Interpretable and Controllable Face Reenactment Using GANs
Tripathy, S., Kannala, J., & Rahtu, E. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093474 · Cited by 88
“Then we calculated the balanced accuracy [15] and F-score [15] of detecting the presence and absence of AUs in the generated images in comparison to those of the driver images.”
ResultIslam, R., Abdel-Raheem, E., & Tarique, M. (2022) · Computer Methods and Programs in Biomedicine Update · doi:10.1016/j.cmpbup.2022.100074 · Cited by 53
Metadata
- Authors
- Jiao, Y., & Du, P.
- Venue
- Quantitative Biology
- Publisher
- Wiley
- Published
- 2016-12
- Volume
- 4
- Issue
- 4
- Pages
- 320-330
- ISSN
- 2095-4689 (ISSN-L)
- 2095-4697
- OpenAlex
- W2559588458
- Cited by
- 307
- References
- 60
- FWCI
- 4.97
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.