The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
The Role of Balanced Training and Testing Data Sets for Binary Classifiers in Bioinformatics
Open accessAbstract
Training and testing of conventional machine learning models on binary classification problems depend on the proportions of the two outcomes in the relevant data sets. This may be especially important in practical terms when real-world applications of the classifier are either highly imbalanced or occur in unknown proportions. Intuitively, it may seem sensible to train machine learning models on data similar to the target data in terms of proportions of the two binary outcomes. However, we show that this is not the case using the example of prediction of deleterious and neutral phenotypes of human missense mutations in human genome data, for which the proportion of the binary outcome is unknown. Our results indicate that using balanced training data (50% neutral and 50% deleterious) results in the highest balanced accuracy (the average of True Positive Rate and True Negative Rate), Matthews correlation coefficient, and area under ROC curves, no matter what the proportions of the two phenotypes are in the testing data. Besides balancing the data by undersampling the majority class, other techniques in machine learning include oversampling the minority class, interpolating minority-class data points and various penalties for misclassifying the minority class. However, these techniques are not commonly used in either the missense phenotype prediction problem or in the prediction of disordered residues in proteins, where the imbalance problem is substantial. The appropriate approach depends on the amount of available data and the specific problem at hand.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
MUSCLE: multiple sequence alignment with high accuracy and high throughput
Edgar, R. C. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh340 · Cited by 44,662
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Biometry: The Principles and Practice of Statistics in Biological Research
Sokal, R. R., & Rohlf, F. J. (1969) · Cited by 20,963
A method and server for predicting damaging missense mutations.
Adzhubei, I. A., Schmidt, S., Peshkin, L., Ramensky, V. E., Gerasimova, A., Bork, P., Kondrashov, A. S., & Sunyaev, S. R. (2010) · Nature Methods · doi:10.1038/nmeth0410-248 · Cited by 13,194
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,122
MUSCLE: a multiple sequence alignment method with reduced time and space complexity
Edgar, R. C. (2004) · BMC Bioinformatics · doi:10.1186/1471-2105-5-113 · Cited by 8,970
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 12×
“In Figure 4c, the MCC of train_50 achieves the best results for most of the testing data sets; train_30 is just a big higher for testing at 0.2 and 0.3, and train_70 is a bit higher at 0.9.”
MethodInfluentialADASYN: Adaptive synthetic sampling approach for imbalanced learning
He, H., Bai, Y., Garcia, E. A., & Li, S. (2008) · 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) · doi:10.1109/ijcnn.2008.4633969 · Cited by 4,521
Borderline-SMOTE : A New Over-Sampling Method in Imbalanced Data Sets Learning
Han, H., Wang, W.-Y., & Mao, B.-H. (2005) · Lecture notes in computer science · doi:10.1007/11538059_91 · Cited by 4,331
Pearson, K. (1900) · The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440009463897 · Cited by 4,069
A hidden Markov model for predicting transmembrane helices in protein sequences.
Sonnhammer, E. L. L., von HEIJNE, G., & Krogh, A. (1998) · PubMed · Cited by 2,505
Human non-synonymous SNPs: server and survey
Ramensky, V. (2002) · Nucleic Acids Research · doi:10.1093/nar/gkf493 · Cited by 2,280
Prediction and Functional Analysis of Native Disorder in Proteins from the Three Kingdoms of Life
Ward, J. J., Sodhi, J. S., McGuffin, L. J., Buxton, B. F., & Jones, D. T. (2004) · Journal of Molecular Biology · doi:10.1016/j.jmb.2004.02.002 · Cited by 2,017
Dosztanyi, Z., Csizmok, V., Tompa, P., & Simon, I. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti541 · Cited by 1,960
The foundations of cost-sensitive learning
Elkan, C. (2001) · Cited by 1,945
Making Large-Scale Support Vector Machine Learning Practical
Joachims, T. (1998) · Advances in Kernel Methods · doi:10.7551/mitpress/1130.003.0015 · Cited by 1,613
UniProt Knowledgebase: a hub of integrated protein data
Magrane, M., & Consortium, U. (2011) · Database · doi:10.1093/database/bar009 · Cited by 1,463
Learning to Classify Text Using Support Vector Machines
Joachims, T. (2002) · Springer eBooks · doi:10.1007/978-1-4615-0907-3 · Cited by 1,366
(1976) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1976.4309452 · Cited by 1,078
Protein database searches using compositionally adjusted substitution matrices
Altschul, S. F., Wootton, J. C., Gertz, E. M., Agarwala, R., Morgulis, A., Schäffer, A. A., & Yu, Y. (2005) · The FEBS Journal · doi:10.1111/j.1742-4658.2005.04945.x · Cited by 1,060
Dosztányi, Z., Csizmók, V., Tompa, P., & Simon, I. (2005) · Journal of Molecular Biology · doi:10.1016/j.jmb.2005.01.071 · Cited by 1,019
A Multiple Resampling Method for Learning from Imbalanced Data Sets
Estabrooks, A., Jo, T., & Japkowicz, N. (2004) · Computational Intelligence · doi:10.1111/j.0824-7935.2004.t01-1-00228.x · Cited by 1,014
Capriotti, E., Calabrese, R., & Casadio, R. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl423 · Cited by 943
Length-dependent prediction of protein intrinsic disorder
Peng, K., Radivojac, P., Vucetic, S., Dunker, A. K., & Obradovic, Z. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-208 · Cited by 928
SNAP: predict effect of non-synonymous polymorphisms on function
Bromberg, Y., & Rost, B. (2007) · Nucleic Acids Research · doi:10.1093/nar/gkm238 · Cited by 863
PrDOS: prediction of disordered protein regions from amino acid sequence
Ishida, T., & Kinoshita, K. (2007) · Nucleic Acids Research · doi:10.1093/nar/gkm363 · Cited by 838
Improving Identification of Difficult Small Classes by Balancing Class Distribution
Laurikkala, J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-48229-6_9 · Cited by 748
Functional annotations improve the predictive score of human disease-related mutations in proteins
Calabrese, R., Capriotti, E., Fariselli, P., Martelli, P. L., & Casadio, R. (2009) · Human Mutation · doi:10.1002/humu.21047 · Cited by 690
The DISOPRED server for the prediction of protein disorder
Ward, J. J., McGuffin, L. J., Bryson, K., Buxton, B. F., & Jones, D. T. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth195 · Cited by 688
Class imbalances versus small disjuncts
Jo, T., & Japkowicz, N. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007737 · Cited by 683
Yang, Z. R., Thomson, R., McNeil, P., & Esnouf, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti534 · Cited by 644
SDM--a server for predicting effects of mutations on protein stability and malfunction
Worth, C. L., Preissner, R., & Blundell, T. L. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr363 · Cited by 565
ESpritz: accurate and fast prediction of protein disorder
Walsh, I., Martin, A. J. M., Di Domenico, T., & Tosatto, S. C. E. (2012) · Bioinformatics · doi:10.1093/bioinformatics/btr682 · Cited by 532
Loss of Protein Structure Stability as a Major Causative Factor in Monogenic Disease
Yue, P., Li, Z., & Moult, J. (2005) · Journal of Molecular Biology · doi:10.1016/j.jmb.2005.08.020 · Cited by 526
An instance-weighting method to induce cost-sensitive trees
Ting, K. M. (2002) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2002.1000348 · Cited by 504
PMUT: a web-based tool for the annotation of pathological mutations on proteins
Ferrer-Costa, C., Gelpi, J. L., Zamakola, L., Parraga, I., de la Cruz, X., & Orozco, M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti486 · Cited by 496
AL2CO: calculation of positional conservation in a protein sequence alignment
Pei, J., & Grishin, N. V. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.8.700 · Cited by 483
SNPs3D: Candidate gene and SNP selection for association studies
Yue, P., Melamud, E., & Moult, J. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-166 · Cited by 464
Velez, D. R., White, B. C., Motsinger, A. A., Bush, W. S., Ritchie, M. D., Williams, S. M., & Moore, J. H. (2007) · Genetic Epidemiology · doi:10.1002/gepi.20211 · Cited by 384
Machine Learning from Imbalanced Data Sets 101
Provost, F. J. (2008) · Faculty Digital Archive (New York University Florence) · Cited by 378
The effect of class distribution on classifier learning: an empirical study
Weiss, G. M., & Provost, F. J. (2001) · doi:10.7282/t3-vpfw-sf95 · Cited by 352
Cost-sensitive learning methods for imbalanced data
Thai-Nghe, N., Gantner, Z., & Schmidt-Thieme, L. (2010) · The 2010 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596486 · Cited by 342
Making sense of cancer genomic data
Chin, L., Hahn, W. C., Getz, G., & Meyerson, M. (2011) · Genes & Development · doi:10.1101/gad.2017311 · Cited by 336
A three-state prediction of single point mutations on protein stability changes
Capriotti, E., Fariselli, P., Rossi, I., & Casadio, R. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-s2-s6 · Cited by 328
Identification and Analysis of Deleterious Human SNPs
Yue, P., & Moult, J. (2006) · Journal of Molecular Biology · doi:10.1016/j.jmb.2005.12.025 · Cited by 265
SNAP predicts effect of mutations on protein function
Bromberg, Y., Yachdav, G., & Rost, B. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn435 · Cited by 261
Alber, T., Sun, D. P., Nye, J. A., Muchmore, D. C., & Matthews, B. W. (1987) · Biochemistry · doi:10.1021/bi00387a002 · Cited by 249
Ferrer-Costa, C., Orozco, M., & de la Cruz, X. (2002) · Journal of Molecular Biology · doi:10.1006/jmbi.2001.5255 · Cited by 211
nsSNPAnalyzer: identifying disease-associated nonsynonymous single nucleotide polymorphisms
Bao, L., Zhou, M., & Cui, Y. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki372 · Cited by 207
Zhang, T., Faraggi, E., Xue, B., Dunker, A. K., Uversky, V. N., & Zhou, Y. (2012) · Journal of Biomolecular Structure and Dynamics · doi:10.1080/073911012010525022 · Cited by 177
Easy retrieval of single amino-acid polymorphisms and phenotype information using SwissVar
Mottaz, A., David, F. P. A., Veuthey, A., & Yip, Y. L. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq028 · Cited by 165
Sequence‐based prediction of pathological mutations
Ferrer‐Costa, C., Orozco, M., & de la Cruz, X. (2004) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.20252 · Cited by 163
E-MSD: an integrated data resource for bioinformatics
Golovin, A. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh078 · Cited by 158
Bao, L., & Cui, Y. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti365 · Cited by 155
XIII. Electric and magnetic spectroscopy
Arkadiew, W. (1925) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786442508634724 · Cited by 142
Evaluation of disorder predictions in CASP9
Monastyrskyy, B., Fidelis, K. A., Moult, J., Tramontano, A. L., & Kryshtafovych, A. (2011) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.23161 · Cited by 119
Sequence feature-based prediction of protein stability changes upon amino acid substitutions
Teng, S., Srivastava, A. K., & Wang, L. (2010) · BMC Genomics · doi:10.1186/1471-2164-11-s2-s5 · Cited by 114
Predicting deleterious nsSNPs: an analysis of sequence and structural attributes
Dobson, R., Munroe, P. B., Caulfield, M. J., & Saqi, M. A. S. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-217 · Cited by 102
A neural network method for prediction of β-turn types in proteins using evolutionary information
Kaur, H., & Raghava, G. P. S. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth322 · Cited by 101
PreDisorder: ab initio sequence-based prediction of protein disordered regions
Deng, X., Eickholt, J., & Cheng, J. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-436 · Cited by 97
E-MSD: an integrated data resource for bioinformatics
Velankar, S. S. (2004) · Nucleic Acids Research · doi:10.1093/nar/gki058 · Cited by 77
Wei, Q., Wang, L., Wang, Q., Kruger, W. D., & Dunbrack, R. L. (2010) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.22722 · Cited by 68
Self-citationDeleterious SNP prediction: be mindful of your training data!
Care, M. A., Needham, C. J., Bulpitt, A. J., & Westhead, D. R. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btl649 · Cited by 63
Lac repressor genetic map in real space
Pace, H. C., Kercher, M. A., Lu, P., Markiewicz, P., Miller, J. H., Chang, G. A., & Lewis, M. (1997) · Trends in Biochemical Sciences · doi:10.1016/s0968-0004(97)01104-3 · Cited by 59
Support vector machines for the classification and prediction of β‐turn types
Cai, Y., Liu, X., Xu, X.-B., & Chou, K. (2002) · Journal of Peptide Science · doi:10.1002/psc.401 · Cited by 53
Wainreb, G., Ashkenazy, H., Bromberg, Y., Starovolsky-Shitrit, A., Haliloǧlu, T., Ruppin, E., Avraham, K. B., Rost, B., & Ben‐Tal, N. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq528 · Cited by 40
Capriotti, E., Arbiza, L., Casadio, R., Dopazo, J., Dopazo, H., & Martı́-Renom, M. A. (2007) · Human Mutation · doi:10.1002/humu.20628 · Cited by 35
Masica, D. L., Sosnay, P. R., Cutting, G. R., & Karchin, R. (2012) · Human Mutation · doi:10.1002/humu.22110 · Cited by 27
Cost-Sensitive Learning via Priority Sampling to Improve the Return on Marketing and CRM Investment
Cui, G., Wong, M. L., & Wan, X. (2012) · Journal of Management Information Systems · doi:10.2753/mis0742-1222290110 · Cited by 26
Cited by
A guide to machine learning for biologists
Greener, J. G., Kandathil, S. M., Moffat, L., & Jones, D. T. (2022) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-021-00407-0 · Cited by 2,127
Chicco, D., Tötsch, N., & Jurman, G. (2021) · BioData Mining · doi:10.1186/s13040-021-00244-z · Cited by 893
Deep learning based tissue analysis predicts outcome in colorectal cancer
Bychkov, D., Linder, N., Turkki, R., Nordling, S., Kovanen, P. E., Verrill, C., Walliander, M., Lundin, M., Haglund, C., & Lundin, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-21758-3 · Cited by 644
Khan, A. A., Chaudhari, O., & Chandra, R. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122778 · Cited by 530
Shah, H. A., Saeed, F., Yun, S., Park, J.-H., Paul, A., & Kang, J.-M. (2022) · IEEE Access · doi:10.1109/access.2022.3184113 · Cited by 337
Metadata
- Authors
- Wei, Q., & Dunbrack, R. L.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-09
- Volume
- 8
- Issue
- 7
- Pages
- e67863
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W1964421503
- Cited by
- 291
- References
- 74
- FWCI
- 13.99
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.