The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Precision-Recall versus Accuracy and the Role of Large Data Sets
Open accessAbstract
Practitioners of data mining and machine learning have long observed that the imbalance of classes in a data set negatively impacts the quality of classifiers trained on that data. Numerous techniques for coping with such imbalances have been proposed, but nearly all lack any theoretical grounding. By contrast, the standard theoretical analysis of machine learning admits no dependence on the imbalance of classes at all. The basic theorems of statistical learning establish the number of examples needed to estimate the accuracy of a classifier as a function of its complexity (VC-dimension) and the confidence desired; the class imbalance does not enter these formulas anywhere. In this work, we consider the measures of classifier performance in terms of precision and recall, a measure that is widely suggested as more appropriate to the classification of imbalanced data. We observe that whenever the precision is moderately large, the worse of the precision and recall is within a small constant factor of the accuracy weighted by the class imbalance. A corollary of this observation is that a larger number of examples is necessary and sufficient to address class imbalance, a finding we also illustrate empirically.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
(1992) · Cited by 23,431
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007735 · Cited by 4,203
Natural language processing with Python
Bird, S., Klein, E., & Loper, E. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,955
The class imbalance problem: A systematic study1
Japkowicz, N., & Stephen, S. (2002) · Intelligent Data Analysis · doi:10.3233/ida-2002-6504 · Cited by 3,295
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 7×
“For K-Nearest Neighbor, we performed validation to determine k . k was set to 5 for most of the data sets, except for letter ( k = 3).”
MethodInfluentialOn the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (2015) · Measures of Complexity · doi:10.1007/978-3-319-21852-6_3 · Cited by 3,105
Exploratory Undersampling for Class-Imbalance Learning
Liu, X.-Y., Wu, J., & Zhou, Z.-H. (2009) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.2007853 · Cited by 2,465
On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (1971) · Theory of Probability & Its Applications · doi:10.1137/1116025 · Cited by 2,448
Learnability and the Vapnik-Chervonenkis dimension
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1989) · Journal of the ACM · doi:10.1145/76359.76371 · Cited by 1,839
Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning
Lemaître, G., Nogueira, F., & Aridas, C. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.06570 · Cited by 1,514
Mining with rarity: a unifying framework
Weiss, G. M. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007734 · Cited by 1,393
Neural network learning theoretical foundations
Anthony, M., & Bartlett, P. L. (2009) · Cited by 1,310
Applying Support Vector Machines to Imbalanced Datasets
Akbani, R., Kwek, S., & Japkowicz, N. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30115-8_7 · Cited by 1,134
Learning when training data are costly: the effect of class distribution on tree induction
Weiss, G. M., & Provost, F. (2003) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1199 · Cited by 921
KNN approach to unbalanced data distributions: a case study involving information extraction
Ziang, J. (2003) · International Conference on Machine Learning · Cited by 772
Large Language Models in Machine Translation
Brants, T., Popat, A. C., Xu, P., Och, F. J., & Dean, J. B. (2007) · Cited by 574
Partially Supervised Classification of Text Documents
Liu, B., Lee, W. S., Yu, P. S., & Li, X. (2002) · Cited by 534
AUC Optimization vs. Error Rate Minimization
Cortes, C., & Mohri, M. (2003) · Cited by 530
A general lower bound on the number of examples needed for learning
Ehrenfeucht, A., Haussler, D., Kearns, M., & Valiant, L. (1989) · Information and Computation · doi:10.1016/0890-5401(89)90002-3 · Cited by 403
Exploratory Under-Sampling for Class-Imbalance Learning
Liu, X.-Y., Wu, J., & Zhou, Z.-H. (2006) · Sixth International Conference on Data Mining (ICDM'06) · doi:10.1109/icdm.2006.68 · Cited by 271
A Neyman–Pearson Approach to Statistical Learning
Scott, C. D., & Nowak, R. D. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.856955 · Cited by 159
A general lower bound on the number of examples needed for learning
Ehrenfeucht, A., Haussler, D. H., Kearns, M. J., & Valiant, L. G. (1988) · Conference on Learning Theory · doi:10.5555/93025.93068 · Cited by 138
Learning from Imbalanced Data: Evaluation Matters
Raeder, T., Forman, G., & Chawla, N. V. (2011) · Intelligent systems reference library · doi:10.1007/978-3-642-23166-7_12 · Cited by 77
The Optimal Sample Complexity of PAC LearningPreprint
Hanneke, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.00473 · Cited by 54
Learnability of Bipartite Ranking Functions
Agarwal, S., & Roth, D. (2005) · Lecture notes in computer science · doi:10.1007/11503415_2 · Cited by 29
Learning Abductive Reasoning Using Random Examples
Juba, B. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10099 · Cited by 26
Self-citation
Cited by
Koay, A. M. Y., Ko, R. K. L., Hettema, H., & Radke, K. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00753-1 · Cited by 131
How Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)
Schlicker, N., Baum, K., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2022) · doi:10.31234/osf.io/qhwvx · Cited by 79
Park, J., Na, H. J., & Kim, H. (2024) · IEEE Access · doi:10.1109/access.2024.3519219 · Cited by 10
Metadata
- Authors
- Juba, B., & Le, H. S.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 4039-4048
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2904542630
- Cited by
- 318
- References
- 30
- FWCI
- 10.82
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.