A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results (e.g., due to data shift), which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores (AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler–classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 3×
“To observe the effect of within-class clusters, we also generated synthetic data sets using scikit-learn [49].”
InfluentialLIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“[24,25].”
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 2×
“The SMOTE (Synthetic Minority Over-sampling Technique) [29] generates new samples along the segments connecting a minority sample ( seed ) with its nearest minority neighbors ( co-seeds ).”
(1996) · Cited by 20,179
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007735 · Cited by 4,203 · cited 1×
“The SMOTE-TomekLink [30] handles this problem by a post-filtering step, which searches for pairs of samples whose members belong to different classes but are closer to each other than to any of the elements of their own class.”
Encyclopedia of Machine Learning
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,465
Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.187 · Cited by 1,890 · cited 1×
“Therefore, it is recommended to repeat the procedure a few times and to average the results to get a more accurate estimate of the model’s performance [21].”
Learning from Imbalanced Data Sets
Fernández, A., García, S., Galar, M., Prati, R. C., Krawczyk, B., & Herrera, F. (2018) · doi:10.1007/978-3-319-98074-4 · Cited by 1,349 · cited 2×
“For example, if the imbalance ratio is high, accuracy (1) is determined almost exclusively by the success of the classification of the majority class samples [15].”
Cited by
Fisher, T. B., Saini, G., Rekha, T. S., Krishnamurthy, J., Bhattarai, S., Callagy, G., Webber, M., Janssen, E. A. M., Kong, J., & Aneja, R. (2024) · Breast Cancer Research · doi:10.1186/s13058-023-01752-y · Cited by 42
“The tile histology classification performance was evaluated by a stratified eightfold cross-validation strategy [23–25].”
Method
Metadata
- Authors
- Szeghalmy, S., & Fazekas, A.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2023-02-20
- Volume
- 23
- Issue
- 4
- Pages
- 2333
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W4321373744
- Cited by
- 162
- References
- 46
- FWCI
- 14.23
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.