A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results (e.g., due to data shift), which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores (AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler–classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 3×
“To observe the effect of within-class clusters, we also generated synthetic data sets using scikit-learn [49].”
InfluentialLIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“[24,25].”
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 2×
“The SMOTE (Synthetic Minority Over-sampling Technique) [29] generates new samples along the segments connecting a minority sample ( seed ) with its nearest minority neighbors ( co-seeds ).”
(1996) · Cited by 20,179
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007735 · Cited by 4,203 · cited 1×
“The SMOTE-TomekLink [30] handles this problem by a post-filtering step, which searches for pairs of samples whose members belong to different classes but are closer to each other than to any of the elements of their own class.”
Encyclopedia of Machine Learning
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,465
Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.187 · Cited by 1,890 · cited 1×
“Therefore, it is recommended to repeat the procedure a few times and to average the results to get a more accurate estimate of the model’s performance [21].”
Learning from Imbalanced Data Sets
Fernández, A., García, S., Galar, M., Prati, R. C., Krawczyk, B., & Herrera, F. (2018) · doi:10.1007/978-3-319-98074-4 · Cited by 1,349 · cited 2×
“For example, if the imbalance ratio is high, accuracy (1) is determined almost exclusively by the success of the classification of the majority class samples [15].”
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
Multilayer perceptrons for classification and regression
Murtagh, F. (1991) · Neurocomputing · doi:10.1016/0925-2312(91)90023-5 · Cited by 1,014
Zhou, Z.-H. (2021) · Springer eBooks · doi:10.1007/978-981-15-1967-3 · Cited by 820 · cited 3×
“…where the vertices have an attribute, the edges be-tween the vertices with the result of this test (with the corresponding value of the given attribute) are labeled, while the leaf tips the decision itself (that is, the value of the attribute representing the corresponding class) [24,27].”
Bansal, M., Goyal, A., & Choudhary, A. (2022) · Decision Analytics Journal · doi:10.1016/j.dajour.2022.100071 · Cited by 768 · cited 1×
“Based on the literature, the nature of the data determines which classifier can be used more successfully without applying any oversampling [56,57].”
Sáez, J. A., Luengo, J., Stefanowski, J., & Herrera, F. (2015) · Information Sciences · doi:10.1016/j.ins.2014.08.051 · Cited by 627 · cited 1×
“The SMOTE-IPF [31] also applies post-filtering, deleting samples from the data set whose classification is not clear based on the votes of the members of an ensemble classifier.”
Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement
Forman, G., & Scholz, M. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1882471.1882479 · Cited by 368 · cited 1×
“There are several measures for evaluating the performance of classifiers [50], but many of them are sensitive to the difference between the size of the classes, so the degree of imbalance must also be considered for their proper interpretation.”
Fernández, A., García, S., del Jesus, M. J., & Herrera, F. (2008) · Fuzzy Sets and Systems · doi:10.1016/j.fss.2007.12.023 · Cited by 310 · cited 1×
“We examined the effect of the two validation techniques on data set collections of the KEEL repository [41,42], which are specifically recommended for investigating imbalanced classification problems.”
Moreno-Torres, J. G., Saez, J. A., & Herrera, F. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2199516 · Cited by 306 · cited 4×
“The difference between the two techniques is known from the literature [20], but there is a lack of extensive studies on the performance of different sampler–classifier combinations when using DOB-SCV instead of SCV for validation.”
InfluentialKovács, G. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.105662 · Cited by 289
Evaluation Measures of the Classification Performance of Imbalanced Data Sets
Gu, Q., Zhu, L., & Cai, Z. (2009) · Communications in Computer and Information Science · doi:10.1007/978-3-642-04962-0_53 · Cited by 254 · cited 1×
“It is more remarkable that the F1 also has a moderate negative correlation with the IR for all classifiers, although F1 is one of the most commonly used measures for imbalanced data sets [55].”
Distribution-balanced stratified cross-validation for accuracy estimation
Zeng, X., & Martinez, T. R. (2000) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/095281300146272 · Cited by 235 · cited 1×
“They suggested that nearby points belonging to the same class should be placed in different folds [22], but the goal is better achieved by the DOB-SCV proposed by Moreno-Torres et al. The DOB-SCV moves a randomly selected sample and its k nearest neighbors into different folds and repeats this…”
Sensor-Based Datasets for Human Activity Recognition – A Systematic Review of Literature
De-La-Hoz-Franco, E., Ariza-Colpas, P., Quero, J. M., & Espinilla, M. (2018) · IEEE Access · doi:10.1109/access.2018.2873502 · Cited by 209 · cited 1×
“…and from the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but not exclusively, used in elderly monitoring systems or medical…”
Smote-variants: A python implementation of 85 minority oversampling techniques
Kovács, G. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.100 · Cited by 197
Fernández, A., del Jesus, M. J., & Herrera, F. (2009) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2008.11.004 · Cited by 194 · cited 1×
“We examined the effect of the two validation techniques on data set collections of the KEEL repository [41,42], which are specifically recommended for investigating imbalanced classification problems.”
Singh, A. P., Ranjan, R. K., & Tiwari, A. (2021) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2021.1907795 · Cited by 182 · cited 1×
“A popular example from the financial field is bank card fraud detection [1,2], where there is a strong imbalance to deal with since the number of frauds is negligible compared to regular transactions.”
Cross-validation Metrics for Evaluating Classification Performance on Imbalanced Data
Wardhani, N. W. S., Rochayani, M. Y., Iriany, A., Sulistyono, A. D., & Lestantyo, P. (2019) · 2019 International Conference on Computer, Control, Informatics and its Applications (IC3INA) · doi:10.1109/ic3ina48034.2019.8949568 · Cited by 166 · cited 1×
“…has to be characterized by a single value, the use of F 1-scores (3), the area under the ROC curve AUC ROC (it can be estimated easily by using the trapezoidal-rule), which is considered robust even against imbalance [51] or perhaps G -mean scores (4), is typical for imbalanced data sets.”
López, V., Fernández, A., & Herrera, F. (2014) · Information Sciences · doi:10.1016/j.ins.2013.09.038 · Cited by 160 · cited 7×
“[19], where V DOB _ SCV and V SCV are the mean performance scores of a sampler–classifiers estimated by the DOB-SCV and SCV methods, respectively ( V ∈ { F 1, AUC , G , Acc } ).”
InfluentialNakamura, M., Kajiwara, Y., Otsuka, A., & Kimura, H. (2013) · BioData Mining · doi:10.1186/1756-0381-6-16 · Cited by 157 · cited 1×
“LVQ-SMOTE [37] uses a special method because the synthetic samples are derived from reference data sets.”
CCR: A combined cleaning and resampling algorithm for imbalanced data classification
Koziarski, M., & Woźniak, M. (2017) · International Journal of Applied Mathematics and Computer Science · doi:10.1515/amcs-2017-0050 · Cited by 129 · cited 1×
“CCR [33] uses a less drastic solution, cleaning the environments of the minority samples by pushing the nearby majority ones outside a circle with a certain radius.”
Adaptive weighted imbalance learning with application to abnormal activity recognition
Gao, X., Chen, Z., Tang, S., Zhang, Y., & Li, J. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.09.064 · Cited by 112 · cited 1×
“…the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but not exclusively, used in elderly monitoring systems or medical applications.”
García, V., Sánchez, J. S., Marqués, A. I., Florencia, R., & Rivera, G. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.113026 · Cited by 108
Chen, Y., Chang, R., & Guo, J. (2021) · IEEE Access · doi:10.1109/access.2021.3068316 · Cited by 107 · cited 1×
“…industrial field, it is worth mentioning fault detection [5], and from the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but…”
ProWSyn: Proximity Weighted Synthetic Oversampling Technique for Imbalanced Data Set Learning
Barua, S., Islam, M. M., & Murase, K. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37456-2_27 · Cited by 102 · cited 1×
“The ProWSyn attempts to generate the right amount of synthetic samples around the minority elements, considering how far they are from the decision boundary [35].”
BackgroundA survey on unsupervised learning for wearable sensor-based activity recognition
Ige, A. O., & Noor, M. H. M. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109363 · Cited by 99
An Introduction to Statistical Analysis in Research
Weaver, K. F., Morales, V., Dunn, S. L., Godde, K., & Weaver, P. F. (2017) · Wiley eBooks · doi:10.1002/9781119454205 · Cited by 95
An efficient fraud detection framework with credit card imbalanced data in financial services
El-Naby, A. A., Hemdan, E. E., & Omera, A. E. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13434-6 · Cited by 78
A comprehensive data‐level investigation of cancer diagnosis on imbalanced data
Gupta, S., & Gupta, M. K. (2021) · Computational Intelligence · doi:10.1111/coin.12452 · Cited by 48 · cited 1×
“From the medical field, countless diagnostic tasks could be mentioned [3,4] where the data sets often do not adequately represent reality.”
Applications of Machine Learning Algorithms and Performance Comparison: A Review
Abdualgalil, B., & Abraham, S. (2020) · International Conference on Emerging Trends in Information Technology and Engineering (ICETITE) · doi:10.1109/ic-etite47903.2020.490 · Cited by 45 · cited 1×
“Based on the literature, the nature of the data determines which classifier can be used more successfully without applying any oversampling [56,57].”
A quasi-linear SVM combined with assembled SMOTE for imbalanced data classification
Zhou, B., Yang, C., Guo, H., & Hu, J. (2013) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2013.6707035 · Cited by 45 · cited 1×
“The Assembled-SMOTE [34] connects minority samples close to the decision boundary with minority samples further away from it, reducing the chance of the new samples falling among the majority ones.”
A Class-Imbalanced Deep Learning Fall Detection Algorithm Using Wearable Sensors
Zhang, J., Li, J., & Wang, W. (2021) · Sensors · doi:10.3390/s21196511 · Cited by 41 · cited 1×
“…the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but not exclusively, used in elderly monitoring systems or medical applications.”
An over-sampling technique with rejection for imbalanced class learning
Lee, J., Kim, N.-R., & Lee, J.-H. (2015) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1145/2701126.2701181 · Cited by 37 · cited 1×
“While according to Lee’s method, the samples generated in the wrong location should be rejected immediately [32].”
A study of data pre-processing techniques for imbalanced biomedical data classification
Xiang, D., Zhou, W., Xiang, Y., Zhang, J., & Liu, S. (2020) · International Journal of Bioinformatics Research and Applications · doi:10.1504/ijbra.2020.109103 · Cited by 33
Link, J., Perst, T., Stoeve, M., & Eskofier, B. M. (2022) · Sensors · doi:10.3390/s22072560 · Cited by 24 · cited 1×
“…the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but not exclusively, used in elderly monitoring systems or medical applications.”
A minority oversampling approach for fault detection with heterogeneous imbalanced data
Liu, J. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115492 · Cited by 22 · cited 1×
“Finally, from the industrial field, it is worth mentioning fault detection [5], and from the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are…”
From Kinect skeleton data to hand gesture recognition with radar
Li, J., Shrestha, A., Kernec, J. L., & Fioranelli, F. (2019) · The Journal of Engineering · doi:10.1049/joe.2019.0557 · Cited by 14 · cited 1×
“…and from the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but not exclusively, used in elderly monitoring systems or medical…”
Face in the Game: Using Facial Action Units to Track Expertise in Competitive Video Game Play
Guglielmo, G., Blom, P. M., Klincewicz, M., Čule, B., & Spronck, P. H. M. (2022) · IEEE Conference on Games (CoG) · doi:10.1109/cog51982.2022.9893599 · Cited by 6 · cited 1×
“Finally, from the industrial field, it is worth mentioning fault detection [5], and from the area of human–machine interaction, we would highlight gesture, emotion [6] recognition and the various areas of activity recognition [7–9], sports [10], gaming [11], and fall detectors [12,13], which are primarily, but not exc…”
Cited by
Fisher, T. B., Saini, G., Rekha, T. S., Krishnamurthy, J., Bhattarai, S., Callagy, G., Webber, M., Janssen, E. A. M., Kong, J., & Aneja, R. (2024) · Breast Cancer Research · doi:10.1186/s13058-023-01752-y · Cited by 42
“The tile histology classification performance was evaluated by a stratified eightfold cross-validation strategy [23–25].”
Method
Metadata
- Authors
- Szeghalmy, S., & Fazekas, A.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2023-02-20
- Volume
- 23
- Issue
- 4
- Pages
- 2333
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W4321373744
- Cited by
- 162
- References
- 46
- FWCI
- 14.23
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.