The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels
Open accessAbstract
This study examines the efficacy of Random Forest and XGBoost classifiers in conjunction with three upsampling techniques—SMOTE, ADASYN, and Gaussian noise upsampling (GNUS)—across datasets with varying class imbalance levels, ranging from moderate to extreme (15% to 1% churn rate). Employing metrics such as F1 score, ROC AUC, PR AUC, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa, this research provides a comprehensive evaluation of classifier performance under different imbalance scenarios, focusing on applications in the telecommunications domain. The findings highlight that tuned XGBoost paired with SMOTE (Tuned_XGB_SMOTE) consistently achieves the highest F1 score and robust performance across all imbalance levels. SMOTE emerged as the most effective upsampling method, particularly when used with XGBoost, whereas Random Forest performed poorly under severe imbalance. ADASYN showed moderate effectiveness with XGBoost but underperformed with Random Forest, and GNUS produced inconsistent results. This study underscores the impact of data imbalance, with MCC, Kappa, and F1 scores fluctuating significantly, whereas ROC AUC and PR AUC remained relatively stable. Moreover, rigorous statistical analyses employing the Friedman test and Nemenyi post hoc comparisons confirmed that the observed improvements in F1 score, PR-AUC, Kappa, and MCC were statistically significant (p < 0.05), with Tuned_XGB_SMOTE significantly outperforming Tuned_RF_GNUS. While differences in ROC-AUC were not significant, the consistency of these results across multiple performance metrics underscores the reliability of our framework, offering a statistically validated and attractive solution for model selection in imbalanced classification scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-00a · Cited by 11,032
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
(2017) · HAL · Cited by 10,512
Larian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618 · cited 1×
“In contrast, OPTUNA is a more sophisticated optimization framework that leverages Bayesian Optimization and Tree-structured Parzen Estimators (TPEs) to iteratively refine the search process based on prior evaluations [65,66].”
(2012) · Cited by 8,009
(2005) · PubMed · Cited by 7,011
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 1×
“This characteristic makes MCC especially valuable for evaluating imbalanced datasets, where metrics like accuracy often fail to provide reliable insights [69,70].”
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
ADASYN: Adaptive synthetic sampling approach for imbalanced learning
He, H., Bai, Y., Garcia, E. A., & Li, S. (2008) · 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) · doi:10.1109/ijcnn.2008.4633969 · Cited by 4,521
Borderline-SMOTE : A New Over-Sampling Method in Imbalanced Data Sets Learning
Han, H., Wang, W.-Y., & Mao, B.-H. (2005) · Lecture notes in computer science · doi:10.1007/11538059_91 · Cited by 4,331
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007735 · Cited by 4,203
Ensemble based systems in decision making
Polikar, R. (2006) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2006.1688199 · Cited by 2,938
Learning from imbalanced data: open challenges and future directions
Krawczyk, B. (2016) · Progress in Artificial Intelligence · doi:10.1007/s13748-016-0094-0 · Cited by 2,459
Addressing the Curse of Imbalanced Training Sets: One-Sided Selection.
Kubát, M., & Matwin, S. (1997) · Cited by 2,230
García, S., Fernández, A., Luengo, J., & Herrera, F. (2010) · Information Sciences · doi:10.1016/j.ins.2009.12.010 · Cited by 2,229
Asymptotic Properties of Nearest Neighbor Rules Using Edited Data
Wilson, D. L. (1972) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1972.4309137 · Cited by 2,130
Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning
Lemaître, G., Nogueira, F., & Aridas, C. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.06570 · Cited by 1,514
Learning from Imbalanced Data Sets
Fernández, A., García, S., Galar, M., Prati, R. C., Krawczyk, B., & Herrera, F. (2018) · doi:10.1007/978-3-319-98074-4 · Cited by 1,349
A Survey of Predictive Modeling on Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · ACM Computing Surveys · doi:10.1145/2907070 · Cited by 1,129
CatBoost: unbiased boosting with categorical features
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09516 · Cited by 1,110
Statistical Comparisons of Classifiers over Multiple Data Sets
DemšarJanez (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248548 · Cited by 1,074
A gradient boosting method to improve travel time prediction
Zhang, Y., & Haghani, A. (2015) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2015.02.019 · Cited by 832
An experimental comparison of classification algorithms for imbalanced credit scoring data sets
Brown, I., & Mues, C. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2011.09.033 · Cited by 730
De Caigny, A., Coussement, K., & De Bock, K. W. (2018) · European Journal of Operational Research · doi:10.1016/j.ejor.2018.02.009 · Cited by 524
A comparison of machine learning techniques for customer churn prediction
Vafeiadis, T., Diamantaras, K. I., Sarigiannidis, G., & Chatzisavvas, K. C. (2015) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2015.03.003 · Cited by 471
Customer churn prediction in telecom using machine learning in big data platform
Ahmad, A. K., Jafar, A., & Aljoumaa, K. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0191-6 · Cited by 421
Customer churn prediction using improved balanced random forests
Xie, Y., Li, X., Ngai, E. W. T., & Ying, W. (2009) · Expert Systems with Applications · doi:10.1016/j.eswa.2008.06.121 · Cited by 412
Ullah, I., Raza, B., Malik, A. K., Imran, M., Islam, S. U., & Kim, S. W. (2019) · IEEE Access · doi:10.1109/access.2019.2914999 · Cited by 393
Customer churn prediction system: a machine learning approach
Lalwani, P., Mishra, M. K., Chadha, J. S., & Sethi, P. (2022) · Computing · doi:10.1007/s00607-021-00908-y · Cited by 326
Amin, A., Anwar, S., Adnan, A., Nawaz, M., Howard, N., Qadir, J., Hawalah, A., & Hussain, A. (2016) · IEEE Access · doi:10.1109/access.2016.2619719 · Cited by 310
Customer churn prediction in telecommunication industry using data certainty
Amin, A., Al-Obeidat, F., Shah, B., Adnan, A., Loo, J., & Anwar, S. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2018.03.003 · Cited by 248
Application of XGBoost algorithm in the optimization of pollutant concentration
Li, J., An, X., Li, Q., Wang, C., Yu, H., Zhou, X., & Geng, Y.-A. (2022) · Atmospheric Research · doi:10.1016/j.atmosres.2022.106238 · Cited by 245
Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers
Wyner, A. J., Olson, M., Bleich, J., & Mease, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.07676 · Cited by 242
Manzoor, A., Qureshi, M. A., Kidney, E., & Longo, L. (2024) · IEEE Access · doi:10.1109/access.2024.3402092 · Cited by 232
Liu, H., Dougherty, E. R., Dy, J. G., Torkkola, K., Tuv, E., Peng, H., Ding, C., Long, F., Berens, M., Parsons, L., et al. (2005) · IEEE Intelligent Systems · doi:10.1109/mis.2005.105 · Cited by 224
Predicting customer churn in mobile networks through analysis of social groups
Richter, Y., Yom-Tov, E., & Slonim, N. (2010) · Proceedings of the 2010 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972801.64 · Cited by 165
Amin, A., Shah, B., Khattak, A. M., Moreira, F., Ali, G., Rocha, Á., & Anwar, S. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.08.015 · Cited by 119
Imani, M., & Arabnia, H. R. (2023) · Technologies · doi:10.3390/technologies11060167 · Cited by 105
Self-citationEarly Diagnosis of Hepatocellular Carcinoma Using Machine Learning Method
Zhang, Z., Tan, J.-X., Wang, F., Dao, F., Zhang, Z., & Lin, H. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.00254 · Cited by 98
Improved customer choice predictions using ensemble methodsPreprint
van Wezel, M., & Potharst, R. (2005) · RePEc: Research Papers in Economics · doi:10.1016/j.ejor.2006.05.029 · Cited by 91
Ensemble Methods in Customer Churn Prediction: A Comparative Analysis of the State-of-the-Art
Bogaert, M., & Delaere, L. (2023) · Mathematics · doi:10.3390/math11051137 · Cited by 85
Cohen’s kappa is a weighted average
Warrens, M. J. (2011) · Statistical Methodology · doi:10.1016/j.stamet.2011.06.002 · Cited by 70
Customer churn prediction in imbalanced datasets with resampling methods: A comparative study
Haddadi, S. J., Farshidvard, A., Silva, F. D. S., dos Reis, J. C., & Reis, M. S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123086 · Cited by 70
Churn Prediction in Telecommunication Using Data Mining Technology
Rahul, J., & Usharani, T. (2011) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2011.020204 · Cited by 63
A Proposed Churn Prediction Model
Nasr, M., Shaaban, E. M., Helmy, Y., & Khedr, A. E. (2019) · Figshare · doi:10.6084/m9.figshare.7763183 · Cited by 56
Churn Prediction in Telecommunication Industry Using Rough Set Approach
Amin, A., Shehzad, S., Khan, C., Ali, I., & Anwar, S. (2014) · Studies in computational intelligence · doi:10.1007/978-3-319-10774-5_8 · Cited by 50
Baghla, S., & Gupta, G. (2022) · Microprocessors and Microsystems · doi:10.1016/j.micpro.2022.104680 · Cited by 49
Customers Churn Prediction using Artificial Neural Networks (ANN) in Telecom Industry
Khan, Y., Shafiq, S., Naeem, A., Ahmed, S., Safwan, N., & Hussain, S. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0100918 · Cited by 48
Sequence Mining for Customer Behaviour Predictions in Telecommunications
Eichinger, F., Nauck, D. D., & Klawonn, F. (2006) · doi:10.5445/ir/1000005948 · Cited by 31
Imani, M., Ghaderpour, Z., Joudaki, M., & Beikmohammadi, A. (2024) · International Conference on Web Research (ICWR) · doi:10.1109/icwr61162.2024.10533320 · Cited by 31
Self-citationRadosavljevik, D., van der Putten, P., & Larsen, K. (2010) · Transactions on machine learning and data mining · Cited by 27
Metadata
- Authors
- Imani, M., Beikmohammadi, A., & Arabnia, H. R.
- Venue
- Technologies
- Publisher
- MDPI AG
- Published
- 2025-02-20
- Volume
- 13
- Issue
- 3
- Pages
- 88
- ISSN
- 2227-7080 (ISSN-L)
- OpenAlex
- W4407843617
- Cited by
- 256
- References
- 64
- FWCI
- 106.53
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.