Mining with rarity: a unifying framework
Abstract
Rare objects are often of great interest and great value. Until recently, however, rarity has not received much attention in the context of data mining. Now, as increasingly complex real-world problems are addressed, rarity, and the related problem of imbalanced data, are taking center stage. This article discusses the role that rare classes and rare cases play in data mining. The problems that can result from these two forms of rarity are described in detail, as are methods for addressing these problems. These descriptions utilize examples from existing research. So that this article provides a good survey of the literature on rarity in data mining. This article also demonstrates that rare classes and rare cases are very similar phenomena---both forms of rarity are shown to cause similar problems during data mining and benefit from the same remediation methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The inspection of the share of epistemic uncertainty of the input data and its correspondence with the model's output confidence can inform the user and eventually trigger rejection of the model's output.
“In this case, the inspection of the share of epistemic uncertainty (namely, the uncertainty due to lack of knowledge) of the input data and its correspondence with the model's output confidence can inform the user and eventually trigger his/her rejection of the model's output [370, 371].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [370]
Cited by
Sorted by citations
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · see citing works
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Borderline-SMOTE : A New Over-Sampling Method in Imbalanced Data Sets Learning
Han, H., Wang, W.-Y., & Mao, B.-H. (2005) · Lecture notes in computer science · doi:10.1007/11538059_91 · Cited by 4,331 · see citing works
Survey on deep learning with class imbalance
Johnson, J. M., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0192-5 · Cited by 3,032 · see citing works
A Survey of Predictive Modeling on Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · ACM Computing Surveys · doi:10.1145/2907070 · Cited by 1,129 · see citing works
On the Class Imbalance Problem
Guo, X., Yin, Y., Dong, C., Yang, G., & Zhou, G. (2008) · 2008 Fourth International Conference on Natural Computation · doi:10.1109/icnc.2008.871 · Cited by 495 · see citing works
KBA: kernel boundary alignment considering imbalanced data distribution
Wu, G., & Chang, E. Y. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.95 · Cited by 380 · see citing works
Amin, A., Anwar, S., Adnan, A., Nawaz, M., Howard, N., Qadir, J., Hawalah, A., & Hussain, A. (2016) · IEEE Access · doi:10.1109/access.2016.2619719 · Cited by 310 · see citing works
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · see citing works
On the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Varshney, K. R., & Alemzadeh, H. (2017) · Big Data · doi:10.1089/big.2016.0051 · Cited by 211 · see citing works
“sample [41], a safe fail mechanism is to always go for manual examination.”
BackgroundKassraian-Fard, P., Matthis, C., Balsters, J. H., Maathuis, M. H., & Wenderoth, N. (2016) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2016.00177 · Cited by 132 · see citing works
Condran, S., Bewong, M., Islam, M. Z., Maphosa, L., & Zheng, L. (2022) · IEEE Access · doi:10.1109/access.2022.3188649 · Cited by 99 · see citing works
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · see citing works
Mugnier, A., Mila, H., Guiraud, F., Brévaux, J., Lecarpentier, M., Martinez, C., Mariani, C., Adib-Lesaux, A., Chastant-Maillard, S., Saegerman, C., et al. (2019) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2019.104746 · Cited by 70 · see citing works
Meaningless comparisons lead to false optimism in medical machine learning
DeMasi, O., Kording, K., & Recht, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0184604 · Cited by 63 · see citing works
Predicting employee absenteeism for cost effective interventions
Lawrance, N., Petrides, G., & Guerry, M.-A. (2021) · Decision Support Systems · doi:10.1016/j.dss.2021.113539 · Cited by 40 · see citing works
Sample Subset Optimization for Classifying Imbalanced Biological Data
Yang, P., Zhang, Z., Zhou, B. B., & Zomaya, A. Y. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-20847-8_28 · Cited by 31 · see citing works
Classification with LSTM Networks in User Behaviour Analytics with Unbalanced Environment
Parshutin, S., Kirshners, A., Kornijenko, Y., Zabiniako, V., Gasparovica-Asite, M., & Rozkalns, A. (2021) · Automatic Control and Computer Sciences · doi:10.3103/s0146411621010077 · Cited by 7 · see citing works
- Authors
- Weiss, G. M.
- Affiliations
- AT&T (United States) (US)
- Corresponding author
- Weiss, Gary M.
- ORCID
- Weiss, Gary M.
- Venue
- ACM SIGKDD Explorations Newsletter
- Published
- 2004-06
- Volume
- 6
- Issue
- 1
- Pages
- 7-19
- ISSN
- 1931-0145 (ISSN-L)
- 1931-0153
- OpenAlex
- W1965895350
- Cited by
- 1,393
- References
- 0
- Citations by year
- 2026: 26 · 2025: 38 · 2024: 43 · 2023: 53 · 2022: 61 · 2021: 75 · 2020: 86 · 2019: 93 · 2018: 71 · 2017: 68 · 2016: 69 · 2015: 76 · 2014: 85 · 2013: 69 · 2012: 99
- FWCI
- 32.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Explorations Newsletter (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining Algorithms and Applications
- Keywords
- rarity, rare classes, rare case, class imbalance, data mining, remediation strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
