Mining needle in a haystack: classifying rare classes via two-phase rule induction
The DOI matches its Crossref record.
Abstract
Learning models to classify rarely occurring target classes is an important problem with applications in network intrusion detection, fraud detection, or deviation detection in general. In this paper, we analyze our previously proposed two-phase rule induction method in the context of learning complete and precise signatures of rare classes. The key feature of our method is that it separately conquers the objectives of achieving high recall and high precision for the given target class. The first phase of the method aims for high recall by inducing rules with high support and a reasonable level of accuracy. The second phase then tries to improve the precision by learning rules to remove false positives in the collection of the records covered by the first phase rules. Existing sequential covering techniques try to achieve high precision for each individual disjunct learned. In this paper, we claim that such approach is inadequate for rare classes, because of two problems: splintered false positives and error-prone small disjuncts. Motivated by the strengths of our two-phase design, we design various synthetic data models to identify and analyze the situations in which two state-of-the-art methods, RIPPER and C4.5 rules, either fail to learn a model or learn a very poor model. In all these situations, our two-phase approach learns a model with significantly better recall and precision levels. We also present a comparison of the three methods on a challenging real-life network intrusion detection dataset. Our method is significantly better or comparable to the best competitor in terms of achieving better balance between recall and precision.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
“Issues that arise due to imbalanced class distributions have been addressed in the data mining and machine learning literature [Joshi et al. 2001; 2002; Chawla et al. 2004; Phua et al. 2004; Weiss and Hirsh 1998; Vilalta and Ma 2002].”
Self-citationMining with rarity: a unifying framework
Weiss, G. M. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007734 · Cited by 1,393
Network Anomaly Detection: Methods, Systems and Tools
Bhuyan, M. H., Bhattacharyya, D. K., & Kalita, J. K. (2014) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2013.052213.00046 · Cited by 1,348
Feature bagging for outlier detection
Lazarevic, A., & Kumar, V. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081891 · Cited by 656
Self-citationIncremental Local Outlier Detection for Data Streams
Pokrajac, D., Lazarevic, A., & Latecki, L. J. (2007) · 2007 IEEE Symposium on Computational Intelligence and Data Mining · doi:10.1109/cidm.2007.368917 · Cited by 386
Metadata
- Authors
- Joshi, M. V., Agarwal, R. C., & Kumar, V.
- Venue
- Proceedings of the 2001 ACM SIGMOD international conference on Management of data
- Publisher
- ACM
- Published
- 2001-05
- Pages
- 91-102
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2170335877
- Cited by
- 128
- References
- 0
- FWCI
- 6.26
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Mining needle in a haystack: classifying rare classes via two-phase rule induction. https://app.cochat.ai/papers/10.1145/375663.375673
- The paper
- Joshi, M. V., Agarwal, R. C., & Kumar, V. (2001). Mining needle in a haystack: classifying rare classes via two-phase rule induction. Proceedings of the 2001 ACM SIGMOD international conference on Management of data. https://doi.org/10.1145/375663.375673
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.