The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems
Abstract
Categorical data fields characterized by a large number of distinct values represent a serious challenge for many classification and regression algorithms that require numerical inputs. On the other hand, these types of data fields are quite common in real-world data mining applications and often contain potentially relevant information that is difficult to represent for modeling purposes.This paper presents a simple preprocessing scheme for high-cardinality categorical data that allows this class of attributes to be used in predictive models such as neural networks, linear and logistic regression. The proposed method is based on a well-established statistical method (empirical Bayes) that is straightforward to implement as an in-database procedure. Furthermore, for categorical attributes with an inherent hierarchical structure, like ZIP codes, the preprocessing scheme can directly leverage the hierarchy by blending statistics at the various levels of aggregation.While the statistical methods discussed in this paper were first introduced in the mid 1950's, the use of these methods as a preprocessing step for complex models, like neural networks, has not been previously discussed in any literature.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Hierarchical Clustering Schemes
Johnson, S. C. (1967) · Psychometrika · doi:10.1007/bf02289588 · Cited by 4,897
Bayes and Empirical Bayes Methods for Data Analysis
(2001) · Technometrics · doi:10.1198/tech.2001.s608 · Cited by 2,098
Methods for Statistical Data Analysis of Multivariate Observations.
Sen, P. K., & Gnanadesikan, R. (1978) · Journal of the American Statistical Association · doi:10.2307/2286626 · Cited by 897
An Empirical Bayes Approach to Statistics
Robbins, H. E. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-0919-5_26 · Cited by 838
Improving Text Classification by Shrinkage in a Hierarchy of Classes
McCallum, A., Rosenfeld, R., Mitchell, T., & Ng, A. Y. (2022) · doi:10.1184/r1/21708647 · Cited by 471
Estimating probabilities: a crucial task in machine learning
Cestnik, B. (1990) · European Conference on Artificial Intelligence · Cited by 459
Analysis of Categorical Data: Dual Scaling and Its Applications
Gokhale, D. V. (1983) · Technometrics · doi:10.1080/00401706.1983.10487834 · Cited by 414
On estimating probabilities in tree pruning
Cestnik, B., & Ivan Bratko, A. (1991) · Lecture notes in computer science · doi:10.1007/bfb0017010 · Cited by 162
Cited by
CatBoost: unbiased boosting with categorical features
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09516 · Cited by 1,110
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Similarity encoding for learning with dirty categorical variables
Cerda, P., Varoquaux, G., & Kégl, B. (2018) · Machine Learning · doi:10.1007/s10994-018-5724-2 · Cited by 267
Duckworth, C., Chmiel, F. P., Burns, D. K., Zlatev, Z. D., White, N. M., Daniels, T. W. V., Kiuber, M., & Boniface, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-02481-y · Cited by 109
Sachan, S., Almaghrabi, F., Yang, J.-B., & Xu, D.-L. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115597 · Cited by 66
Metadata
- Authors
- Micci-Barreca, D.
- Venue
- ACM SIGKDD Explorations Newsletter
- Published
- 2001-07
- Volume
- 3
- Issue
- 1
- Pages
- 27-32
- ISSN
- 1931-0145 (ISSN-L)
- 1931-0153
- OpenAlex
- W1990836268
- Cited by
- 276
- References
- 11
- FWCI
- 2.07
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. https://app.cochat.ai/papers/10.1145/507533.507538
- The paper
- Micci-Barreca, D. (2001). A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. ACM SIGKDD Explorations Newsletter. https://doi.org/10.1145/507533.507538
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
