The DOI matches its DataCite record; 32 of the 36 references it cites were found in a registry or an index.
Entity Embeddings of Categorical Variables
Open accessAbstract
We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They are, however, not limited to image processing and analysis but are outperforming other approaches in areas like natural language processing [3, 4, 5], speech recognition and synthesis [6, 7] 2 , and in the analysis of unstructured, tabular-type data using entity embeddings [8, 9].”
From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis
Kruskal, J. B. (1964) · Psychometrika · doi:10.1007/bf02289565 · Cited by 7,270
Measures of Multivariate Skewness and Kurtosis with Applications
Mardia, K. V. (1970) · Biometrika · doi:10.1093/biomet/57.3.519 · Cited by 5,125
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
Yang, B., Yih, W.-T., He, X., Gao, J., & Deng, L. (2014) · arXiv · doi:10.48550/arxiv.1412.6575 · Cited by 2,252
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581
Deep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A., Kingsbury, B. E. D., & Ramabhadran, B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,083
Learning Structured Embeddings of Knowledge Bases
Bordes, A., Weston, J., Collobert, R., & Bengio, Y. (2011) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v25i1.7917 · Cited by 862
Learning distributed representations of concepts.
Hinton, G. E. (1989) · eScholarship (California Digital Library) · Cited by 827
Metric spaces and positive definite functions
Schoenberg, I. J. (1938) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1938-1501980-0 · Cited by 712
Bordes, A., Glorot, X., Weston, J., & Bengio, Y. (2014) · Machine Learning · doi:10.1007/s10994-013-5363-6 · Cited by 675
Learning to Represent Knowledge Graphs with Gaussian Embedding
He, S., Liu, K., Ji, G., & Zhao, J. (2015) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2806416.2806502 · Cited by 402
A latent factor model for highly multi-relational data
Jenatton, R., Roux, N. L., Bordes, A., & Obozinski, G. (2012) · Cited by 373
Empirical evaluation and combination of advanced language modeling techniques
Mikolov, T., Deoras, A., Kombrink, S., Burget, L., & Černocký, J. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-242 · Cited by 278
On the distortion required for embedding finite metric spaces into normed spaces
Matoušek, J. (1996) · Israel Journal of Mathematics · doi:10.1007/bf02761110 · Cited by 99
Constructive Approximation of Discontinuous Functions by Neural Networks
Llanas, B., Sánchez, S. L., & Sáinz, F. J. (2008) · Neural Processing Letters · doi:10.1007/s11063-007-9070-9 · Cited by 49
Learning Hierarchical Structures with Linear Relational Embedding
Paccanaro, A., & Hinton, G. E. (2001) · Cited by 20
Paccanaro, A., & Hinton, G. E. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.857906 · Cited by 17
Structured Embedding via Pairwise Relations and Long-Range Interactions in Knowledge Base
Wu, F. Q., Song, J., Yang, Y., Li, X., Zhang, Z. M., & Zhuang, Y. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9391 · Cited by 11
Cited by
Sorted by citations
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058
Aas, K., Jullum, M., & Løland, A. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103502 · Cited by 1,264
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,400
Fastai: A Layered API for Deep Learning
Howard, J., & Gugger, S. (2020) · Information · doi:10.3390/info11020108 · Cited by 1,055
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2018) · NORA - Norwegian Open Research Archives · Cited by 1,012
Metadata
- Authors
- Guo, C., & Berkhahn, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1604.06737
- OpenAlex
- W2464785945
- Cited by
- 308
- References
- 36
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 39 · 2023: 33 · 2022: 41 · 2021: 65 · 2020: 59 · 2019: 33 · 2018: 16 · 2017: 2 · 2016: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- entity embeddings, categorical variables, one-hot encoding, data sparsity, categorical data visualization, clustering, generalization, Kaggle competition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
