Linear Algebraic Structure of Word Senses, with Applications to Polysemy
Open accessAbstract
Word embeddings are ubiquitous in NLP and information retrieval, but it is unclear what they represent when the word is polysemous. Here it is shown that multiple word senses reside in linear superposition within the word embedding and simple sparse coding can recover vectors that approximately capture the senses. The success of our approach, which applies to several embedding methods, is mathematically explained using a variant of the random walk on discourses model (Arora et al., 2016). A novel aspect of our technique is that each extracted word sense is accompanied by one of about 2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense. Discourse atoms can be of independent interest, and make the method potentially more useful. Empirical tests are used to verify and support the theory.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Documents can be conceived of as a random walk in a discourse space, where words are more likely to follow other words if they are closer in an embedding space.
“Those papers conceive of documents being a "random walk" in a discourse space, where words are more likely to follow other words if they are closer to them in an embedding space.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
Section 2, “Justification for Linearity Assertion” (page not provided), describes a slow random walk through discourse space that generates nearby words. But it says word probabilities depend on closeness to the current discourse vector, not directly on closeness to the preceding word’s embedding. (LLM, conf=0.85)
Experimental — accuracy not yet measuredUnder the random walk model, a particular relationship follows for the embedding of a word and the embeddings of the words that appear in its contexts.
“Crucially for ALC, Arora et al. (2018) also proves that under this model, a particular relationship will follow for the embedding of a word and the embeddings of the words that appear in the contexts around it.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
The matrix A downweights the contributions of very common but uninformative words when averaging.
“That A serves to downweight the contributions of very common (but uninformative) words when averaging.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
If the random walk model holds, a researcher can obtain an embedding for a word by averaging the embeddings of the words around it and multiplying them by a particular square matrix A.
“The main result from Arora et al. (2018) shows the following: if the random walk model holds, the researcher can obtain an embedding for word w (e.g., "bill") by taking the average of the embeddings of the words around w (u w Þ and multiplying them by a particular square matrix A .”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
Embedding representations can be used to solve analogy reasoning tasks, which implies that the vectors capture relational meaning between words.
“For example, embedding representations can be used to solve analogy reasoning tasks, implying the vectors do indeed capture relational meaning between words (e.g., Arora et al. 2018;Mikolov et al. 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
The Introduction and Section 4.2 discuss how embeddings represent word senses and test that structure, but they do not address solving word-analogy tasks. Searches for analogy reasoning and relational meaning found no passage in our copy supporting the cited example or its inference. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,595 · see citing works
$rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
Aharon, M., Elad, M., & Bruckstein, A. M. (2006) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2006.881199 · Cited by 9,464 · see citing works
(2010) · Cited by 8,261 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · see citing works
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642 · see citing works
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · Cited by 2,913 · see citing works
From Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,845 · see citing works
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981658.981684 · Cited by 2,411 · see citing works
(2009) · Cited by 1,900 · see citing works
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581 · see citing works
Automatic word sense discrimination
Schütze, H. (1998) · Cited by 1,269 · see citing works
Topics in semantic representation
Griffiths, T. L., Steyvers, M., & Tenenbaum, J. B. (2007) · Psychological Review · doi:10.1037/0033-295x.114.2.211 · Cited by 1,122 · see citing works
Improving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,071 · see citing works
Hierarchical Topic Models and the Nested Chinese Restaurant Process
Griffiths, T. L., Jordan, M. I., Tenenbaum, J. B., & Blei, D. M. (2003) · Cited by 912 · see citing works
Three new graphical models for statistical language modelling
Mnih, A., & Hinton, G. E. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273577 · Cited by 569 · see citing works
Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space
Neelakantan, A., Shankar, J., Passos, A., & McCallum, A. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1113 · Cited by 441 · see citing works
Multi-Prototype Vector-Space Models of Word Meaning
Reisinger, J., & Mooney, R. J. (2010) · Cited by 347 · see citing works
Sparse Overcomplete Word Vector Representations
Faruqui, M., Tsvetkov, Y., Yogatama, D., Dyer, C., & Smith, N. A. (2015) · doi:10.3115/v1/p15-1144 · Cited by 159 · see citing works
N-gram Counts and Language Models from the Common Crawl
Buck, C., Heafield, K., & van Ooyen, B. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3uyduafd26my · Cited by 154 · see citing works
Clustering and Diversifying Web Search Results with Graph-Based Word Sense Induction
Marco, A. D., & Roberto Navigli, A. (2012) · Computational Linguistics · doi:10.1162/coli_a_00148 · Cited by 151 · see citing works
Skip-Gram - Zipf + Uniform = Vector Additivity
Gittens, A., Achlioptas, D., & Mahoney, M. W. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1007 · Cited by 76 · see citing works
SemEval-2013 Task 11: Word Sense Induction and Disambiguation within an End-User Application
Roberto Navigli, A., & Vannella, D. (2013) · Joint Conference on Lexical and Computational Semantics · Cited by 65 · see citing works
The role of named entities in web people search
Artiles, J., Amigó, E., & Gonzalo, J. A. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699571.1699582 · Cited by 54 · see citing works
RAND-WALK: A Latent Variable Model Approach to Word EmbeddingsPreprint
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2015) · arXiv (Cornell University) · Cited by 35 · see citing works
Self-citationScott, N. (1961) · Bulletin of the School of Oriental and African Studies · doi:10.1017/s0041977x00092156 · Cited by 20 · see citing works
SMALLbox - An Evaluation Framework for Sparse Representations and Dictionary Learning Algorithms
Damnjanovic, I., Davies, M. E. P., & Plumbley, M. D. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15995-4_52 · Cited by 16 · see citing works
The mechanism of additive composition
Tian, R., Okazaki, N., & Inui, K. (2017) · Machine Learning · doi:10.1007/s10994-017-5634-8 · Cited by 9 · see citing works
WoSIT: A Word Sense Induction Toolkit for Search Result Clustering and Diversification
Vannella, D., Flati, T., & Roberto Navigli, A. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5012 · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 14
- Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embed- dings. In In Proceedings of International Conference on Learning Representations.
- Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic lan- guage model. Journal of Machine Learning Research, pages 1137-1155.
- Samuel Brody and Mirella Lapata. 2009. Bayesian word sense induction. In Proceedings of the 12th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 103-111.
- Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicogra- phy. Computational linguistics, pages 22-29.
- Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press.
- Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. SensEmbed: Learning sense embeddings for word and relational similarity. In Pro- ceedings of Association for Computational Linguis- tics, pages 95-105.
- Suresh Manandhar, Ioannis P Klapaftis, Dmitriy Dligach, and Sameer S Pradhan. 2010. SemEval 2010: Task 14: Word sense induction & disambiguation. In Pro- ceedings of the 5th International Workshop on Seman- tic Evaluation, pages 63-68.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Cor- rado, and Jeff Dean. 2013a. Distributed represen- tations of words and phrases and their composition- ality. In Advances in Neural Information Processing Systems, pages 3111-3119.
- Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017. Ge- ometry of polysemy. In Proceedings of International Conference on Learning Representations.
- Brian Murphy, Partha Pratim Talukdar, and Tom M. Mitchell. 2012. Learning effective and interpretable semantic models using non-negative sparse embed- ding. In Proceedings of the 24th International Confer- ence on Computational Linguistics, pages 1933-1950.
- Arvind Neelakantan, Jeevan Shankar, Re Passos, and Andrew Mccallum. 2014. Efficient nonparametric estimation of multiple embeddings per word in vec- tor space. In Proceedings of Conference on Empiri- cal Methods in Natural Language Processing, pages 1059-1069.
- Andrew Rosenberg and Julia Hirschberg. 2007. V- measure: A conditional entropy-based external clus- ter evaluation measure. In Conference on Empirical Methods in Natural Language Processing and Confer- ence on Computational Natural Language Learning, pages 410-420.
- Hinrich Schutze. 1998. Automatic word sense discrimi- nation. Computational Linguistics, pages 97-123.
- Peter D. Turney and Patrick Pantel. 2010. From fre- quency to meaning: Vector space models of seman- tics. Journal of Artificial Intelligence Research, pages 141-188. Wikimedia. 2012. English Wikipedia dump. Accessed March 2015.
Cited by
Sorted by citations
Learning Gender-Neutral Word Embeddings
Zhao, J., Zhou, Y., Li, Z., Wang, W., & Chang, K.-W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1521 · Cited by 376 · see citing works
A Latent Variable Model Approach to PMI-based Word Embeddings
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00106 · Cited by 286 · see citing works
Self-citationDistributional Semantics and Linguistic Theory
Boleda, G. (2019) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-011619-030303 · Cited by 297 · see citing works
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 240 · see citing works
SenseBERT: Driving Some Sense into BERT
Levine, Y., Lenz, B., Dagan, O., Ram, O., Padnos, D., Sharir, O., Shalev‐Shwartz, S., Shashua, A., & Shoham, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.423 · Cited by 179 · see citing works
Probabilistic FastText for Multi-Sense Word Embeddings
Athiwaratkun, B., Wilson, A. G., & Anandkumar, A. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1001 · Cited by 176 · see citing works
SECNLP: A survey of embeddings in clinical natural language processing
Kalyan, K. S., & Sangeetha, S. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103323 · Cited by 111 · see citing works
Cultural cartography with word embeddings
Stoltz, D. S., & Taylor, M. A. (2021) · Poetics · doi:10.1016/j.poetic.2021.101567 · Cited by 100 · see citing works
Semantic Structure and Interpretability of Word Embeddings
Senel, L. K., Utlu, I., Yucesoy, V., Koç, A., & Çukur, T. (2018) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2018.2837384 · Cited by 100 · see citing works
Papakyriakopoulos, O., Hegelich, S., Serrano, J. C. M., & Marco, F. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372843 · Cited by 95 · see citing works
Feature Purification: How Adversarial Training Performs Robust Deep Learning
Allen-Zhu, Z., & Li, Y. (2022) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs52979.2021.00098 · Cited by 96 · see citing works
Self-citationInterPLM: discovering interpretable features in protein language models via sparse autoencoders
Simon, E. P., & Zou, J. (2025) · Nature Methods · doi:10.1038/s41592-025-02836-7 · Cited by 89 · see citing works
Machine Learning as a Model for Cultural Learning: Teaching an Algorithm What it Means to be Fat
Arseniev‐Koehler, A., & Foster, J. G. (2022) · Sociological Methods & Research · doi:10.1177/00491241221122603 · Cited by 74 · see citing works
Embedding Regression: Models for Context-Specific Description and Inference
RODRIGUEZ, P. L., SPIRLING, A., & STEWART, B. M. (2023) · American Political Science Review · doi:10.1017/s0003055422001228 · Cited by 70 · see citing works
Exploring Semantic Properties of Sentence Embeddings
Zhu, X., Li, T., & de Melo, G. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2100 · Cited by 65 · see citing works
Aceves, P., & Evans, J. A. (2023) · Organization Science · doi:10.1287/orsc.2023.1686 · Cited by 64 · see citing works
Accelerating Innovation Through Analogy Mining
Hope, T., Chan, J., Kittur, A., & Shahaf, D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098038 · Cited by 64 · see citing works
Theoretical Foundations and Limits of Word Embeddings: What Types of Meaning can They Capture?
Arseniev‐Koehler, A. (2022) · Sociological Methods & Research · doi:10.1177/00491241221140142 · Cited by 64 · see citing works
SCDV : Sparse Composite Document Vectors using soft clustering over distributional representations
Mekala, D., Gupta, V., Paranjape, B., & Karnick, H. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1069 · Cited by 54 · see citing works
Word2Sense: Sparse Interpretable Word Embeddings
Panigrahi, A., Simhadri, H. V., & Bhattacharyya, C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1570 · Cited by 51 · see citing works
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasksPreprint
Jastrzȩbski, S., Leśniak, D., & Czarnecki, W. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02170 · Cited by 46 · see citing works
Improving Entity Linking through Semantic Reinforced Entity Embeddings
Hou, F., Wang, R., He, J., & Zhou, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.612 · Cited by 42 · see citing works
Adaptive cross-contextual word embedding for word polysemy with unsupervised topic modeling
Li, S., Pan, R., Luo, H., Liu, X., & Zhao, G. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106827 · Cited by 40 · see citing works
Integrating topic modeling and word embedding to characterize violent deaths
Arseniev‐Koehler, A., Cochran, S. D., Mays, V. M., Chang, K.-W., & Foster, J. G. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2108801119 · Cited by 40 · see citing works
Metrics and mechanisms: Measuring the unmeasurable in the science of science
Wu, L., Kittur, A., Youn, H., Milojević, S., Leahey, E., Fiore, S. M., & Ahn, Y. (2022) · Journal of Informetrics · doi:10.1016/j.joi.2022.101290 · Cited by 41 · see citing works
Mohanty, S. D., Biggers, B., Sayedahmed, S., Pourebrahim, N., Goldstein, E. B., Bunch, R. L., Chi, G., Sadri, F., McCoy, T., & Cosby, A. G. (2021) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2020.102032 · Cited by 37 · see citing works
Yun, Z., Chen, Y., Olshausen, B. A., & LeCun, Y. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.1 · Cited by 38 · see citing works
RAND-WALK: A Latent Variable Model Approach to Word EmbeddingsPreprint
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2015) · arXiv (Cornell University) · Cited by 35 · see citing works
Self-citationFuture Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33 · see citing works
Polysemy—Evidence from Linguistics, Behavioral Science, and Contextualized Language Models
Haber, J., & Poesio, M. (2023) · Computational Linguistics · doi:10.1162/coli_a_00500 · Cited by 40 · see citing works
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
Nanda, N., Lee, A., & Wattenberg, M. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.2 · Cited by 55 · see citing works
Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings
Yaghoobzadeh, Y., Kann, K., Hazen, T. J., Agirre, E., & Schütze, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1574 · Cited by 30 · see citing works
Probing Lexical Ambiguity: Word Vectors Encode Number and Relatedness of Senses
Beekhuizen, B. F., Armstrong, B. C., & Stevenson, S. (2021) · Cognitive Science · doi:10.1111/cogs.12943 · Cited by 27 · see citing works
Toy, T. (2023) · AI & Society · doi:10.1007/s00146-023-01786-y · Cited by 25 · see citing works
Ontology-Aware Token Embeddings for Prepositional Phrase Attachment
Dasigi, P., Ammar, W., Dyer, C., & Hovy, E. H. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1191 · Cited by 24 · see citing works
Aceves, P., & Evans, J. A. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01815-w · Cited by 24 · see citing works
Investigating Capsule Network and Semantic Feature on Hyperplanes for Text Classification
Du, C., Sun, H., Wang, J., Qi, Q., Liao, J., Wang, C., & Ma, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1043 · Cited by 21 · see citing works
Şenel, L. K., Şahi̇nuç, F., Yücesoy, V., Schütze, H., Çukur, T., & Koç, A. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102925 · Cited by 20 · see citing works
Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations
Berend, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.683 · Cited by 19 · see citing works
Arseniev‐Koehler, A., Foster, J. G., Mays, V. M., Chang, K.-W., & Cochran, S. D. (2021) · American Journal of Public Health · doi:10.2105/ajph.2021.306312 · Cited by 18 · see citing works
Using Multi-Sense Vector Embeddings for Reverse Dictionaries
Hedderich, M. A., Yates, A., Klakow, D., & de Melo, G. (2019) · International Conference on Computational Semantics (IWCS) · doi:10.18653/v1/w19-0421 · Cited by 18 · see citing works
LessLex: Linking Multilingual Embeddings to SenSe Representations of LEXical Items
Colla, D., Mensa, E., & Radicioni, D. P. (2020) · Computational Linguistics · doi:10.1162/coli_a_00375 · Cited by 19 · see citing works
Chang, T.-Y., Chi, T.-C., Tsai, S.-C., & Chen, Y.-N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03348 · Cited by 18 · see citing works
Robust Word Vectors: Context-Informed Embeddings for Noisy Texts
Malykh, V. A., Logacheva, V., & Khakhulin, T. (2018) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/w18-6108 · Cited by 17 · see citing works
Imparting interpretability to word embeddings while preserving semantic structure
Şenel, L. K., Utlu, İ., Şahi̇nuç, F., Özaktaş, H. M., & Koç, A. (2020) · Natural Language Engineering · doi:10.1017/s1351324920000315 · Cited by 17 · see citing works
Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval
Jang, K.-R., Kang, J., Hong, G., Myaeng, S.-H., Park, J., Yoon, T., & Seo, H.-C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.78 · Cited by 17 · see citing works
300-sparsans at SemEval-2018 Task 9: Hypernymy as interaction of sparse attributes
Berend, G., Makrai, M., & Földiák, P. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1152 · Cited by 16 · see citing works
Learning and Evaluating Sparse Interpretable Sentence Embeddings
Trifonov, V., Ganea, O.-E., Potapenko, A., & Hofmann, T. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5422 · Cited by 16 · see citing works
RG-Flow: a hierarchical and explainable flow model based on renormalization group and sparse prior
Hu, H.-Y., Wu, D., You, Y., Olshausen, B. A., & Chen, Y. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac8393 · Cited by 16 · see citing works
Webson, A., Chen, Z., Eickhoff, C., & Pavlick, E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.335 · Cited by 15 · see citing works
Kulahcioglu, T., & de Melo, G. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413534 · Cited by 15 · see citing works
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2015) · arXiv (Cornell University) · Cited by 15 · see citing works
Self-citationOn knowing a gene: A distributional hypothesis of gene function
Kwon, J. J., Pan, J., Gonzalez, G., Hahn, W. C., & Žitnik, M. (2024) · Cell Systems · doi:10.1016/j.cels.2024.04.008 · Cited by 12 · see citing works
The high-dimensional space of human diseases built from diagnosis records and mapped to genetic loci
Jia, G., Li, Y., Zhong, X. Y., Wang, K., Pividori, M., Alomairy, R., Esposito, A., Ltaief, H., Terao, C. C., Akiyama, M., et al. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00453-y · Cited by 11 · see citing works
Self-citationExtending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications
Chang, H.-S., Agrawal, A., & McCallum, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16857 · Cited by 12 · see citing works
Interpreting Embedding Spaces by Conceptualization
Simhi, A., & Markovitch, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.106 · Cited by 12 · see citing works
PolyLM: Learning about Polysemy through Language Modeling
Ansell, A., Bravo-Márquez, F., & Pfahringer, B. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.45 · Cited by 11 · see citing works
Integrating topic modeling and word embedding to characterize violent deathsPreprint
Arseniev‐Koehler, A., Cochran, S. D., Mays, V. M., Chang, K.-W., & Foster, J. G. (2020) · doi:10.31235/osf.io/nkyaq · Cited by 11 · see citing works
Karell, D., Sachs, J. D., & Barrett, R. (2024) · Poetics · doi:10.1016/j.poetic.2024.101966 · Cited by 10 · see citing works
Out-of-distribution generalization via composition: A lens through induction heads in Transformers
Song, J., Xu, Z., & Zhong, Y. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2417182122 · Cited by 11 · see citing works
Robust to Noise Models in Natural Language Processing Tasks
Malykh, V. A. (2019) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p19-2002 · Cited by 10 · see citing works
Efficient Graph-based Word Sense Induction by Distributional Inclusion Vector Embeddings
Chang, H.-S., Agrawal, A., Ganesh, A., Desai, A., Mathur, V., Hough, A. A., & McCallum, A. (2018) · doi:10.18653/v1/w18-1706 · Cited by 10 · see citing works
Proceedings of the Workshop on Stylistic Variation
(2017) · doi:10.18653/v1/w17-49 · Cited by 10 · see citing works
SciCo: Hierarchical Cross-Document Coreference for Scientific ConceptsPreprint
Cattan, A., Johnson, S., Weld, D. S., Dagan, I., Beltagy, I., Downey, D., & Hope, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08809 · Cited by 16 · see citing works
FluxSpace: Disentangled Semantic Editing in Rectified Flow Models
Dalva, Y., Venkatesh, K., & Yanardag, P. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01221 · Cited by 13 · see citing works
Knowledge Graph Construction and Representation Method for Potato Diseases and Pests
Yang, W., Yang, S., Wang, G., Liu, Y., Lü, J., & Yuan, W. (2023) · Agronomy · doi:10.3390/agronomy14010090 · Cited by 9 · see citing works
Spot the Odd Man Out: Exploring the Associative Power of Lexical Resources
Stanovsky, G., & Hopkins, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1182 · Cited by 9 · see citing works
Discovering Stylistic Variations in Distributional Vector Space Models via Lexical Paraphrases
Niu, X., & Carpuat, M. J. (2017) · doi:10.18653/v1/w17-4903 · Cited by 8 · see citing works
Discovering Universal Geometry in Embeddings with ICA
Yamagiwa, H., Oyama, M., & Shimodaira, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.283 · Cited by 8 · see citing works
What Company Do Semantically Ambiguous Words Keep? Insights from Distributional Word Vectors.
Beekhuizen, B. F., Milić, S., Armstrong, B. C., & Stevenson, S. (2018) · TSpace (University of Toronto) · Cited by 8 · see citing works
Aspectuality Across Genre: A Distributional Semantics Approach
Kober, T., Alikhani, M., Stone, M., & Steedman, M. J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.401 · Cited by 8 · see citing works
A Cluster-Matching-Based Method for Video Face Recognition
Mendes, P. R. C., Busson, A. J. G., Colcher, S., Schwabe, D., Guedes, Á. L. V., & Laufer, C. (2020) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3428658.3430967 · Cited by 8 · see citing works
A Simple Approach to Learn Polysemous Word EmbeddingsPreprint
Sun, Y., Rao, N., & Ding, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01793 · Cited by 8 · see citing works
Mark my Word: A Sequence-to-Sequence Approach to Definition ModelingPreprint
Mickus, T., Paperno, D., & Constant, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05715 · Cited by 8 · see citing works
A Novel Named Entity Recognition Scheme for Steel E-Commerce Platforms Using a Lite BERT
Chen, M., Luo, X., Shen, H., Huang, Z., & Peng, Q. (2021) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2021.017491 · Cited by 8 · see citing works
One Representation per Word - Does it make Sense for Composition?
Kober, T., Weeds, J., Wilkie, J., Reffin, J., & Weir, D. R. (2017) · doi:10.18653/v1/w17-1910 · Cited by 7 · see citing works
You Shall Know the Most Frequent Sense by the Company it Keeps
Hauer, B. D., Luan, Y., & Kondrak, G. (2019) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icosc.2019.8665531 · Cited by 7 · see citing works
Semantic Features Based N-Best Rescoring Methods for Automatic Speech Recognition
Liu, C., Zhang, P., Ta, L., & Yan, Y. (2019) · Applied Sciences · doi:10.3390/app9235053 · Cited by 7 · see citing works
Enhancing Medical Word Sense Inventories Using Word Sense Induction: A Preliminary Study
Dong, Q., & Wang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-71055-2_13 · Cited by 7 · see citing works
Word Embedding Visualization Via Dictionary LearningPreprint
Zhang, J., Chen, Y., Cheung, B., & Olshausen, B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03833 · Cited by 7 · see citing works
BERT as a Teacher: Contextual Embeddings for Sequence-Level RewardPreprint
Schmidt, F., & Hofmann, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02738 · Cited by 8 · see citing works
Robust Handling of Polysemy via Sparse Representations
Mahabal, A., Roth, D., & Mittal, S. (2018) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2031 · Cited by 6 · see citing works
Distributional Models of Category Concepts Based on Names of Category Members
Westera, M., Gupta, A., Boleda, G., & Padó, S. (2021) · Cognitive Science · doi:10.1111/cogs.13029 · Cited by 6 · see citing works
Preposition Sense Disambiguation and Representation
Gong, H., Mu, J., Bhat, S., & Viswanath, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1180 · Cited by 6 · see citing works
Restricted Isometry Property under High CorrelationsPreprint
Kasiviswanathan, S. P., & Rudelson, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05510 · Cited by 6 · see citing works
Superbloom: Bloom filter meets TransformerPreprint
Anderson, J. R., Huang, Q., Krichene, W., Rendle, S., & Zhang, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04723 · Cited by 8 · see citing works
When can Wasserstein GANs minimize Wasserstein DistancePreprint
Li, Y., & Dou, Z. (2020) · arXiv (Cornell University) · Cited by 6 · see citing works
Self-citationTopology of Word Embeddings: Singularities Reflect PolysemyPreprint
Jakubowski, A., Gašić, M., & Zibrowius, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09413 · Cited by 6 · see citing works
Beyond the Benchmarks: Toward Human-Like Lexical Representations
Stevenson, S., & Merlo, P. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796741 · Cited by 5 · see citing works
Accelerating Innovation Through Analogy MiningPreprint
Hope, T., Chan, J., Kittur, A., & Shahaf, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05585 · Cited by 5 · see citing works
Sociolinguistic Properties of Word EmbeddingsPreprint
Arseniev‐Koehler, A., & Foster, J. G. (2020) · doi:10.31235/osf.io/b8kud · Cited by 5 · see citing works
Li, M., Zheng, A., Song, M., Jin, F., Pang, M., Zhang, Y., Wu, Y. N., Li, X., Zhao, M., & Li, Z. (2025) · Ageing Research Reviews · doi:10.1016/j.arr.2025.102692 · Cited by 4 · see citing works
Geometry and Analogies: A Study and Propagation Method for Word Representations
Khalife, S., Liberti, L., & Vazirgiannis, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-31372-2_9 · Cited by 4 · see citing works
Bilingual dictionary generation and enrichment via graph exploration
Goel, S., Gracia, J., & Forcada, M. L. (2022) · Semantic Web · doi:10.3233/sw-222899 · Cited by 4 · see citing works
Measuring Media Criticism with ALC Word Embeddings
Barrie, C., Ketchley, N., Siegel, A. A., & Bagdouri, M. (2025) · Political Analysis · doi:10.1017/pan.2025.10012 · Cited by 4 · see citing works
Multi-resolution Hashing for Fast Pairwise Summations
Charikar, M., & Siminelakis, P. (2019) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2019.00051 · Cited by 4 · see citing works
CoKE : Word Sense Induction Using Contextualized Knowledge Embeddings.
Ramprasad, S., & Maddox, J. (2019) · AAAI Spring Symposium Combining Machine Learning with Knowledge Engineering · Cited by 4 · see citing works
- Authors
- Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A.
- Affiliations
- Princeton University (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 483-495
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2295508206
- Cited by
- 186
- References
- 31
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 9 · 2023: 13 · 2022: 14 · 2021: 26 · 2020: 32 · 2019: 30 · 2018: 23 · 2017: 12 · 2016: 4 · 2015: 3
- FWCI
- 15.53
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1527371, 1317308)
- Office of Naval Research (ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word embeddings, polysemy, word senses, linear superposition, sparse coding, co-occurrence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
