The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Linear Algebraic Structure of Word Senses, with Applications to Polysemy
Open accessAbstract
Word embeddings are ubiquitous in NLP and information retrieval, but it is unclear what they represent when the word is polysemous. Here it is shown that multiple word senses reside in linear superposition within the word embedding and simple sparse coding can recover vectors that approximately capture the senses. The success of our approach, which applies to several embedding methods, is mathematically explained using a variant of the random walk on discourses model (Arora et al., 2016). A novel aspect of our technique is that each extracted word sense is accompanied by one of about 2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense. Discourse atoms can be of independent interest, and make the method potentially more useful. Empirical tests are used to verify and support the theory.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Those papers conceive of documents being a "random walk" in a discourse space, where words are more likely to follow other words if they are closer to them in an embedding space.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Crucially for ALC, Arora et al. (2018) also proves that under this model, a particular relationship will follow for the embedding of a word and the embeddings of the words that appear in the contexts around it.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
The cited source says what the claim attributes to it.
In the citing paper
“That A serves to downweight the contributions of very common (but uninformative) words when averaging.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
The cited source says what the claim attributes to it.
In the citing paper
“The main result from Arora et al. (2018) shows the following: if the random walk model holds, the researcher can obtain an embedding for word w (e.g., "bill") by taking the average of the embeddings of the words around w (u w Þ and multiplying them by a particular square matrix A .”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, embedding representations can be used to solve analogy reasoning tasks, implying the vectors do indeed capture relational meaning between words (e.g., Arora et al. 2018;Mikolov et al. 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2018
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,529
$rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
Aharon, M., Elad, M., & Bruckstein, A. M. (2006) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2006.881199 · Cited by 9,464
(2010) · Cited by 8,261
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642
From Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,949
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · Cited by 2,913
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981658.981684 · Cited by 2,411
(2009) · Cited by 1,900
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581
Automatic word sense discrimination
Schütze, H. (1998) · Cited by 1,269
Topics in semantic representation
Griffiths, T. L., Steyvers, M., & Tenenbaum, J. B. (2007) · Psychological Review · doi:10.1037/0033-295x.114.2.211 · Cited by 1,122
Improving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,071
Hierarchical Topic Models and the Nested Chinese Restaurant Process
Griffiths, T. L., Jordan, M. I., Tenenbaum, J. B., & Blei, D. M. (2003) · Cited by 912
Three new graphical models for statistical language modelling
Mnih, A., & Hinton, G. E. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273577 · Cited by 569
Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space
Neelakantan, A., Shankar, J., Passos, A., & McCallum, A. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1113 · Cited by 454
Multi-Prototype Vector-Space Models of Word Meaning
Reisinger, J., & Mooney, R. J. (2010) · Cited by 347
Sparse Overcomplete Word Vector Representations
Faruqui, M., Tsvetkov, Y., Yogatama, D., Dyer, C., & Smith, N. A. (2015) · doi:10.3115/v1/p15-1144 · Cited by 165
N-gram Counts and Language Models from the Common Crawl
Buck, C., Heafield, K., & van Ooyen, B. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3uyduafd26my · Cited by 154
Clustering and Diversifying Web Search Results with Graph-Based Word Sense Induction
Marco, A. D., & Roberto Navigli, A. (2012) · Computational Linguistics · doi:10.1162/coli_a_00148 · Cited by 151
Skip-Gram - Zipf + Uniform = Vector Additivity
Gittens, A., Achlioptas, D., & Mahoney, M. W. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1007 · Cited by 76
SemEval-2013 Task 11: Word Sense Induction and Disambiguation within an End-User Application
Roberto Navigli, A., & Vannella, D. (2013) · Joint Conference on Lexical and Computational Semantics · Cited by 65
The role of named entities in web people search
Artiles, J., Amigó, E., & Gonzalo, J. A. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699571.1699582 · Cited by 54
RAND-WALK: A Latent Variable Model Approach to Word EmbeddingsPreprint
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2015) · arXiv (Cornell University) · Cited by 35
Self-citationScott, N. (1961) · Bulletin of the School of Oriental and African Studies · doi:10.1017/s0041977x00092156 · Cited by 20
SMALLbox - An Evaluation Framework for Sparse Representations and Dictionary Learning Algorithms
Damnjanovic, I., Davies, M. E. P., & Plumbley, M. D. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15995-4_52 · Cited by 16
The mechanism of additive composition
Tian, R., Okazaki, N., & Inui, K. (2017) · Machine Learning · doi:10.1007/s10994-017-5634-8 · Cited by 9
WoSIT: A Word Sense Induction Toolkit for Search Result Clustering and Diversification
Vannella, D., Flati, T., & Roberto Navigli, A. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5012 · Cited by 3
Flagged first, then by citations
Research works printed but not linked: 3
- Bengio Yoshua, 2003, Journal of Machine Learning Research, pages, 1137
- Manandhar Suresh, 2010, In Proceedings of the 5th International Workshop on Semantic Evaluation, pages, 63
- Navigli Roberto, 2013, In Second Joint Conference on Lexical and Computational Semantics, pages, 193
Cited by
Sorted by citations
Learning Gender-Neutral Word Embeddings
Zhao, J., Zhou, Y., Li, Z., Wang, W. W., & Chang, K.-W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1521 · Cited by 426
A Latent Variable Model Approach to PMI-based Word Embeddings
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00106 · Cited by 286
Self-citationDistributional Semantics and Linguistic Theory
Boleda, G. (2019) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-011619-030303 · Cited by 328
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 330
SenseBERT: Driving Some Sense into BERT
Levine, Y., Lenz, B., Dagan, O., Ram, O., Padnos, D., Sharir, O., Shalev‐Shwartz, S., Shashua, A., & Shoham, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.423 · Cited by 195
Metadata
- Authors
- Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A.
- Affiliations
- Princeton University (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 483-495
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2295508206
- Cited by
- 186
- References
- 31
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 9 · 2023: 13 · 2022: 14 · 2021: 26 · 2020: 32 · 2019: 30 · 2018: 23
- FWCI
- 15.53
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1527371, 1317308)
- Office of Naval Research (ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word embeddings, polysemy, word senses, linear superposition, sparse coding, co-occurrence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
