A Latent Variable Model Approach to PMI-based Word Embeddings
Open accessAbstract
Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of Mnih and Hinton (2007). The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by Mikolov et al. (2013a) and many subsequent papers. Experimental support is provided for the generative model assumptions, the most important of which is that latent word vectors are fairly uniformly dispersed in space.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Documents can be conceived of as a random walk in a discourse space, where words are more likely to follow other words if they are closer in an embedding space.
“Those papers conceive of documents being a "random walk" in a discourse space, where words are more likely to follow other words if they are closer to them in an embedding space.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Arora et al. 2016
Section 2, “Generative model and its properties” (page not supplied), models text generation as a random walk of a latent discourse vector that makes nearby words arise under similar discourses. But Section 1.1, “Related work” (page not supplied), distinguishes this model from one in which the next word’s probability depends directly on its embedding-space distance from the previous word. The claim conflates those two models. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 31 · Cited by 286
Research works printed but not linked: 22
- Jacob Andreas and Dan Klein. 2014. When and why are log-linear models self-normalizing? In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics.
- Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2015. A latent variable model approach to PMI-based word embeddings. Technical report, ArXiV. http://arxiv.org/abs/1502.03520. · arXiv
- David Belanger and Sham M. Kakade. 2015. A linear dynamical system model for text. In Proceedings of the 32nd International Conference on Machine Learn- ing.
- David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning.
- Shay B. Cohen, Karl Stratos, Michael Collins, Dean P. Foster, and Lyle Ungar. 2012. Spectral learning of latent-variable PCFGs. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1.
- Ronan Collobert and Jason Weston. 2008a. A uni- fied architecture for natural language processing: Deep neural networks with multitask learning. In Proceed- ings of the 25th International Conference on Machine Learning.
- Ronan Collobert and Jason Weston. 2008b. A uni- fied architecture for natural language processing: Deep neural networks with multitask learning. In Proceed- ings of the 25th International Conference on Machine Learning.
- Scott C. Deerwester, Susan T Dumais, Thomas K. Lan- dauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science.
- John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research.
- Amir Globerson, Gal Chechik, Fernando Pereira, and Naftali Tishby. 2007. Euclidean embedding of co- occurrence data. Journal of Machine Learning Re- search.
- Tatsunori B. Hashimoto, David Alvarez-Melis, and Tommi S. Jaakkola. 2016. Word embeddings as met- ric recovery in semantic spaces. Transactions of the Association for Computational Linguistics.
- Thomas Hofmann. 1999. Probabilistic latent semantic analysis. In Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence.
- Daniel Hsu, Sham M. Kakade, and Tong Zhang. 2012. A spectral algorithm for learning hidden markov models. Journal of Computer and System Sciences.
- Omer Levy and Yoav Goldberg. 2014a. Linguistic regu- larities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Compu- tational Natural Language Learning.
- Omer Levy and Yoav Goldberg. 2014b. Neural word em- bedding as implicit matrix factorization. In Advances in Neural Information Processing Systems.
- Yariv Maron, Michael Lamar, and Elie Bienenstock. 2010. Sphere embedding: An application to part-of- speech induction. In Advances in Neural Information Processing Systems.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Cor- rado, and Jeff Dean. 2013b. Distributed represen- tations of words and phrases and their composition- ality. In Advances in Neural Information Processing Systems.
- Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the Confer- ence of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies.
- Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th International Conference on Machine Learning.
- Christos H. Papadimitriou, Hisao Tamaki, Prabhakar Raghavan, and Santosh Vempala. 1998. Latent se- mantic indexing: A probabilistic analysis. In Proceed- ings of the 7th ACM SIGACT-SIGMOD-SIGART Sym- posium on Principles of Database Systems.
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word rep- resentation. Proceedings of the Empiricial Methods in Natural Language Processing.
- Peter D. Turney and Patrick Pantel. 2010. From fre- quency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research.
- Authors
- Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A.
- Affiliations
- Princeton University (US)
- Corresponding author
- Arora, Sanjeev
- Li, Yuanzhi
- Liang, Yingyu
- Ma, Tengyu
- Risteski, Andrej
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2016-12
- Volume
- 4
- Pages
- 385-399
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2518186251
- Cited by
- 286
- References
- 31
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 17 · 2023: 43 · 2022: 34 · 2021: 39 · 2020: 58 · 2019: 37 · 2018: 26 · 2017: 9 · 2015: 2
- FWCI
- 19.77
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Science Foundation (1527371, 1317308)
- Office of Naval Research (ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- word embeddings, latent variable models, pointwise mutual information, generative models, word co-occurrence statistics, hyperparameter optimization, word analogy, Word2Vec, gloves
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
