A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
Open accessAbstract
Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, Sanjeev Arora. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Simply averaging the embeddings of context words is not sufficient to give an embedding for a word in different senses.
“Unfortunately, they will not-this is shown empirically in Khodak et al. (2018) and in our Trump/trump example below.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Khodak et al. 2018
Section 4.1 reports that à la carte embeddings significantly outperform simple averaging of context-word embeddings on a rare-word similarity task. The paper also constructs embeddings for individual senses in its Synset Embeddings section, but the passages do not show an empirical comparison establishing that averaging alone is insufficient for distinguishing a word’s different senses. Page numbers are unavailable in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measuredKhodak et al. (2018) demonstrate that this strategy achieves state-of-the-art and near state-of-the-art performance on a wide variety of natural language processing tasks at a fraction of the computational cost of the alternatives.
“They also demonstrate that this strategy achieves state-of-the-art and near state-ofthe-art performance on a wide variety of natural language processing tasks (e.g., learning the embedding of a word using only its definition, learning meaningful n-grams, classification tasks, etc.) at a fraction of the computational cost of the alternatives.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Khodak et al. 2018
The Abstract reports state-of-the-art results on a nonce task and some document-classification tasks, while the classification evaluation describes results competitive with recent methods. The Introduction calls à la carte embedding resource-efficient and contrasts it with computationally more expensive approaches, but our copy does not establish that it costs a fraction as much as the alternatives across the cited range of tasks. (LLM, conf=0.83)
Experimental — accuracy not yet measuredSimply averaging embeddings overexaggerates common components associated with frequent words.
“As implied above, the intuition is that simply averaging embeddings overexaggerates common components associated with frequent (e.g., "stop") words.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Khodak et al. 2018
A large corpus can be used to estimate the transformation matrix A.
“The idea is that a large corpus (generally the corpus the embeddings were originally trained on, such as Wikipedia) can be used to estimate the transformation matrix A.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Khodak et al. 2018
Estimating A is a one-time cost after which each new word embedding can be computed à la carte, rather than retraining an entire corpus for a single word.
“This is a one time cost after which each new word embedding can be computed à la carte (thus the name), rather than needing to retrain an entire corpus just to get the embedding for a single word.”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Khodak et al. 2018
Khodak et al. (2018) found empirically that the learned A in a large corpus recovers the original word vectors with high accuracy (greater than 0.90 cosine similarity).
“Well beyond this toy example, Khodak et al. (2018) finds empirically that the learned b Ain a large corpus recovers the original word vectors with high accuracy (greater than 0.90 cosine similarity).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Khodak et al. 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · see citing works
Flagged first, then by citations
Research works printed but not linked: 43
- Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language model- ing. In Proc. EACL.
- Sanjeev Arora, Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018a. A compressed sensing view of unsupervised text embeddings, bag-of-n- grams, and lstms. In Proc. ICLR.
- Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to pmi-based word embeddings. TACL.
- Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018b. Linear algebraic struc- ture of word senses, with applications to polysemy. TACL.
- Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence em- beddings. In Proc. ICLR.
- Danushka Bollegala, David Weir, and John Carroll. 2014. Learning to predict distributions of words across domains. In Proc. ACL.
- Daniel Cer, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Seman- tic textual similarity multilingual and cross-lingual focused evaluation. In Proc. SemEval.
- Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at word- vectors.org. In Proc. ACL: System Demonstrations.
- Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press.
- Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, and Lawrence Carin. 2017. Learning generic sentence representations using convolutional neural networks. In Proc. EMNLP.
- Aurélie Herbelot and Marco Baroni. 2017. High-risk learning: Acquiring new word vectors from tiny data. In Proc. EMNLP.
- Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proc. NAACL.
- Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proc. KDD.
- Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. Sensembed: Learning sense embeddings for word and relational similarity. In Proc. ACL-IJCNLP.
- Mikhail Khodak, Andrej Risteski, Christiane Fell- baum, and Sanjeev Arora. 2017. Automated word- net construction using word embeddings. In Proc. Workshop on Sense, Concept and Entity Represen- tations and their Applications.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urta- sun, and Sanja Fidler. 2015. Skip-thought vectors. In Adv. NIPS.
- Helen Langone, Benjamin R. Haskell, and George A. Miller. 2004. Annotating wordnet. In Proc. Work- shop on Frontiers in Corpus Annotation.
- Xin Li and Dan Roth. 2002. Learning question classi- fiers. In Proc. COLING.
- Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence represen- tations. In Proc. ICLR.
- Thang Luong, Richard Socher, and Christopher Man- ning. 2013. Better word representations with re- cursive neural networks for morphology. In Proc. CoNLL.
- Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proc. ACL-HLT.
- Julian McAuley, Rahul Pandey, and Jure Leskovec. 2015. Inferring networks of substitutable and com- plementary products. In Proc. KDD.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeffrey Dean. 2013. Distributed represen- tations of words and phrases and their composition- ality. In Adv. NIPS.
- Andrea Moro and Roberto Navigli. 2015. Semeval- 2015 task 13: Multilingual all-words sense disam- biguation and entity linking. In Proc. SemEval.
- Jiaqi Mu and Pramod Viswanath. 2018. All-but-the- top: Simple and effective post-processing for word representations. In Proc. ICLR.
- Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. Semeval-2013 task 12: Multilingual word sense disambiguation. In Proc. SemEval.
- Sebastian Pado, Aurelie Herbelot, Max Kisselew, and Jan Snajder. 2016. Predictability of distributional semantics in derivational word formation. In Proc. COLING.
- Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embed- dings using compositional n-gram features. In Proc. NAACL.
- Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proc. ACL.
- Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proc. ACL.
- Jeffrey Pennington, Richard Socher, and Christo- pher D. Manning. 2014. Glove: Global vectors for word representation. In Proc. EMNLP.
- Adam Poliak, Pushpendre Rastogia, M. Patrick Martin, and Benjamin Van Durme. 2017. Efficient, com- positional, order-sensitive n-gram embeddings. In Proc. EACL.
- Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. ArXiv.
- Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017. Neural sequence learning models for word sense disambiguation. In Proc. EMNLP.
- Sascha Rothe and Hinrich Schütze. 2015. Autoex- tend: Extending word embeddings to embeddings for synsets and lexemes. In Proc. ACL-IJCNLP.
- Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over a sentiment treebank. In Proc. EMNLP.
- Dingquan Wang, Nanyun Peng, and Kevin Duh. 2017. A multi-task learning approach to adapting bilin- gual word embeddings for cross-lingual named en- tity recognition. In Proc. IJCNLP.
- Sida Wang and Christopher D. Manning. 2012. Base- lines and bigrams: Simple, good sentiment and topic classification. In Proc. ACL.
- Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emo- tions in language. Proc. LREC.
- John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sen- tence embeddings. In Proc. ICLR.
- Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. 2017. Starspace: Embed all the things! ArXiv.
- Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the em- bedding of words and entities for named entity dis- ambiguation. In Proc. CoNLL.
- Wenpeng Yin and Hinrich Schutze. 2014. An explo- ration of embeddings for generalized phrases. In Proc. ACL 2014 Student Research Workshop.
Cited by
Sorted by citations
Ash, E., & Hansen, S. (2023) · Annual Review of Economics · doi:10.1146/annurev-economics-082222-074352 · Cited by 197 · see citing works
Embedding Regression: Models for Context-Specific Description and Inference
RODRIGUEZ, P. L., SPIRLING, A., & STEWART, B. M. (2023) · American Political Science Review · doi:10.1017/s0003055422001228 · Cited by 70 · see citing works
Self-citation
- Authors
- Khodak, M., Saunshi, N., Liang, Y., Ma, T., Stewart, B., & Arora, S.
- Affiliations
- Princeton University (US)
- University of Wisconsin–Madison (US)
- Meta (United States) (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 12-22
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2897994656
- Cited by
- 111
- References
- 1
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 13 · 2023: 17 · 2022: 13 · 2021: 12 · 2020: 16 · 2019: 17 · 2018: 4
- FWCI
- 5.04
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1302518, 1527371)
- Office of Naval Research (ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- word embeddings, word representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
