The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Open accessAbstract
Recent trends suggest that neural-network-inspired word embedding models outperform traditional count-based distributional models on word similarity and analogy detection tasks. We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
The Approximation of One Matrix by Another of Lower Rank
Eckart, C., & Young, G. (1936) · Psychometrika · doi:10.1007/bf02288367 · Cited by 3,800
Word association norms, mutual information, and lexicography
Church, K., & Hanks, P. (1990) · doi:10.5555/89086.89095 · Cited by 3,590
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
From Frequency to Meaning: Vector Space Models of Semantics
Turney, P. D., & Pantel, P. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2934 · Cited by 2,866
Cited by
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications
Cai, H., Zheng, V. W., & Chang, K. C.-C. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2807452 · Cited by 2,034
Comparing different supervised machine learning algorithms for disease prediction
Uddin, S., Khan, A., Hossain, M. E., & Moni, M. A. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-1004-8 · Cited by 1,627
Deep Neural Networks for Learning Graph Representations
Cao, S., Lu, W., & Xu, Q. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10179 · Cited by 1,014
CHURCH, K. W. (2017) · Natural Language Engineering · doi:10.1017/s1351324916000334 · Cited by 974
Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1141 · Cited by 943
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821
Self-citationUnsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features
Pagliardini, M., Gupta, P., & Jaggi, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1049 · Cited by 678
Learning Distributed Representations of Sentences from Unlabelled Data
Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566
“Examples include machine translation (Sutskever et al., 2014), image captioning (Mao et al., 2015) and dialogue systems (Serban et al., 2015).”
BackgroundEnd-to-End Neural Ad-hoc Ranking with Kernel Pooling
Xiong, C., Dai, Z., Callan, J., Liu, Z., & Power, R. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080809 · Cited by 546
Evaluation methods for unsupervised word embeddings
Schnabel, T., Labutov, I., Mimno, D., & Joachims, T. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1036 · Cited by 540
A Survey of Cross-lingual Word Embedding Models
Ruder, S., Vulić, I., & Søgaard, A. (2018) · Apollo (University of Cambridge) · doi:10.17863/cam.30462 · Cited by 510
The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings
Kozlowski, A. C., Taddy, M., & Evans, J. A. (2019) · American Sociological Review · doi:10.1177/0003122419877135 · Cited by 499
Attributed Social Network Embedding
Liao, L., He, X., Zhang, H., & Chua, T.-S. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2819980 · Cited by 440
Analogical Reasoning on Chinese Morphological and Semantic Relations
Li, S., Zhao, Z., Hu, R., Li, W., Liu, T., & Du, X. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2023 · Cited by 438
Mandera, P., Keuleers, E., & Brysbaert, M. (2017) · Journal of Memory and Language · doi:10.1016/j.jml.2016.04.001 · Cited by 428
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora
Hamilton, W. L., Clark, K., Leskovec, J., & Jurafsky, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1057 · Cited by 355
Advances in Pre-Training Distributed Word Representations
Mikolov, T., Grave, É., Bojanowski, P., Puhrsch, C., & Joulin, A. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4b3prw5a5tze · Cited by 354
Learning Gender-Neutral Word Embeddings
Zhao, J., Zhou, Y., Li, Z., Wang, W., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1521 · Cited by 345
How to Train good Word Embeddings for Biomedical NLP
Chiu, B., Crichton, G., Korhonen, A., & Pyysalo, S. (2016) · Proceedings of the 15th Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/w16-2922 · Cited by 344
An Introduction to Neural Information Retrieval
Mitra, B., & Craswell, N. (2018) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000061 · Cited by 317
Semantic memory: A review of methods, models, and current challenges
Kumar, A. A. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01792-x · Cited by 314
Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings
Vulić, I., & Moens, M.-F. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767752 · Cited by 294
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04207 · Cited by 283
Self-citationA Survey of Cross-lingual Word Embedding Models
Ruder, S., Vulić, I., & Søgaard, A. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11640 · Cited by 266
Embeddings for Word Sense Disambiguation: An Evaluation Study
Iacobacci, I., Pilehvar, M. T., & Navigli, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1085 · Cited by 253
Learning Deep Network Representations with Adversarially Regularized Autoencoders
Yu, W., Zheng, C., Cheng, W., Aggarwal, C. C., Song, D., Zong, B., Chen, H., & Wang, W. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220000 · Cited by 188
Knowledge Base Completion: Baselines Strike Back
Kadlec, R., Bajgar, O., & Kleindienst, J. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2609 · Cited by 173
“In light of these results we think that the field would benefit from a large-scale empirical comparative study of different KBC algorithms, similar to a recent study of word embedding models (Levy et al., 2015).”
ResultKleyko, D., Rachkovskij, D., Osipov, E., & Rahimi, A. (2023) · ACM Computing Surveys · doi:10.1145/3558000 · Cited by 141
Altszyler, E., Ribeiro, S., Sigman, M., & Fernández Slezak, D. (2017) · Consciousness and Cognition · doi:10.1016/j.concog.2017.09.004 · Cited by 128
A Timely Intervention: Tracking the Changing Meanings of Political Concepts with Word Vectors
Rodman, E. (2020) · Political Analysis · doi:10.1017/pan.2019.23 · Cited by 112
Context-aware Path Ranking for Knowledge Base Completion
Mazumder, S., & Liu, B. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/166 · Cited by 35
How a Stylistic, Machine-Generated Thesaurus Impacts a Writer's Process
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 on Creativity and Cognition · doi:10.1145/3325480.3326573 · Cited by 32
Automatic generation of short answer questions for reading comprehension assessment
HUANG, Y., & HE, L. (2016) · Natural Language Engineering · doi:10.1017/s1351324915000455 · Cited by 30
Keuchenius, A., Törnberg, P., & Uitermark, J. (2021) · PLOS ONE · doi:10.1371/journal.pone.0256696 · Cited by 27
A structured distributional model of sentence meaning and processing
Chersoni, E., Santus, E., Pannitto, L., Lenci, A., Blache, P., & Huang, C.-R. (2019) · Natural Language Engineering · doi:10.1017/s1351324919000214 · Cited by 19
A Paradigm for Democratizing Artificial Intelligence Research
Moreau, E., Vogel, C., & Barry, M. (2019) · Intelligent Systems Reference Library · doi:10.1007/978-3-030-15939-9_8 · Cited by 10
Metadata
- Authors
- Levy, O., Goldberg, Y., & Dagan, I.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2015-12
- Volume
- 3
- Pages
- 211-225
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W1615991656
- Cited by
- 1,308
- References
- 27
- FWCI
- 164.23
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.