CoChat
Sign Up
PreprintCited by 24,533Topic ModelingNatural Language Processing TechniquesSentiment Analysis and Opinion Mining

Distributed Representations of Words and Phrases and their Compositionality

Open access
Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg, Dean, Jeffrey·2013·arXiv
DOI 10.48550/arxiv.1310.4546arXiv 1310.4546
Open this paper in CoChat — free accountOpen PDF

Abstract

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1310.4546

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Earlier word embedding models, e.g., word2vec and GloVe, predominantly relied on syntactic contexts for training.

    “Earlier word embedding models, e.g., word2vec [60] and GloVe [61], predominantly relied on syntactic contexts for training.”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [60]

    The Introduction (p. 1) says Skip-gram trains by predicting nearby words, and §5 (p. 7) describes predicting surrounding words in a sentence. That supports the word2vec portion's reliance on word context, but the paper does not characterize those training contexts as predominantly syntactic. GloVe has a separate citation and is not assessed here. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Continuous word representations capture not only lexical but also semantic properties of tokens, contrary to string similarity-based approaches.

    “Contrary to string similarity-based approaches, continuous word representations capture not only lexical but also semantic properties of tokens.”

    From A Method for the Calculation of Parallel Passages for Buddhist Chinese Sources Based on Million-scale Nearest Neighbor Search (2020) · cites this paper as Mikolov et al. (2013)

    The Abstract (p. 1) says continuous vector representations capture syntactic and semantic word relationships, and the Introduction (p. 1) gives an example of a semantic analogy. The paper does not compare these representations with string similarity-based approaches or establish the claimed contrast. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Mikolov et al. (2013) demonstrated that unweighted averaging of word embeddings performs well for the representation of short phrases.

    “Mikolov et al. (2013) demonstrated that unweighted averaging of word embeddings performs well for the representation of short phrases.”

    From A Method for the Calculation of Parallel Passages for Buddhist Chinese Sources Based on Million-scale Nearest Neighbor Search (2020) · cites this paper as Mikolov et al. (2013)

    Section 5 (p. 7) reports that adding word vectors can meaningfully combine words, and the Introduction (p. 2) gives examples whose sums are close to phrase vectors. The paper does not demonstrate unweighted averaging as a general method for representing short phrases; it also learns phrase vectors by treating phrases as individual tokens. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Even during the pre-train and fine-tune paradigm, there was some use of pre-trained LMs exemplified by word2vec and GloVe, but they were used for only a limited portion of the final model parameters.

    “Even during this stage, there was some use of pre-trained LMs exemplified by word2vec[85,86] and GloVe[95], but they were used for only a limited portion of the final model parameters.”

    From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [86]

    The Introduction (p. 1) and Conclusion (p. 8) identify this work as training Skip-gram word representations, supporting the citation as an example of word2vec. The paper does not describe word2vec as a pre-trained language model or establish that its representations occupied only a limited portion of a final model’s parameters. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • These earlier word embedding models were trained on vast amounts of unlabeled data, aiming to capture intricate grammatical and semantic rules.

    “These models were trained on vast amounts of unlabeled data, aiming to capture intricate grammatical and semantic rules.”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [60]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 19

  1. [1][1] Yoshua Bengio, R´ ejean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The Journal of Machine Learning Research , 3:1137-1155, 2003. -
  2. [2][2] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning , pages 160-167. ACM, 2008. -
  3. [3][3] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML , 513-520, 2011. -
  4. [4][4] Michael U Gutmann and Aapo Hyv¨ arinen. Noise-contrasti ve estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research , 13:307-361, 2012. -
  5. [5][5] Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on , pages 5528-5531. IEEE, 2011. -
  6. [6][6] Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget and Jan Cernocky. Strategies for Training Large Scale Neural Network Language Models. In Proc. Automatic Speech Recognition and Understanding , 2011. -
  7. [7][7] Tomas Mikolov. Statistical Language Models Based on Neural Networks. PhD thesis, PhD Thesis, Brno University of Technology , 2012. -
  8. [9][9] Tomas Mikolov, Wen-tau Yih and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT , 2013. -
  9. [10][10] Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. Advances in neural information processing systems , 21:1081-1088, 2009. -
  10. [11][11] Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426 , 2012. - · arXiv
  11. [12][12] Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics , pages 246-252, 2005. -
  12. [13][13] David E Rumelhart, Geoffrey E Hintont, and Ronald J Williams. Learning representations by backpropagating errors. Nature , 323(6088):533-536, 1986. -
  13. [14][14] Holger Schwenk. Continuous space language models. Computer Speech and Language , vol. 21, 2007. -
  14. [15][15] Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 26th International Conference on Machine Learning (ICML) , volume 2, 2011. -
  15. [16][16] Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2012. -
  16. [17][17] Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pages 384-394. Association for Computational Linguistics, 2010. -
  17. [18][18] Peter D. Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. In Journal of Artificial Intelligence Research , 37:141-188, 2010. -
  18. [19][19] Peter D. Turney. Distributional semantics beyond words: Supervised learning of analogy and paraphrase. In Transactions of the Association for Computational Linguistics (TACL) , 353-366, 2013. -
  19. [20][20] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of the Twenty-Second international joint conference on Artificial Intelligence-Volume Volume Three , pages 2764-2770. AAAI Press, 2011.

Cited by

Sorted by citations

Authors
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2013
Volume
26
Pages
3111-3119
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1310.4546
arXiv
1310.4546
OpenAlex
W2153579005
Cited by
24,533
References
1
Citations by year
2026: 114 · 2025: 872 · 2024: 1388 · 2023: 1965 · 2022: 2239 · 2021: 3514 · 2020: 3787 · 2019: 3870 · 2018: 2785 · 2017: 1949 · 2016: 1194 · 2015: 658 · 2014: 176 · 2013: 7 · 2012: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
distributed word representations, skip-gram model, phrase representation, negative sampling, frequent terms, idioms, word order, training speed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review