A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
Open accessAbstract
In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Almost all previous approaches still need to run the original Softmax function, either on full vector or reduced one; exceptions are training with Noise Contrastive Estimation and inference with Self-Normalized Softmax.
“Almost all of the approaches still need to run the original Softmax function, either on full vector or reduced one. There are two exceptions that don't need to compute the softmax normalization term: training with Noise Contrastive Estimation and inference with Self-Normalized Softmax.”
From Online normalizer calculation for softmax (2018) · cites this paper as [10]
Section 3 (p. 3) says NCE training avoids explicit normalization, supporting one named exception. Section 2.2 (p. 3) describes the cost of computing over the full vocabulary, but the paper does not identify Self-Normalized Softmax inference as another exception or establish that these are the only two exceptions. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 21 · Cited by 315
Research works printed but not linked: 20
- Bengio, Yoshua and Sen´ ecal, Jean-S´ ebastien. Quick training of probabilistic neural nets by importance sampling. In AISTATS'03 , 2003.
- Bengio, Yoshua and Sen´ ecal, Jean-S´ ebastien. Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Transactions on Neural Networks , 19(4):713-722, 2008.
- Bengio, Yoshua, Ducharme, R´ ejean, and Vincent, Pascal. A neural probabilistic language model. In NIPS , pp. 932-938, 2000.
- Bengio, Yoshua, Ducharme, Rejean, Vincent, Pascal, and Jauvin, Christian. A neural probabilistic language model. Journal of Machine Learning Research , 3:1137-1155, 2003.
- Berger, A.L., Pietra, V.J.D., and Pietra, S.A.D. A maximum entropy approach to natural language processing. Computational linguistics , 22(1):39-71, 1996.
- Bird, S., Klein, E., and Loper, E. Natural language processing with Python . O'Reilly Media, 2009.
- Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning , 2008.
- Gutmann, M. and Hyv¨ arinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS2010) , 2010.
- Gutmann, M.U. and Hyv¨ arinen, A. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research , 13:307-361, 2012.
- Maas, A.L. and Ng, A.Y. A probabilistic model for semantic word vectors. In NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning , 2010.
- Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and ˇ Cernock` y, J. Empirical evaluation and combination of advanced language modeling techniques. In Proceedings of Interspeech , pp. 605-608, 2011.
- Mnih, A. and Hinton, G. Three new graphical models for statistical language modelling. Proceedings of the 24th International Conference on Machine Learning , pp. 641-648, 2007.
- Mnih, A., Yuecheng, Z., and Hinton, G. Improving a statistical language model through non-linear prediction. Neurocomputing , 72(7-9):1414-1418, 2009.
- Mnih, Andriy and Hinton, Geoffrey. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems , volume 21, 2009.
- Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In AISTATS'05 , pp. 246-252, 2005.
- Pihlaja, M., Gutmann, M., and Hyv¨ arinen, A. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI2010). , 2010.
- Schwenk, Holger and Gauvain, Jean-Luc. Training neural network language models on very large corpora. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing , pp. 201-208, 2005.
- Socher, R., Lin, C.C., Ng, A.Y., and Manning, C.D. Parsing natural scenes and natural language with recursive neural networks. In International Conference on Machine Learning (ICML) , 2011.
- Stolcke, A. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing , volume 2, pp. 901-904, 2002.
- Turian, J., Ratinov, L., and Bengio, Y. Word representations: A simple and general method for semisupervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pp. 384-394, 2010.
- Authors
- Mnih, A., & Teh, Y. W.
- Affiliations
- Gatsby Computational Neuroscience Unit (GB)
- ORCID
- Teh, Yee Whye
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2012
- Pages
- 419-426
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1206.6426
- OpenAlex
- W2120861206
- Cited by
- 315
- References
- 21
- Citations by year
- 2025: 6 · 2024: 10 · 2023: 10 · 2022: 16 · 2021: 30 · 2020: 28 · 2019: 32 · 2018: 38 · 2017: 25 · 2016: 32 · 2015: 51 · 2014: 26 · 2013: 9 · 2012: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- noise contrastive estimation, training efficiency, importance sampling, Penn Treebank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
