One billion word benchmark for measuring progress in statistical language modeling
Open accessAbstract
We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned Kneser-Ney 5-gram model achieves perplexity 67.6; a combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
After training for 10 epochs on the 1B Word Benchmark, the average forward and backward perplexities is 39.7, compared to 30.0 for the forward CNN-BIG-LSTM.
“After training for 10 epochs on the 1B Word Benchmark (Chelba et al., 2014), the average forward and backward perplexities is 39.7, compared to 30.0 for the forward CNN-BIG-LSTM.”
From Deep Contextualized Word Representations (2018) · cites this paper as Chelba et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 25 · Cited by 627
Research works printed but not linked: 25
- Y. Bengio, R. Ducharme, and P. Vin- cent. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155.
- T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. 2007. Large language models in machine translation. In Proceedings of EMNLP.
- A. Emami. 2006. A Neural Syntactic Lan- guage Model. Ph.D. thesis, Johns Hopkins University.
- J. T. Goodman. 2001b. Classes for fast maximum entropy training. In Proceedings of ICASSP.
- F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. 1991. A Dynamic Language Model for Speech Recognition. In Proceedings of the DARPA Workshop on Speech and Natural Language.
- C. Chelba, T. Brants, W. Neveitt, and P. Xu. 2010. Study on Interaction between Entropy Pruning and Kneser-Ney Smoothing. In Proceedings of Interspeech.
- Chen and Goodman1996] S. F. Chen and J. T. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of ACL.
- S. F. Chen. 2009. Shrinking exponential lan- guage models. In Proceedings of NAACL-HLT.
- S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. In IEEE Transactions on Acous- tics, Speech and Signal Processing.
- and Ney1995] R. Kneser and H. Ney. 1995. Im- proved Backing-Off For M-Gram Language Modeling. In Proceedings of ICASSP.
- T. Mikolov, S. Kombrink, L. Bur- get, J. Černocký, and S. Khudanpur. 2011. Exten- sions ofl Recurrent Neural Network Language Model. In Proceedings of ICASSP.
- T. Mikolov, A. Deoras, S. Kom- brink, L. Burget, and J. Černocký. 2011a. Empirical Evaluation and Combination of Advanced Language Modeling Techniques. In Proceedings of Interspeech.
- T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Černocký. 2011b. Strategies for Training Large Scale Neural Network Language Mod- els. In Proceedings of ASRU.
- T. Mikolov. 2012. Statistical Language Models based on Neural Networks. Ph.D. thesis, Brno University of Technology.
- A. Mnih and G. Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML.
- Morin and Bengio2005] F. Morin and Y. Bengio. 2005. Hierarchical Probabilistic Neural Network Language Model. In Proceedings of AISTATS.
- B. Roark, M. Saralar, M. Collins, and M. Johnson. 2004. Discriminative language model- ing with conditional random fields and the perceptron algorithm. In Proceedings of ACL.
- R. Rosenfeld. 1994. Adaptive Statis- tical Language Modeling: A Maximum Entropy Ap- proach. Ph.D. thesis, Carnegie Mellon University.
- H. Schwenk. 2007. Continuous space language models. Computer Speech and Language, vol. 21.
- A. Stolcke. 1998. Entropy-based Pruning of Back-off Language Models. In Proceedings of News Transcription and Understanding Workshop.
- M. Sundermeyer, R. Schluter, and H. Ney. 2012. LSTM Neural Networks for Lan- guage Modeling. In Proceedings of Interspeech.
- Y. W. Teh. 2006. A hierarchical Bayesian lan- guage model based on PitmanYor processes. In Pro- ceedings of Coling/ACL.
- Y. Wu, H. Yamamoto, X. Lu, S. Matsuda, C. Hori, and H. Kashioka. 2012. Factored Recur- rent Neural Network Language Model in TED Lecture Transcription. In Proceedings of IWSLT.
- Puyang Xu, A. Gunawardana, and S. Khudanpur. 2011. Efficient Subsampling for Train- ing Complex Language Models. In Proceedings of EMNLP.
- Zweig and Makarychev2013] G. Zweig and K. Makarychev. 2013. Speed Regularization and Optimality in Word Classing. In Proceedings of ICASSP.
- Authors
- Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T.
- Affiliations
- Google (United States) (US)
- Venue
- Interspeech 2014
- Publisher
- ISCA
- Published
- 2014-09-14
- Pages
- 2635-2639
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W1539309091
- Cited by
- 627
- References
- 25
- Citations by year
- 2026: 3 · 2025: 23 · 2024: 24 · 2023: 51 · 2022: 62 · 2021: 90 · 2020: 103 · 2019: 115 · 2018: 68 · 2017: 31 · 2016: 33 · 2015: 19 · 2014: 5
- FWCI
- 17.27
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- statistical language models, benchmark corpus, perplexity, Kneser-Ney smoothing, n-gram language models, recurrent neural network language models, cross-entropy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
