The DOI matches its DataCite record.
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Open accessAbstract
We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned Kneser-Ney 5-gram model achieves perplexity 67.6; a combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is critical to use a document-level corpus rather than a shuffled sentence-level corpus such as the Billion Word Benchmark in order to extract long contiguous sequences.
“It is critical to use a document-level corpus rather than a shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to extract long contiguous sequences.”
From BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018) · cites this paper as Chelba et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 26
- Y. Bengio, R. Ducharme, and P. Vin- cent. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155.
- T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. 2007. Large language models in machine translation. In Proceedings of EMNLP.
- A. Emami. 2006. A Neural Syntactic Lan- guage Model. Ph.D. thesis, Johns Hopkins University.
- J. T. Goodman. 2001b. Classes for fast maximum entropy training. In Proceedings of ICASSP.
- F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. 1991. A Dynamic Language Model for Speech Recognition. In Proceedings of the DARPA Workshop on Speech and Natural Language.
- C. Chelba, T. Brants, W. Neveitt, and P. Xu. 2010. Study on Interaction between Entropy Pruning and Kneser-Ney Smoothing. In Proceedings of Interspeech.
- Chen and Goodman1996] S. F. Chen and J. T. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of ACL.
- S. F. Chen. 2009. Shrinking exponential lan- guage models. In Proceedings of NAACL-HLT.
- S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. In IEEE Transactions on Acous- tics, Speech and Signal Processing.
- and Ney1995] R. Kneser and H. Ney. 1995. Im- proved Backing-Off For M-Gram Language Modeling. In Proceedings of ICASSP.
- T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur. 2010. Recurrent neu- ral network based language model. In Proceedings of Interspeech.
- T. Mikolov, S. Kombrink, L. Bur- get, J. Černocký, and S. Khudanpur. 2011. Exten- sions ofl Recurrent Neural Network Language Model. In Proceedings of ICASSP.
- T. Mikolov, A. Deoras, S. Kom- brink, L. Burget, and J. Černocký. 2011a. Empirical Evaluation and Combination of Advanced Language Modeling Techniques. In Proceedings of Interspeech.
- T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Černocký. 2011b. Strategies for Training Large Scale Neural Network Language Mod- els. In Proceedings of ASRU.
- T. Mikolov. 2012. Statistical Language Models based on Neural Networks. Ph.D. thesis, Brno University of Technology.
- A. Mnih and G. Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML.
- Morin and Bengio2005] F. Morin and Y. Bengio. 2005. Hierarchical Probabilistic Neural Network Language Model. In Proceedings of AISTATS.
- B. Roark, M. Saralar, M. Collins, and M. Johnson. 2004. Discriminative language model- ing with conditional random fields and the perceptron algorithm. In Proceedings of ACL.
- R. Rosenfeld. 1994. Adaptive Statis- tical Language Modeling: A Maximum Entropy Ap- proach. Ph.D. thesis, Carnegie Mellon University.
- H. Schwenk. 2007. Continuous space language models. Computer Speech and Language, vol. 21.
- A. Stolcke. 1998. Entropy-based Pruning of Back-off Language Models. In Proceedings of News Transcription and Understanding Workshop.
- M. Sundermeyer, R. Schluter, and H. Ney. 2012. LSTM Neural Networks for Lan- guage Modeling. In Proceedings of Interspeech.
- Y. W. Teh. 2006. A hierarchical Bayesian lan- guage model based on PitmanYor processes. In Pro- ceedings of Coling/ACL.
- Y. Wu, H. Yamamoto, X. Lu, S. Matsuda, C. Hori, and H. Kashioka. 2012. Factored Recur- rent Neural Network Language Model in TED Lecture Transcription. In Proceedings of IWSLT.
- Puyang Xu, A. Gunawardana, and S. Khudanpur. 2011. Efficient Subsampling for Train- ing Complex Language Models. In Proceedings of EMNLP.
- Zweig and Makarychev2013] G. Zweig and K. Makarychev. 2013. Speed Regularization and Optimality in Word Classing. In Proceedings of ICASSP.
Cited by
Sorted by citations
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304
Symbolic Discovery of Optimization Algorithms
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.06675 · Cited by 303
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01951 · Cited by 90
Metadata
- Authors
- Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2013
- Pages
- 2635-2639
- ISSN
- 2308-457X (ISSN-L)
- arXiv
- 1312.3005
- OpenAlex
- W1539309091
- Cited by
- 1,051
- References
- 0
- Citations by year
- 2026: 75 · 2025: 70 · 2024: 62 · 2023: 81 · 2022: 80 · 2021: 120 · 2020: 139 · 2019: 183 · 2018: 107 · 2017: 58 · 2016: 46 · 2015: 23 · 2014: 6
- FWCI
- 25.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- statistical language models, benchmark corpus, perplexity, Kneser-Ney smoothing, n-gram language models, recurrent neural network language models, cross-entropy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
