The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Strategies for training large scale neural network language models
Abstract
We describe how to effectively train neural network based language models on large data sets. Fast convergence during training and better overall performance is observed when the training data are sorted by their relevance. We introduce hash-based implementation of a maximum entropy model, that can be trained as a part of the neural network model. This leads to significant reduction of computational complexity. We achieved around 10% relative reduction of word error rate on English Broadcast News speech recognition task, against large 4-gram model trained on 400M tokens.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“The architecture is called the Recurrent neural network based language model (RNN LM), and we have described it in [7] and [5].”
MethodSelf-citationBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2×
“For example, in [15] H = 100 is used when the amount of the training data is over 600M words.”
BackgroundMethodLearning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 1×
“In the context of simple RNN based language models, it has previously been investigated by Elman [18].”
MethodExtensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
Self-citation(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533223 · Cited by 1,566
(2005) · Cited by 823
A maximum entropy approach to adaptive statistical language modelling
Rosenfeld, R. (1996) · Computer Speech & Language · doi:10.1006/csla.1996.0011 · Cited by 542
Continuous space language models
Schwenk, H. (2007) · Computer Speech & Language · doi:10.1016/j.csl.2006.09.003 · Cited by 516
Empirical evaluation and combination of advanced language modeling techniques
Mikolov, T., Deoras, A., Kombrink, S., Burget, L., & Černocký, J. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-242 · Cited by 278
Self-citationClasses for fast maximum entropy training
Goodman, J. (2002) · 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221) · doi:10.1109/icassp.2001.940893 · Cited by 219
Structured Output Layer neural network language model
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J., & Yvon, F. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947610 · Cited by 136
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J. (2005) · doi:10.3115/1220575.1220601 · Cited by 123
The IBM Attila speech recognition toolkit
Soltau, H., Saon, G., & Kingsbury, B. E. D. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700829 · Cited by 119
Can artificial neural networks learn language models?
Xu, W. H., & Rudnicky, A. I. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-50 · Cited by 95
Shrinking exponential language models
Chen, S. F. (2009) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1620754.1620822 · Cited by 85
Efficient estimation of maximum entropy language models with n-gram features: an SRILM extension
Alumäe, T., & Kurimo, M. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-519 · Cited by 43
Zweig, G., Nguyen, P., Compernolle, D. V., Demuynck, K., Atlas, L., Clark, P., Sell, G., Wang, M., Sha, F., Hynek Hermansky, B., et al. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947490 · Cited by 37
Scaling shrinkage-based language models
Chen, S. F., Mangu, L., Ramabhadran, B., Sarikaya, R., & Sethy, A. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5373380 · Cited by 36
A Fast Re-scoring Strategy to Capture Long-Distance Dependencies
Deoras, A., Mikolov, T., & Church, K. (2011) · Cited by 33
Self-citationEfficient Subsampling for Training Complex Language Models
Xu, P., Gunawardana, A., & Khudanpur, S. P. (2011) · Cited by 24
Cited by
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Self-citationDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317
“However, these same architectures have been shown to be unstable to small, well sought, perturbations of the images.”
BackgroundGeometric Deep Learning: Going beyond Euclidean data
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., & Vandergheynst, P. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2693418 · Cited by 3,661
Wide & Deep Learning for Recommender Systems
Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., et al. (2016) · Proceedings of the 1st Workshop on Deep Learning for Recommender Systems · doi:10.1145/2988450.2988454 · Cited by 3,535
Deep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
Metadata
- Authors
- Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J.
- Venue
- 2011 IEEE Workshop on Automatic Speech Recognition & Understanding
- Publisher
- IEEE
- Published
- 2011-12
- Pages
- 196-201
- OpenAlex
- W1965154800
- Cited by
- 509
- References
- 20
- FWCI
- 15.83
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.