Efficient Training and Evaluation of Recurrent Neural Network Language Models for Automatic Speech Recognition
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Recurrent neural network language models (RNNLMs) are becoming increasingly popular for a range of applications including automatic speech recognition. An important issue that limits their possible application areas is the computational cost incurred in training and evaluation. This paper describes a series of new efficiency improving approaches that allows RNNLMs to be more efficiently trained on graphics processing units (GPUs) and evaluated on CPUs. First, a modified RNNLM architecture with a nonclass-based, full output layer structure (F-RNNLM) is proposed. This modified architecture facilitates a novel spliced sentence bunch mode parallelization of F-RNNLM training using large quantities of data on a GPU. Second, two efficient RNNLM training criteria based on variance regularization and noise contrastive estimation are explored to specifically reduce the computation associated with the RNNLM output layer softmax normalisation term. Finally, a pipelined training algorithm utilizing multiple GPUs is also used to further improve the training speed. Initially, RNNLMs were trained on a moderate dataset with 20M words from a large vocabulary conversational telephone speech recognition task. The training time of RNNLM is reduced by up to a factor of 53 on a single GPU over the standard CPU-based RNNLM toolkit. A 56 times speed up in test time evaluation on a CPU was obtained over the baseline F-RNNLMs. Consistent improvements in both recognition accuracy and perplexity were also obtained over C-RNNLMs. Experiments on Google's one billion corpus also reveals that the training of RNNLM scales well.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 3×
“For more efficient lattice rescoring using RNNLMs, appropriate approximation schemes, for example, based on clustering among complete histories [17] can be used.”
MethodLSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · cited 1×
“X. Liu is with the Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Shatin, Hong Kong (e-mail: xyliu@se.cuhk.edu.hk).”
BackgroundOne Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013) · arXiv · doi:10.48550/arxiv.1312.3005 · Cited by 1,051
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627
Fast and Robust Neural Network Joint Models for Statistical Machine Translation
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., & Makhoul, J. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1129 · Cited by 481 · cited 3×
“As the number of classes is normally significantly smaller than the full output layer size, a speed up of both training and evaluation can be achieved [11].”
BackgroundInfluentialA Fast and Simple Algorithm for Training Neural Probabilistic Language Models
Mnih, A., & Teh, Y. W. (2012) · arXiv · doi:10.48550/arxiv.1206.6426 · Cited by 315
Improved neural network based language modelling and adaptation
Park, J., Liu, X., Gales, M. J. F., & Woodland, P. C. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-342 · Cited by 81
Self-citationLarge scale hierarchical neural network language models
Kuo, H.-K., Arısoy, E., Emami, A., & Vozila, P. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-458 · Cited by 26
Cited by
Indoor positioning and wayfinding systems: a survey
Kunhoth, J., Karkar, A., Al-Maadeed, S., & Al-Ali, A. (2020) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-020-00222-0 · Cited by 270
Distilling the Knowledge From Handcrafted Features for Human Activity Recognition
Chen, Z., Zhang, L., Cao, Z., & Guo, J. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2789925 · Cited by 160
“The RNN has been reported to be efficient on various topics such as speech recognition [29] and text generation [30].”
Wang, Q., Xu, W., Huang, X., & Yang, K. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.03.006 · Cited by 82
On-device machine learning An algorithms and learning theory perspective
Dhar, S., Guo, J., Liu, J., Tripathi, S., Kurup, U., & Shah, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00623 · Cited by 47
Metadata
- Authors
- Chen, X., Liu, X., Wang, Y., Gales, M. J. F., & Woodland, P. C.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2016-11
- Volume
- 24
- Issue
- 11
- Pages
- 2146-2157
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W2496955520
- Cited by
- 58
- References
- 9
- FWCI
- 8.58
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.