The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
From Feedforward to Recurrent LSTM Neural Networks for Language Modeling
Abstract
Language models have traditionally been estimated based on relative frequencies, using count statistics that can be extracted from huge amounts of text data. More recently, it has been found that neural networks are particularly powerful at estimating probability distributions over word sequences, giving substantial improvements over state-of-the-art count models. However, the performance of neural network language models strongly depends on their architectural structure. This paper compares count models to feedforward, recurrent, and long short-term memory (LSTM) neural network variants on two large-vocabulary speech recognition tasks. We evaluate the models in terms of perplexity and word error rate, experimentally validating the strong correlation of the two quantities, which we find to hold regardless of the underlying type of the language model. Furthermore, neural networks incur an increased computational complexity compared to count models, and they differently model context dependences, often exceeding the number of words that are taken into account by count based approaches. These differences require efficient search methods for neural networks, and we analyze the potential improvements that can be obtained when applying advanced algorithms to the rescoring of word lattices on large-scale setups.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 1×
“While the RNN architecture itself facilitates the use of long range history information, in [37] it was found that standard”
MethodRecurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 2×
“Recurrent neural network LMs as introduced in [9] follow a very similar topology compared to feedforward neural networks.”
BackgroundA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
(1992) · doi:10.5555/176313.176316 · Cited by 2,892
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · 9th International Conference on Artificial Neural Networks: ICANN '99 · doi:10.1049/cp:19991218 · Cited by 2,337
An empirical study of smoothing techniques for language modeling
Chen, S. F., & Goodman, J. (1999) · Computer Speech & Language · doi:10.1006/csla.1999.0128 · Cited by 2,066
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963
Self-citationStochastic Gradient Descent Tricks
Bottou, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_25 · Cited by 1,929
Neural networks for pattern recognition
Marshall, J. A. (1995) · Neural Networks · doi:10.1016/0893-6080(95)90002-0 · Cited by 1,889
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Katz, S. (1987) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1987.1165125 · Cited by 1,637
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
Improved backing-off for M-gram language modeling
Kneser, R., & Ney, H. (2002) · 1995 International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1995.479394 · Cited by 1,473
Self-citation(2005) · Cited by 823
Two decades of statistical language modeling: where do we go from here?
Rosenfeld, R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880083 · Cited by 735
Tandem connectionist feature extraction for conventional HMM systems
Hermansky, H., Ellis, D. P. W., & Sharma, S. (2002) · 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100) · doi:10.1109/icassp.2000.862024 · Cited by 693
Mangu, L., Brill, E., & Stolcke, A. (2000) · Computer Speech & Language · doi:10.1006/csla.2000.0152 · Cited by 646
Continuous space language models
Schwenk, H. (2007) · Computer Speech & Language · doi:10.1016/j.csl.2006.09.003 · Cited by 516
Strategies for training large scale neural network language models
Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163930 · Cited by 509
RNNLM - Recurrent Neural Network Language Modeling Toolkit
Mikolov, T., Kombrink, S., Deoras, A., Burget, L., & Černocký, J. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · Cited by 256
Joint Language and Translation Modeling with Recurrent Neural Networks
Auli, M., Galley, M., Quirk, C., & Zweig, G. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1106 · Cited by 253
Testing the correlation of word error rate and perplexity
Klakow, D., & Peters, J. (2002) · Speech Communication · doi:10.1016/s0167-6393(01)00041-3 · Cited by 244
Classes for fast maximum entropy training
Goodman, J. (2002) · 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221) · doi:10.1109/icassp.2001.940893 · Cited by 219
Recurrent neural network based language modeling in meeting recognition
Kombrink, S., Mikolov, T., Karafiát, M., & Burget, L. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-720 · Cited by 198
Algorithms for bigram and trigram word clustering
Martin, S., Liermann, J., & Ney, H. (1998) · Speech Communication · doi:10.1016/s0167-6393(97)00062-9 · Cited by 191
Self-citationDynamic programming search for continuous speech recognition
Ney, H., & Ortmanns, S. (1999) · IEEE Signal Processing Magazine · doi:10.1109/79.790984 · Cited by 178
Self-citationTranslation Modeling with Bidirectional Recurrent Neural Networks
Sundermeyer, M., Alkhouli, T., Wuebker, J., & Ney, H. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1003 · Cited by 176
Self-citationDeep Neural Network Language Models
Arısoy, E., Sainath, T. N., Kingsbury, B. E. D., & Ramabhadran, B. (2012) · Cited by 174
Comparison of feedforward and recurrent neural network language models
Sundermeyer, M., Oparin, I., Gauvain, J.-L., Freiberg, B., Schlüter, R., & Ney, H. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639310 · Cited by 167
Self-citationProgress in dynamic programming search for LVCSR
Ney, H., & Ortmanns, S. (2000) · Proceedings of the IEEE · doi:10.1109/5.880081 · Cited by 111
Self-citationEfficient lattice rescoring using recurrent neural network language models
Liu, X., Wang, Y.-Q., Chen, X., Gales, M., & Woodland, P. C. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854535 · Cited by 91
RASR - The RWTH Aachen University Open Source Speech Recognition Toolkit
Rybach, D., Hahn, S., Lehnen, P., Nolden, D., Sundermeyer, M., Tüske, Z., Wiesler, S., Schlüter, R., & Ney, H. (2011) · RWTH Publications (RWTH Aachen) · Cited by 77
Self-citationStructured Output Layer Neural Network Language Models for Speech Recognition
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J.-L., & Yvon, F. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2215599 · Cited by 76
Lattice decoding and rescoring with long-Span neural network language models
Sundermeyer, M., Tüske, Z., Schlüter, R., & Ney, H. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-167 · Cited by 45
Self-citationRecognition performance of a structured language model
Chelba, C. I., & Jelinek, F. (1999) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1999-397x · Cited by 38
Forming Word Classes by Statistical Clustering for Statistical Language Modelling
Kneser, R., & Ney, H. (1993) · Springer eBooks · doi:10.1007/978-94-011-1769-2_15 · Cited by 37
Self-citationrwthlm — the RWTH aachen university neural network language modeling toolkit
Sundermeyer, M., Schlüter, R., & Ney, H. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-475 · Cited by 34
Self-citationA Fast Re-scoring Strategy to Capture Long-Distance Dependencies
Deoras, A., Mikolov, T., & Church, K. (2011) · Cited by 33
Use of contexts in language model interpolation and adaptation
Liu, X., Gales, M., & Woodland, P. C. (2012) · Computer Speech & Language · doi:10.1016/j.csl.2012.06.004 · Cited by 32
Performance analysis of Neural Networks in combination with n-gram language models
Oparin, I., Sundermeyer, M., Ney, H., & Gauvain, J. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6289044 · Cited by 29
Self-citationMeasuring the Influence of Long Range Dependencies with Neural Network Language Models
Le, H.-S., Allauzen, A., & Yvon, F. (2012) · Cited by 25
Speed regularization and optimality in word classing
Zweig, G., & Makarychev, K. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639271 · Cited by 24
Cited by
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Supervised Speech Separation Based on Deep Learning: An Overview
Wang, D., & Chen, J. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2842159 · Cited by 1,524
Deep Recurrent Neural Networks for Hyperspectral Image Classification
Mou, L., Ghamisi, P., & Zhu, X. X. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2016.2636241 · Cited by 1,383
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
Oh, S. L., Ng, E. Y. K., Tan, R. S., & Acharya, U. R. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2018.06.002 · Cited by 729
Metadata
- Authors
- Sundermeyer, M., Ney, H., & Schluter, R.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2015-03
- Volume
- 23
- Issue
- 3
- Pages
- 517-529
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W1985258458
- Cited by
- 507
- References
- 43
- FWCI
- 38.79
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.