The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Context dependent recurrent neural network language model
Abstract
Recurrent neural network language models (RNNLMs) have recently demonstrated state-of-the-art performance across a variety of tasks. In this paper, we improve their performance by providing a contextual real-valued input vector in association with each word. This vector is used to convey contextual information about the sentence being modeled. By performing Latent Dirichlet Allocation using a block of preceding text, we achieve a topic-conditioned RNNLM. This approach has the key advantage of avoiding the data fragmentation associated with building multiple topic models on different data subsets. We report perplexity results on the Penn Treebank data, where we achieve a new state-of-the-art. We further apply the model to the Wall Street Journal speech recognition task, where we observe improvements in word-error-rate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
Self-citationKaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/jmlr.2003.3.4-5.993 · Cited by 3,024
An empirical study of smoothing techniques for language modeling
Chen, S. F., & Goodman, J. (1999) · Computer Speech & Language · doi:10.1006/csla.1999.0128 · Cited by 2,066
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
Self-citation(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533223 · Cited by 1,566
Three new graphical models for statistical language modelling
Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273577 · Cited by 563
Learning Recurrent Neural Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2011) · Cited by 525
Continuous space language models
Schwenk, H. (2007) · Computer Speech & Language · doi:10.1016/j.csl.2006.09.003 · Cited by 516
Strategies for training large scale neural network language models
Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163930 · Cited by 509
Self-citationA cache-based natural language model for speech recognition
Kuhn, R., & De Mori, R. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.56193 · Cited by 495
Exploiting latent semantic information in statistical language modeling
Bellegarda, J. R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880084 · Cited by 349
Empirical evaluation and combination of advanced language modeling techniques
Mikolov, T., Deoras, A., Kombrink, S., Burget, L., & Černocký, J. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-242 · Cited by 278
Self-citationThe subspace Gaussian mixture model—A structured model for speech recognition
Povey, D., Burget, L., Agarwal, M., Akyazi, P., Kai, F., Ghoshal, A., Glembek, O., Goel, N., Karafiát, M., Rastrow, A., et al. (2011) · Computer Speech & Language · doi:10.1016/j.csl.2010.06.003 · Cited by 276
Trigger-based language models: a maximum entropy approach
Lau, R., Rosenfeld, R., & Roukos, S. (1993) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1993.319225 · Cited by 172
Modeling long distance dependence in language: topic mixtures versus dynamic cache models
Iyer, R. M., & Mari Ostendorf, E. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.736328 · Cited by 157
Structured Output Layer neural network language model
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J., & Yvon, F. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947610 · Cited by 136
Khudanpur, S. P., & Jun, W. (2000) · Computer Speech & Language · doi:10.1006/csla.2000.0149 · Cited by 95
Shrinking exponential language models
Chen, S. F. (2009) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1620754.1620822 · Cited by 85
Random forests and the data sparseness problem in language modeling
Xu, P., & Jelinek, F. (2006) · Computer Speech & Language · doi:10.1016/j.csl.2006.01.003 · Cited by 63
Performance prediction for exponential language models
Chen, S. F. (2009) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1620754.1620820 · Cited by 56
Robust decision tree state tying for continuous speech recognition
Reichl, W., & Chou, W. (2000) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.861375 · Cited by 56
A joint language model with fine-grain syntactic tags
Filimonov, D., & Harper, M. P. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699648.1699657 · Cited by 32
Self-supervised discriminative training of statistical language models
Xu, P., Karakos, D., & Khudanpur, S. P. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5373401 · Cited by 30
Measuring the Influence of Long Range Dependencies with Neural Network Language Models
Le, H.-S., Allauzen, A., & Yvon, F. (2012) · Cited by 25
Discriminative Language Modeling with Linguistic and Statistically Derived Features
Arısoy, E., Saraçlar, M., Brian Roark, A., & Izhak Shafran, J. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2162323 · Cited by 24
Personalizing model M for voice-search
Zweig, G., & Chang, S. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-243 · Cited by 14
Self-citationCache neural network language models based on long-distance dependencies for a spoken dialog system
Zamora-Martínez, F., España-Boquera, S., Castro-Bleda, M. J., & Mori, R. D. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6289041 · Cited by 12
Cited by
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
Metadata
- Authors
- Mikolov, T., & Zweig, G.
- Venue
- 2012 IEEE Spoken Language Technology Workshop (SLT)
- Publisher
- IEEE
- Published
- 2012-12
- Pages
- 234-239
- OpenAlex
- W1999965501
- Cited by
- 560
- References
- 33
- FWCI
- 20.43
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.