Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
“Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model N ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural networks [14, 2].”
BackgroundSelf-citationDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Matching Networks for One Shot Learning
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04080 · Cited by 854
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
“, 2000), recurrent neural networks (Elman, 1990; Mikolov et al., 2010) and LSTMs (Hochreiter and Schmidhuber, 1997; Graves, 2013).”
BackgroundMethodInfluentialGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
“To manage the structured, combinatorial action space, the agent uses an auto-regressive policy 7,10,11 and recurrent pointer network 12 .”
BackgroundMethodTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
“However, different from truncated BPTT, our method caches a sequence of hidden states instead of the last one, and should be applied together with the relative positional encoding technique described in Section 3.3.”
BackgroundMethodInfluentialA survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
“ss layers is enabled by imparting two gating units within a layer (equation (5)). The idea of a gating mechanism was inspired by Long Short Term Memory (LSTM) based on Recurrent Neural Networks (RNN) [156], [157]. The aggregation of information by combining the lth layer and previous lj layers information creates a re…”
BackgroundMethodLong short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
“In language modeling, a conventional RNN has obtained significant reduction in perplexity over standard n -gram models [6] and an LSTM RNN model has shown improvements over conventional RNN LMs [7].”
MethodRecent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Self-citationOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Federated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565
“In addition, there also exists several DNN networks and architectures tailored to process the varying natures of input data, e.g., Multilayer Perceptron (MLP) [55], Convolutional Neural Network (CNN) [56] typically for CV tasks, and Recurrent Neural Network (RNN) [57] usually for sequential tasks.”
BackgroundAttention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
“Language modeling is the first task where RNNs have achieved substantial success (Mikolov et al., 2010; 2011; Pascanu et al., 2013).”
BackgroundAttention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification
Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., & Xu, B. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2034 · Cited by 2,012
“Traditional relation classification methods that employ handcrafted features from lexical resources, are usually based on pattern matching, and have achieved high performance (Bunescu ∗Correspondence author: zhenyu.qi@ia.ac.cn and Mooney, 2005; Mintz et al., 2009; Rink and Harabagiu, 2010).”
MethodPersonalized Top-N Sequential Recommendation via Convolutional Sequence Embedding
Tang, J., & Wang, K. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159656 · Cited by 1,975
“While RNN has shown to have an impressive capability in modeling sequences [18], its sequentially connected network structure may not work well under sequential recommendation setting.”
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937
“Matrix-vector multiplication (M × V) is a basic building block in a wide range of neural networks and deep learning applications.”
BackgroundMethodLSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963
“On the contrary we observed that convergence was considerably slowed down by normalization.”
BackgroundSession-Based Recommendation with Graph Neural Networks
Wu, S., Tang, Y., Zhu, Y., Wang, L., Xie, X., & Tan, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301346 · Cited by 1,532
“Among numerous language models, the recurrent neural network (RNN) has been the most successful one in modeling sentences (Mikolov et al. 2010) and has been flourishingly applied in various natural language processing tasks, such as machine translation (Cho et al. 2014), conversation machine (Serban…”
BackgroundA Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
“Since much of NLP is dependent on the order of words or other elements such as phonemes or sentences, it is useful to have memory of the previous elements when processing new ones [29]–[31].”
BackgroundDeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
“In recent years, neural language models that use recurrent neural networks have been shown to be highly effective across various NLP tasks [3, 25] language model, a LSTM-based one can encode more intricate patterns and maintain long-range state over a sequence [34].”
BackgroundInfluentialDeep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data
Karpatne, A., Atluri, G., Faghmous, J. H., Steinbach, M., Banerjee, A., Ganguly, A., Shekhar, S., Samatova, N., & Kumar, V. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2720168 · Cited by 1,752
“able to incorporate the sequential context of time in speech and language processing [44].”
BackgroundBuilding End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I., Sordoni, A., Bengio, Y., Courville, A., & Pineau, J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9883 · Cited by 1,615
“Nevertheless, for probabilistic language models word perplexity is a well-established performance metric (Bengio et al. 2003; Mikolov et al. 2010), and has been suggested for generative dialogue models previously (Pietquin and Hastie 2013).”
BackgroundMethodInfluentialA State-of-the-Art Survey on Deep Learning Theory and Architectures
Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., Hasan, M., Van Essen, B. C., Awwal, A. A. S., & Asari, V. K. (2019) · Electronics · doi:10.3390/electronics8030292 · Cited by 1,620
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
Self-citationCross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
“While recurrent neural networks obtain state-of-the-art performance on language modeling benchmarks (Mikolov et al., 2010; Jozefowicz et al., 2016), Transformer models are also very competitive (Dai et al., 2019).”
BackgroundLanguage Modeling with Gated Convolutional Networks
Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08083 · Cited by 1,080
“Recently, neural networks (Bengio et al., 2003; Mikolov et al., 2010; Jozefowicz et al., 2016) outperform classical n-gram language models (Kneser & Ney, 1995; Chen & Goodman, 1996).”
BackgroundA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,491
“Neural Language Models (NLM) address the n -gram data sparsity issue through parameterization of words as vectors (word embeddings) and using them as inputs to a neural network (Bengio, Ducharme, and Vincent 2003; Mikolov et al. 2010).”
MethodInfluentialCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530
“Neural Language Models (NLM) address the n -gram data sparsity issue through parameterization of words as vectors (word embeddings) and using them as inputs to a neural network (Bengio, Ducharme, and Vincent 2003; Mikolov et al. 2010).”
MethodInfluentialAn Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · International Conference on Machine Learning · Cited by 2,051
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development
Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123618 · Cited by 1,452
“The neural language model (NLM) [34], [35] started in 2010 has driven researchers to model complex natural language and multiple tasks [36], [37] with a wide range of deep neural network structures such as multilayer perceptron (MLP), convolutional neural network (CNN) and recurrent neural network (RNN), and thus form…”
MethodSequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426
“…dimensions which are important when training a model for text generation: the exposure bias which can adversely affect generation at test time, the ability to fully back-propagate gradients (including with respect to the chosen inputs at each time step), and a loss operating at the sequence level.”
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
“Our work is also inspired by the recent success of neural language modeling (Bengio et al., 2003; Mikolov et al., 2010; Mikolov, 2012), which shows that recurrent neural networks are rather effective models for natural language.”
BackgroundMolecular de-novo design through deep reinforcement learning
Olivecrona, M., Blaschke, T., Engkvist, O., & Chen, H. (2017) · Journal of Cheminformatics · doi:10.1186/s13321-017-0235-x · Cited by 1,324
“The Recurrent Neural Network (RNN) is commonly used as a generative model for data of sequential nature, and have been used successfully for tasks such as natural language processing [11] and music generation [12].”
MethodSequence Transduction with Recurrent Neural Networks
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,170
“Furthermore, these results demonstrate the use of long-range memory to perform such actions as closing parentheses after many intervening characters (Sutskever et al., 2011), or using delayed strokes to identify handwritten characters from pen trajectories (Graves et al., 2008).”
BackgroundDeep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
“In contrast to typical neural networks that have structures for a feed-forward network, RNNs can use the temporal memory of networks and yield significant performance improvements in natural language processing, recognition, handwriting recognition, speech recognition and generation tasks (24, 25).”
BackgroundDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,251
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1176 · Cited by 1,232
“The RLM is trained by backpropagation through time (Mikolov et al., 2010).”
BackgroundMethodInfluentialCui, P., Wang, X., Pei, J., & Zhu, W. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2849727 · Cited by 1,274
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · International Conference on Machine Learning · Cited by 1,783
Deng, L., Hinton, G., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639344 · Cited by 1,178
“Recently, however, recurrent neural networks have achieved excellent results at language modeling [41] and the use of multiple hidden layers has allowed recurrent neural networks to outperform all other methods on TIMIT [21].”
BackgroundLong Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1053 · Cited by 1,172
“…or vanishing gradients (Bengio et al., 1994), leading to models with gated activation functions (Hochreiter and Schmidhuber, 1997; Cho et al., 2014), and more advanced architectures that enhance the information flow within the network (Koutnı́k et al., 2014; Chung et al., 2015; Yao et al., 2015).”
BackgroundLong Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
“This template is naturally suited for many NLP tasks such as language modeling [2, 64, 65], machine translation [66, 67, 68], speech recognition [69, 70, 71, 72], image captioning [73].”
BackgroundMethodInfluentialDenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.494 · Cited by 1,137
“These models have previously been used in language modeling [2, 15, 33, 44], where they are known to learn powerful long-term interactions [22].”
BackgroundMethodConvolutional Neural Network Architectures for Matching Natural Language Sentences
Hu, B., Lu, Z., Li, H., & Chen, Q. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03244 · Cited by 1,098
“Relation to Recursive Models Our convolutional model differs from Recurrent Neural Network (RNN, [15]) and Recursive Auto-Encoder (RAE, [21]) in several important ways.”
MethodInfluentialBrain intelligence: go beyond artificial intelligence
Lu, H., Li, Y., Chen, M., Kim, H., & Serikawa, S. (2018) · Mobile Networks and Applications · doi:10.1007/s11036-017-0932-8 · Cited by 1,153
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
Lin, Y., Han, S., Mao, H., Wang, Y., & Dally, W. J. (2017) · arXiv · doi:10.48550/arxiv.1712.01887 · Cited by 1,134
“We validate our approach on three types of machine learning tasks: image classification with ResNet and AlexNet [20, 21] on Cifar10 [22] and ImageNet [23], language modeling with 2-layer LSTM [24, 25] on Penn Treebank dataset [26, 27], and speech recognition [28] with 5-layer LSTM on AN4 [29] and 7-layer GRU on Libris…”
BackgroundMethodBao, W., Yue, J., & Rao, Y. (2017) · PLOS ONE · doi:10.1371/journal.pone.0180944 · Cited by 1,124
Joint Deep Modeling of Users and Items Using Reviews for Recommendation
Zheng, L., Noroozi, V., & Yu, P. S. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018665 · Cited by 1,075
“04 78 3v 1 [c s. LG ] 17 J an 2 CCS Concepts •Information systems→Collaborative filtering; Recommender systems; •Computing methodologies → Neural networks; Keywords Recommender Systems, Deep Learning, Convolutional Neural Networks, Rating Prediction”
Interactive Attention Networks for Aspect-Level Sentiment Classification
Ma, D., Li, S., Zhang, X., & Wang, H. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/568 · Cited by 1,069
Efficient Neural Architecture Search via Parameter Sharing
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 517
Deep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079
EIE: Efficient Inference Engine on Compressed Deep Neural Network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca.2016.30 · Cited by 1,077
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013) · arXiv · doi:10.48550/arxiv.1312.3005 · Cited by 1,051
Self-citationxDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
Lian, J., Zhou, X., Zhang, F., Chen, Z., Xie, X., & Sun, G. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220023 · Cited by 1,040
“it feedback. 5.2 Recommender Systems with Deep Learning Deep learning techniques have achieved great success in computer vision [10, 19], speech recognition [1, 15] and natural language understanding [6, 29]. As a result, an increasing number of researchers are interested in employing DNNs for recommender systems. 5.2…”
BackgroundInfluentialPredicting the Next Location: A Recurrent Model with Spatial and Temporal Contexts
Liu, Q., Wu, S., Wang, L., & Tan, T. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9971 · Cited by 975
“Recently, Recurrent Neural Networks (RNN) not only has been successfully applied in word embedding for sentence modeling (Mikolov et al. 2010; 2011a; 2011b), but also shows promising performance for sequential click prediction (Zhang et al.”
BackgroundMethodInfluentialDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1152 · Cited by 978
An overview of online fake news: Characterization, detection, and discussion
Zhang, X., & Ghorbani, A. A. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.03.004 · Cited by 997
Normalizing Flows for Probabilistic Modeling and Inference
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., & Lakshminarayanan, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02762 · Cited by 318
“, 2013, 2014), sequence models (Mikolov et al., 2010; Graves, 2013; Sutskever et al., 2014), and image/video models (Theis and Bethge, 2015; Oord et al.”
Background(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
“The creation of the Recurrent Neural Network (RNN) [8], which allowed for the modeling of sequential data, including language, was a crucial milestone.”
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
Marcheggiani, D., & Titov, I. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1159 · Cited by 907
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879
“sequence to sequence learning. Several neural language models such as neural probabilistic language model [11], log-bilinear models [105], skip-gram models [98], and recurrent neural networks (RNNs) [99] have been proposed for learning sequence to sequence tasks. RNNs have widely been used in various sequence learning…”
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874
“The generation model proposed in this paper is based on a recurrent NN architecture (Mikolov et al., 2010) in which a 1-hot encoding w t of a token w t is input at each time step t conditioned on a recurrent hidden layer h t and outputs the probability distribution of the next token w t +1 .”
BackgroundMethodPiech, C., Spencer, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., & Sohl-Dickstein, J. (2015) · arXiv · doi:10.48550/arxiv.1506.05908
“Recurrent neural networks are competitive or state-of-the-art for several time series tasks–for instance, speech to text [15], translation [22], and image captioning [17]–where large amounts of training data are available.”
BackgroundA survey on deep learning based bearing fault diagnosis
Hoang, D.-T., & Kang, H.-J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.06.078 · Cited by 866
Sak, H., Senior, A., & Beaufays, F. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.1128 · Cited by 838
“In language modeling, a conventional RNN has obtained very significant reduction of perplexity over standard n-gram models [13].”
MethodMultimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs
Jin, Z., Cao, J., Guo, H., Zhang, Y., & Luo, J. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123454 · Cited by 812
“Specifically, convolutional neural networks (CNN) have shown their powerful abilities on image representation [18, 27] while recurrent neural networks (RN-N) are widely employed recently in sentence representation [3, 22].”
BackgroundExploring the Limits of Language Modeling
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 823
“Much work has been done on small and large scale RNN-based LMs (Mikolov et al., 2010; Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
BackgroundVisualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845
“These include language modeling (22), handwriting recognition and generation (9), machine translation (28; 1), speech recognition (10), video analysis (7) and image captioning (30; 19).”
BackgroundPalangi, H., Deng, L., Shen, Y., Gao, J., He, X., Chen, J., Song, X., & Ward, R. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2520371 · Cited by 618
“To resolve the gradient explosion problem we use gradient re-normalization method described in [35], [24].”
MethodPipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., & Zaharia, M. (2019) · Proceedings of the 27th ACM Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 818
“…(ILSVRC12) [44] dataset; 2) Translation, using the WMT16 English to German dataset for training, and the “newstest2014” dataset for validation; 3) Language Modeling, using the Penn Treebank (PTB) [41] dataset; and 4) Video Captioning (S2VT), using the Microsoft Video description corpus (MSVD) [11].”
MethodZhang, J., Zhu, Y., Zhang, X., Ye, M., & Yang, J. (2018) · Journal of Hydrology · doi:10.1016/j.jhydrol.2018.04.065 · Cited by 816
“LSTM model have already been used as a very advanced model in the field of Artificial Intelligence and Deep Learning, such as in nature language processing (Mikolov et al., 2010; Sundermeyer et al., 2012), speech recognition (Graves and Jaitly, 2014), machine translation (Sutskever et al.”
MethodFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
BackgroundPTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks
Tang, J., Qu, M., & Mei, Q. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783307 · Cited by 723
“Result : word embeddings (cid:126)w . while iter ≤ T do • sample an edge from E ww and draw K negative edges, and update the word embeddings; • sample an edge from E wd and draw K negative edges, and update the word and document embeddings; • sample an edge from E wl and draw K negative edges, and…”
BackgroundFadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2707140 · Cited by 810
“New optimization methods to train generative RNNs that modify stochastic gradient descent have appeared in the literature recently [38]–[40].”
Training Spiking Neural Networks Using Lessons From Deep Learning
Eshraghian, J. K., Ward, M., Neftci, E. O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., & Lu, W. D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3308088 · Cited by 782
“I N T R O D U C T I O N Deep learning has solved numerous problems in computer vision [1], [2], [3], [4], [5], [6], speech recognition [7], [8], [9], and natural language processing [10], [11], [12], [13], [14].”
BackgroundA Network-based End-to-End Trainable Task-oriented Dialogue System
Wen, T.-H., Vandyke, D., Mrkšić, N., Gasic, M., Rojas Barahona, L. M., Su, P.-H., Ultes, S., & Young, S. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1042 · Cited by 778
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801
“2, which adds a side-channel of information to the basic recurrent network of [38].”
BackgroundHybrid CTC/Attention Architecture for End-to-End Speech Recognition
Watanabe, S., Hori, T., Kim, S., Hershey, J. R., & Hayashi, T. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2763455 · Cited by 800
“…th-order Markov assumption) as an m - gram model, i.e., Although recurrent neural network language models (RNNLMs) can avoid this conditional independence assumption issue [23], it makes the decoding complex, and RNNLMs are often combined with m -gram language models based on a rescoring technique.”
MethodBetter Word Representations with Recursive Neural Networks for Morphology
Luong, M.-T., Socher, R., & Manning, C. D. (2013) · Conference on Computational Natural Language Learning · Cited by 1,020
Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
Zhou, H., Huang, M., Zhang, T., Zhu, X., & Liu, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11325 · Cited by 763
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
“It may also yield more expressive or creative captions if it is used to overcome the limitations of grammars or templates (as shown by the example of Midge; Mitchell et al., 2012).”
BackgroundMethodA Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.-Y., Gao, J., & Dolan, B. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1020 · Cited by 764
“We give a brief overview of the Recurrent Language Model (RLM) (Mikolov et al., 2010) architecture that our models extend.”
BackgroundMethodInfluentialA Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Soplin, N. E. Y., Yamamoto, R., Wang, X., et al. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 727
“In the decoding stage, the decoder predicts the next token given the speech feature X and the previous predicted tokens using beam search, which combines the scores of S2S, CTC and the RNN language model (LM) [22] as follows:”
Method(2017) · International Conference on Machine Learning · Cited by 758
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Dupoux, E., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 737
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Serban, I., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A., & Bengio, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10983 · Cited by 697
“Under this assumption, the RNN Language Model (RNNLM) [18], the simplest possible generative RNN for discrete sequences, parametrizes the output distribution using the softmax function applied to an affine transformation of the hidden state hm.”
BackgroundMethodInfluentialA tutorial survey of architectures, algorithms, and applications for deep learning
Deng, L. (2014) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2013.9 · Cited by 725
“More recent work on neural network LM with deep architectures can be found in [51, 200–203].”
BackgroundResultExtracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
“While the unsupervised objectives used to train these models vary, one popular choice is a “next-step prediction” objective [4, 27, 39, 48].”
BackgroundEvaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719
“The first is nexttoken prediction [46], which models p(xi | x1, .”
BackgroundAdversarial Training Methods for Semi-Supervised Text Classification
Miyato, T., Dai, A. M., & Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07725 · Cited by 179
“Following Dai and Le [4], we initialized the word embedding m atrix and LSTM weights with a pre-trained recurrent language model [3, 19] that was train ed on both labeled and unlabeled examples.”
MethodConvolutional Neural Networks over Tree Structures for Programming Language Processing
Mou, L., Li, G., Zhang, L., Wang, T., & Jin, Z. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10139 · Cited by 675
“ith data in k-dimensional space; but it fails to capture tree-structural information as in programs. Another example is the recurrent neural network, which can be regarded as a time-decaying network (Mikolov et al. 2010). Hence, it is typically suitable for one-dimensional data, but structural information is also lost…”
BackgroundA Novel Neural Source Code Representation Based on Abstract Syntax Tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., & Liu, X. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00086 · Cited by 655
“We use Recurrent Neural Network (RNN) [31] to encode statements and the sequential dependency between the statements into a vector.”
MethodXie, J., Yu, F. R., Huang, T., Xie, R., Liu, J., Wang, C., & Liu, Y. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2866942 · Cited by 700
“However, recurrent NN [99], [100] is a stateful network, which can use internal state (memory) to handle sequential data.”
BackgroundGated Self-Matching Networks for Reading Comprehension and Question Answering
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1018 · Cited by 666
“First, the question and passage are processed by a bi-directional recurrent network (Mikolov et al., 2010) separately.”
BackgroundModeling Relation Paths for Representation Learning of Knowledge Bases
Lin, Y., Liu, Z., Luan, H., Sun, M., Rao, S., & Liu, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1082 · Cited by 657
“RNN is a recent neural-based model for semantic composition (Mikolov et al., 2010).”
Background
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.