Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 401–500 of 1,058
Predictive Uncertainty Estimation via Prior Networks
Malinin, A., & Gales, M. (2018) · Apollo (University of Cambridge) · doi:10.17863/cam.35237 · Cited by 163
Generative AI-Driven Semantic Communication Networks: Architecture, Technologies, and Applications
Liang, C., Du, H., Sun, Y., Niyato, D. T., Kang, J., Zhao, D., & Imran, M. A. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3435524 · Cited by 245
“Particularly, Seq2Seq models are designed to handle variable-length input and output sequences with the architectures such as long short-term memory (LSTM) [36], recurrent neural networks (RNN) [37] and gated recurrent unit (GRU) [38].”
Learning a Recurrent Visual Representation for Image Caption Generation
Chen, X., & Zitnick, C. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5654 · Cited by 160
“For instance RNN language models often find difficultly in learning long distance relations [3, 24] without specialized gating units [12].”
BackgroundMethodInfluentialBoosting coverage-based fault localization via graph-based representation learning
Lou, Y., Zhu, Q., Dong, J., Li, X., Sun, Z., Hao, D., Zhang, L., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468580 · Cited by 163
“entities based on suspiciousness scores of existing SBFL, code complexity, and code history; similarly, the latest learning-to-combine technique DeepFL [41], utilizes neural networks [49, 55] to combine suspiciousness scores of spectrum-based FL and mutation-based FL [50, 52, 78], code complexity, and text similarity.”
BackgroundMultichannel Variable-Size Convolution for Sentence Classification
Yin, W., & Schütze, H. (2015) · Proceedings of the Nineteenth Conference on Computational Natural Language Learning · doi:10.18653/v1/k15-1021 · Cited by 162
“For example, Chen et al. (2013) compared HLBL (Mnih and Hinton, 2009), SENNA (Collobert and Weston, 2008), Turian (Turian et al., 2010) and Huang (Huang et al., 2012), showing great variance in quality and characteristics of the semantics captured by the tested embedding versions.”
BackgroundAffect-LM: A Neural Language Model for Customizable Affective Text Generation
Ghosh, S., Chollet, M., Laksana, E., Morency, L.-P., & Scherer, S. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1059 · Cited by 157
Generating Steganographic Text with LSTMs
Fang, T., Jaggi, M., & Argyraki, K. (2017) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p17-3017 · Cited by 216
Luo, T.-J., Zhou, C.-L., & Chao, F. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2365-1 · Cited by 161
“To solve the two major challenges, this paper introduces a deep recurrent neural network (RNN) architecture for the classification based on FB-CSP algorithm [28, 29].”
BackgroundMethodArtificial Neural Network Algorithms for 3D Printing
Mahmood, M. A., Visan, A. I., Ristoscu, C., & Mihailescu, I. N. (2020) · Materials · doi:10.3390/ma14010163 · Cited by 161
“Therefore, it is used in long short-term memory that can regenerate the simulation results accurately [44].”
BackgroundA Neural Probabilistic Model for Context Based Citation Recommendation
Huang, W., Wu, Z., Liang, C., Mitra, P., & Giles, C. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9528 · Cited by 157
“…such as parsing (Collobert and Weston 2008; Socher et al. 2013a), sentiment analysis (Socher et al. 2013b), language modeling (Bengio et al. 2003; Mnih and Hinton 2007; Mikolov et al. 2010; Mnih and Teh 2012; Huang et al. 2012a; Mikolov et al. 2013), and machine translation (Gao et al. 2014).”
BackgroundTrain Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161
Language Models with Transformers
Wang, C., Li, M., & Smola, A. J. (2019) · arXiv · doi:10.48550/arxiv.1904.09408 · Cited by 161
“Recurrent neural networks (RNNs) (Mikolov et al., 2010) memorize the sequential context in carefully designed cells.”
BackgroundMethodInfluentialA Neural Click Model for Web Search
Borisov, A., Markov, I., de Rijke, M., & Serdyukov, P. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2883033 · Cited by 187
Sengupta, A., & Roy, K. (2017) · Applied Physics Reviews · doi:10.1063/1.5012763 · Cited by 158
An end-to-end model for rice yield prediction using deep learning fusion
Chu, Z., & Yu, J. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105471 · Cited by 158
Deep Learning on Mobile and Embedded Devices: State-of-the-art, Challenges, and Future Directions
Chen, Y., Zheng, B., Zhang, Z., Wang, Q., Shen, C., & Zhang, Q. (2021) · ACM Computing Surveys · doi:10.1145/3398209 · Cited by 159
“Recent years have witnessed the rapid development of deep learning with successful applications in a wide range of areas, including image classification [91, 169], speech recognition [33], object detection [149], semantic segmentation [113], and natural language processing (NLP) [125].”
BackgroundA Neural Network Approach to Jointly Modeling Social Networks and Mobile Trajectories
Yang, C. H., Sun, M., Zhao, W. X., Liu, Z., & Chang, E. Y. (2017) · ACM Transactions on Information Systems · doi:10.1145/3041658 · Cited by 171
“RNN and its variant GRU, which aim at sequential modeling, have been successfully applied in sentence modeling [Mikolov et al. 2010], speech signal modeling [Chung et al. 2014], and sequential click prediction [Zhang et al. 2014b].”
MethodNeural Word Segmentation Learning for Chinese
Cai, D., & Zhao, H. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1039 · Cited by 161
Learning to Understand Phrases by Embedding the Dictionary
Hill, F., Cho, K., Korhonen, A., & Bengio, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00080 · Cited by 171
“RNNs (with LSTMs) have achieved state-of-the-art performance in language modelling (Mikolov et al., 2010) image caption generation (Kiros et al., 2014), approach state-of-the-art performance in machine translation (Bahdanau et al., 2015).”
BackgroundGenerative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157
“Inspired by the neural language modeling objective for text [11], which models the likelihood of a sequence of tokens to appear as a legit language, APC is trained to understand what a reasonable spectrogram should look like and encode such information in the representations.”
MethodDeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detection
Cao, B., Zheng, L., Zhang, C., Yu, P. S., Piscitello, A., Zulueta, J., Ajilore, O., Ryan, K., & Leow, A. D. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098086 · Cited by 145
“In the first stage, each 2This is for privacy concerns, because malicious person may be able to unscramble and recover the texts using such information. view of the time series is separately modeled by a Recurrent Neural Network (RNN) [34, 44].”
MethodInfluentialRecurrent neural networks for classifying relations in clinical notes
Luo, Y. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.006 · Cited by 152
“In general domain NLP, a growing number of studies have successfully used recurrent neural networks (RNNs) combined with word embedding [22] on tasks including language modeling [23], text classification [24–27], question answering [25,26,28,29], machine translation [25,30–32], named entity recognition [33–36], and re…”
A Review on IoT Deep Learning UAV Systems for Autonomous Obstacle Detection and Collision Avoidance
Fraga-Lamas, P., Ramos, L., Mondéjar-Guerra, V., & Fernández-Caramés, T. M. (2019) · Remote Sensing · doi:10.3390/rs11182144 · Cited by 153
“For such a purpose, the use of Recurrent Neural Network (RNN) architectures [66] provides the ability to learn temporal dependencies from the information of an arbitrarily long sequence of observations.”
MethodPersonalized Graph Neural Networks With Attention Mechanism for Session-Aware Recommendation
Zhang, M., Wu, S., Gao, M., Jiang, X., Xu, K., & Wang, L. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3031329 · Cited by 176
“Recently, deep neural networks have become the most successful methods in modeling sequence, such as machine translation[20], [21], [22], and conversation machine[23].”
MethodMICU: Image super-resolution via multi-level information compensation and U-net
Chen, Y., Xia, R., Yang, K., & Zou, K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123111 · Cited by 164
A Model of Coherence Based on Distributed Sentence Representation
Li, J., & Hovy, E. H. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1218 · Cited by 178
Hierarchical Recurrent Neural Network for Document Modeling
Lin, R., Liu, S., Yang, M., Li, M., Zhou, M., & Li, S. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1106 · Cited by 176
“As shown in Figure 1, similar to the conventional recurrent neural network, for the sentence j, our network has two input layers xsj and hsj−1. xsj is the current sentence representation, and hsj−1 is the history information vector before sentence j.”
BackgroundMethodInfluentialData-driven Optical Fiber Channel Modeling: A Deep Learning Approach
Wang, D., Song, Y., Li, J., Qin, J., Yang, T., Zhang, M., Chen, X., & Boucouvalas, A. C. (2020) · Journal of Lightwave Technology · doi:10.1109/jlt.2020.2993271 · Cited by 168
“Among the existing DL algorithms, recurrent neural networks (RNNs) contain cyclic connections and have achieved great success in sequential data-related problems, such as caption generation, speech recognition, and machine translation [21].”
BackgroundCyclical Annealing Schedule: A Simple Approach to Mitigating KL VanishingPreprint
Fu, H., Li, C., Liu, X., Gao, J., Çelikyılmaz, A., & Carin, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.10145 · Cited by 261
“To generate a text sequence of length T , x = [ x 1 , · · · , x T ] , neural language models (Mikolov et al., 2010) generate every token x t conditioned on the previously generated tokens: where x t indicates all tokens before t .”
BackgroundWen, T.-H., Gasic, M., Kim, D., Mrksic, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4639 · Cited by 151
A Language-Independent Neural Network for Event Detection
Feng, X., Huang, L., Tang, D., Ji, H., Qin, B., & Liu, T. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2011 · Cited by 156
See-Through-Text Grouping for Referring Image Segmentation
Chen, D.-J., Jia, S., Lo, Y.-C., Chen, H.-T., & Liu, T.-L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00755 · Cited by 181
“The field of natural language processing (NLP) has developed several useful language models for extracting language features [2, 30, 31, 32].”
BackgroundAutomatic Human Sleep Stage Scoring Using Deep Neural Networks
Malafeev, A., Laptev, D., Bauer, S., Omlin, X., Wierzbicka, A., Wichniak, A., Jernajczyk, W., Riener, R., Buhmann, J., & Achermann, P. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00781 · Cited by 151
“In the last few years, recurrent neural networks (RNNs) have demonstrated better performance than “classical” machine learning methods on datasets with a temporal structure (Mikolov et al., 2010; Graves et al., 2013; Karpathy and Fei-Fei, 2015).”
BackgroundA Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version
Serban, I. V., Lowe, R., Henderson, P., Charlin, L., & Pineau, J. (2018) · Dialogue & Discourse · doi:10.5087/dad.2018.101 · Cited by 150
“Another metric proposed to evaluate probabilistic language models (Bengio et al., 2003; Mikolov et al., 2010) that has seen significant recent use for evaluating end-to-end dialogue systems is word perplexity (Pietquin and Hastie, 2013; Serban et al.”
BackgroundSyntaLinker: automatic fragment linking with deep conditional transformer neural networks
Yang, Y., Zheng, S., Su, S., Zhao, C., Xu, J., & Chen, H. (2020) · Chemical Science · doi:10.1039/d0sc03126g · Cited by 146
Coevolutionary Recommendation Model: Mutual Learning between Ratings and Reviews
Lu, Y., Dong, R., & Smyth, B. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186158 · Cited by 187
“Mostwidely used neural network structures, including convolutional neural networks [17, 20], recurrent neural networks [27, 37], and neural turing machine [44], have shown promising results in various natural language processing benchmarks.”
BackgroundLong short-term memory recurrent neural network architectures for Urdu acoustic modeling
Zia, T., & Zahid, U. (2018) · International Journal of Speech Technology · doi:10.1007/s10772-018-09573-7 · Cited by 162
“This approach is primarily inspired from the remarkable successes of DNNs over swallow networks particularly for object recognition, speech recognition and natural language modeling (Graves et al. 2013a; Krizhevsky et al. 2012; Mikolov et al. 2010).”
BackgroundFew Sample Knowledge Distillation for Efficient Network Compression
Li, T., Li, J., Liu, Z., & Zhang, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01465 · Cited by 149
Urban ride-hailing demand prediction with multiple spatio-temporal information fusion network
Jin, G., Cui, Y., Zeng, L., Tang, H., Feng, Y., & Huang, J. (2020) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2020.102665 · Cited by 159
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Mao, J., Wei, X., Yang, Y., Wang, J., Huang, Z., & Yuille, A. L. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.291 · Cited by 148
“For natural language, the Recurrent Neural Network (RNN [13, 36]) and the Long-Short Term Memory (LSTM [20]) achieve the state-of-the-art performance in many NLP tasks such as machine translation [21, 8, 47] and speech recognition [37].”
MethodDeep Learning Based Supervised Image Classification Using UAV Images for Forest Areas Classification
Haq, M. A., Rahaman, G., Baral, P., & Ghosh, A. (2020) · Journal of the Indian Society of Remote Sensing · doi:10.1007/s12524-020-01231-3 · Cited by 174
“Different forms of deep learning methods such as convolutional neural networks (Farabet et al. 2013), recurrent neural networks (Mikolov et al. 2010), deep belief networks (Hinton et al. 2006) and Stacked Auto-encoders (Masci et al. 2011) exist and these neural network architectures have been…”
BackgroundAn evidential classifier based on Dempster-Shafer theory and deep learning
Zheng, T., Xu, P., & Denœux, T. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.066 · Cited by 182
“In recent years, with the explosive development of deep learning [29], several models have been developed for precise classification, such as convolutional neural networks (CNNs) [25, 31, 65], recurrent neural networks [33, 39, 40], graph neural networks [53, 54], and deep autoencoders [63, 64].”
BackgroundLi, Y., Sun, K., Yao, Q., & Wang, L. (2023) · Energy · doi:10.1016/j.energy.2023.129604 · Cited by 159
Efficient softmax approximation for GPUs
Grave, É., Joulin, A., Cissé, M., Grangier, D., & Jeǵou, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04309 · Cited by 132
“Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units.”
BackgroundMethodInfluentialDeepCache: A Deep Learning Based Framework For Content Caching
Narayanan, A., Verma, S. M., Ramadan, E., Babaie, P., & Zhang, Z.-L. (2018) · ACM SIGCOMM Workshop on Network Meets AI & ML (NetAI) · doi:10.1145/3229543.3229555 · Cited by 156
“RNNs have shown their unchallenged dominance in the area of natural language processing [15], machine language translation [2], speech recognition [7], and image captioning [8].”
BackgroundA Recurrent Encoder-Decoder Network for Sequential Face Alignment
Peng, X., Feris, R. S., Wang, X., & Metaxas, D. N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_3 · Cited by 146
“1 https://sites.google.com/site/xipengcshomepage/ project/face-alignment Recurrent neural networks (RNNs) are widely employed in the literature of speech recognition [28] and natural language processing [27].”
BackgroundCotfas, L., Delcea, C., Roxin, I., Ioanăş, C., Gherai, D. S., & Tajariol, F. (2021) · IEEE Access · doi:10.1109/access.2021.3059821 · Cited by 167
“The Long Short-Term Memory (LSTM) [102] is a type of Recurrent Neural Network (RNN).”
MethodDeep Learning for Automated Extraction of Primary Sites From Cancer Pathology Reports
Qiu, J. X., Yoon, H.-J., Fearn, P. A., & Tourassi, G. D. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2700722 · Cited by 146
Dressing as a Whole: Outfit Compatibility Learning Based on Node-wise Graph Neural Networks
Cui, Z., Li, Z., Wu, S., Zhang, X.-Y., & Wang, L. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313444 · Cited by 164
“Li et al. extracted multi-modal information from items and evaluated the outfit compatibility with an Recurrent Neural Network (RNN) [17].”
BackgroundMedical Semantic Similarity with a Neural Language Model
De Vine, L., Zuccon, G., Koopman, B., Sitbon, L., & Bruza, P. (2014) · Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management · doi:10.1145/2661829.2661974 · Cited by 145
“This model is employed for the task of measuring semantic similarity between medical concepts, a task that is central to a number of techniques in medical informatics and information retrieval.”
MethodBuilding Program Vector Representations for Deep Learning
Peng, H. M., Mou, L., Li, G., Liu, Y., Zhang, L., & Jin, Z. G. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-25159-2_49 · Cited by 160
“Recurrent neural network (RNN) is introduced in order to capture long-term dependencies [40].”
BackgroundRevisiting Recurrent Neural Networks for robust ASR
Vinyals, O., Ravuri, S. V., & Povey, D. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288816 · Cited by 143
“More recently, applications in Language Modeling [13] and advances in optimization [14] have seen state of the art performance by the usage of RNNs for sequence modeling.”
BackgroundBayesian Recurrent Neural Networks
Fortunato, M., Blundell, C., & Vinyals, O. (2017) · arXiv · doi:10.48550/arxiv.1704.02798 · Cited by 145
“Another related technique is dynamic evaluation (Mikolov et al., 2010), which trains an RNN during evaluation of the model with a fixed learning rate.”
BackgroundMethodInfluentialComparison of feedforward and recurrent neural network language models
Sundermeyer, M., Oparin, I., Gauvain, J.-L., Freiberg, B., Schlüter, R., & Ney, H. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639310 · Cited by 167
“Index Terms— Automatic speech recognition, feedforward neural networks, recurrent neural networks”
BackgroundLearning to Parse and Translate Improves Neural Machine Translation
Eriguchi, A., Tsuruoka, Y., & Cho, K. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2012 · Cited by 144
“Unlike a usual recurrent language model (see, e.g., Mikolov et al., 2010), an RNNG simultaneously models both tokens and their tree-based composition.”
A Capsule Network for Recommendation and Explaining What You Like and Dislike
Li, C., Quan, C., Peng, L., Qi, Y., Deng, Y., & Wu, L. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331216 · Cited by 156
“posed to model contextual information from reviews for better recommendation with deep learning techniques [3 ,4 24 27 39]. Convolutional neural networks (CNN) [16] and Recurrent Neural Network (RNN) [9, 21] are mainly used to composite the semantic contextual information into the continuous real-valued vector represe…”
MethodProtein sequence design with deep generative models
Wu, Z., Johnston, K. E., Arnold, F. H., & Yang, K. K. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.004 · Cited by 135
Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
Cai, H., Lin, J., Lin, Y., Liu, Z., Tang, H., Wang, H., Zhu, L., & Han, S. (2022) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3486618 · Cited by 195
“Tradition NLP models are based on Recurrently Neural Network (RNN) and Convolutional Neural Network (CNN) [126, 148, 191, 210, 271, 350].”
MethodLong and Short-Term Recommendations with Recurrent Neural Networks
Devooght, R., & Bersini, H. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3079628.3079670 · Cited by 142
“Recurrent neural networks (RNN) are commonly used for language modeling, where they are trained to learn sequences of words [13].”
BackgroundGeng, T., Wu, C., Zhang, Y., Tan, C., Xie, C., You, H., Herbordt, M., Lin, Y., & Li, A. (2021) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3466752.3480113 · Cited by 171
“Conventional deep learning paradigms such as Convolution Neural Networks (CNNs) [26] and Recurrent Neural Networks (RNNs) [33] have been demonstrated to be quite efficient, but primarily for applications using Euclidean data [15, 16, 42, 44].”
BackgroundBlockQNN: Efficient Block-Wise Neural Network Architecture Generation
Zhong, Z., Yang, Z., Deng, B., Yan, J., Wu, W., Shao, J., & Liu, C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2969193 · Cited by 163
“in language modeling [57] and video analytics [58], we choose to explore recurrent networks in our problem.”
BackgroundHierarchical Neural Network Generative Models for Movie Dialogues.Preprint
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2015) · arXiv (Cornell University) · Cited by 143
Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model
Yang, R., Mentzer, F., Gool, L. V., & Timofte, R. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3043590 · Cited by 176
“, language modeling [21], [22] and video analysis [23], some recurrent auto-encoders were proposed for representation learning on time-series tasks, such as machine translation [24], [25] and captioning [26], etc.”
BackgroundOptimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
Li, C., Gao, X., Li, Y., Peng, B., Li, X., Zhang, Y., & Gao, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.378 · Cited by 138
Transformers with convolutional context for ASR
Mohamed, A., Okhonko, D., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11660 · Cited by 137
“Speech Recognition systems have experienced many advances over the past decade, with neural acoustic and language models leading to impressive new levels of performance across many challenging tasks [1, 2].”
BackgroundCombining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 138
“The embeddings are learned using the recurrent neural network language model (RNNLM) (Mikolov et al. 2010; 2013).”
MethodRazavi‐Far, R., Hallaji, E., Farajzadeh-Zanjani, M., Saif, M., Kia, S. H., Henao, H., & Capolino, G. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2873546 · Cited by 143
“b) Long Short-Term Memory Network: LSTM is an advanced type of RNN [35], which benefits from an intrinsic memory (or cell), in which predictions are memorized over time to perform a more accurate decision making w.”
BackgroundInfluentialUnsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation
Zhao, T., Lee, K., & Eskenazi, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1101 · Cited by 139
“We used the version pre-processed by Mikolov (Mikolov et al., 2010).”
Missing value imputation in multivariate time series with end-to-end generative adversarial networks
Zhang, Y., Zhou, B., Cai, X., Guo, W., Ding, X., & Yuan, X. (2020) · Information Sciences · doi:10.1016/j.ins.2020.11.035 · Cited by 159
Generative Low-Bitwidth Data Free Quantization
Xu, S., Li, H., Zhuang, B., Liu, J., Cao, J., Liang, C., & Tan, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_1 · Cited by 186
Ensemble Distribution Distillation
Gales, M., Malinin, A., & Mlodozeniec, B. (2019) · arXiv (Cornell University) · doi:10.17863/cam.49348 · Cited by 114
Cross-Lingual Word Embeddings for Low-Resource Language Modeling
Adams, O., Makarucha, A. J., Neubig, G., Bird, S., & Cohn, T. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1088 · Cited by 163
Use of Deep Learning for Characterization of Microfluidic Soft Sensors
Han, S., Kim, T., Kim, D., Park, Y.-L., & Jo, S. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2792684 · Cited by 136
“We introduced a hierarchical recurrent sensing network, a type of recurrent neural network (RNN) model, in the calibration stage for estimating the magnitude and the location of a contact pressure simultaneously, since RNNs have demonstrated remarkable advances for modeling temporal data over the past decade [16], [17…”
MethodStock Prediction Using Convolutional Neural Network
Chen, S., & He, H. (2018) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/435/1/012026 · Cited by 167
“Series: Materials Science and 012026 doi:10.1088/1757-899X/435/1/012026 and Recurrent neural networks [14-17].”
BackgroundInfluentialNeural information retrieval: at the end of the early years
Onal, K. D., Zhang, Y., Altingovde, I. S., Rahman, M. M., Karagoz, P., Braylan, A., Dang, B., Chang, H.-L., Kim, H., McNamara, Q., et al. (2018) · Information Retrieval Journal · doi:10.1007/s10791-017-9321-y · Cited by 136
“This work provides an analysis of CLSM vectors for queries similar to the word embedding space analysis in Mikolov et al. (2013c). Mitra (2015) found that offsets between CLSM query vectors can represent intent transition patterns.”
BackgroundMethodInfluentialInvestigating the use of recurrent motion modelling for speech gesture generation
Ferstl, Y., & McDonnell, R. (2018) · Proceedings of the 18th International Conference on Intelligent Virtual Agents · doi:10.1145/3267851.3267898 · Cited by 132
“, [27]), specifically tasks such as machine translation (e.”
BackgroundKong, Q., Xu, Y., Wang, W., & Plumbley, M. D. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3014737 · Cited by 171
“Recurrent neural networks (RNNs) [42] are kinds of neural networks that can store history information in their hidden states, and thus capture long term dependency of sequential data.”
BackgroundMethodPivot Based Language Modeling for Improved Neural Domain Adaptation
Ziser, Y., & Reichart, R. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1112 · Cited by 161
“We start with a brief description of LSTM based language modeling (LSTM-LM, (Mikolov et al., 2010)) and then describe how PBLM modifies that model in order to learn pivot-based representations that are aware of the structure of the input text.”
MethodA language-independent neural network for event detection
Feng, X., Qin, B., & Liu, T. (2018) · Science China Information Sciences · doi:10.1007/s11432-017-9359-x · Cited by 152
LSTM-CNN Hybrid Model for Text Classification
Zhang, J., Li, Y., Tian, J., & Li, T. (2018) · IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) · doi:10.1109/iaeac.2018.8577620 · Cited by 160
“Neural network models, such as Convolutional Neural Network(CNN)[8] and Recurrent Neural Network(RNN)[9] are used for text classification tasks, and the performance of neural network models are better than traditional machine learning methods.”
MethodCan We Use Split Learning on 1D CNN Models for Privacy Preserving Training?
Abuadbba, A., Kim, K., Kim, M., Thapa, C., Camtepe, S. A., Gao, Y., Kim, H., & Nepal, . (2020) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3320269.3384740 · Cited by 181
DeltaRNN: A Power-efficient Recurrent Neural Network Accelerator
Gao, C., Neil, D., Ceolini, E., Liu, S.-C., & Delbruck, T. (2018) · Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3174243.3174261 · Cited by 133
“Recurrent Neural Networks (RNNs) are fully-connected single- or multi-layered networks with complex neurons that have multiple memory states and enabled state-of-art accuracies in tasks involving temporal sequences [27] such as automatic speech recognition [2, 9] and natural language processing [23].”
BackgroundTree-RNN: Tree structural recurrent neural network for network traffic classification
Ren, X., Gu, H., & Wei, W. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114363 · Cited by 151
Inter-patient ECG classification with convolutional and recurrent neural networks
Guo, L., Sim, G., & Matuszewski, B. J. (2019) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2019.06.001 · Cited by 158
Natural Language Generation Using Deep Learning to Support MOOC Learners
Li, C., & Xing, W. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-020-00235-x · Cited by 133
“RNN has been widely applied to explore its capability of text generation and has shown great potential in NLG tasks (Fedus et al. 2018; Mikolov et al. 2010; Potash et al. 2015; Sordoni et al. 2015).”
BackgroundMethodEnhancing heart disease prediction using a self-attention-based transformer model
Rahman, A., Alsenani, Y. S., Zafar, A., Ullah, K., Rabie, K. M., & Shongwe, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-51184-7 · Cited by 172
“One of the first models 39 in the field of recurrent processing was introduced as LSTM, a sort of RNN with gated units.”
MethodBayesian Recurrent Neural Network for Language Modeling
Chien, J.-T., & Ku, Y.-C. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2499302 · Cited by 132
“Finally, the conclusions are drawn in Section VI.”
BackgroundMethodInfluentialTraining End-to-End Dialogue Systems with the Ubuntu Dialogue Corpus
Lowe, R., Pow, N., Serban, I. V., Charlin, L., Liu, C., & Pineau, J. (2017) · Dialogue & Discourse · doi:10.5087/dad.2017.102 · Cited by 127
Attention-based context-aware sequential recommendation model
Yuan, W., Wang, H., Yu, X., Liu, N., & Li, Z. (2019) · Information Sciences · doi:10.1016/j.ins.2019.09.007 · Cited by 142
A Review on Human-Computer Interaction and Intelligent Robots
Ren, F., & Bao, Y. (2020) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622019300052 · Cited by 164
Dropout Inference in Bayesian Neural Networks with Alpha-divergences
Li, Y., & Gal, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02914 · Cited by 101
“…to obtain state-ofthe-art results on many tasks (Krizhevsky et al., 2012; Szegedy et al., 2014; Sutskever et al., 2014; Sundermeyer et al., 2012; Mikolov et al., 2010; Kalchbrenner & Blunsom, 2013), and in many pipelines these models have replaced the more traditional Bayesian probabilistic…”
BackgroundLarge-Vocabulary Continuous Speech Recognition Systems: A Look at Some Recent Advances
Saon, G., & Chien, J. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2197156 · Cited by 151
GLA: Global–Local Attention for Image Description
Li, L., Tang, S., Zhang, Y., Deng, L., & Tian, Q. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2751140 · Cited by 140
Structured Output Layer neural network language model
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J., & Yvon, F. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947610 · Cited by 152
“This topic has recently gained much attention in the domain of speech recognition [3, 4].”
BackgroundA study of sentiment analysis using deep learning techniques on Thai Twitter data
Vateekul, P., & Koomsubha, T. (2016) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse.2016.7748849 · Cited by 165
“With these word vectors trained by word2vec, a group of words having similar meaning also have similar word vectors.”
MethodImage Captioning with Deep Bidirectional LSTMs and Multi-Task Learning
Wang, C., Yang, H., & Meinel, C. (2018) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3115432 · Cited by 130
“It starts by introducing Recurrent Neural Network (RNN) which equips neural networks with memories, followed by the review of recently proposed approaches on the image captioning task.”
MethodInfluentialNPENAS: Neural Predictor Guided Evolution for Neural Architecture Search
Wei, C., Niu, C., Tang, Y., Wang, Y., Hu, H., & Liang, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3151160 · Cited by 164
“NASBench-NLP contains 14k RNN-like neural architectures, trained and validated on the Penn Tree Bank (PTB) [54] dataset.”
MethodDeep Learning and Visualization for Identifying Malware Families
Sun, G., & Qian, Q. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2018.2884928 · Cited by 146
“Recurrent neural networks (RNN) [7] are the strongest neural networks and have been applied to language models [8], online handwriting recognition and generation [9], and speech recognition [10].”
BackgroundAn Attentional Recurrent Neural Network for Personalized Next Location Recommendation
Guo, Q., Sun, Z., Zhang, J., & Theng, Y. L. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5337 · Cited by 132
“Recent studies apply Neural Network (NN) in next location recommendation, thanks to its strong capability of capturing sequential information (Hochreiter and Schmidhuber 1997; Mikolov et al. 2010; Zhang et al. 2014; Huang et al. 2019).”
BackgroundA survey on text generation using generative adversarial networks
de Rosa, G. H., & Papa, J. P. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108098 · Cited by 129
“Additionally, previous sequences must be carried to the future during learning, as the language model predicts data based on past information [3].”
Background
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.