Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 501–600 of 1,058
Improved Word Representation Learning with Sememes
Niu, Y., Xie, R., Liu, Z., & Sun, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1187 · Cited by 131
“Distributed word representations are powerful and have been widely utilized in many NLP tasks, including neural language models (Bengio et al., 2003; Mikolov et al., 2010), machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), parsing (Chen and Manning, 2014) and text classification…”
Learning Private Neural Language Modeling with Attentive Aggregation
Ji, S., Pan, S., Long, G., Li, X., Jiang, J., & Huang, Z. H. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852464 · Cited by 225
“used a simple recurrent neural network-based language model for speech recognition [15].”
MethodXie, Z., Sun, Z., Jin, L., Ni, H., & Lyons, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2732978 · Cited by 159
Generative Adversarial Networks for Spatio-temporal Data: A Survey
Gao, N., Xue, H., Shao, W., Zhao, S., Qin, K. K., Prabowo, A., Rahaman, M. S., & Salim, F. D. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3474838 · Cited by 127
“e usually not independently generated as in traditional data. With the prevalence of deep learning, many neural networks (e.g., Convolutional Neural Network (CNN) [74], Recurrent Neural Network (RNN) [100], Autoencoder (AE) [57], Graph Convolutional Authors’ address: Nan Gao, nan.gao@rmit.edu.au; Hao Xue, hao.xue@rmit…”
BackgroundInfluentialLearning Memory Access PatternsPreprint
Hashemi, M., Swersky, K., Smith, J., Ayers, G., Litz, H., Chang, J., Kozyrakis, C., & Ranganathan, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02329 · Cited by 240
“Notably, speech recognition (Hinton et al., 2012) and natural language processing (Mikolov et al., 2010).”
BackgroundMethodA topic-enhanced word embedding for Twitter sentiment classification
Ren, Y., Wang, R., & Ji, D. (2016) · Information Sciences · doi:10.1016/j.ins.2016.06.040 · Cited by 138
“With the revival of deep learning [5, 15, 25], incorporating the continuous representation of a word as feature has been used for parsing [46], language modeling [4, 30, 32] and named entity recognition [54, 59], representing each word as a matrix and composing words using iterated matrix multiplication.”
MethodHierarchical Temporal Convolutional Networks for Dynamic Recommender Systems
You, J., Wang, Y., Pal, A., Eksombatchai, P., Rosenburg, C., & Leskovec, J. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313747 · Cited by 136
“t number of interactions, is in fact a very strong baseline [12, 23]. Recent years have witnessed the power of sequence-based deep learning models in computer vision [22], natural language processing [19], and graph structured data [37]. In addition, recent works have applied sequence models for dynamic recommender sy…”
MethodData Analytics in Industry 4.0: A Survey
Duan, L., & Xu, L. D. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10190-0 · Cited by 183
“It has different architectures, including graph neural network (Scarselli et al., 2008), recurrent neural network (Mikolov et al., 2010), and convolutional neural network (Lawrence et al., 1997), for applications in computer vision, speech recognition, and natural language processing with…”
Zhong, H., Yu, S. S., Trinh, H. M., Lv, Y., Yuan, R. B., & Wang, Y. (2023) · Measurement · doi:10.1016/j.measurement.2022.112421 · Cited by 142
An Analysis of Neural Language Modeling at Multiple Scales
Merity, S., Keskar, N. S., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08240 · Cited by 123
“In Mikolov et al. (2010), the Penn Treebank dataset (Marcus et al., 1994) was processed to generate a character-level language modeling dataset.”
MethodDistributed Compressive Sensing: A Deep Learning Approach
Palangi, H., Ward, R. K., & Deng, L. (2016) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2557301 · Cited by 137
“It has been used extensively in time sequence modelling [30], [31], [32], [33], [34], [35], [36], [37], [38].”
MethodLong short-term memory RNN for biomedical named entity recognition
Lyu, C., Chen, B., Ren, Y., & Ji, D. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1868-5 · Cited by 129
“Recurrent neural network (RNN) [16] and its variants long-short term memory (LSTM) [17] have been successfully used in various sequence prediction problems, such as general domain NER [18, 19], language modeling [20, 21] and speech recognition [22].”
cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information
Cao, S., Lu, W., Zhou, J. W., & Li, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12029 · Cited by 138
“Existing approaches only concentrated on learning such representations based on contextual information (Mikolov et al. 2010; 2013b; Ling et al. 2015; Levy, Goldberg, and Ramat-Gan 2014; Pennington, Socher, and Manning 2014) where words are regarded as atomic tokens.”
BackgroundIndustrial Big Data Analytics for Prediction of Remaining Useful Life Based on Deep Learning
Yan, H., Wan, J., Zhang, C., Tang, S., Hua, Q., & Wang, Z. (2018) · IEEE Access · doi:10.1109/access.2018.2809681 · Cited by 123
“One of the most obvious advantages of deep learning is its ability to extract the features automatically such as convolutional neural network (CNN) and recurrent neural network (RNN) [36].”
BackgroundA Neural Architecture for Automated ICD Coding
Xie, P., & Xing, E. P. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1098 · Cited by 159
“Due to the capacity of capturing long-range semantics in texts, LSTM is widely used for language modeling and sequence encoding 19 , 20 .”
BackgroundA Framework for Fall Detection Based on OpenPose Skeleton and LSTM/GRU Models
Lin, C.-B., Dong, Z., Kuan, W.-K., & Huang, Y. (2020) · Applied Sciences · doi:10.3390/app11010329 · Cited by 154
“The two are defined as [29] ht = σh(Vht−1 + Uhxt + bh) (9)”
BackgroundInfluentialA Deep Neural Network Model for Speaker Identification
Ye, F., & Yang, J. J. (2021) · Applied Sciences · doi:10.3390/app11083603 · Cited by 144
“In language modeling, a traditional RNN model has obtained a significant reduction in perplexity over standard n-gram models [25].”
MethodWang, Q., Wang, X., Meng, Y., Zhou, Y. Z., & Wang, H. (2023) · Sustainable Cities and Society · doi:10.1016/j.scs.2023.104432 · Cited by 136
Discourse Complements Lexical Semantics for Non-factoid Answer Reranking
Jansen, P. A., Surdeanu, M., & Clark, P. E. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1092 · Cited by 126
“In particular, we use the recurrent neural network language model (RNNLM) of Mikolov et al. (2013; 2010).”
MethodDeviceMien: network device behavior modeling for identifying unknown IoT devices
Ortiz, J., Crawford, C., & Le, F. (2019) · IEEE/ACM International Conference on Internet-of-Things Design and Implementation (IoTDI) · doi:10.1145/3302505.3310073 · Cited by 153
“RNNs have been used to successfullymodel sequences, such as handwriting recognition [10], language modeling [22] and acoustic modeling [12].”
MethodPredicting citation counts based on deep neural network learning techniques
Abrishami, A., & Aliakbary, S. (2019) · RePEc: Research Papers in Economics · Cited by 124
Evaluating Word Representation Features in Biomedical Named Entity Recognition Tasks
Tang, B., Cao, H., Wang, X., Chen, Q., & Xu, H. (2014) · BioMed Research International · doi:10.1155/2014/240403 · Cited by 122
“According to a review by Turian et al. [20], WR features can be divided into three categories: (1) clustering-based methods such as Brown clustering [27]; (2) distributional representations, such as LSA [23], LDA [24], and random indexing [25]; and (3) word embeddings (also called distributed…”
MethodRetweet Prediction with Attention-based Deep Neural Network
Zhang, Q., Gong, Y., Wu, J., Huang, H., & Huang, X. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983809 · Cited by 135
“[45] integrates three types of attention: the bottom-up attention, the object-level top-down attention, and the part-level top-down attention to apply visual attention to fine-grained classification task based on deep neural network.”
BackgroundLanguage Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data
Pratapa, A., Bhat, G., Choudhury, M., Sitaram, S., Dandapat, S., & Bali, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1143 · Cited by 130
“In Adel et al.(2014b; 2014a; 2013) RNNLMs are combined with n-gram based models, or converted to backoff models, giving improvements in perplexity and mixed error rate.”
MethodOn Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing
Ahmad, W. U., Zhang, Z., Ma, X., Hovy, E. H., Chang, K.-W., & Peng, N. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1253 · Cited by 158
“RNN is a prevalent family of models for many NLP tasks and has demonstrated compelling performances (Mikolov et al., 2010; Sutskever et al., 2014; Peters et al., 2018).”
BackgroundHe, K., Chen, X., Wu, Q., Yu, S., & Zhou, Z. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.3020582 · Cited by 136
Application of Deep Learning on Millimeter-Wave Radar Signals: A Review
Abdu, F. J., Zhang, Y., Fu, M., Li, Y., & Deng, Z. (2021) · Sensors · doi:10.3390/s21061951 · Cited by 151
“The output of RNNs constitutes the present inputs and the earlier outputs embedded in their hidden state h [93].”
BackgroundLSTM, WaveNet, and 2D CNN for nonlinear time history prediction of seismic responses
Ning, C., Xie, Y., & Sun, L. (2023) · Engineering Structures · doi:10.1016/j.engstruct.2023.116083 · Cited by 128
Learning User and Product Distributed Representations Using a Sequence Model for Sentiment Analysis
Chen, T., Xu, R., He, Y., Xia, Y., & Wang, X. (2016) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2016.2572539 · Cited by 133
“Models for learning distributed representations of knowledge have been proposed at different granularity levels, including word sense level [37], [38], [39], word level [15], [40], [41], [42], phrase level [22], [43], [44], sentence level [17], [19], [45], discourse level [46] and document level [16].”
BackgroundEvolutionary neural AutoML for deep learning
Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K., & Miikkulainen, R. P. (2019) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3321707.3321721 · Cited by 157
“d using recurrent (LSTM) layers as the basic building block. Since comments are essentially an ordered list of words, recurrent layers (having been shown to be effective at processing sequential data [36]) were a natural choice. In order for the words to be given as input into a recurrent network, they must be convert…”
MethodSoftware Vulnerability Discovery via Learning Multi-Domain Knowledge Bases
Lin, G., Zhang, J., Luo, W., Pan, L., Vel, O. D., Montague, P. S., & Xiang, Y. (2019) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2019.2954088 · Cited by 151
Kim, T. S., & Sohn, S. Y. (2020) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-020-01630-w · Cited by 131
“Because RNN has a vanishing or increasing gradient tendency during back propagation, it has difficulty of capturing long-term dependencies (Mikolov et al. 2010).”
BackgroundBilingually-constrained Phrase Embeddings for Machine Translation
Zhang, J., Liu, S., Li, M., Zhou, M., & Zong, C. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1011 · Cited by 136
“…has seen a strong interest in adapting and applying DNN to many tasks, such as word alignment (Yang et al., 2013), translation confi-dence estimation (Mikolov et al., 2010; Liu et al., 2013; Zou et al., 2013), phrase reordering prediction (Li et al., 2013), translation modelling (Auli et al., 2013;…”
MethodIntegrating and Evaluating Neural Word Embeddings in Information Retrieval
Zuccon, G., Koopman, B., Bruza, P., & Azzopardi, L. A. (2015) · Australasian Document Computing Symposium · doi:10.1145/2838931.2838936 · Cited by 121
“A NEURAL TRANSLATION LANGUAGE MODEL A variety of neural network-based language models(2) have emerged as effective approaches for generating representations of words [4, 26, 22].”
BackgroundLexicon-Free Conversational Speech Recognition with Neural Networks
Maas, A. L., Xie, Z., Jurafsky, D., & Ng, A. Y. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1038 · Cited by 145
Neural Discourse Structure for Text Categorization
Ji, Y., & Smith, N. A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1092 · Cited by 157
“For all datasets, we use the same preprocessing steps, mostly following recent work on language modeling (e.g., Mikolov et al., 2010).”
MethodSharma, A., Liu, X., & Yang, X. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.05.019 · Cited by 128
“Recently, RNNs have been demonstrated to achieve significant results on sequential data and have been applied in different fields like natural language processing [30], [34], [20], computer vision [32], [16], [39], multi-modal [22], [11], [15] and robotics [28].”
BackgroundHourly and Daily Urban Water Demand Predictions Using a Long Short-Term Memory Based Model
Li, M., Zheng, F., Tao, R., Zhang, Q., & Kapelan, Z. S. (2020) · Journal of Water Resources Planning and Management · doi:10.1061/(asce)wr.1943-5452.0001276 · Cited by 132
Okewu, E., Adewole, P., Misra, S., Maskeliūnas, R., & Damaševičius, R. (2021) · Applied Artificial Intelligence · doi:10.1080/08839514.2021.1922847 · Cited by 159
“…2016), (Bengio, Courville, and Vincent 2013), (Sak, Senior, and Beaufays 2014b), (Li and Wu 2015), (Zen and Sak 2015), (Gers and Schmidhuber 2001), (Mikolov et al. 2010), (Bengio, Boulanger-Lewandowski, and Pascanu 2013), (Campagni et al. 2015), (Ferreira and Andrade 2014), (Palmer 2013),…”
InfluentialRecurrent Neural Networks for Snapshot Compressive Imaging
Cheng, Z., Chen, B., Lu, R., Wang, Z., Zhang, H., Meng, Z., & Yuan, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3161934 · Cited by 150
“…originally developed to capture temporal correlations for text and speech, e.g., [48], [49], are becoming increasingly popular for video or sprectral tasks, such as video deblurring [50], super-resolution [51], [52], object segmentation [53], and hyspectral image classification [54] and fusion [55].”
BackgroundVision Transformer for Pansharpening
Meng, X., Wang, N., Shao, F., & Shutao Li, A. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3168465 · Cited by 143
“Unlike the recurrent neural network (RNN) [34], [35] or CNN [36], [37], the transformer tends to build up a long-distance dependency which not only can attend to the whole image but can degrade to function such as CNN.”
BackgroundLarge language models: a survey of their development, capabilities, and applications
Yadagiri, A., & Pakray, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02310-4 · Cited by 214
Long, S., Tan, J., Mao, B., Tang, F., Li, Y., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3526606 · Cited by 165
“In 2014, Google released the Seq2Seq model [56], [57], [58], [59], [60], [61], [62], [63], [64] based on Recurrent Neural Networks (RNNs) [67], [68], [69], [70] and Long Short-Term Memory networks (LSTMs) [71], [72], [73], [74], [75], showcasing significant improvements in tasks like machine…”
Towards Automated ICD Coding Using Deep Learning
Shi, H., Xie, P., Hu, Z., Zhang, M., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04075 · Cited by 113
“Due to the capacity of capturing long-range semantics in texts, LSTM is widely used for language modeling and sequence encoding 19 , 20 .”
BackgroundTransformer-Transducer: End-to-End Speech Recognition with Self-AttentionPreprint
Yeh, C.-F., Mahadeokar, J., Kalgaonkar, K., Wang, Y.-Q., Le, D., Jain, M., Schubert, K., Fuegen, C., & Seltzer, M. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12977 · Cited by 169
“For language modeling, both statistical n-gram models [2] and more recently, neural-network-based models [3] aim to model purely the connection between word tokens.”
Yan, M. Y., Gustad, L. T., & Nytrø, Ø. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab236 · Cited by 119
Radical-Enhanced Chinese Character Embedding
Sun, Y., Lin, L., Yang, N., Ji, Z., & Wang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12640-1_34 · Cited by 137
“Mikolov et al. introduce the Recurrent neural network language models (RNNLMs) (Mikolov et al., 2010), Continuous Bag-of-Word (CBOW) and skip-gram model (Mikolov et al., 2013a) to learn embedding for words and phrases.”
BackgroundMethodChoi, J. W., Ku, Y., Yoo, B. W., Kim, J. A., Lee, D. S., Chai, Y. J., Kong, H., & Kim, H. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0189259 · Cited by 129
Fluency Boost Learning and Inference for Neural Grammatical Error Correction
Ge, T., Wei, F., & Zhou, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1097 · Cited by 119
“During model inference, we set beam size to 5 and decode 1-best result with a 2-layer GRU RNN language model (Mikolov et al., 2010) through shallow fusion (G¨ulc¸ehre et al., 2015) with weight β = 0 .”
MethodDeep Leaming for Stock Market Prediction Using Technical Indicators and Financial News Articles
Vargas, M. R., dos Anjos, C. E. M., Bichara, G. L. G., & Evsukoff, A. G. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489208 · Cited by 109
“This is shown by the lower performance of the I-RNN model, which uses only a set of technical inputs, compared to SI-RCNN model, that uses a hybrid input.”
MethodInfluentialThe Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting SecretsPreprint
Carlini, N., Liu, C., Kos, J., Erlingsson, Ú., & Song, D. X. (2018) · arXiv (Cornell University) · Cited by 124
Estimation of vegetation indices for high-throughput phenotyping of wheat using aerial imaging
Khan, Z., Eichi, V. R., Haefele, S. M., Garnett, T., & Miklavcic, S. J. (2018) · Plant Methods · doi:10.1186/s13007-018-0287-6 · Cited by 119
“In future work, a more complex DNN model could be designed to account for causal relationships of the data by using recurrent neural networks [38].”
BackgroundAttention-based bidirectional GRU networks for efficient HTTPS traffic classification
Liu, X., You, J., Wu, Y., Li, T., Li, L., Zhang, Z., & Ge, J. (2020) · Information Sciences · doi:10.1016/j.ins.2020.05.035 · Cited by 121
“Recurrent neural networks (RNNs) [47] have been successfully applied in many fields.”
BackgroundLearning Continuous Phrase Representations for Translation Modeling
Gao, J., He, X., Yih, W.-T., & Deng, L. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1066 · Cited by 117
“…recent work on continuous space language models, which estimate the probability of a word sequence in a continuous space (Bengio et al. 2003; Mikolov et al. 2010), have advanced the state of the art in language modeling, outperforming the traditional n-gram model on speech recognition…”
BackgroundPredicting citation counts based on deep neural network learning techniques
Abrishami, A., & Aliakbary, S. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.02.011 · Cited by 167
“In recent years, neural network has found significant applications in text processing (Mikolov et al., 2010; Lai et al., 2015; Severyn and Moschitti, 2015b), image processing (Simonyan and Zisserman, 2014; Karpathy and Fei-Fei, 2017; Krizhevsky et al.”
BackgroundConverting Your Thoughts to Texts: Enabling Brain Typing via Deep Feature Learning of EEG Signals
Zhang, X., Yao, L., Sheng, Q. Z., Kanhere, S. S., Gu, T., & Zhang, D. (2018) · 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom) · doi:10.1109/percom.2018.8444575 · Cited by 112
Deep learning-based fabric defect detection: A review
Kahraman, Y., & Durmuşoğlu, A. (2022) · Textile Research Journal · doi:10.1177/00405175221130773 · Cited by 144
“Recurrent neural networks (RNNs) 26,27 consists of a rich set of architecture and are the basic network architecture.”
BackgroundSupervised Segmentation of Un-Annotated Retinal Fundus Images by Synthesis
Zhao, H., Li, H., Maurer‐Stroh, S., Guo, Y., Deng, Q., & Cheng, L. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2854886 · Cited by 119
“On the other hand, the so called recurrent neural network or RNN has emerged in a wide range of applications such as speech recognition [34], language modeling [35], imaging captioning [36], and image generation [37].”
BackgroundArtificial neural network approaches for disaster management: A literature review
Guha, S., Jana, R. K., & Sanyal, M. K. (2022) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2022.103276 · Cited by 140
Self-Adaptive Hierarchical Sentence Model
Zhao, H., Lu, Z., & Poupart, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.05070 · Cited by 47
Cascade Dynamics Modeling with Attention-based Recurrent Neural Network
Wang, Y., Shen, H., Liu, S., Gao, J., & Cheng, X. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/416 · Cited by 115
“Recently, researchers found that Recurrent Neural Network (RNN) offers a convenient and effective tool for cascade modeling [Mikolov et al., 2010; Wang et al., 2017].”
BackgroundStrategies for Training Large Vocabulary Neural Language Models
Chen, W., Grangier, D., & Auli, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1186 · Cited by 115
Adversarial Generation of Natural Language
Subramanian, S., Rajeswar, S., Dutil, F., Pal, C., & Courville, A. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2629 · Cited by 112
“Recurrent Neural Networks (RNNs), particularly Long short-term memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Networks (Cho et al., 2014) are powerful models that have been successful at modeling sequential data (Graves and Schmidhuber, 2009; Mikolov et al., 2010).”
BackgroundBot, K., & Borges, J. G. (2022) · Inventions · doi:10.3390/inventions7010015 · Cited by 130
“Some of the popular algorithms for regression tasks are: naive Bayes [29], decision trees [30], classification and regression trees [31], random forest [32], deep neural network [33], Gaussian processes [34], neural networks [35], genetic algorithms [36], recurrent neural networks [37], and maximum entropy [38].”
MethodSentiment analysis on Bangla text using extended lexicon dictionary and deep learning algorithms
Bhowmik, N. R., Arifuzzaman, M., & Mondal, M. R. H. (2022) · Array · doi:10.1016/j.array.2021.100123 · Cited by 132
Survey of Graph Neural Networks and Applications
Liang, F., Qian, C., Yu, W., GRIFFITH, D. W. T., & Golmie, N. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/9261537 · Cited by 165
“The existing neural networks (Convolutional Neural Networks (CNNs) [1], Recurrent Neural Networks (RNNs) [2], etc.) have been devoted to di ff erent problem domains (IoT, etc.), supporting a variety of tasks (prediction, classi fi cation, identi fi cation, tracking, codesign, and data generation,…”
BackgroundOn the Convergence Rate of Training Recurrent Neural Networks
Allen-Zhu, Z., Li, Y., & Song, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12065 · Cited by 104
“These networks are widely used in practice for natural language processing, language generation, machine translation, speech recognition, video and music processing, and many other tasks [11, 12, 27, 35, 36, 44, 52, 54].”
BackgroundThe third ‘CHiME’ speech separation and recognition challenge: Analysis and outcomes
Barker, J., Marxer, R., Emmanuel Vincent, A., & Watanabe, S. (2016) · Computer Speech & Language · doi:10.1016/j.csl.2016.10.005 · Cited by 114
“…hypotheses using either a DNN-LM (Vu et al., 2015), LSTM-LM (El-Desoky Mousa et al., 2015) or, most commonly, a conventional recurrent neural network language model (RNN-LM; Mikolov et al., 2010) (Tachioka et al., 2015; Yoshioka et al., 2015; Sivasankaran et al., 2015; Pfeifenberger et al., 2015).”
CharNER: Character-Level Named Entity Recognition
Kuru, O., Can, O. A., & Yüret, D. (2016) · International Conference on Computational Linguistics · Cited by 164
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.2306 · Cited by 114
“This objective encourages aligned image-sentences pairs to have a higher score than misaligned pairs, by a margin.”
A Probabilistic Model for Learning Multi-Prototype Word Embeddings
Tian, F., Dai, H., Bian, J., Gao, B., Zhang, R., Chen, E., & Liu, T. (2014) · International Conference on Computational Linguistics · Cited by 113
Probabilistic Prediction of Regional Wind Power Based on Spatiotemporal Quantile Regression
Yu, Y., Han, X., Yang, M. Y., & Yang, J. (2020) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2020.2992945 · Cited by 122
“Hence, the RNN appears the vanishing gradient and loses the ability to link to the long-term information [39].”
BackgroundRecurrent conditional random field for language understanding
Yao, K., Peng, B., Zweig, G., Yu, D., Li, X., & Gao, F. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854368 · Cited by 112
CTS-LSTM: LSTM-based neural networks for correlatedtime series prediction
Wan, H., Guo, S., Yin, K., Liang, X., & Lin, Y. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105239 · Cited by 118
Chinese Poetry Generation with Planning based Neural NetworkPreprint
Wang, Z., He, W. P., Wu, H. R., Wu, H., Li, W., Wang, H., & Chen, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09889 · Cited by 166
“We use a Recurrent Neural Network Language Model (RNNLM) (Mikolov et al., 2010) to predict the subsequent keywords according to the preceding sequence of keywords: ki = argmaxk P (k|k1:i−1), where ki is the i-th keyword and k1:i−1 is the preceding keywords sequence.”
MethodSun, Q., & Ge, Z. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2869899 · Cited by 114
“As for the dynamic issue, recurrent neural network (RNN) was proposed and has made a great breakthrough in the field of natural language processing, which shows an excellent ability in dealing with time series [22].”
BackgroundSECNLP: A survey of embeddings in clinical natural language processing
Kalyan, K. S., & Sangeetha, S. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103323 · Cited by 111
“tion in clinical records. Viewed the problem as sequence labelling task where the proposed method predicts the appropriate labels for the words. Experimented with different word embeddings like RNNLM [86] trained on Broad News Corpus and CBOW trained on Google news corpus. Evaluation on 2014 i2b2 challenge data set [1…”
BackgroundMethodPressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J. (2024) · Healthcare · doi:10.3390/healthcare12080825 · Cited by 155
“Powered by natural language processing (NLP) and machine learning, LLMs leverage complex neural network architectures [3].”
Background“How knowledge workers think generative AI will (not) transform their industries,”
Woodruff, A., Shelby, R., Kelley, P. G., Rousso-Schindler, S., Smith-Loud, J., & Wilcox, L. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642700 · Cited by 105
“Since then, researchers developed new kinds of neural networks that can generate novel content — including Generative Adversarial Networks [65], Recurrent Neural Networks (RNNs) [107], Variational Autoencoders (VAEs) [89], and diffusion generation models [144] — marking technical advances that set the stage for genera…”
BackgroundS-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking
Yang, Y., & Chang, M.-W. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1049 · Cited by 110
“Recently, many structured learning models based on neural networks have been proposed and are widely used in language modeling (Bengio et al., 2006; Mikolov et al., 2010), sentiment classification (Socher et al.”
Semi-Supervised Word Sense Disambiguation Using Word Embeddings in General and Specific Domains
Taghipour, K., & Ng, H. T. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1035 · Cited by 107
“Examples of such neural networks are (log-)linear networks (Mikolov et al., 2013a), deeper feed-forward neural networks (Bengio et al., 2003; Collobert and Weston, 2008), or recurrent neural networks (Mikolov et al., 2010).”
BackgroundThree Classes of Deep Learning Architectures and Their Applications: A Tutorial Survey
Deng, L. (2012) · APSIPA Transactions on Signal and Information Processing · Cited by 168
Inter-Weighted Alignment Network for Sentence Pair Modeling
Shen, G., Yang, Y., & Deng, Z. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1122 · Cited by 123
“For sentence modeling, RNN (Elman, 1990; Mikolov et al., 2010) and CNN (Kim, 2014) are both powerful and widely used.”
MethodClinical Assistant Diagnosis for Electronic Medical Record Based on Convolutional Neural Network
Yang, Z., Huang, Y., Jiang, Y., Sun, Y., Zhang, Y.-J., & Luo, P. (2018) · Scientific Reports · doi:10.1038/s41598-018-24389-w · Cited by 121
An automatic sorting system for unwashed eggs using deep learning
Nasiri, A., Omid, M., & Taheri‐Garavand, A. (2020) · Journal of Food Engineering · doi:10.1016/j.jfoodeng.2020.110036 · Cited by 126
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing
Klyuchnikov, N., Trofimov, I., Artemova, E., Salnikov, M., Fedorov, M. V., Filippov, A., & Burnaev, E. V. (2022) · IEEE Access · doi:10.1109/access.2022.3169897 · Cited by 165
Neural-based time series forecasting of loss of coolant accidents in nuclear power plants
Radaideh, M. I., Pigg, C., Kozłowski, T., Deng, Y., & Qu, A. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113699 · Cited by 132
“…between the hidden recurrent neurons and the output neurons, which can be used to calculate the output of the hidden layer as follows RNNs demonstrated successes in sequential data such as language modeling Mikolov et al. (2010) and time series forecasting Connor et al. (1994); Dorffner (1996).”
Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models
DeRose, J. F., Wang, J., & Berger, M. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028976 · Cited by 109
“It is common to use recurrent neural networks (RNNs) [34], and variants such as Long Short Term Memory (LSTM) [17] or Gated Recurrent Units (GRUs) [8], for this task to learn contextualized word-level embeddings, given contextindependent embeddings as input, e.”
BackgroundIndustry 4.0 Applications for Medical/Healthcare Services
Paul, S., Riffat, M., Yasir, A., Mahim, M. N., Sharnali, B. Y., Naheen, I. T., Rahman, A., & Kulkarni, A. (2021) · Journal of Sensor and Actuator Networks · doi:10.3390/jsan10030043 · Cited by 142
“Structured health information indicates data that contain lexicons involving demographic, medication, procedures, laboratory test, and diagnosis data [132], while unstructured information consists of text-based documents, such as medical notes from nurses and doctors.”
BackgroundMethodDeepMath - Deep Sequence Models for Premise Selection
Alemi, A., Chollet, F., Eén, N., Irving, G., Szegedy, C., & Urban, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04442 · Cited by 109
“Given a large set of premises P , an ATP system A with given resource limits, and a new conjecture C, predict those premises from P that will most likely lead to an automatically constructed proof of C by A.”
BackgroundHistory, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109
“• Neural Language Models: NLMs [56, 57, 69] leverage neural networks to predict the probabilities of subsequent words within sequences.”
Higuchi, T., Ito, N., Araki, S., Yoshioka, T., Delcroix, M., & Nakatani, T. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2665341 · Cited by 109
“(32) From this update rule, we can see that the hyperparameter η controls the contribution from the observations and the a priori expectation because Ψ(ν )f is set at (η −M − 1)×Ψ(ν )f ′, where Ψ(ν )f ′ denotes the a priori expectation of the spatial correlation matrix.”
BackgroundAutomatic Generation of Text Descriptive Comments for Code Blocks
Liang, Y., & Zhu, K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11963 · Cited by 106
“Existing work [8, 17, 30] has used Recurrent Neural Network to generate sentences.”
MethodFar-Field Automatic Speech Recognition
Haeb‐Umbach, R., Heymann, J., Drude, L., Watanabe, S., Delcroix, M., & Nakatani, T. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3018668 · Cited by 152
“There exist N-gram LMs and neural LMs, such as recurrent neural network (RNN) LM [102].”
BackgroundAnalyzing DistilBERT for Sentiment Classification of Banking Financial News
Dogra, V., Singh, A. D., Verma, S., Kavita, K., Jhanjhi, N. Z., & Talib, M. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-16-3153-5_53 · Cited by 131
Trellis Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06682 · Cited by 90
“After Mikolov et al. (2010) processed the corpus, the PTB dataset contains 888K words for training, 70K for validation and 79K for testing, where each sentence is marked with an <eos> tag at its end.”
MethodGeneralizing Eye Tracking With Bayesian Adversarial Learning
Wang, K., Zhao, R., Su, H., & Ji, Q. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01218 · Cited by 110
“More recently, appearance-based methods [24, 25, 26, 27, 28] with deep learning [29, 30, 31, 32] are the dominant approaches because of their improved performance over traditional model-based/appearance-based methods.”
BackgroundRMAF: Relu-Memristor-Like Activation Function for Deep Learning
Yu, Y., Adu, K., Tashi, N., Anokye, P., Wang, X., & Ayidzoe, M. A. (2020) · IEEE Access · doi:10.1109/access.2020.2987829 · Cited by 123
Surrogate optimization of deep neural networks for groundwater predictions
Müller, J., Park, J., Sahu, R. K., Varadharajan, C., Arora, B., Faybishenko, B. A., & Agarwal, D. (2020) · Journal of Global Optimization · doi:10.1007/s10898-020-00912-0 · Cited by 112
“g \backward in time" in a fully connected RNN may either vanish or blow up, which leads to oscillating weights or long training times. RNNs have been used in applications like speech recognition [58], text generation [59], and time series forecasting in hydrology [60, 61], where system memory and context are crit…”
MethodInfluentialInteractive neural machine translation
Peris, Á., Domingo, M., & Casacuberta, F. (2016) · Computer Speech & Language · doi:10.1016/j.csl.2016.12.003 · Cited by 131
“From here, continuous models have been used profusely in a wide range of tasks like language modelling (Mikolov et al., 2010; Schwenk, 2007; Sundermeyer et al., 2012), handwritten text recognition (Graves et al., 2009) or automatic speech recognition (Graves et al., 2013).”
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.