Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 601–700 of 1,058
Dict2vec : Learning Word Embeddings using Lexical Dictionaries
Tissier, J., Gravier, C., & Habrard, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1024 · Cited by 107
“Recurrent neural networks are another way to exploit the context of a word by considering the sequence of words preceding it (Mikolov et al., 2010; Sutskever et al., 2011).”
BackgroundRecurrent Neural Network for Partial Discharge Diagnosis in Gas-Insulated Switchgear
Nguyen, M.-T., Nguyen, V.-H., Yun, S.-J., & Kim, Y. (2018) · Energies · doi:10.3390/en11051202 · Cited by 107
“When compared to traditional neural network structures, such as those of a fully-connected neural network and a convolutional neural network, an RNN model uses a recurrence formula during every time step in order to consider sequential information [35].”
MethodEnhancing Attention-Based LSTM With Position Context for Aspect-Level Sentiment Classification
Zeng, J., Ma, X., & Zhou, K. (2019) · IEEE Access · doi:10.1109/access.2019.2893806 · Cited by 114
“Recurrent neural networks (RNNs) [11], [45], [46] have become a cornerstone for many natural language processing (NLP) applications.”
BackgroundMethodModified deep learning and reinforcement learning for an incentive-based demand response model
Wen, L., Zhou, K., Li, J., & Wang, S. (2020) · Energy · doi:10.1016/j.energy.2020.118019 · Cited by 107
Neural Models for Information RetrievalPreprint
Mitra, B., & Craswell, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.01509 · Cited by 219
“Shift-invariant neural operations Convolutional [89, 103, 113, 114] and recurrent [67, 83, 135, 173] architectures are commonplace in most deep learning applications.”
BackgroundLe, Q. V. (2015) · Cited by 150
A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
McDermott, E., Sak, H., & Variani, E. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003790 · Cited by 96
“The most popular of these is “Shallow Fusion” [5, 6, 7, 8, 9], which is simple loglinear interpolation between the scores from the end-to-end model and the separately-trained LM.”
BackgroundLearning Private Neural Language Modeling with Attentive Aggregation
Ji, S., Pan, S., Long, G., Li, X., Jiang, J. B., & Huang, Z. H. (2018) · UTS ePRESS (University of Technology Sydney) · Cited by 106
Clinical and Surgical Applications of Large Language Models: A Systematic Review
Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13113041 · Cited by 107
“Through iterative exposure to a training dataset, LLMs refine their predictive abilities by anticipating subsequent words in a sequence, evaluating their predictions against actual outcomes, and adjusting their parameters accordingly until they achieve a high level of proficiency [4,5].”
Privacy-Preserving Federated Learning With Malicious Clients and Honest-but-Curious Servers
Le, J., Zhang, D., Lei, X., Jiao, L., Zeng, K., & Liao, X. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3295949 · Cited by 125
“DEEP learning has demonstrated good performance in many fields, such as image classification [1], [2], pattern recognition [3] and language processing [4].”
BackgroundAN OVERVIEW OF POPULAR DEEP LEARNING METHODS
Coşkun, M., Yıldırım, Ö., Uçar, A., & Demir, Y. (2017) · European Journal of Technic · doi:10.23884/ejt.2017.7.2.11 · Cited by 126
PredCNN: Predictive Learning with Cascade Convolutions
Xu, Z., Wang, Y., Long, M., & Wang, J. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/408 · Cited by 122
“[ Ranzato et al. , 2014 ] defined a recurrent architecture for spatial and temporal correlation discovery en-lightened by language modeling technique [ Mikolov et al. , 2010 ] .”
MethodStreaming Video QoE Modeling and Prediction: A Long Short-Term Memory Approach
Eswara, N., Ashique, S., Panchbhai, A., Chakraborty, S., Sethuram, H. P., Kuchi, K., Kumar, A., & Channappayya, S. S. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2895223 · Cited by 124
“LSTMs have been successfully used to address complex challenges in applications such as sequence labeling [40], visual recognition [41], image captioning [42] and machine translation [43].”
MethodAdelani, D. I., Mai, H., Fang, F., Nguyen, H. H., Yamagishi, J., & Echizen, I. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-44041-1_114 · Cited by 123
Deep Learning for Predictive Business Process Monitoring: Review and Benchmark
Rama-Maneiro, E., Vidal, J. C., & Lama, M. (2021) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2021.3139807 · Cited by 220
“We have identified the following encoding formats: • Continuous [13], [29], [35], [43], [44]: this encoding technique is inspired by the training of neural language models [71].”
MethodA Neural Knowledge Language Model
Ahn, S., Choi, H., Pärnamaa, T., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00318 · Cited by 101
“Specifically, in the recurrent neural network language model (RNNLM) (Mikolov et al., 2010), the computational complexity is linearly dependent on the number of vocabulary words.”
BackgroundInfluentialYousefi, M., Yousefi, M., Ferreira, R. P. M., Kim, J. H., & Fogliatto, F. S. (2017) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2017.10.002 · Cited by 106
Multi-Behavior Sequential Transformer Recommender
Yuan, E., Guo, W., He, Z., Guo, H., Liu, C., & Tang, R. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532023 · Cited by 148
“Sequential recommenders are designed to model the sequential dynamics in user behaviors to which various sequence models have been applied, including Recurrent Neural Network (RNN) [28], Convolutional Neural Network (CNN) [22], and Transformer [36].”
MethodA Word Embedding Approach to Predicting the Compositionality of Multiword Expressions
Salehi, B., Cook, P. F., & Baldwin, T. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1099 · Cited by 138
“…et al. (2013a) and Mikolov et al. (2013b), using the WORD2VEC package.2 WORD2VEC uses a log-linear model inspired by the original RNNLM approach of Mikolov et al. (2010), in two forms: (1) a continuous bagof-words (“CBOW”) model, whereby all words in a context window are averaged in a single…”
BackgroundMethodIdentifying Nontechnical Power Loss via Spatial and Temporal Deep Learning
Bhat, R. R., Trevizan, R. D., Sengupta, R., Li, X., & Bretas, A. S. (2016) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2016.0052 · Cited by 111
“Deep Learning has recently achieved unprecedented performance records in areas such as computer vision [21], speech [22] and natural language processing (NLP) [23], which motivates its application to the NTL identification problem.”
BackgroundUnsupervised Recurrent Neural Network Grammars
Kim, Y., Rush, A., Yu, L., Kuncoro, A., Dyer, C., & Melis, G. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1114 · Cited by 98
“(2016) differs considerably from the usual language modeling setup (Mikolov et al., 2010) since we model each sentence independently and use a much larger vocabulary (see §3.”
MethodInfluentialCognition Digital Twins for Personalized Information Systems of Smart Cities: Proof of Concept
Du, J., Zhu, Q., Shi, Y., Wang, Q., Lin, Y., & Zhao, D. (2020) · Journal of Management in Engineering · doi:10.1061/(asce)me.1943-5479.0000740 · Cited by 102
“RNN was selected because it has been proven to be a highly effective way of analyzing time series data (Connor et al. 1994).”
MethodInfluentialEnsemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie ReviewsPreprint
Mesnil, G., Mikolov, T., Ranzato, M., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5335 · Cited by 160
“One can use advanced machine learning techniques such as recurrent neural networks and their variations (Mikolov et al., 2010; Socher et al., 2011), however it is not clear if these provide any significant gain over simple bag-of-words and bag-of-ngram techniques (Pang & Lee, 2008; Wang & Manning,…”
BackgroundMethodSelf-citationA Read-Write Memory Network for Movie Story Understanding
Na, S., Lee, S., Kim, J., & Kim, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.80 · Cited by 100
“In the era of deep learning, one prevailing approach to model sequential input is to use recurrent neural networks (RNNs) [17] which store the given information into a hidden memory and update it over time.”
BackgroundMulti-scale Information Diffusion Prediction with Reinforced Recurrent Networks
Yang, C. H., Tang, J., Sun, M., Cui, G., & Liu, Z. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/560 · Cited by 115
“Recurrent Networks RNNs have shown their effectiveness in sequential data modeling in many areas, such as natural language processing [Mikolov et al., 2010].”
MethodA recurrent neural network for classification of unevenly sampled variable stars
Naul, B., Bloom, J. S., Pérez, F., & van der Walt, S. (2018) · eScholarship (California Digital Library) · Cited by 101
E.T.-RNN: Applying Deep Learning to Credit Loan Applications
Babaev, D., Savchenko, M., Tuzhilin, A., & Umerenkov, D. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330693 · Cited by 140
“-steps in addition to the current input in order to produce next output. This approach is naturally suited for many NLP tasks including text classification, machine translation and language modelling [24]. Our Embedding-Transctional RNN (E.T.-RNN) architecture is presented in Figure 1 and is inspired by the NLP method…”
MethodGated Word-Character Recurrent Language Model
Miyamoto, Y., & Cho, K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1209 · Cited by 108
“For the word–character hybrid models, we applied a pre-training procedure to encourage the model to use both representations.”
MethodA Latent Variable Recurrent Neural Network for Discourse-Driven Language Models
Ji, Y., Haffari, G., & Eisenstein, J. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1037 · Cited by 162
An analysis of large language models: their impact and potential applications
Bharathi Mohan, G., Prasanna Kumar, R., Vishal Krishh, P., Keerthinathan, A., Lavanya, G., Meghana, M. K. U., Sulthana, S., & Doss, S. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02120-8 · Cited by 99
“However, unless equipped with powerful artificial intelligence (AI) algorithms, machines cannot naturally grasp the abilities of understanding and communicating in the form of human language [2].”
Mousa, A. E.-D., & Schuller, B. W. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1096 · Cited by 116
“Novel state-of-the-art LMs are based on ANNs like RNNs that provide long-span probabilities conditioned on all predecessor words (Mikolov et al., 2010; Kombrink et al., 2011).”
BackgroundAthiwaratkun, B., & Wilson, A. G. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1151 · Cited by 129
“Other popular approaches use feedforward (Bengio et al., 2003) and recurrent neural network language models (Mikolov et al., 2010, 2011b; Collobert and We-ston, 2008) to predict missing words in sentences, producing hidden layers that can act as word embeddings that encode semantic information.”
Graña, D., Azevedo, L., & Liu, M. (2019) · Geophysics · doi:10.1190/geo2019-0405.1 · Cited by 104
Toward Remote Sensing Image Retrieval Under a Deep Image Captioning Perspective
Hoxha, G., Melgani, F., & Demir, B. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3013818 · Cited by 107
“Later, we present the multimodal RNN. 1) Simple Recurrent Neural Networks: In NLP community, RNNs have shown great success in different tasks such as speech recognition [34] , [35].”
BackgroundA Review on Deep-Learning-Based Cyberbullying Detection
Hasan, M. T., Hossain, M. A. E., Mukta, M. S. H., Akter, A., Ahmed, M., & Islam, S. (2023) · Future Internet · doi:10.3390/fi15050179 · Cited by 103
“If the problem demands a discriminative model, Convolutional Neural Network (CNN) [43] and Recurrent Neural Network (RNN) [44] might be chosen.”
BackgroundMethodKhanpour, H., Guntakandla, N., & Nielsen, R. D. (2016) · University of North Texas Digital Library (University of North Texas) · Cited by 127
Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation
Xiong, Y., Kim, H. J., & Singh, V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00793 · Cited by 104
“In fact, recurrent neural networks (RNNs) have been studied in language modeling [31], speech recognition [37], image captioning generation [55], motion capture [47], and machine translation [5, 57] since the 1990s and recent work has substantially built upon the early formulations.”
BackgroundLearning Generic Sentence Representations Using Convolutional Neural Networks
Gan, Z., Pu, Y., Henao, R., Li, C., He, X., & Carin, L. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1254 · Cited by 96
Software defect prediction employing BiLSTM and BERT-based semantic feature
Uddin, M. N., Li, B., Ali, Z., Kefalas, P., Khan, I. U., & Zada, I. (2022) · Soft Computing · doi:10.1007/s00500-022-06830-5 · Cited by 115
Machine Learning in Manufacturing towards Industry 4.0: From ‘For Now’ to ‘Four-Know’
Chen, T., Sampath, V., May, M. C., Shan, S., Jorg, O. J., Martín, J. J. A., Stamer, F., Fantoni, G., Tosello, G., & Calaon, M. (2023) · Applied Sciences · doi:10.3390/app13031903 · Cited by 130
“Typically, CNNs [74] and Recurrent Neural networks (RNNs) [75] are used for image data and time series data, respectively.”
MethodTransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer
Deng, J., Yang, Z., Liu, D., Chen, T., Zhou, W., Zhang, Y., Li, H., & Ouyang, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3296823 · Cited by 135
“Compared with recurrent units in RNNs [46], [47] and LSTMs [48], the core component of Transformer, i.e., attention modules, show remarkable superiority in long-term sequence modeling.”
Ontology-Based Multilabel Text Classification of Construction Regulatory Documents
Zhou, P., & El-Gohary, N. (2016) · Journal of Computing in Civil Engineering · doi:10.1061/(asce)cp.1943-5487.0000530 · Cited by 96
“Examples of neural network algorithms include the feedfor-ward neural network algorithm (Bengio et al. 2003) and the recurrent neural network algorithm (Mikolov et al. 2010).”
MethodRNNDROP: A novel dropout for RNNS in ASR
Moon, T., Choi, H., Lee, H., & Song, I. (2015) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404775 · Cited by 124
“Recurrent neural networks (RNNs) have been successfully applied to many applications including speech recognition [1], handwriting recognition [2] and language understanding [3] for modeling temporal dependencies in data.”
MethodRecurrent Filter Learning for Visual Tracking
Yang, T., & Chan, A. B. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.235 · Cited by 91
A comprehensive survey on convolutional neural network in medical image analysis
Yao, X., Wang, X., Wang, S., & Zhang, Y. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09634-7 · Cited by 121
“In 1997, the CNN model is presented [49, 63, 86].”
MethodConvolutional gated recurrent neural network incorporating spatial features for audio tagging
Xu, Y., Kong, Q., Huang, Q., Wang, W., & Plumbley, M. D. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966291 · Cited by 112
Hybrid-Model-Based Intelligent Optimization of Ironmaking Process
Zhou, H., Zhang, H., & Yang, C. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2903770 · Cited by 99
“exhibits outstanding performance in language recognition due to its advantage in storing history information [11], [12], which corresponds to the characteristics of continuity and large iner-”
BackgroundEfficient softmax approximation for GPUs
Grave, É., Joulin, A., Cissé, M., & Jeǵou, H. (2017) · International Conference on Machine Learning · Cited by 219
Generating Sentences from Disentangled Syntactic and Semantic Spaces
Bao, Y., Zhou, H., Huang, S., Li, L., Mou, L., Vechtomova, O., Dai, X.-Y., & Chen, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1602 · Cited by 93
“Bowman et al. (2016) successfully applied VAE in the NLP domain, showing that VAE improves recurrent neural network (RNN)-based language modeling (RNN-LM, Mikolov et al., 2010); that VAE allows sentence sampling and sentence interpolation in the continuous latent space.”
MethodCalibrating the adaptive learning rate to improve convergence of ADAM
Tong, Q., Liang, G., & Bi, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.014 · Cited by 103
“We use three datasets for image classifications: MNIST, CIFAR-10 and CIFAR-100 and two datasets for LSTM language mod-350 els: Penn Treebank dataset (PTB) and the WikiText-2 (WT2) dataset.”
InfluentialData augmentation for self-paced motor imagery classification with C-LSTM.
Freer, D., & Yang, G.-Z. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab57c0 · Cited by 96
“for language processing [32, 33], including speech recognition.”
Research Progress of RNN Language Model
Xiao, J., & Zhou, Z. (2020) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica50127.2020.9182390 · Cited by 119
“In 2010, Tomas mikolv proposed to apply the cyclic neural network model to language modeling (RNNLM) [2], and applied it to various NLP tasks such as speech recognition, machine translation, etc., and achieved the best results at that time, showing the outstanding advantages of the recurrent neural…”
MethodThe Multi-Temporal Urban Development SpaceNet Dataset
Etten, A. V., Hogan, D., Martinez-Manso, J., Shermeyer, J., Weir, N., & Lewis, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00633 · Cited by 149
“Recurrent Neural Networks (RNNs) [20] and Long-Short Term Memory networks (LSTMs) [12].”
BackgroundHierarchical Attention Networks for Cyberbullying Detection on the Instagram Social Network
Cheng, L., Guo, R., Silva, Y., Hall, D., & Liu, H. (2019) · Proceedings of the 2019 SIAM International Conference on Data Mining · doi:10.1137/1.9781611975673.27 · Cited by 95
“3.1 Bidirectional GRU-RNN To model the continuous temporal phenomena, we use the bi-directional GRU [1] based RNN to encode the sequence of words and comments.”
MethodInfluentialDynamic Entity Representations in Neural Language Models
Ji, Y., Tan, C., Martschat, S., Choi, Y., & Smith, N. A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1195 · Cited by 94
“Our starting point for language modeling is a recurrent neural network (Mikolov et al., 2010), which defines where W h and b are parameters of the model (along with word embeddings x t ), LSTM is the widely used recurrent function known as “long short-term memory” (Hochreiter and Schmidhuber, 1997),…”
BackgroundTian, Y., Peng, S., Zhang, X., Rodemann, T., Tan, K. C., & Jin, Y. (2020) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2020.3022339 · Cited by 99
“3, the reverse Polish expression is first tokenized by encoding each notation in a one-hot representation [47], i.e., a binary vector with one dimension being 1 and the rest being 0.”
BackgroundCheng, Z., Lu, R., Wang, Z., Zhang, H., Chen, B., Meng, Z., & Yuan, X. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_16 · Cited by 119
“RNNs, originally developed to capture temporal correlations for text and speech, e.g., [9, 14], are becoming increasingly popular for video tasks, such as deblurring [32], super-resolution [11, 31] and object segmentation [45].”
MethodIntegrating external knowledge into aspect-based sentiment analysis using graph neural network
Gu, T., Zhao, H., He, Z., Li, M., & Ying, D. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110025 · Cited by 112
Barman, D., Guo, Z., & Conlan, O. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100545 · Cited by 92
Interleaved text/image Deep Mining on a large-scale radiology database
Shin, H.-C., Lu, L., Kim, L., Seff, A., Yao, J., & Summers, R. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298712 · Cited by 94
“7While RNN [37, 46] is the popular choice for learning language models [3, 31], deep CNN [28, 24] is more suitable for image classification.”
MethodDeep Evolutionary Networks with Expedited Genetic Algorithms for Medical Image Denoising
Liu, P., Basha, M. D. E., Li, Y., Xiao, Y., Sanelli, P. C., & Fang, R. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.03.004 · Cited by 103
Deep Learning-Based Named Entity Recognition and Knowledge Graph Construction for Geological Hazards
Fan, R., Wang, L., Yan, J., Song, W., Zhu, Y., & Chen, X. (2019) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9010015 · Cited by 104
“The first is the NN-CRF architecture [31–34], in which CNNs/RNNs are used to learn the vector representation at each word position.”
BackgroundSynthesizing benchmarks for predictive modeling
Cummins, C., Petoumenos, P., Wang, Z., & Leather, H. (2017) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo.2017.7863731 · Cited by 114
“We use the Long Short-Term Memory (LSTM) architecture of Recurrent Neural Network [17, 18] to learn a character-level language model over the corpus of OpenCL compute kernels.”
MethodA Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond
Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., & Liu, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3277122 · Cited by 175
“[4] K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in Proc.”
MethodInfluentialApplication of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
“A language model 184–186 is a probabilistic model that models the joint probability distribution of tokens (meaningful units of text, such as words or sub-words or morphemes) in a sequence, that is, the probabilities of how words and phrases are used in sequences.”
Morphological Analysis for Unsegmented Languages using Recurrent Neural Network Language Model
Morita, H., Kawahara, D., & Kurohashi, S. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1276 · Cited by 93
“The point of the paper is that we have tackled the chicken-and-egg problem, not by using a lan-2292 guage model of raw word sequences, but by using a semantically generalized language model based on word embeddings, RNNLM (Recurrent Neural Network Language Model) (Mikolov et al., 2010; Mikolov et al., 2011).”
BackgroundMethodUnsupervised Topic Modeling for Short Texts Using Distributed Representations of Words
Sridhar, V. K. R. (2015) · Workshop on Vector Space Modeling for Natural Language Processing (VSM-NLP) · doi:10.3115/v1/w15-1526 · Cited by 97
“…in the context of a deep neural network framework for a given task (Bengio et al., 2003; Collobert and Weston, 2008; Bengio et al., 2009; Turian et al., 2010; Mikolov et al., 2010), recent work in (Mikolov et al., 2013) has also shown that they can also be induced by using simple log-linear models.”
BackgroundLearning to Segment Moving Objects
Tokmakov, P., Schmid, C., & Alahari, K. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1122-2 · Cited by 137
“, [25, 47], are becoming increasingly popular for visual data.”
BackgroundSun, C., Chen, C., Li, W., Fan, J., & Chen, W. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2937558 · Cited by 98
“For example, the convolutional neural network (CNN) and deep belief network (DBN) have been proved to have good capacity in feature extraction [9], [10], and the recurrent neural network (RNN) exhibit powerful effects on time series signal processing, such as NLP, action recognition [14], [15].”
BackgroundPapastratis, I., Dimitropoulos, K., Konstantinidis, D., & Daras, P. (2020) · IEEE Access · doi:10.1109/access.2020.2993650 · Cited by 93
“The proposed text encoder is a RNN Language Model (RNNLM) [35], [36] that models the probability of a word occurrence under the condition of its previous words in a sentence, i.e., it aims to learn the structure and syntax of sign language.”
MethodDropKey for Vision Transformer
Li, B., Hu, Y., Nie, X., Han, C., Jiang, X., Guo, T., & Liu, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02174 · Cited by 137
“Dropout Dropout is a common technique for improving the generalizaibility of neural networks, e.g. , CNNs [16], RNNs [21] and GNNs [24].”
MethodSarkar, K., Shiuly, A., & Dhal, K. G. (2023) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2023.134212 · Cited by 107
R-Transformer: Recurrent Neural Network Enhanced Transformer
Wang, Z., Ma, Y., Liu, Z., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05572 · Cited by 88
“For example, in language modeling task, x t is the character/word in a textual sentence and y is the character/word at next position (Mikolov et al., 2010); in session-based recommendation, x t is the user-item interaction in a session and y is the future item that users will interact with (Hidasi…”
BackgroundInfluentialXu, P., & Sarikaya, R. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853573 · Cited by 103
“The RNN based language model (LM) is one of the bigger successes of using RNN in recent years [2, 3] – on top of the standard feed-forward NN based LM [4, 5], recurrent con2014 IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP)”
MethodShan, C., Weng, C., Wang, G., Su, D., Luo, M., Yu, D., & Xie, L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682490 · Cited by 86
“The LM fusion [8, 13, 14, 15, 16, 17, 18, 19] is another way to alleviate this issue.”
BackgroundTu, Y.-H., Du, J., & Lee, C. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2940662 · Cited by 94
“The language models are 5-gram with Kneser-Ney (KN) smoothing [46] for the first-pass decoding and the simple RNN-based language model [47] for rescoring.”
MethodSelf-Attention-Based BiLSTM Model for Short Text Fine-Grained Sentiment Classification
Xie, J., Chen, B., Gu, X., Liang, F., & Xu, X. (2019) · IEEE Access · doi:10.1109/access.2019.2957510 · Cited by 110
Web page classification: a survey of perspectives, gaps, and future directions
Hashemi, M. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08373-8 · Cited by 91
“104], language modeling [60], machine translation [9, 63, 79, 81], question answering [62–64, 87],”
BackgroundVision Transformer Hashing for Image Retrieval
Dubey, S. R., Singh, S. K., & Chu, W.-T. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859900 · Cited by 114
“Whereas, Recurrent Neural Networks (RNNs) are used for sequential data [30].”
Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural NetworkPreprint
Wang, P., Qian, Y., Soong, F. K., He, L., & Zhao, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.06168 · Cited by 147
Is Machine Learning in Power Systems Vulnerable?
Chen, Y., Tan, Y., & Deka, D. (2018) · IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) · doi:10.1109/smartgridcomm.2018.8587547 · Cited by 115
Gkoumas, D., Li, Q., Lioma, C., Yu, Y., & Song, D. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.09.005 · Cited by 117
“Considering human language contains time-series and thus requiring fusing time-varying signals, a recent trend is to exploit LSTMs and RNNs to fuse unimodal representations at the feature level [12, 47].”
MethodInfluentialA Taxonomy of Empathetic Response Intents in Human Social Conversations
Welivita, A., & Pu, P. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.429 · Cited by 87
Early stopping by correlating online indicators in neural networks
Ferro, M. V., Doval, Y., Ribadas, F. J., & Bilbao, V. M. D. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.11.035 · Cited by 116
Mustapha, K. B. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103066 · Cited by 118
A Neural Language Model for Query Auto-Completion
Park, D. H., & Chiba, R. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080758 · Cited by 94
“As neural language models have been reported to perform well [2, 10, 11], we follow this direction.”
BackgroundSelf-Knowledge Distillation in Natural Language Processing
Hahn, S., & Choi, H. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_050 · Cited by 98
“PTB was made by (Marcus et al., 1993), and we use the pre-processed version by (Mikolov et al., 2010).”
MethodTime Series Domain Adaptation via Sparse Associative Structure Alignment
Cai, R., Chen, J., Li, Z., Chen, W., Zhang, K., Ye, J., Li, Z., Yang, X., & Zhang, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16846 · Cited by 86
Adversarial Attacks Against Deep Generative Models on Data: A Survey
Sun, H., Zhu, T., Zhang, Z., Jin, D., Xiong, P., & Zhou, W. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130903 · Cited by 90
“A number of surveys have also been conducted on DDMs, particularly CNNs and RNNs, see e.g., [28], [29], [30], [31], [32].”
BackgroundPhysics-informed deep learning model in wind turbine response prediction
Li, X., & Zhang, W. (2021) · Renewable Energy · doi:10.1016/j.renene.2021.12.058 · Cited by 96
Understanding and Improving Knowledge DistillationPreprint
Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Ed, H., & Jain, S. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03532 · Cited by 165
“We use Penn Tree Bank (PTB) dataset for word-level language modeling task using the standard train/validation/test split by (Mikolov et al., 2010).”
MethodMind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01951 · Cited by 90
“We then explore a simple way of keeping our models up-to-date and mitigate this temporal degradation by performing dynamic evaluation (Mikolov et al., 2010; Krause et al., 2018, 2019, §5) incrementally on streams of new data.”
MethodApplication of Convolutional Neural Network in Natural Language Processing
Wang, W., & Gang, J. (2018) · 2018 International Conference on Information Systems and Computer Aided Education (ICISCAE) · doi:10.1109/iciscae.2018.8666928 · Cited by 80
Pan, C., Tan, J., & Feng, D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.027 · Cited by 95
Du, S., Wang, J., Wang, M., Yang, J., Zhang, C., Zhao, Y. S., & Song, H. (2022) · Energy · doi:10.1016/j.energy.2022.126121 · Cited by 100
Vision–Language Model for Visual Question Answering in Medical Imagery
Bazi, Y., Rahhal, M. M. A., Bashmal, L., & Zuair, M. (2023) · Bioengineering · doi:10.3390/bioengineering10030380 · Cited by 87
Bootstrapping language acquisition
Abend, O., Kwiatkowski, T., Smith, N. J., Goldwater, S. J., & Steedman, M. J. (2017) · Cognition · doi:10.1016/j.cognition.2017.02.009 · Cited by 125
Recurrent neural network language model adaptation for multi-genre broadcast speech recognition
Chen, X., Tan, T., Liu, X., Lanchantin, P., Wan, M., Gales, M., & Woodland, P. C. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-696 · Cited by 91
“In state-of-the-art ASR systems, RNNLMs are often linearly interpolated with n-gram LMs to obtain both a good context coverage and strong generalisation [1, 3, 17, 18, 19, 20].”
BackgroundHuo, X., & Li, M. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/265 · Cited by 97
Decoding of finger trajectory from ECoG using deep learning
Xie, Z., Schwartz, O., & Prasad, A. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aa9dbe · Cited by 100
“At the same time, recurrent neural networks (RNN) are excellent tools for capturing patterns in sequences of data, such as speech, text, connected hand writing among others [53, 54, 61].”
Background
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.