Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 201–300 of 1,058
Representation learning for dynamic graphs: A survey
Kazemi, S. M., Goel, R., Jain, K., Kobyzev, I., Sethi, A., Forsyth, P., & Poupart, P. (2019) · arXiv · doi:10.48550/arxiv.1905.11485 · Cited by 338
“With the success of RNNs in several synchronous sequence modeling problems [160, 9, 94, 158, 103, 200], where the duration between any two consecutive items in the sequence is considered equal, and several asynchronous sequence modeling problems [45, 143, 64, 167, 260, 100], RNNs have been the most…”
BackgroundDeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334
“The most popular ones are based on recurrent neural networks (RNNs) (Elman, 1990; Mikolov et al., 2010), in particular Long Short-Term Memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997).”
BackgroundHuang, T., Zhang, Z., & Zhang, J. (2019) · Proceedings of the 13th ACM Conference on Recommender Systems · doi:10.1145/3298689.3347043 · Cited by 311
“easily be used in Internet companies. 2.2 Deep Learning based CTR Models Deep learning has achieved great success in many research fields such as computer vision[5, 14] and natural language processing[2, 18]. As a result, many deep learning based CTR models have also been proposed in recent years[1, 4, 6, 15, 22, 23,…”
BackgroundInfluentialBeneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 321
“Historically, the focus of statistical language models has been to create syntactically coherent text using architectures such as n-grams models (Stolcke, 2002) or auto-regressive recurrent architectures (Bengio et al., 2003; Mikolov et al., 2010; Sundermeyer et al., 2012).”
BackgroundMalware Detection with Deep Neural Network Using Process Behavior
Tobiyama, S., Yamaguchi, Y., Shimada, H., Ikuse, T., & Yagi, T. (2016) · 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2016.151 · Cited by 312
“proposed Recurrent Neural Network Language Model (RNNLM) which is the language model using a simple RNN [6].”
MethodApplication of deep learning to cybersecurity: A survey
Mahdavifar, S., & Ghorbani, A. A. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.02.056 · Cited by 325
The microsoft 2016 conversational speech recognition system
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953159 · Cited by 326
“2, 3, 4, 5, 6, 7], it is only recently that they have emerged as the best models for speech recognition. Surprisingly, this is the case for both acoustic modeling [8, 9, 10, 11] and language modeling [12, 13]. In comparison to standard feed-forward MLPs or DNNs, these acoustic models have the ability to model a large…”
BackgroundMethodInfluentialLSTM with sentence representations for document-level sentiment classification
Rao, G., Huang, W., Feng, Z., & Cong, Q. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.04.045 · Cited by 326
Learning Video Object Segmentation with Visual Memory
Tokmakov, P., Alahari, K., & Schmid, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.480 · Cited by 316
“, [17, 27], are becoming increasingly popular for visual data.”
BackgroundVideo (language) modeling: a baseline for generative models of natural videos
Ranzato, M., Szlam, A. D., Bruna, J., Mathieu, M., Collobert, R., & Chopra, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6604 · Cited by 254
“In order to design a system that tackles these tasks, we draw inspiration from classical methods from natural language processing, namely n-grams, neural net language models (Bengio et al., 2003) and recurrent neural networks (Mikolov et al., 2010).”
MethodInfluentialSpoken language understanding using long short-term memory neural networks
Yao, K., Peng, B., Zhang, Y., Yu, D., Zweig, G., & Shi, Y. (2014) · 2014 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2014.7078572 · Cited by 317
“In recent years, neural network based approaches have demonstrated outstanding performance in a variety of natural language processing tasks [1] [2] [3] [4] [5] [6] [7] [8].”
BackgroundRNN-Stega: Linguistic Steganography Based on Recurrent Neural Networks
Yang, Z.-L., Guo, X.-Q., Chen, Z.-M., Huang, Y.-F., & Zhang, Y.-J. (2019) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2871746 · Cited by 322
“Recurrent Neural Network [39] is a special artificial neural network model.”
BackgroundRecurrent neural networks for language understanding
Yao, K., Zweig, G., Hwang, M.-Y., Shi, Y., & Yu, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-569 · Cited by 315
“In recent years, Recurrent Neural Network language models (RNN-LMs) have demonstrated outstanding performance in a variety of natural language processing tasks [1, 2, 3, 4, 5, 6].”
BackgroundMethodLiu, Q., Zhou, F., Hang, R., & Yuan, X. (2017) · Remote Sensing · doi:10.3390/rs9121330 · Cited by 318
Neural Optimizer Search with Reinforcement Learning
Bello, I., Zoph, B., Vasudevan, V., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1709.07417 · Cited by 319
“Our last experiment is to evaluate our optimizers on the Penn Treebank (PTB) language modeling task (Marcus et al., 1993; Mikolov et al., 2010).”
MethodDeep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2020) · arXiv · doi:10.48550/arxiv.2001.07416 · Cited by 314
Bilingual Word Representations with Monolingual Quality in Mind
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing · doi:10.3115/v1/w15-1521 · Cited by 315
Ask the GRU: Multi-task Learning for Deep Text Recommendations
Bansal, T., Belanger, D., & McCallum, A. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959180 · Cited by 306
“Existing hybrid methods have used unsupervised learning objectives on text content to regularize the parameters of the recommendation model [4, 23, 24].”
MethodTERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
Liu, A. T., Li, S.-W., & Lee, H.-Y. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3095662 · Cited by 312
“The APC approach uses an autoregressive model to encode temporal information of past acoustic sequence; the model then predicts future frames like a recurrent-based LM [32] while conditioning on past frames.”
MethodA Fast and Simple Algorithm for Training Neural Probabilistic Language Models
Mnih, A., & Teh, Y. W. (2012) · arXiv · doi:10.48550/arxiv.1206.6426 · Cited by 315
“In spite of its simplicity, the LBL model has been shown to outperform n -grams, though the more complex neural language models (Mikolov et al., 2010; Mnih et al., 2009) can outperform it.”
BackgroundPredicting COVID-19 in China Using Hybrid AI Model
Zheng, N., Du, S., Wang, J., Zhang, H., Cui, W., Kang, Z., Yang, T., Lou, B., Chi, Y., Long, H., et al. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2990162 · Cited by 313
“Compared with traditional RNN [12], LSTM exhibits better performance in capturing the long-term dependencies of sequences and is therefore suitable for the classification, processing, and prediction of long-sequence data [13]–[16].”
MethodZaremba, W., & Sutskever, I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4615 · Cited by 283
“We clip the norm of the gradients (normalized by minibatch size) at 5 (Mikolov et al., 2010).”
Gated Neural Networks for Targeted Sentiment Analysis
Zhang, M., Zhang, Y., & Vo, D.-T. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10380 · Cited by 304
“Recurrent neural networks (RNN) have been shown effective in modeling sentences (Mikolov et al. 2010; Graves 2012; Cho et al. 2014b; Sutskever, Vinyals, and Le 2014), capturing long-range dependencies between words.”
MethodA recurrent neural network based microscopic car following model to predict traffic oscillation
Zhou, M., Qu, X., & Li, X. (2017) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2017.08.027 · Cited by 309
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 297
“In the past, researchers have developed and applied various neural architectures for NLP, including convolutional neural networks (Kalchbrenner et al., 2014; Kim, 2014), recurrent neural networks (Mikolov et al., 2010, 2011; Graves, 2013), and recursive neural networks (Socher et al., 2010, 2011).”
BackgroundAWB-GCN: A Graph Convolutional Network Accelerator with Runtime Workload Rebalancing
Geng, T., Li, A., Shi, R., Wu, C., Wang, T., Li, Y., Haghi, P., Tumeo, A., Che, S., Reinhardt, S., et al. (2020) · 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro50266.2020.00079 · Cited by 301
“Deep learning paradigms such as Convolutional Neural Networks (CNNs) [1] and Recurrent Neural Networks (RNNs) [2] have been applied to a wide range of applications including image classification, video processing, speech recognition, and natural language processing [3]–[7].”
BackgroundSemantically Enhanced Software Traceability Using Deep Learning Techniques
Guo, J., Cheng, J., & Cleland-Huang, J. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.9 · Cited by 232
“In particular, RNN and its variants have produced significant breakthroughs in many NLP tasks including language modeling [49], machine translation [6], and semantic entailment [66].”
BackgroundSurvey on deep learning for radiotherapy
Meyer, P., Noblet, V., Mazzara, C., & Lallement, A. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2018.05.018 · Cited by 302
Temporal Analysis of Language through Neural Language Models
Kim, Y., Chiu, Y.-I., Hanaki, K., Hegde, D., & Petrov, S. (2014) · Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science · doi:10.3115/v1/w14-2517 · Cited by 305
“Despite its simplicity— and thus, computational efficiency—compared to other NLMs, Mikolov et al. (2013b) note that the Skip-gram is competitive with other vector space models in the Semantic-Syntactic Word Relation-ship test set when trained on the same data.”
BackgroundPredicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory
Wang, X., Liu, Y., SUN, C., Wang, B., & Wang, X. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1130 · Cited by 297
Learning to Assemble Neural Module Tree Networks for Visual Grounding
Liu, D., Zhang, H., Zha, Z.-J., & Wu, F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00477 · Cited by 287
“Thanks to the development of deep visual features [9] and language models [27], we can extend the grounding systems from fixed-size inventory of words to open-vocabulary [12] or even descriptive and relational phrases [41, 31].”
BackgroundSharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303
“We analyze two datasets commonly used for language modeling, Penn Treebank (PTB) (Marcus et al., 1993; Mikolov et al., 2010) and Wikitext-2 (Wiki) (Merity et al., 2017).”
MethodSharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · arXiv · doi:10.48550/arxiv.1805.04623 · Cited by 302
Multichannel Audio Source Separation With Deep Neural Networks
Nugraha, A. A., Liutkus, A., & Vincent, E. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2580946 · Cited by 300
“Table IV shows the performance comparison using the DNN+sMBR back-end trained with enhanced multi-condition data followed by 5-gram KN smoothing and RNN-LM rescoring.”
MethodA Review on Deep Learning Techniques for Video Prediction
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045007 · Cited by 301
Chinese Poetry Generation with Recurrent Neural Networks
Zhang, X., & Lapata, M. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1074 · Cited by 298
Bayesian Deep-Learning-Based Health Prognostics Toward Prognostics Uncertainty
Peng, W., Ye, Z.-S., & Chen, N. (2020) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2907440 · Cited by 294
“RNNs are sequence-based models for learning and representing sequential features, especially for applications with data expressed in sequential forms [36]–[38].”
BackgroundPolyglot: Distributed Word Representations for Multilingual NLP
Al‐Rfou, R., Perozzi, B., & Skiena, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.1662 · Cited by 289
“Recent developments have led to state-of-art performance in several NLP tasks such as language modeling (Bengio et al., 2006; Mikolov et al., 2010), and syntactic tasks such as sequence tagging (Collobert et al.”
BackgroundA survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296
“RNN is a specific subclass of discriminative deep learning model which are designed to capture temporal dependencies among input data [41].”
Intelligent Computing: The Latest Advances, Challenges, and Future
Zhu, S., Yu, T., Xu, T., Chen, H., Dustdar, S., Gigan, S., Gunduz, D., Hossain, E., Jin, Y., Lin, F., et al. (2023) · Intelligent Computing · doi:10.34133/icomputing.0006 · Cited by 290
“Graph neural networks, such as graph convolutional networks [45], RNNs [46], graph attention networks [47], and graph residual networks [48], evolve from the technical framework of deep learning.”
Image Captioning with Deep Bidirectional LSTMs
Wang, C., Yang, H., Bartz, C., & Meinel, C. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2964299 · Cited by 287
“Inspired by the success use of CNN [14, 45] and Recurrent Neural Network [24, 25, 1].”
MethodHybrid deep neural networks for face emotion recognition
Jain, N., Kumar, S., Kumar, A., Shamsolmoali, P., & Zareapoor, M. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.04.010 · Cited by 294
““RNNs recently have greatly succeeded in handling sequential data such as speech recognition [23] , natural language processing [24,25] , action recognition [26] , and so on.”
BackgroundAre deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292
“We replicate their experiments using the Recurrent Neural Network LanguageModel toolkit (RNNLM6) [25], which includes a hidden layer size of 300 and 1,000 direct connections.”
BackgroundMethodDeep learning for detecting financial statement fraud
Craja, P., Kim, A., & Lessmann, S. (2020) · Decision Support Systems · doi:10.1016/j.dss.2020.113421 · Cited by 290
Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring
Dong, F., Zhang, Y., & Yang, J. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1017 · Cited by 277
“Both recurrent neural networks (Williams and Zipser, 1989; Mikolov et al., 2010) and convolutional neural networks (LeCun et al., 1998; Kim, 2014) have been used for modelling input essays.”
MethodGeneralization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285
“Neural language models (LMs) typically solve two subproblems: (1) mapping sentence prefixes to fixed-sized representations, and (2) using these representations to predict the next word in the text (Bengio et al., 2003; Mikolov et al., 2010).”
BackgroundLong Short-Term Memory Neural Networks for Chinese Word Segmentation
Chen, X., Qiu, X., Zhu, C., Liu, P., & Huang, X. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1141 · Cited by 274
Deep learning based computer-aided diagnosis systems for diabetic retinopathy: A survey
Asiri, N., Hussain, M., Al Adel, F., & Alzaidi, N. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.07.009 · Cited by 279
Empirical evaluation and combination of advanced language modeling techniques
Mikolov, T., Deoras, A., Kombrink, S., Burget, L., & Černocký, J. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-242 · Cited by 278
“Emami has proposed a Syntactical NNLM [8] that aims to incorporate linguistic features into the neural network model.”
BackgroundSelf-citationAn Iterative BP-CNN Architecture for Channel Decoding
Liang, F., Shen, C., & Wu, F. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2794062 · Cited by 276
“ork without memory and state information. In the deep learning literature, there is another celebrated network form called recurrent neural networks (RNN), which is mainly used in language processing [29]. The exploitation of the RNN structure with memory in BP decoding is an important and interesting topic that we pl…”
BackgroundDeep Air Learning: Interpolation, Prediction, and Feature Analysis of Fine-Grained Air Quality
Qi, Z., Wang, T., Song, G., Hu, W., Li, X., & Zhang, Z. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2823740 · Cited by 274
“Logistic Regression 5 [33], L 1 -regularized Logistic Regression with the same spatio-temporal constraints (ST), Neural Network, Autoencoder, Laplacian Regression (LapReg) [34], Autoregressive Integrated Moving Average (ARIMA) [35], and Recurrent Neural Network (RNN) [36] in the experiments.”
InfluentialMultilingual Models for Compositional Distributed Semantics
Hermann, K. M., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1006 · Cited by 268
“They have received a lot of attention in recent years (Collobert and We-ston, 2008; Mnih and Hinton, 2009; Mikolov et al., 2010, inter alia ) and have achieved state of the art performance in language modelling.”
BackgroundA survey on the application of recurrent neural networks to statistical language modeling
De Mulder, W., Bethard, S., & Moens, M.-F. (2015) · Computer Speech & Language · doi:10.1016/j.csl.2014.09.005 · Cited by 272
Cyclical Annealing Schedule: A Simple Approach to Mitigating
Fu, H., Li, C., Liu, X., Gao, J., Celikyilmaz, A., & Carin, L. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1021 · Cited by 230
“To generate a text sequence of length T , x = [ x 1 , · · · , x T ] , neural language models (Mikolov et al., 2010) generate every token x t conditioned on the previously generated tokens: where x t indicates all tokens before t .”
BackgroundA brief survey of machine learning methods and their sensor and IoT applications
Shanthamallu, U. S., Spanias, A., Tepedelenlioglu, C., & Stanley, M. (2017) · 2017 8th International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa.2017.8316459 · Cited by 264
“Networks whose output depends on present and past inputs, namely recurrent neural networks (RNNs) [169,172,173], have also been used in several applications.”
MethodAndroid malware detection based on system call sequences and LSTM
Xiao, X., Zhang, S., Mercaldo, F., Hu, G., & Sangaiah, A. K. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5104-0 · Cited by 268
“The work [22] proposes an RNN language model which makes greater progress than the previous works.”
BackgroundMethodResultHST-LSTM: A Hierarchical Spatial-Temporal Long-Short Term Memory Network for Location Prediction
Kong, D., & Wu, F. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/324 · Cited by 259
Social media sentiment analysis based on COVID-19
Nemes, L., & Kiss, A. (2021) · Journal of Information and Telecommunication · doi:10.1080/24751839.2020.1790793 · Cited by 268
Lexicon Infused Phrase Embeddings for Named Entity Resolution
Passos, A., Kumar, V., & McCallum, A. (2015) · Cited by 277
Deep learning for tactile understanding from visual and haptic data
Gao, Y., Hendricks, L. A., Kuchenbecker, K. J., & Darrell, T. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487176 · Cited by 264
“For understanding 1-D time series, recurrent networks have also achieved great success on tasks such as language modeling [28] and handwriting recognition [16].”
BackgroundZero-Resource Translation with Multi-Lingual Neural Machine Translation
Firat, O., Sankaran, B., Al-Onaizan, Y., Yarman Vural, F. T., & Cho, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1026 · Cited by 261
Joint Language and Translation Modeling with Recurrent Neural Networks
Auli, M., Galley, M., Quirk, C., & Zweig, G. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1106 · Cited by 253
“In this paper we focus on recurrent neural network architectures, which have recently advanced the state of the art in language modeling (Mikolov et al., 2010; Mikolov et al., 2011a; Mikolov, 2012), outperforming multi-layer feed-forward based networks in both perplexity and word error rate in…”
BackgroundMethodRNNLM - Recurrent Neural Network Language Modeling Toolkit
Mikolov, T., Kombrink, S., Deoras, A., Burget, L., & Černocký, J. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · Cited by 256
Self-citationFrom Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning
Song, J., Guo, Y., Gao, L., Li, X., Hanjalic, A., & Shen, H. T. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2851077 · Cited by 259
“The RNN language model (RNNLM) [36] parameterized the output distribution by applying a softmax function onto the previous hidden state ht−1.”
MethodInfluentialUsing artificial intelligence to create value in insurance
Riikkinen, M., Saarijärvi, H., Sarlin, P., & Lähteenmäki, I. (2018) · International Journal of Bank Marketing · doi:10.1108/ijbm-01-2017-0015 · Cited by 258
Qian, J., Song, Z., Yao, Y., Zhu, Z., & Zhang, X. (2022) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2022.104711 · Cited by 257
Large language models for human–robot interaction: A review
Zhang, C., Chen, J., Li, J., Peng, Y., & Mao, Z. (2023) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2023.100131 · Cited by 254
Ensemble deep learning for speech recognition
Deng, L., & Platt, J. C. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-433 · Cited by 248
“Deep learning has gone a long way from the early small experiments [17][12][13][25][24][2] using the fully-connected DNN to increasingly larger experiments [16][39][11][29][33] [18] and to the exploitation of more advanced deep architectures such as the deep CNN [30][9][31] which takes into account invariant propertie…”
MethodCreative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257
“A sentence-level language model [23] defnes the probability distribution over the next word, from within a predefned vocabulary, conditioned on the left context (i.”
BackgroundAdaptive Input Representations for Neural Language Modeling
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 48
“Recently, much progress has been made by neural methods (Bengio et al., 2003; Mikolov et al., 2010) based on LSTMs (J ´ ozefowicz et al., 2016), gated convolutional networks (Dauphin et al., 2017) and self-attentional networks (Al-Rfou et al., 2018).”
MethodMalware classification with LSTM and GRU language models and a character-level CNN
Athiwaratkun, B., & Stokes, J. W. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952603 · Cited by 239
“The authors propose either using a standard recurrent neural network (RNN) [7] or an echo state network (ESN) [8] as the language model.”
MethodAn Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253
Xu, S., & Niu, R. (2018) · Computers & Geosciences · doi:10.1016/j.cageo.2017.10.013 · Cited by 249
“RNNs have been applied with great success in studies involving natural language processing and audio processing (Graves, 2013; Mikolov et al., 2010; Parveen et al., 2000).”
MethodDetection of Suicide Ideation in Social Media Forums Using Deep Learning
Tadesse, M. M., Lin, H., Xu, B., & Yang, L. (2019) · Algorithms · doi:10.3390/a13010007 · Cited by 249
“The growing success of word embedding [12,13] and deep neural networks are reflected in outperforming more traditional machine learning systems for suicide risk assessments.”
BackgroundTao, L., Varshney, V., & Li, Y. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.1c01031 · Cited by 246
DeepNano: Deep recurrent neural networks for base calling in MinION nanopore reads
Boža, V., Brejová, B., & Vinař, T. (2017) · PLOS ONE · doi:10.1371/journal.pone.0178751 · Cited by 246
“ase caller DeepNano, we opt to use recurrent neural networks, which have stellar results for speech recognition (Graves et al., 2013), machine translation (Sutskever et al., 2014), language modeling (Mikolov et al., 2010), and other sequence processing tasks. Note that neural networks were previously used for base cal…”
MethodToward Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., Yu, D., & Zweig, G. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2756440 · Cited by 245
“This is the case for both acoustic modeling [28]–[33] and language modeling [34]–[37].”
BackgroundZilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 74
A Novel Deep Learning-Based Collaborative Filtering Model for Recommendation System
Fu, M., Qu, H., Yi, Z., Lu, L., & Liu, Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2795041 · Cited by 243
“For example: a) RNN [35] to consider the temporal relations between the history items of user and b) CNN [34] to address overall historical data of users and items; 2) consider the content information, such as text, images, and video via appending additional view of content into our multiview neural networks; and 3) i…”
InfluentialCompositional Vector Space Models for Knowledge Base Completion
Neelakantan, A., Roth, B., & McCallum, A. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1016 · Cited by 230
A survey on automatic image caption generation
Bai, S., & An, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.080 · Cited by 241
“Recurrent Neural Networks are known to have difficulties in learning long term dependencies [103] , [104] .”
BackgroundAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training WorkloadsPreprint
Jeon, M., Venkataraman, S., Phanishayee, A., Qian, J., Xiao, W., & Yang, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.05758 · Cited by 246
“Jobs are based on recently proposed learning architectures like convolutional neural networks [31], LSTMs [43] and RNNs [35].”
BackgroundA Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
“The impressive performance of LLMs can be attributed to their Transformer architecture, large-scale pre-training, and scaling laws.”
BackgroundGenerative Models for De Novo Drug Design
Tong, X., Liu, X., Tan, X., Li, X., Jiang, J., Xiong, Z., Xu, T., Jiang, H., Qiao, N., & Zheng, M. (2021) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.1c00927 · Cited by 233
Deep learning for stock market prediction from financial news articles
Vargas, M. R., de Lima, B. S. L. P., & Evsukoff, A. G. (2017) · 2017 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA) · doi:10.1109/civemsa.2017.7995302 · Cited by 229
Word Embedding for Understanding Natural Language: A Survey
Li, Y., & Yang, T. (2018) · Studies in Big Data · doi:10.1007/978-3-319-53817-4_4 · Cited by 226
Why gradient clipping accelerates training: A theoretical justification for adaptivity
Sra, S., He, T., Sra, S., & Jadbabaie, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11881 · Cited by 79
“We run language modeling on the Penn Treebank (PTB) (Mikolov et al., 2010) dataset with AWDLSTM models (Merity et al., 2018).”
MethodInfluentialDo Multi-Sense Embeddings Improve Natural Language Understanding?
Li, J., & Jurafsky, D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1200 · Cited by 232
“Neural embedding learning frameworks represent each token with a dense vector representation, optimized through predicting neighboring words or decomposing co-occurrence matrices (Bengio et al., 2006; Collobert and Weston, 2008; Mnih and Hinton, 2007; Mikolov et al., 2013; Mikolov et al., 2010; Pennington et al., 2014…”
MethodAspect-Based Sentiment Analysis: A Survey of Deep Learning Methods
Liu, H., Chatterjee, I., Zhou, M., Lu, X. S., & Abusorrah, A. (2020) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2020.3033302 · Cited by 231
“They can be divided into: 1) CNN [24], [87], [88]; 2) RNN [25], [89] that includes standard RNN, LSTM [26], and GRU [27]; 3) RecNN [28]; and 4) Memory Network (MN) [29].”
MethodProstate cancer detection using deep convolutional neural networks
Yoo, S., Gujrathi, I., Haider, M. A., & Khalvati, F. (2019) · Scientific Reports · doi:10.1038/s41598-019-55972-4 · Cited by 230
End-to-End Speech Recognition: A Survey
Prabhavalkar, R., Hori, T., Sainath, T. N., Schlüter, R., & Watanabe, S. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3328283 · Cited by 223
“3) RNN-LM: A recurrent neural network (RNN) LM was introduced to exploit longer contextual information over N − 1 previous tokens using recurrent connections [289].”
MethodHow to Construct Deep Recurrent Neural Networks
Pascanu, R., Gülçehre, Ç., Cho, K., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6026 · Cited by 228
“In language modeling, the learning rate starts from an initial value and is halved each time the validation cost does not decrease significantly (Mikolov et al. , 2010).”
BackgroundCCLearner: A Deep Learning-Based Clone Detection Approach
Li, L., Feng, H., Zhuang, W., Meng, N., & Ryder, B. (2017) · 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2017.46 · Cited by 214
“The tool first used recurrent neural network [44] to map program tokens to continuous-valued vectors, and then used recursive neural network [45] to combine the vectors with extracted syntactic features to train a classifier.”
MethodMorariu, C., Morariu, O., Răileanu, S., & Borangiu, T. (2020) · Computers in Industry · doi:10.1016/j.compind.2020.103244 · Cited by 226
A critical review of state‐of‐the‐art chatbot designs and applications
Luo, B., Lau, R. Y. K., Li, C., & Si, Y. (2022) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1434 · Cited by 224
“Based on the report by Mikolov et al. (2010) on RNN language modeling, which is an RNN-based algorithm for speech recognition, Wen et al. (2015) introduced a novel neural-based approach for NLG.”
MethodConvolutional neural tensor network architecture for community-based question answering
Qiu, X., & Huang, X. (2015) · International Conference on Artificial Intelligence · Cited by 284
LSTM-MSNet: Leveraging Forecasts on Sets of Related Time Series With Multiple Seasonal Patterns
Bandara, K., Bergmeir, C., & Hewamalage, H. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2985720 · Cited by 224
“…LSTMs. RNNs, and in particular LSTMs, have been embraced by many fields that involve sequence modeling tasks, such as natural language processing [41], speech recognition [42], and image generation [43], and, more recently, have received a great amount of attention in time-series research…”
BackgroundRecognizing functions in binaries with neural networks
Shin, E. C. R., Song, D. X., & Moazzezi, R. (2015) · USENIX Security Symposium · Cited by 322
Ullah, F. U. M., Ullah, A., Haq, I. U., Rho, S., & Baik, S. W. (2020) · IEEE Access · doi:10.1109/access.2019.2963045 · Cited by 222
“RNNs [42] typically experience short-term memory problems, particularly when processing and storing a long sequence from the earlier to the later steps.”
Background
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.