Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 901–1,000 of 1,058
Optimizing a quantum reservoir computer for time series prediction
Kutvonen, A., Fujii, K., & Sagawa, T. (2020) · Scientific Reports · doi:10.1038/s41598-020-71673-9 · Cited by 106
Syntactic and Semantic Features For Code-Switching Factored Language Models
Adel, H., Vu, N. T., Kirchhoff, K., Telaar, D., & Schultz, T. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2389622 · Cited by 67
Dependable Fire Detection System with Multifunctional Artificial Intelligence Framework
Park, J.-H., Lee, S.-G., Yun, S., Kim, H., & Kim, W.-T. (2019) · Sensors · doi:10.3390/s19092025 · Cited by 68
“For example, the Convolutional Neural Network (CNN) requires image data to recognize objects and the Recurrent Neural Network (RNN) is suitable for processing sequential data, including voice and text [12,13].”
MethodMulti-Channel Speech Recognition: LSTMs All the Way Through
Erdoğan, H., Hayashi, T., Hershey, J. R., Hori, T., Hori, C., Hsu, W.-N., Kim, S., Roux, J. L., Meng, Z., & Watanabe, S. (2016) · Cited by 61
Artificial Intelligence in Optical Communications: From Machine Learning to Deep Learning
Wang, D., & Zhang, M. (2021) · Frontiers in Communications and Networks · doi:10.3389/frcmn.2021.656786 · Cited by 75
“Different from feedforward neural networks, RNNs containing cyclic connections aim to provide neural networks with memory, meaning that the outputs are not only related by the current inputs but also the formerly available information (Mikolov et al., 2010).”
BackgroundA novel multi-step ahead forecasting model for flood based on time residual LSTM
Zou, Y., Wang, J., Peng, L., & Li, Y. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.129521 · Cited by 76
Deep Learning in Finance: A Survey of Applications and Techniques
Mienye, E. D., Jere, N. R., Obaido, G., Mienye, I. D., & Aruleba, K. (2024) · AI · doi:10.3390/ai5040101 · Cited by 116
Jin, C., Pok, G., Lee, Y., Park, H., Kim, K. D., Yun, U., & Ryu, K. H. (2014) · Energy Conversion and Management · doi:10.1016/j.enconman.2014.11.010 · Cited by 70
Large scale recurrent neural network on GPU
Li, B., Zhou, E., Huang, B., Duan, J., Wang, Y., Xu, N., Zhang, J., & Yang, H. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889433 · Cited by 76
“Moreover, such a property also demonstrates that the recurrent neural language model is able to learn more patterns from less training data compared with the N-gram model [23].”
ResultShip Classification in High-Resolution SAR Images via Transfer Learning with Small Training Dataset
Lu, C., & Li, W. (2018) · Sensors · doi:10.3390/s19010063 · Cited by 70
“In recent years, many new models had been proposed, such as Alexnet [24], Resnet [34], VGG16 [22], etc.”
BackgroundMethodTweet modeling with LSTM recurrent neural networks for hashtag recommendation
Li, J., Xu, H., He, X., Deng, J., & Sun, X. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727385 · Cited by 75
“RNNs for NLP: RNNs are powerful models that have shown great promise in many natural language processing (NLP) tasks such as language modeling, question answering, sentiment classification, and machine translation.”
BackgroundConditional Generation and Snapshot Learning in Neural Dialogue Systems
Wen, T.-H., Gašić, M., Mrkšić, N., Rojas-Barahona, L. M., Su, P.-H., Ultes, S., Vandyke, D., & Young, S. J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1233 · Cited by 94
“Interest in generating natural language using NNs can be attributed to the success of RNN LMs for large vocabulary speech recognition (Mikolov et al., 2010; Mikolov et al., 2011).”
BackgroundDeep learning with knowledge transfer for explainable anomaly prediction in wind turbines
Chatterjee, J., & Dethlefs, N. (2020) · Wind Energy · doi:10.1002/we.2510 · Cited by 77
Quantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65
“Recurrent neural network language models (LMs) can learn to predict upcoming words with remarkably low perplexity (Mikolov et al., 2010; Jozefowicz et al., 2016; Radford et al., 2019).”
Adel, H., Vu, N. T., & Schultz, T. (2013) · Cited by 65
Exploring Different Dimensions of Attention for Uncertainty Detection
Adel, H., & Schütze, H. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1003 · Cited by 72
Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks
Yan, C., & He, Y. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3389738 · Cited by 74
“This is the proposed approach using a deep model architecture in Figure 13 (implemented using Keras [11]), which combines signals from both sequence modeling using RNN, as well as the characteristics of input tables captured by single-operator predictions (Section 4).”
MethodInfluentialSeqTrans: Automatic Vulnerability Fix Via Sequence to Sequence Learning
Chi, J., Qu, Y., Liu, T., Zheng, Q., & Yin, H. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3156637 · Cited by 95
“Unlike Recurrent Neural Network (RNN) [49] or Long Short Term Memory (LSTM) [50] models, transformer relies entirely on the self-attention mechanism to draw global dependencies between input and output data.”
Attention-Based CNN-RNN Arabic Text Recognition from Natural Scene Images
Butt, H., Raza, M. R., Ramzan, M. J., Ali, M. J., & Haris, M. (2021) · Forecasting · doi:10.3390/forecast3030033 · Cited by 74
“Shahin presented a regression-based model to recognize Arabic printed text on paper [29], which could be linear or nonlinear.”
BackgroundMethodQin, H., & Liu, G. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.12.001 · Cited by 72
Lee, M.-C., Chang, J.-W., Yeh, S., Chia, T.-L., Liao, J.-S., & Chen, X. (2022) · Neural Computing and Applications · doi:10.1007/s00521-021-06828-4 · Cited by 77
“For time series data with extremely high correlation between the context, it is relatively suitable to use recurrent neural network (RNN) [13].”
BackgroundMethodDRlinker: Deep Reinforcement Learning for Optimization in Fragment Linking Design
Tan, Y., Dai, L., Huang, W., Guo, Y., Zheng, S., Lei, J., Chen, H., & Yang, Y. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00982 · Cited by 84
Kreutzer, J., Schamoni, S., & Riezler, S. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3037 · Cited by 66
“…Penning-ton et al. (2014) inter alia ), or by producing state-of-the-art performance in applications such as language modeling (Bengio et al. (2003), Mikolov et al. (2010), inter alia ) or statistical machine translation (Kalchbrenner and Blunsom (2013), Bah-danau et al. (2015), inter alia ).”
BackgroundLiu, S., Li, T., Ding, H., Tang, B., Wang, X., Chen, Q., Yan, J., & Zhou, Y. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01155-x · Cited by 64
“…to predict medical events, As medical data of each patient is time series data, the typical time series analysis methods such as machine learning methods based on manually-crafted features [1] and Autoregressive Integrated Moving Average (ARIMA) models applied for medical event prediction [2].”
MethodA Neural Approach to Source Dependence Based Context Model for Statistical Machine Translation
Chen, K., Zhao, T., Yang, M., Liu, L., Tamura, A., Wang, R., Utiyama, M., & Sumita, E. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2772846 · Cited by 70
Audio-based snore detection using deep neural networks
Xie, J., Aubert, X. L., Long, X., van Dijk, J., Arsenali, B., Fonseca, P. M., & Overeem, S. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105917 · Cited by 75
Comparison and Analysis of RNN-LSTMs and CNNs for Social Reviews Classification
Bodapati, S., Bandarupally, H., Shaw, R. N., & Ghosh, A. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-33-6919-1_4 · Cited by 76
Multi-classification of arrhythmias using a HCRNet on imbalanced ECG datasets
Luo, X., Yang, L., Cai, H., Tang, R., Chen, Y., & Li, W. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106258 · Cited by 67
EditSum: A Retrieve-and-Edit Framework for Source Code Summarization
Li, J. A., Li, Y., Li, G., Hu, X., Xia, X., & Jin, Z. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678724 · Cited by 62
“Iyer et al. [12] used a Recurrent Neural Network (RNN) [37] with an attention mechanism as a decoder to generate code summaries and achieved good results on C# and SQL summaries.”
MethodNeural Architecture Search Benchmarks: Insights and Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · IEEE Access · doi:10.1109/access.2023.3253818 · Cited by 64
“The benchmark generated around 14k models and their training performance on the Penn Tree Bank (PTB) dataset [103], 4k of them are trained thrice with different settings, and the rest are trained only once.”
MethodRecurrent Neural Networks with External Memory for Spoken Language Understanding
Peng, B., Yao, K., Li, J., & Wong, K. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-25207-0_3 · Cited by 68
“Specifically, recurrent neural networks (RNNs) based methods have shown strong performances, in language modeling [16], language understanding [25], and machine translation [7,5] tasks.”
BackgroundContinuous N-gram Representations for Authorship Attribution
Sari, Y., Vlachos, A., & Stevenson, M. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2043 · Cited by 63
Personalized Research Paper Recommendation using Deep Learning
Hassan, H. A. M. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3079628.3079708 · Cited by 68
“For instance, in [15] and [12], the authors showed that LSTM can be used to build a language model and assess semantic similarity between sentences.”
BackgroundA hierarchical and regional deep learning architecture for image description generation
Kinghorn, P., Zhang, L., & Shao, L. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.09.013 · Cited by 69
Implicit Heterogeneous Features Embedding in Deep Knowledge Tracing
Yang, H., & Cheung, L. P. (2017) · Cognitive Computation · doi:10.1007/s12559-017-9522-0 · Cited by 66
“Recently, deep RNNs such as the long short-term memory (LSTM) and the gated recurrent unit (GRN) have been intensively investigated and proved their effectiveness in solving various real-world applications such as speech recognition, language modeling, and knowledge tracing [8, 20, 34, 38].”
BackgroundSequential Learning-Based Energy Consumption Prediction Model for Residential and Commercial Sectors
Haq, I. U., Ullah, A., Khan, S. U., Khan, N., Lee, M. Y., Rho, S., & Baik, S. W. (2021) · Mathematics · doi:10.3390/math9060605 · Cited by 70
“In ConvLSTM, the functionality and architecture are the same as LSTM, but the ConvLSTM takes input in 3D tensors fashion, and it preserves the spatial information [45].”
BackgroundDepth-Gated Recurrent Neural NetworksPreprint
Yao, K., Cohn, T., Vylomova, K., Duh, K., & Dyer, C. (2015) · arXiv (Cornell University) · Cited by 67
Training DNNs with Hybrid Block Floating PointPreprint
Oliveira, D. L. D., Lin, T., Jaggi, M., & Falsafi, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01526 · Cited by 93
“We also evaluate language modeling tasks with Penn Tree Bank(PTB) dataset [24].”
MethodLearning Sense-specific Word Embeddings By Exploiting Bilingual Resources
Guo, J., Che, W., Wang, H., & Liu, T. (2014) · International Conference on Computational Linguistics · Cited by 63
Goel, P., Kulshreshtha, D., Jain, P., & Shukla, K. K. (2017) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/w17-5207 · Cited by 62
“Using Recurrent Neural Networks (RNN) has become a very common technique for various NLP based tasks like language modeling (Mikolov et al., 2010).”
MethodLee, D.-H., Lim, M., Park, H.-S., Kang, Y., Park, J., Jang, G., & Kim, J. (2017) · China Communications · doi:10.1109/cc.2017.8068761 · Cited by 68
“A LM based on Recurrent Neural Network (RNN) using word vectors that express words as vectors is applied to N-gram rescoring and shows a higher performance than the N-gram language model [7][8][9].”
MethodDGeoSegmenter: A dictionary-based Chinese word segmenter for the geoscience domain
Qiu, Q., Xie, Z., Wu, L., & Li, W. (2018) · Computers & Geosciences · doi:10.1016/j.cageo.2018.08.006 · Cited by 63
Ham, S. W., Cho, J.-H., Park, S., & Kim, D. (2021) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981211003896 · Cited by 63
“As long as the personal mobility prediction problem handles spatiotemporal data, the recurrent neural network (RNN) model can provide an advantage in predictive ability (9).”
BackgroundEnergon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse Attention
Zhou, Z., Liu, J., Gu, Z., & Sun, G. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3170848 · Cited by 92
“Unlike traditional DNNs [27], [34], [56], transformer models are generally built by stacking several transformer blocks.”
MethodHarnessing large vision and language models in agriculture: a review
Zhu, H., Qin, S., Su, M., Lin, C., Li, A., & Gao, J. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1579355 · Cited by 120
“Neural Language Models (NLM): NLMs utilize various neural networks and are more effective than SLMs (Bengio et al., 2000; Mikolov et al., 2010; Sundermeyer et al., 2012).”
Numeracy for language models: Evaluating and improving their ability to predict numbers
Spithourakis, G., & Riedel, S. (2018) · UCL Discovery (University College London) · Cited by 63
Su, P.-H., Vandyke, D., Gašić, M., Kim, D., Mrkšić, N., Wen, T.-H., & Young, S. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-456 · Cited by 63
“It has been shown to be successful in various natural language processing tasks such as language modelling [21, 22, 23] and spoken language understanding (SLU) [24].”
BackgroundADMM for Efficient Deep Learning with Global Convergence
Wang, J., Yu, F., Chen, X., & Zhao, L. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330936 · Cited by 86
Towards Query-Efficient Adversarial Attacks Against Automatic Speech Recognition Systems
Wang, Q., Zheng, B., Li, Q. L., Shen, C. Q., & Ba, Z. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3026543 · Cited by 63
“processing [4].”
TBCNN: A Tree-Based Convolutional Neural Network for Programming Language Processing.Preprint
Mou, L., Li, G., Jin, Z. G., Zhang, L., & Wang, T. (2014) · arXiv (Cornell University) · Cited by 63
Feedforward Sequential Memory Networks: A New Structure to Learn Long-term DependencyPreprint
Zhang, S., Liu, C., Jiang, H., Wei, S., Dai, L.-R., & Hu, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.08301 · Cited by 63
“As for RNN-LMs, we have trained a simple RNN with one hidden layer of 600 units using the toolkit in (Mikolov et al., 2010).”
BackgroundMethodMultilingual Models for Compositional Distributed SemanticsPreprint
Hermann, K. M., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.4641 · Cited by 103
“They have received a lot of attention in recent years (Collobert and We-ston, 2008; Mnih and Hinton, 2009; Mikolov et al., 2010, inter alia ) and have achieved state of the art performance in language modelling.”
BackgroundA Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06714 · Cited by 62
“We give a brief overview of the Recurrent Language Model (RLM) (Mikolov et al., 2010) architecture that our models extend.”
BackgroundMethodInfluentialAdaCliP: Adaptive Clipping for Private SGD
Pichapati, V., Suresh, A. T., Yu, F. X., Reddi, S. J., & Kumar, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07643 · Cited by 60
“Machine learning models are widely deployed in various applications such as image classification [1, 2], natural language processing [3, 4], and recommendation systems [5].”
BackgroundHow Powerful are Performance Predictors in Neural Architecture Search?Preprint
White, C., Zela, A., Ru, B., Liu, Y., & Hutter, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01177 · Cited by 178
“In our experiments, we use the NAS-Bench-NLP dataset, which consists of 14 000 architectures drawn uniformly at random and trained on Penn Tree Bank [40] for 50 epochs.”
MethodText Analysis for Psychology: Methods, Principles, and PracticesPreprint
Kennedy, B., Ashokkumar, A., Boyd, R. L., & Dehghani, M. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/h2b8t · Cited by 70
EURASIP Journal on Audio, Speech, and Music Processing
(2013) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1007/13636.1687-4722 · Cited by 63
Learning Simpler Language Models with the Differential State Framework
Ororbia, A. G., Mikolov, T., & Reitter, D. (2017) · Neural Computation · doi:10.1162/neco_a_01017 · Cited by 80
“Simple models, largely based on the original Elman RNN (Elman, 1990), have often been shown to perform quite well in language modeling tasks (Mikolov et al., 2010, 2011).”
MethodSelf-citationCompositional coding capsule network with k-means routing for text classification
Ren, H., & Lu, H. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.05.028 · Cited by 73
Bao, Q., Ni, L., & Liu, J. (2020) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/3373017.3373049 · Cited by 69
“They have used RNN [12] to capture the semantic level connection and Attention [19] to follow the entity and relationship.”
MethodLong-Term Video Question Answering via Multimodal Hierarchical Memory Attentive Networks
Yu, T., Yu, J., Zhou, Y., Huang, Q., & Tian, Q. C. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.2995959 · Cited by 71
IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction
Yang, J., Chen, Y., Du, S., Chen, B., & Príncipe, J. C. (2024) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2024.3359237 · Cited by 78
“Unlike traditional RNNs [80], an LSTM network is well-suited to learn from experience to classify, process and predict time series given very long time lags.”
Backgroundedge2vec: Representation learning using edge semantics for biomedical knowledge discovery
Gao, Z., Fu, G., Ouyang, C., Tsutsui, S., Liu, X., Yang, J. J., Gessner, C. R., Foote, B., Wild, D. J., Ding, Y., et al. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2914-2 · Cited by 83
“to project nodes into a latent space with a pre-dened dimensionality. Recently, deep neural network[28] based representation learning has been widely used in the natural language processing. Word2vec[29] is the deep learning model developed by Google to represent a word in a low dimension dense vector, which Gao et al…”
MethodDeepAM: Migrate APIs with Multi-modal Sequence to Sequence Learning
Gu, X., Zhang, H., Zhang, D., & Kim, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/514 · Cited by 62
Short-Term Bitcoin Price Fluctuation Prediction Using Social Media and Web Search Data
Mittal, A., Dhiman, V., Singh, A., & Prakash, C. (2019) · International Conference on Contemporary Computing · doi:10.1109/ic3.2019.8844899 · Cited by 67
“Recurrent Neural Network is a generalization of feedforward neural network that has an internal memory [15].”
BackgroundKnowledge Enhanced Neural Fashion Trend Forecasting
Ma, Y., Ding, Y., Yang, X., Liao, L., Wong, W. K., & Chua, T. (2020) · International Conference on Multimedia Retrieval · doi:10.1145/3372278.3390677 · Cited by 82
“…the success of deep neural networks in a wide range of tasks, RNN, especially its variant LSTM [12], has shown its superiority in modeling sequential data and achieved superior performance in various applications of NLP [29], speech recognition [8], and also time series forecasting [4, 19, 36].”
BackgroundWriting Stories with Help from Recurrent Neural Networks
Roemmele, M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9810 · Cited by 59
“Mikolov et al. (Mikolov et al. 2010) showed that RNNs encode more accurate language models than traditional n-gram statistics, as measured by performance on a standard speech recognition task.”
MethodUnsupervised Text Normalization Using Distributed Representations of Words and Phrases
Sridhar, V. K. R. (2015) · Workshop on Vector Space Modeling for Natural Language Processing (VSM-NLP) · doi:10.3115/v1/w15-1502 · Cited by 60
“…in the context of a deep neural network framework for a given task (Bengio et al., 2003; Collobert and Weston, 2008; Bengio et al., 2009; Turian et al., 2010; Mikolov et al., 2010), recent work in (Mikolov et al., 2013) has also shown that they can also be induced by using simple log-linear models.”
BackgroundMachine Learning Based Recommendation of Method Names: How Far are We
Jiang, L. C., Liu, H., & Jiang, H. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00062 · Cited by 72
“[27] first employ RNN model [33] for code recommendation and White et al.”
MethodMachine Learning for Detecting Data Exfiltration: A Review
Sabir, B., Ullah, F., Babar, M. A., & Gaire, R. (2021) · ACM Computing Surveys · doi:10.1145/3442181 · Cited by 98
“Neural Network [41] (NN), NN with Multi-layered Perception [42] (MLP), Convolutional NN [43] (CNN), Recurrent NN [44] (RNN), Gated RNN [45] (GRU) and Long Short Term Memory [46] (LSTM).”
Arabic sentiment analysis using recurrent neural networks: a review
Alhumoud, S. O., & Wazrah, A. A. A. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09989-9 · Cited by 73
“In an RNN (Elman 1990; Kobayashi et al. 2010; Ayyadevara 2018), the inputs are a hidden state vector for the previous timestamp ( ht -1 ) and the input state vector at a certain time ( et ) (see Fig.”
MethodMohammed, B., & Gbashi, E. K. (2021) · Engineering and Technology Journal · doi:10.30684/etj.v39i7.1695 · Cited by 65
“The returns loops are recurrent rotation over time or sequence [24].”
BackgroundOn the Reproducibility and Replicability of Deep Learning in Software Engineering
Liu, C., Gao, C., Xia, X., Lo, D., Grundy, J., & Yang, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3477535 · Cited by 61
Multimodal fusion diagnosis of depression and anxiety based on CNN-LSTM model
Xie, W., Wang, C., Lin, Z., Luo, X., Chen, W., Xu, M., Liang, L., Liu, X., Wang, Y., Luo, H., et al. (2022) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102128 · Cited by 72
“We also replace LSTM with Recurrent Neural Network (RNN) (Mikolov et al., 2010) for comparison, and the LSTM module works better.”
MethodLanguage modeling with sum-product networks
Cheng, W., Kok, S., Pham, H. V., Chieu, H. L., & Chai, K. M. A. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-476 · Cited by 61
“We compared our SPNs to an interpolated 5-gram model with modified Kneser-Ney smoothing and no count cutoffs (KN5) [3], the log-bilinear model [4], feedforward neural networks [5], syntactical neural networks [8], recurrent neural networks (RNN) [6], and LDA-augmented RNN [9], all of which are described in Section 1.”
BackgroundMethodInfluentialA Spatial-Temporal-Semantic Neural Network Algorithm for Location Prediction on Moving Objects
Wu, F., Fu, K., Wang, Y., Xiao, Z., & Fu, X. (2017) · Algorithms · doi:10.3390/a10020037 · Cited by 60
“This section first describes two datasets prepared for evaluation and several evaluation metrics, then introduces several compared algorithms, including a feature selection method and three models Finally, Finally, the experiments are conducted and the results are showed.”
BackgroundMethodLiSHT: Non-parametric Linearly Scaled Hyperbolic Tangent Activation Function for Neural Networks
Roy, S. K., Manna, S., Dubey, S. R., & Chaudhuri, B. B. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-31407-0_35 · Cited by 82
Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning
Cheng, J., & Kartsaklis, D. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1177 · Cited by 80
“With regard to compositional settings (discussed in the next section), Hashimoto et al. (2014) use dependencybased word embeddings by employing a hinge loss objective, while Hermann and Blunsom (2013) condition their objectives on the CCG types of the involved words.”
BackgroundTensorLayer: A Versatile Library for Efficient Deep Learning Development
Dong, H., Supratak, A., Mai, L., Liu, F., Oehmichen, A., Yu, S., & Guo, Y. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3129391 · Cited by 61
Residual Convolutional CTC Networks for Automatic Speech RecognitionPreprint
Wang, Y., Deng, X., Pu, S., & Huang, Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07793 · Cited by 92
“In the last few years, Recurrent Neural Networks (RNNs) have been widely used for sequential modeling due to its capability of modeling long history (Mikolov et al., 2010).”
BackgroundWord Emdeddings through Hellinger PCAPreprint
Lebret, R., & Collobert, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5542 · Cited by 99
“er authors: Collobert and Weston (2008) designed a neural language model which eliminates the linear dependency on vocabulary size, Mnih and Hinton (2008) proposed a hierarchical linear neural model, Mikolov et al. (2010) investigated a recurrent neural network architecture for language modeling. Such architectures be…”
BackgroundAfandizadeh, S., Abdolahi, S., & Mirzahossein, H. (2024) · Journal of Advanced Transportation · doi:10.1155/2024/9981657 · Cited by 64
A Deep Architecture for Semantic Parsing
Grefenstette, E., Blunsom, P., de Freitas, N., & Hermann, K. M. (2014) · Proceedings of the ACL 2014 Workshop on Semantic Parsing · doi:10.3115/v1/w14-2405 · Cited by 57
“…(Yih et al., 2011; Lauly et al., 2014; Hermann and Blunsom, 2014a), frame-semantic parsing (Hermann et al., 2014), and machine translation (Mikolov et al., 2010; Kalchbrenner and Blunsom, 2013a), inter alia, combining two empirically successful deep learning models to form a new…”
BackgroundMethodInfluentialRecurrent neural network based language model personalization by social network crowdsourcing
Wen, T.-H., Heidel, A., Lee, H.-Y., Tsao, Y., & Lee, L.-S. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-621 · Cited by 67
“Moreover, RNNLMs [26, 27, 28], which memorizes arbitrary length of histories in a recurrent structure, can model long range dependencies in an elegant way.”
BackgroundFairness in Large Language Models: A Taxonomic Survey
Chu, Z., Wang, Z., & Zhang, W. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3682112.3682117 · Cited by 59
“Language models are computational models with the capacity to comprehend and generate human language [115, 93].”
BackgroundGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
“More advanced methods used various neural network architectures, such as feedforward networks [37] and recurrent networks [38], for the language modeling task.”
Contextual spoken language understanding using recurrent neural networks
Shi, Y., Yao, K., Chen, H., Pan, Y., Hwang, M.-Y., & Peng, B. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178977 · Cited by 65
“In recent years, Recurrent Neural Network (RNN) has demonstrated outstanding performance in a variety of natural language processing tasks [6–14].”
BackgroundSimple Fusion: Return of the Language Model
Stahlberg, F., Cross, J., & Stoyanov, V. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6321 · Cited by 86
“Language model type So far we have used recurrent neural network language models (Mikolov et al., 2010, RNN-LM) with LSTM cells in all our experiments.”
MethodInfluentialMachine Learning Based Neural Network Solving Methods for the FDTD Method
Yao, H. M., & Jiang, L. J. (2018) · Digest - IEEE Antennas and Propagation Society. International Symposium · doi:10.1109/apusncursinrsm.2018.8608745 · Cited by 60
“Recurrent Neural Network (RNN) has been widely utilized to solve problems related to sequential information [5].”
MethodNeural networks for metamodelling the hygrothermal behaviour of building components
Tijskens, A., Roels, S., & Janßen, H. (2019) · Building and Environment · doi:10.1016/j.buildenv.2019.106282 · Cited by 63
Efficient Nearest Neighbor Language Models
He, J., Neubig, G., & Berg-Kirkpatrick, T. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.461 · Cited by 97
“Modern neural language models (NLMs) based on recurrent (Mikolov et al., 2010; Sundermeyer et al., 2012) or selfattentional (Vaswani et al.”
BackgroundWeng, C., Cui, J., Wang, G., Wang, J., Yu, C., Su, D., & Yu, D. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1030 · Cited by 58
“The performance of speech recognition has been improved dramatically since deep neural networks (DNNs) were applied to its main components such as acoustic model [1–6], language model [7, 8], pronunciation model [9], etc.”
MethodOn the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling
Gerz, D., Vulić, I., Ponti, E. M., Reichart, R., & Korhonen, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1029 · Cited by 97
“guage understanding, with applications in speech recognition (Mikolov et al., 2010), text summarisation (Filippova et al.”
BackgroundInfluentialPositionless aspect based sentiment analysis using attention mechanism
Yadav, R. K., Jiao, L., Goodwin, M., & Granmo, O. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107136 · Cited by 72
Learning to Generate Market Comments from Stock Prices
Murakami, S., Watanabe, A., Miyazawa, A., Goshima, K., Yanase, T., Takamura, H., & Miyao, Y. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1126 · Cited by 58
Identifying attack and support argumentative relations using deep learning
Cocarascu, O., & Toni, F. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1144 · Cited by 83
“Recurrent Neural Networks (RNNs) (Elman, 1990; Mikolov et al., 2010) are a type of neural networks in which the hidden layer is connected to itself so that the previous hidden state is used along with the input at the current step.”
BackgroundA Memory Network Approach for Story-Based Temporal Summarization of 360° Videos
Lee, S., Sung, J., Yu, Y., & Kim, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00153 · Cited by 80
A Review-Driven Neural Model for Sequential Recommendation
Li, C., Niu, X., Luo, X., Chen, Z., & Quan, C. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/397 · Cited by 59
Zhu, H., Wang, X., Chen, X., & Zhang, L. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103178 · Cited by 62
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.