Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 801–900 of 1,058
Lee, M., Lee, Y.-H., Song, J., Kim, G., Jo, Y., Min, H., Kim, C. H., & Park, Y. (2020) · eLife · doi:10.7554/elife.49023 · Cited by 81
Götz, F. M., Maertens, R., Loomba, S., & van der Linden, S. (2024) · Psychological Methods · doi:10.1037/met0000540 · Cited by 77
Fully Convolutional Speech Recognition
Zeghidour, N., Xu, Q., Liptchinsky, V., Usunier, N., Synnaeve, G., & Collobert, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06864 · Cited by 75
“In end-to-end speech recognition however, recurrent architectures are still prevalent for acoustic and/or language modeling [5, 6, 7, 8, 9].”
BackgroundPereira, J., & Saraiva, F. D. O. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107085 · Cited by 81
Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework
Banchs, R. E., D’Haro, L. F., & Li, H. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2405751 · Cited by 77
“T HEmathematical metaphor offered by the geometric concept of distance in continuous space with respect to semantics and meaning has been proven to be useful in many monolingual and cross-language natural language processing applications such as document classification [1], information retrieval [2], word similarity […”
BackgroundEncoder-decoder with focus-mechanism for sequence labelling based spoken language understanding
Zhu, S., & Yu, K. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953243 · Cited by 83
“Recently, motivated by a number of very successful continuous-space, neural network and deep learning approaches [5, 6], many neural network architectures have been applied to this task, such as simple recurrent neural networks (RNNs) [7, 8, 9], convolutional neural networks (CNNs) [10], long short-term memory (LSTM)…”
BackgroundPersonalized DeepInf: Enhanced Social Influence Prediction with Deep Learning and Transfer Learning
Leung, C. K.-S., Cuzzocrea, A., Mai, J. J., Deng, D., & Jiang, F. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9005969 · Cited by 86
“For example, DeepCas [21] was developed to study macro social influence patterns using recurrent neural network ( RNN ) [22].”
MethodShen, Z., & Chen, S. (2020) · Security and Communication Networks · doi:10.1155/2020/8858010 · Cited by 101
Chung, S., Moon, S., Kim, J.-H., Kim, J., Lim, S. P., & Chi, S. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105020 · Cited by 99
Kamtam, S. B., Lu, Q., Bouali, F., Haas, O. C. L., & Birrell, S. A. (2024) · Sensors · doi:10.3390/s24123957 · Cited by 124
“Among these, RNN [74,117] and LSTM [55,69,74,118,119] are more suitable for time-series prediction.”
BackgroundA Survey of Robot Intelligence with Large Language Models
Jeong, H., Lee, H., Kim, C., & Shin, S. (2024) · Applied Sciences · doi:10.3390/app14198868 · Cited by 120
Peephole: Predicting Network Performance Before TrainingPreprint
Deng, B., Yan, J., & Lin, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.03351 · Cited by 140
“Inspired by the success of recurrent networks in sequential modeling, e.g . in language modeling [19] and video analytics [25], we choose to explore recurrent networks in our problem.”
DeepCare: A Deep Dynamic Memory Model for Predictive Medicine
Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.00357 · Cited by 74
“Another approach to learn the embedding matrix E is language modeling with an RNN [47].”
BackgroundAugmenting Self-attention with Persistent Memory
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1907.01470 · Cited by 75
“Language modeling has been dominated by neural networks with models either based on feedforward networks [4] or recurrent networks [27].”
BackgroundDoes Multimodality Help Human and Machine for Translation and Image Captioning?
Çağlayan, O., Aransa, W., Wang, Y., Masana, M., García-Martínez, M., Bougares, F., Barrault, L., & van de Weijer, J. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2358 · Cited by 96
Multi-accent speech recognition with hierarchical grapheme based models
Rao, K., & Sak, H. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953071 · Cited by 89
“Similarly, language models have been improved with RNNs [10, 11, 12].”
MethodSequence to Better Sequence: Continuous Revision of Combinatorial Structures.
Mueller, J. W., Gifford, D. K., & Jaakkola, T. (2017) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 116
A Survey on Opinion Mining: From Stance to Product Aspect
Wang, R., Zhou, D., Jiang, M., Si, J., & Yang, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2906754 · Cited by 85
“The RNCRF contains a dependencytree RNN and a Condition Random Field as output layer.”
BackgroundMethodInfluentialA survey for trust-aware recommender systems: A deep learning perspective
Dong, M., Yuan, F., Yao, L., Wang, X., Xu, X., & Zhu, L. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108954 · Cited by 100
EdgeRec: Recommender System on Edge in Mobile Taobao
Gong, Y., Jiang, Z., Feng, Y., Hu, B., Zhao, K., Liu, Q., & Ou, W. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412700 · Cited by 115
“Based on the recurrent computation characteristics of RNN, we model online incoming user behaviors synchronously and real-time as discussed in Sec.”
BackgroundMethodInfluentialWhen the Timeline Meets the Pipeline: A Survey on Automated Cyberbullying Detection
Elsafoury, F., Katsigiannis, S., Pervez, Z., & Ramzan, N. (2021) · IEEE Access · doi:10.1109/access.2021.3098979 · Cited by 73
“[21] and [11] used Long Short Term Memory (LSTM) models, which are a variation of Recurrent Neural Network (RNN) [23] models, to detect cyberbullying.”
MethodUsing Artificial Intelligence for Space Challenges: A Survey
Russo, A., & Lax, G. (2022) · Applied Sciences · doi:10.3390/app12105106 · Cited by 94
A comprehensive review of recommender systems: Transitioning from theory to practice
Raza, S., Rahman, M., Kamawal, S., Toroghi, A., Raval, A., Navah, F., & Kazemeini, A. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100849 · Cited by 193
A Constructive Prediction of the Generalization Error Across Scales
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., & Shavit, N. (2019) · arXiv · doi:10.48550/arxiv.1909.12673 · Cited by 74
“1 DATASETS We evaluate on several datasets commonly used for (word-level) language modeling: Penn Treebank (Mikolov et al., 2010), WikiText-2 (Bradbury et al.”
MethodThe Inside-Outside Recursive Neural Network model for Dependency Parsing
Le, P., & Zuidema, W. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1081 · Cited by 73
“Thanks to recent advances in deep learning, this approach has recently started to look very promising again, with state-of-the-art results in sentiment analysis (Socher et al., 2013), language modelling (Mikolov et al., 2010), and other tasks.”
BackgroundMethodYan, R. (2016) · International Joint Conference on Artificial Intelligence · Cited by 74
Towards Micro-video Understanding by Joint Sequential-Sparse Modeling
Liu, M., Nie, L., Wang, M., & Chen, B. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123341 · Cited by 79
“In addition, it has been very successful in variety of temporal sequence tasks, such as language modeling [12, 29], translation [26], dialog system [11], time series prediction [37], rhythm learning [13], visual question answering [27], handwriting recognition [15], and protein homology detection…”
BackgroundRybalkin, V., Pappalardo, A., Ghaffar, M. M., Gambardella, G., Wehn, N., & Blott, M. (2018) · International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2018.00024 · Cited by 92
Biomedical Named Entity Recognition based on Deep Neutral Network
Yao, L., Liu, H., Liu, Y., Li, X., & Anwar, M. W. (2015) · International Journal of Hybrid Information Technology · doi:10.14257/ijhit.2015.8.8.29 · Cited by 97
“The second way is to view word representation as a part of training a neural network language model [20-23].”
BackgroundPower Load Forecasting Using BiLSTM-Attention
Du, J., Cheng, Y., Zhou, Q., Zhang, J., Zhang, X., & Li, G. (2020) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/440/3/032115 · Cited by 79
“The LSTM neural network evolved from the RNN neural network [21].”
MethodStructured Output Layer Neural Network Language Models for Speech Recognition
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J.-L., & Yvon, F. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2215599 · Cited by 76
“The ability of feed-forward NNLMs to improve system performance with increasing of context is in line with results obtained with recurrent networks that implicitly take into account the full history to predict a given word [26].”
ResultProbabilistic short-term wind power forecasting based on deep neural networks
Wu, W., Chen, K., Qiao, Y., & Lu, Z. (2016) · International Conference on Probabilistic Methods Applied to Power Systems (PMAPS) · doi:10.1109/pmaps.2016.7764155 · Cited by 74
“The implementation of RNN has been quite effective in tasks including speech recognition, linguistic modelling and machine translation [16], and the nature of RNN is suitable for the wind power forecasting task due to its intrinsic temporal dependency [17].”
MethodOgawa, A., & Hori, T. (2017) · Speech Communication · doi:10.1016/j.specom.2017.02.009 · Cited by 77
“However, an (R)NN projects a sparse feature vector into a low-dimensional continuous space (Mikolov et al., 2010, 2011; Hori et al., 2014; Schwenk, 2007; Yao et al., 2013; Mesnil et al., 2013; Yao et al., 2014).”
BackgroundMethodRACE-Net: A Recurrent Neural Network for Biomedical Image Segmentation
Chakravarty, A., & Sivaswamy, J. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2852635 · Cited by 76
“commonly used RNN architecture ([22] for example) where the output of an individual neuron is fed back onto itself to update its hidden state.”
MethodDefinition Modeling: Learning to Define Word Embeddings in Natural Language
Noraset, T., Liang, C., Birnbaum, L., & Downey, D. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.10996 · Cited by 150
“Thedefinition models explored in this paper are based on a recurrent neural network language model (RNNLM) (Mikolov et al. 2010).”
InfluentialCollaborative Filtering with Recurrent Neural NetworksPreprint
Devooght, R., & Bersini, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07400 · Cited by 93
“RNNs are commonly used for language modeling, where they are trained to learn sequences of words [11].”
BackgroundChen, X., Liu, X., Gales, M., & Woodland, P. C. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7179005 · Cited by 87
“In state-of-the-art ASR systems, NNLMs are often linearly interpolated with n-gram LMs to obtain both a good context coverage and strong generalisation [1, 5, 23, 24, 25, 26].”
BackgroundContinuous multilinguality with language vectors
Östling, R., & Tiedemann, J. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2102 · Cited by 72
Statistical downscaling of precipitation using long short-term memory recurrent neural networks
Misra, S., Sarkar, S., & Mitra, P. (2017) · Theoretical and Applied Climatology · doi:10.1007/s00704-017-2307-2 · Cited by 80
“LSTMs have achieved state of the art results in sequence related problems like speech recognition (Hinton et al. 2012; Dahl et al. 2012), language modeling (Mikolov et al. 2010), machine translation (Sutskever et al. 2014; Bahdanau et al. 2014), etc.”
BackgroundMethodAn Advanced Deep Generative Framework for Temporal Link Prediction in Dynamic Networks
Yang, M., Liu, J., Chen, L., Zhao, Z., Chen, X., & Shen, Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2920268 · Cited by 73
“However, these meth-ods merely explore the spatial features of the static network at each timestamp by deep belief network (DBN) [11] or model the network evolving patterns across the time-varying dynamic networks by recurrent neural network (RNN) [12].”
BackgroundRecurrent Convolutional Neural Networks for Discourse Compositionality
Kalchbrenner, N., & Blunsom, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.3584 · Cited by 178
Bloom’s Learning Outcomes’ Automatic Classification Using LSTM and Pretrained Word Embeddings
Shaikh, S., Daudpotta, S. M., & Imran, A. S. (2021) · IEEE Access · doi:10.1109/access.2021.3106443 · Cited by 70
“The major reason for this success is the use of word embeddings [37], [38] and deep neural network architectures [39].”
BackgroundDeep-Learning-Based Diagnosis and Prognosis of Alzheimer’s Disease: A Comprehensive Review
Sharma, R., Goel, T., Tanveer, M., Lin, C., & Murugan, R. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3254209 · Cited by 88
Han, Y. M., Fan, C., Xu, M., Geng, Z., & Zhong, Y. (2019) · Applied Thermal Engineering · doi:10.1016/j.applthermaleng.2019.114072 · Cited by 73
Muffin: testing deep learning libraries via neural architecture fuzzing
Gu, J., Luo, X., Zhou, Y., & Wang, X. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510092 · Cited by 71
“[16] propose a set of similarity metrics and coverage criteria to analyze stateful DL systems such as Recurrent Neural Networks (RNNs) [39].”
BackgroundReal-time Arm Skeleton Tracking and Gesture Inference Tolerant to Missing Wearable Sensors
Liu, Y., Li, Z., Liu, Z., & Wu, K. (2019) · Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3307334.3326109 · Cited by 71
“With the aforementioned inputs, we propose to first apply a recurrent neural network (RNN) to extract the input’s individual dynamic motion features [30] and then gradually concatenate them with additional RNNs for further extracting characteristics from an aggregated view, which are finally used for the gesture infer…”
BackgroundMethodPersistent RNNs: stashing recurrent weights on-chip
Diamos, G., Sengupta, S., Catanzaro, B., Chrzanowski, M., Coates, A., Elsen, E., Engel, J., Hannun, A., & Satheesh, S. (2016) · International Conference on Machine Learning · Cited by 100
Semantic Similarity of Arabic Sentences with Word Embeddings
Nagoudi, E. M. B., & Schwab, D. (2017) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/w17-1303 · Cited by 71
Chen, C., Xing, Z., Liu, Y., & Xiong, K. O. L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2896123 · Cited by 82
“Compared with traditional n-gram language model [22], an RNN-based language model can predict a next word by preceding words with variant distance rather”
MethodA Unified Framework for Multilingual Speech Recognition in Air Traffic Control Systems
Lin, Y., Guo, D., Zhang, J., Chen, Z., & Yang, B. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3015830 · Cited by 75
A Brief Survey of Telerobotic Time Delay Mitigation
Farajiparvar, P., Ying, H., & Pandya, A. D. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.578805 · Cited by 90
“Recurrent Neural Networks (RNNs) was among the first type of neural networks method for time series prediction (Mikolov et al., 2010).”
MethodZhang, H., Xie, Z., Zarei, R., Wu, T., & Chen, K. (2021) · IEEE Access · doi:10.1109/access.2021.3095915 · Cited by 87
“, image recognition and natural language processing [1], [2].”
BackgroundLook Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 338
“Representative traditional language models include HMM (hidden Markov model) [48], n -gram [49], and RNN (recurrent neural networks) [50].”
Li, H., Xi, J., Hsu, C. H. C., Yu, B. X. B., & Zheng, X. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105179 · Cited by 101
Compression fractures detection on CT
Bar, A., Wolf, L., Amitai, O. B., Toledano, E., & Elnekave, E. (2017) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2249635 · Cited by 72
“RNN’s are commonly utilized in tasks such as video classification [20], language modeling [21], and speech recognition [22], among others.”
MethodInvestigation of recurrent neural networks in the field of sentiment analysis
Baktha, K., & Tripathy, B. K. (2017) · International Conference on Communications and Signal Processing (ICCSP) · doi:10.1109/iccsp.2017.8286763 · Cited by 81
“RNNs have proved their superiority in many NLP tasks including learning language models that can outperform ngrams [1-4].”
BackgroundMachine Learning and Radiogenomics: Lessons Learned and Future Directions
Kang, J., Rancati, T., Lee, S., Oh, J. H., Kerns, S. L., Scott, J. G., Schwartz, R. S., Kim, S. Y., & Rosenstein, B. S. (2018) · Frontiers in Oncology · doi:10.3389/fonc.2018.00228 · Cited by 76
“Deep Learning “Deep learning” describes a class of neural networks that has exploded in popularity in the recent years—particularly in the fields of computer vision (105) and natural language processing (106)—as larger training data sets have become available and computational processing resources have become more acc…”
BackgroundSelective Audio Adversarial Example in Evasion Attack on Speech Recognition System
Kwon, H., Kim, Y., Yoon, H., & Choi, D. (2019) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2019.2925452 · Cited by 81
“sive neural networks (RNNs) [14].”
A Unified Tagging Solution: Bidirectional LSTM Recurrent Neural Network with Word EmbeddingPreprint
Wang, P., Qian, Y., Soong, F. K., He, L., & Zhao, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.00215 · Cited by 110
“Word embedding can be obtained by training a neural network language model (Bengio et al., 2006), shallow neural network (Mikolov et al., 2013a; Pennington et al., 2014a; Collobert et al., 2011), or a recurrent neural network (Mikolov et al., 2010).”
BackgroundGenerative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Lyu, H., Lu, H., Mou, L., Li, W., Wright, J., Li, X., Li, X., Zhu, X., Wang, J., Yu, L., et al. (2018) · Remote Sensing · doi:10.3390/rs10030471 · Cited by 70
“The basic RNN with LSTM activation also performed well, with OA values approaching those for the deep-learning approach.”
MethodRecurrent neural network language model adaptation with curriculum learning
Shi, Y., Larson, M., & Jonker, C. M. (2014) · Computer Speech & Language · doi:10.1016/j.csl.2014.11.004 · Cited by 70
Throwing Darts in the Dark? Detecting Bots with Limited Data using Neural Data Augmentation
Jan, S. T. K., Hao, Q., Hu, T., Pu, J., Oswal, S., Wang, G., & Viswanath, B. (2020) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40000.2020.00079 · Cited by 99
“LSTM is a specialized Recurrent Neural Network (RNN) designed to capture the relationships of events in a sequence and is suitable to model sequential data [46], [47].”
BackgroundFull-Scale Information Diffusion Prediction With Reinforced Recurrent Networks
Yang, C. H., Wang, H., Tang, J., Shi, C., Sun, M., Cui, G., & Liu, Z. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3106156 · Cited by 73
“1) Recurrent Networks: RNNs have shown their effectiveness in sequential data modeling in many areas, such as natural language processing [43].”
BackgroundAutomatic Text Input Generation for Mobile Testing
Liu, P., Zhang, X., Pistoia, M., Zheng, Y., Marques, M., & Zeng, L. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.65 · Cited by 70
“[32], the RNN model is significantly more accurate than traditional statistical models such as the n-gram model and the hidden Markov model.”
MethodAutomatically Generating Rhythmic Verse with Neural Networks
Hopkins, J., & Kiela, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1016 · Cited by 70
“, 2006) have garnered signficant research interest for their ability to learn complex syntactic and semantic representations of natural language (Mikolov et al., 2010; Sutskever et al., 2014; Cho et al., 2014; Kim et al., 2015).”
BackgroundTable-to-Text: Describing Table Region With Natural Language
Bao, J., Tang, D., Duan, N., Yan, Z., Lv, Y., Zhou, M., & Zhao, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11944 · Cited by 70
“We implement a table conditional neural language model ( TC-NLM ) baseline, which is based on a recurrent neural network language model (Mikolov et al. 2010).”
MethodChen, S.-J., Subramanian, A. S., Xu, H., & Watanabe, S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1262 · Cited by 78
Zhang, J., Wu, J., Qiu, Y., Song, A., Li, W., Li, X., & Liu, Y. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106517 · Cited by 69
“9(c), the characteristic of the RNN is that it will be affected by the previous input while processing the current input, which can better process the time sequences [58].”
Xu, Z., Chou, J., Zhang, X. S., Luo, Y., Isakova, T., Adekkanattu, P., Ancker, J. S., Jiang, G., Kiefer, R. C., Pacheco, J. A., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103361 · Cited by 69
“Compared to conventional deep learning models such as Recurrent Neural Network (RNN) (Mikolov et al., 2010), MN models can explicitly control parts of information in the memory that will be extracted, which is much more flexible and interpretable.”
MethodAsyMo: scalable and efficient deep-learning inference on asymmetric mobile CPUs
Wang, M., Ding, S., Cao, T., Liu, Y., & Xu, F. (2021) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3447993.3448625 · Cited by 85
“The configurations for the two RNN models are: Char-RNN: hidden_size 512, layers 2, batch_size 1, input_size 65, length 40, cell LSTM; RNN classifier: hidden_size 1024, layers 3, batch_size 1, input_size 512, length 20, cell GRU.”
MethodInfluentialSingle‐Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty
van Schijndel, M., & Linzen, T. (2021) · Cognitive Science · doi:10.1111/cogs.12988 · Cited by 68
Lin, X., Zhu, X., Feng, M., Han, Y. M., & Geng, Z. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.148444 · Cited by 69
A Two-Stream Neural Network for Pose-Based Hand Gesture Recognition
Li, C., Li, S., Gao, Y., Zhang, X., & Li, W. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3126637 · Cited by 75
“(RNNs) [24], [25] to a hand skeleton sequence for classi-”
BackgroundExplainable anomaly detection framework for predictive maintenance in manufacturing systems
Choi, H., Kim, D., Kim, J., Kim, J., & Kang, P. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109147 · Cited by 95
A personalized recommendation framework based on MOOC system integrating deep learning and big data
Li, B., Li, G., Xu, J., Li, X., Liu, X., Wang, M., & Lv, J. (2023) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108571 · Cited by 92
NEMO: Next Career Move Prediction with Contextual Embedding
Li, L., Jing, H., Tong, H., Yang, J., He, Q., & Chen, B.-C. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054200 · Cited by 94
“In order to decode one’s career path from the context vector learned, we take advantage of the Recurrent Neural Network (RNN) due to its success in modeling sequential data [24, 3].”
MethodDual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization
Wang, Y., Xia, Y., Zhao, L., Bian, J., Qin, T., Liu, G., & Liu, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11999 · Cited by 78
“2015) proposed to train language models (Mikolov et al. 2010; Sundermeyer, Schlüter, and Ney 2012) independently with target-side monolingual sentences, and incorporate them into NMT models during decoding by re-scoring the candidate words according to the weighted sum of the scores provided by the translation model a…”
MethodMeaning to Form: Measuring Systematicity as Information
Pimentel, T., McCarthy, A. D., Blasí, D. E., Roark, B., & Cotterell, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1171 · Cited by 92
“, 2018)—and complex distributions Pr(wi | w<i) (Mikolov et al., 2010; Sundermeyer et al., 2012).”
BackgroundReference-Aware Language Models
Yang, Z., Blunsom, P., Dyer, C., & Ling, W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1197 · Cited by 68
“Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1850–1859 Copenhagen, Denmark, September 7–11, 2017. c©2017 Association for Computational Linguistics”
BackgroundRestoring ancient text using deep learning: a case study on Greek epigraphy
Assael, Y., Sommerschield, T., & PRAG, J. R. W. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1668 · Cited by 78
TheoryOn: A Design Framework and System for Unlocking Behavioral Knowledge Through Ontology Learning
Li, J., Larsen, K. R. T., & Abbasi, A. (2020) · MIS Quarterly · doi:10.25300/misq/2020/15323 · Cited by 91
“1998) and recurrent neural networks (RNN) (Mikolov et al. 2010) for modeling non-linear, complex patterns.”
BackgroundInfluentialMultimodal Transformer Network for Hyperspectral and LiDAR Classification
Zhang, Y., Xu, S., Hong, D., Gao, H., Zhang, C., Bi, M., & Li, C. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3283508 · Cited by 77
“In contrast to RNNs that process sequences in a fixed order, transformers use SA to compute a weighted sum of all the input embeddings at each position, where the weights are learned dynamically based on the similarity between each input and all other inputs in the sequence.”
BackgroundLarge language models for medicine: a survey
Zheng, Y., Gan, W., Chen, Z., Qi, Z., Liang, Q., & Yu, P. S. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02318-w · Cited by 131
“Compared to traditional methods dependent on multiple sequence alignment (MSA), HelixFold-Single is an innovative protein structure prediction method to overcome limitations and time constraints.”
Principles of structure building in music, language and animal song
Rohrmeier, M., Zuidema, W. H., Wiggins, G. A., & Scharff, C. (2015) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2014.0097 · Cited by 79
“In computational linguistics, vector grammars (which have a close relation to neural networks models of linguistic structure from the 1990s [150,151]) are experiencing a new wave of excitement following some successes with learning such grammars from data for practical natural language processing tasks [152–155].”
BackgroundChen, X., Song, L., & Yang, X. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2239260 · Cited by 74
PowerLSTM: Power Demand Forecasting Using Long Short-Term Memory Neural Network
Cheng, Y., Xu, C., Mashima, D., Thing, V. L. L., & Wu, Y. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69179-4_51 · Cited by 70
“Under this circumstance, LSTM is designed to solve the challenge of long-term dependences [13], which has demonstrated its practicality and sucess in tasks that are not solvable by other RNNs, such as continual prediction [10], speech recognition [12], language modeling [20], and translation [24].”
BackgroundResearch on text sentiment analysis based on CNNs and SVM
Chen, Y., & Zhang, Z. (2018) · IEEE Conference on Industrial Electronics and Applications (ICIEA) · doi:10.1109/iciea.2018.8398173 · Cited by 83
“In 2010, Mikolov [10,11] et al used the Log-Bilinear model to reduce the deep learning model to an acceptable range, and Word2vec tools for converting words into word vectors was introduced by Google; In the research work in 2011-2013, Socher et al [12-14] proposed a series of classification models based on recursive…”
MethodLiu, H., Liu, Z., Liu, S., Liu, Y., Bin, J., Shi, F., & Dong, H. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2018.2852632 · Cited by 71
“The status of the hidden layer will be updated across all time steps, and the update function can be abstracted by the following equation [32]:”
BackgroundGenerating Commit Messages from Diffs using Pointer-Generator Network
Liu, Q., Liu, Z., Zhu, H., Fan, H., Du, B., & Qian, Y. (2019) · 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00056 · Cited by 64
“, xT ), RNN [27] can learn the semantics of sentence and predict the next word according to previous words.”
BackgroundMultilingual Image Description with Neural Sequence ModelsPreprint
Elliott, D., Frank, S., & Hasler, E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.04709 · Cited by 100
“The core of our model is a Recurrent Neural Network model over word sequences, i.e., a neural language model ( LM ) (Mikolov et al., 2010).”
MethodSun, C., Fan, J., Chen, C., Li, W., & Chen, W. (2019) · IEEE Access · doi:10.1109/access.2019.2933814 · Cited by 77
“For example, the convolutional neural network (CNN) and deep belief network (DBN) exhibit powerful effects on feature extraction [10], [13], the recurrent neural network (RNN) has been proved to have good capacity in time series signal processing [14], [15].”
BackgroundChiu, P., Chang, J.-W., Lee, M.-C., Chen, C., & Lee, D. (2020) · IEEE Access · doi:10.1109/access.2020.2984383 · Cited by 75
“The Recurrent Neural Network (RNN) [53] is relatively suitable for processing sequence data (ie, highly correlated between contexts).”
MethodRobust Few-Shot Learning for User-Provided Data
Lu, J., Jin, S., Liang, J., & Zhang, C. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2984710 · Cited by 73
“DEEP learning has made impressive achievements in a broad spectrum of research fields, including language [1], vision [2] and speech [3].”
BackgroundDi, Y., & Wang, E. (2021) · Rock Mechanics and Rock Engineering · doi:10.1007/s00603-020-02314-w · Cited by 68
“In the previous RNN model, data flow from front to back through the hidden states, and the state of the current time step can also be determined using the state of the subsequent time step (Mikolov et al. 2010).”
BackgroundMethodDesigning self-assembling kinetics with differentiable statistical physics models
Goodrich, C. P., King, E. M., Schoenholz, S. S., Cubuk, E. D., & Brenner, M. P. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2024083118 · Cited by 80
“This method provides a more robust solution for large-scale optimization and is similar in spirit to truncated backpropagation used in language modeling (27) and metalearning (28).”
MethodBorkowski, L. B., Sorini, C. W., & Chattopadhyay, A. (2021) · Computers & Structures · doi:10.1016/j.compstruc.2021.106678 · Cited by 80
Machine fault detection methods based on machine learning algorithms: A review
Ciaburro, G. (2022) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2022534 · Cited by 90
Recurrent Neural Networks for Word Alignment Model
Tamura, A., Watanabe, T., & Sumita, E. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1138 · Cited by 74
“Other weights were randomly initialized to [ − 0 . used the RNNLM Toolkit 7 (Mikolov et al., 2010) with the default options.”
BackgroundMethodPardos, Z. A., Fan, Z., & Jiang, W. (2019) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-019-09218-7 · Cited by 89
“Neural networks have, in contrast, performed well in predicting sequences of implicit or behavioral data, beginning most notably with language [69] and extending to machine translation [94], text classification [59], speech recognition [34], speech synthesis [30], image captioning [68], image generation [36],”
Background
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.