Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 301–400 of 1,058
A survey on empathetic dialogue systems
Ma, Y., Nguyen, K. L., Xing, F. Z., & Cambria, E. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.011 · Cited by 220
Deep learning for smart agriculture: Concepts, tools, applications, and opportunities
Zhu, N., Liu, X., Liu, Z., Hu, K., Yingkuan, W., Tan, J., Huang, M., Zhu, Q., Ji, X., Jiang, Y., et al. (2018) · International journal of agricultural and biological engineering · doi:10.25165/j.ijabe.20181104.4475 · Cited by 284
Yoshioka, T., Ito, N., Delcroix, M., Ogawa, A., Kinoshita, K., Fujimoto, M., Yu, C., Fabian, W. J., Espi, M., Higuchi, T., et al. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404828 · Cited by 222
“In addition to the of fi cial 5K-word vocabulary tri-gram language model, we used an RNN-LM, which has been proven to improve recognition performance in many tasks [21, 22].”
MethodQuestion Answering Using Enhanced Lexical Semantic Models
Yih, W.-T., Chang, M., Meek, C., & Pastusiak, A. (2013) · Cited by 270
Word Embeddings through Hellinger PCA
Lebret, R., & Collobert, R. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1051 · Cited by 214
“…authors: Collobert and Weston (2008) designed a neural language model which eliminates the linear dependency on vocabulary size, Mnih and Hinton (2008) proposed a hierarchical linear neural model, Mikolov et al. (2010) investigated a recurrent neural network architecture for language modeling.”
BackgroundA LSTM based framework for handling multiclass imbalance in DGA botnet detection
Tran, D., Mac, H., Tong, V., Tran, H. A., & Nguyen, L. G. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.11.018 · Cited by 218
Software Engineering Challenges of Deep Learning
Arpteg, A., Brinne, B., Crnkovic-Friis, L., & Bosch, J. (2018) · 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00018 · Cited by 201
“Deep Learning (DL) has received considerable attention in recent years due to its success in areas such as computer vision tasks (e.g., object recognition [1] and image generation [2]) using convolution neural networks, natural language understanding using recurrent neural networks (RNN) [3] and machine strategic thin…”
MethodDeep Convolutional Neural Networks for Predominant Instrument Recognition in Polyphonic Music
Han, Y., Kim, J., & Lee, K. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2632307 · Cited by 179
“Finally, we conclude the paper in section VI.”
BackgroundGlobally Coherent Text Generation with Neural Checklist Models
Kiddon, C., Zettlemoyer, L., & Choi, Y. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1032 · Cited by 219
“Recurrent neural network (RNN) architectures have proven to be well suited for many natural language generation tasks (Mikolov et al., 2010; Mikolov et al., 2011; Sordoni et al., 2015; Xu et al., 2015; Wen et al., 2015; Mei et al., 2016).”
BackgroundAre You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 216
“For natural language, the Recurrent Neural Network (RNN [7, 27]) and the Long Short-Term Memory network (LSTM [12]) are also widely used in machine translation [13, 5, 35] and speech recognition [28].”
MethodCross-Project Transfer Representation Learning for Vulnerable Function Discovery
Lin, G., Zhang, J., Luo, W., Pan, L., Xiang, Y., De Vel, O., & Montague, P. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2821768 · Cited by 217
“In this paper, we leverage the recent advances in recurrent neural networks (RNNs) [14] to reveal deep sequential features that”
MethodA Proposed Sentiment Analysis Deep Learning Algorithm for Analyzing COVID-19 Tweets
Kaur, H., Ahsaan, S. U., Alankar, B., & Chang, V. (2021) · Information Systems Frontiers · doi:10.1007/s10796-021-10135-7 · Cited by 217
“Mikolov et al. (2010) deployed Recurrent Neural Networks with statistical language modeling to forecast the upcoming word within the textual data.”
MethodVAE-Stega: Linguistic Steganography Based on Variational Auto-Encoder
Yang, Z.-L., Zhang, S.-Y., Hu, Y.-T., Hu, Z.-W., & Huang, Y.-F. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3023279 · Cited by 215
“In the field of natural language processing, perplexity is a standard measure for language model testing [40].”
BackgroundSentiment Analysis by Capsules
Wang, Y., Sun, A., Han, J., Liu, Y., & Zhu, X. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186015 · Cited by 210
“A Recurrent Neural Network (RNN) is a class of artificial neural network where connections between units form a directed cycle.”
MethodInfluentialChang, H.-H., Song, H., Yi, Y., Zhang, J., He, H., & Liu, L. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2872441 · Cited by 213
“RNN are widely used in the natural language processing because understanding a sentence highly requires temporal relationship between words [19], [26].”
BackgroundUnderstanding Hidden Memories of Recurrent Neural Networks
Ming, Y., Cao, S., Zhang, R., Li, Z., Chen, Y., Song, Y., & Qu, H. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585721 · Cited by 207
Context-Aware Sequential Recommendation
Liu, Q., Wu, S., Wang, D., Li, Z., & Wang, L. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0135 · Cited by 205
Addressing Failure Prediction by Learning Model Confidence
Corbière, C., Thome, N., Bar-Hen, A., Cord, M., & Pérez, P. (2019) · arXiv · doi:10.48550/arxiv.1910.04851 · Cited by 211
“Deep neural networks have seen a wide adoption, driven by their impressive performance in various tasks including image classification [25], object recognition [43, 33, 37], natural language processing [34, 35], and speech recognition [18, 15].”
BackgroundA survey of quaternion neural networks
Parcollet, T., Morchid, M., & Linarès, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09752-1 · Cited by 209
“With the concurrent growth of the amount of available data and computation power, artificial neural networks (NN) have reached state-of-the-art performances in numerous real-world applications (Mikolov et al. 2010; Simonyan and Zisserman 2014; Graves et al. 2013).”
BackgroundCompositional Morphology for Word Representations and Language Modelling
Botha, J. A., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4273 · Cited by 149
“Conventional back-off n -gram LMs (Chen & Goodman, 1998) and the increasingly popular vector-based LMs (Bengio et al., 2003; Schwenk et al., 2006; Mikolov et al., 2010) use parametrisations that do not explicitly encode morphological regularities among re-lated forms, like abstract , abstraction…”
BackgroundMining Fashion Outfit Composition Using an End-to-End Deep Learning Approach on Set Data
Li, Y., Cao, L., Zhu, J., & Luo, J. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2690144 · Cited by 176
Predicting Domain Generation Algorithms with Long Short-Term Memory Networks
Woodbridge, J., Anderson, H. S., Ahuja, A., & Grant, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00791 · Cited by 205
“In a variety of natural language tasks, recurrent neural networks (RNNs) have been used to capture meaningful temporal relationships among tokens in a sequence [19], [20], [21], [22].”
BackgroundRecurrent neural network based language modeling in meeting recognition
Kombrink, S., Mikolov, T., Karafiát, M., & Burget, L. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-720 · Cited by 198
“Previous experiments in [10] and [1] already showed the advantage of RNN language models on simple ASR systems using limited training data.”
BackgroundMethodResultInfluentialSelf-citationConvolutional Neural Network for Trajectory Prediction
Nikhil, N., & Morris, B. T. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11015-4_16 · Cited by 189
Drug-drug interaction extraction from biomedical texts using long short-term memory network
Sahu, S. K., & Anand, A. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.08.005 · Cited by 206
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205
“The two most benchmarked word-level language modelling datasets either stress the modelling of stand-alone sentences (Billion Word Benchmark from Chelba et al. (2013)) or the modelling of a small selection of short news articles (Penn Treebank processed by Mikolov et al. (2010)).”
BackgroundMultilabel Image Classification With Regional Latent Semantic Dependencies
Zhang, J., Wu, Q., Shen, C., Zhang, J., & Lu, J. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2812605 · Cited by 203
DeepDGA: Adversarially-Tuned Domain Generation and Detection
Anderson, H. S., Woodbridge, J., & Filar, B. (2016) · Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security · doi:10.1145/2996758.2996767 · Cited by 202
Natural Language Processing Advancements By Deep Learning: A Survey
Torfi, A., Shirvani, R. A., Keneshloo, Y., Tavaf, N., & Fox, E. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01200 · Cited by 190
“Most of the popular seq2seq models are minimizing cross-entropy loss as their optimization objective via Teacher Forc-ing (Section III-B).”
BackgroundInfluentialDo Characters Abuse More Than Words?
Mehdad, Y., & Tetreault, J. (2016) · Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w16-3638 · Cited by 198
“To overcome this challenge, we exploit the power of recurrent neural networks (RNNs) (Mikolov et al., 2010) which demonstrated state-of-the-art re-sults for language models with less training data (Mikolov, 2012).”
BackgroundStructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Wang, W., Bi, B., Yan, M., Wu, C., Bao, Z., Xia, J., Peng, L., & Si, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04577 · Cited by 91
“Schmaltz showed that recurrent neural network language models [11] with long short-term memory [12] cells work effectively for word ordering even without any explicit syntactic information.”
BackgroundChiarelli, A. M., Croce, P., Merla, A., & Zappasodi, F. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aaaf82 · Cited by 197
“The DNN employed was not a CNN (encoding spatial or temporal information, Krizhevsky et al (2012) and Kalchbrenner et al (2014)) nor an RNN (encoding the sequential order of features (Hochreiter and Schmidhuber 1997, Mikolov et al 2010)).”
MethodLearning to Compose and Reason with Language Tree Structures for Visual Grounding
Hong, R., Liu, D., Mo, X., He, X., & Zhang, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2911066 · Cited by 194
“Thanks to the advance of visual deep features [6] and neural language models [7], recent studies show promising results on scaling up visual grounding to open-vocabulary scenario, such as thousands of object categories [1], [8], relationships [9], and phrases [10].”
BackgroundA Survey of Event Extraction From Text
Xiang, W., & Wang, B. (2019) · IEEE Access · doi:10.1109/access.2019.2956831 · Cited by 195
“can exploit the potential dependencies in between any two words, either directly or indirectly connected, which enables its wide applications in many NLP tasks [155], including named entity recognition [156], part-of-speech tagging [157], relation extraction [158], sentence parsing [159], sequence”
BackgroundLocation Prediction over Sparse User Mobility Traces Using RNNs: Flashback in Hidden States!
Yang, D., Fankhauser, B., Rosso, P., & Cudre-Mauroux, P. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/302 · Cited by 193
“Recently, Recurrent Neural Networks (RNNs) have been shown as a successful tool to model sequential data [Mikolov et al. , 2010], capturing complex long-and short-term dependency over input sequences.”
MethodControllable protein design with language models
Ferruz, N., & Höcker, B. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00499-z · Cited by 194
Sajedian, I., Kim, J., & Rho, J. (2019) · Microsystems & Nanoengineering · doi:10.1038/s41378-019-0069-y · Cited by 196
“After leaky relu layer no. 10, we used a time distributed layer to prepare the output of the CNN network for the input of the RNN network.”
MethodInfluentialNamuduri, S., Narayanan, B. N., Davuluru, V. S. P., Burton, L., & Bhansali, S. (2020) · Journal of The Electrochemical Society · doi:10.1149/1945-7111/ab67a8 · Cited by 195
Du, J., Vong, C.-M., & Chen, C. L. P. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2969705 · Cited by 194
Das, S., Tariq, A., Santos, T., Kantareddy, S. S., & Banerjee, I. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_4 · Cited by 187
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
Deep learning in generating radiology reports: A survey
Monshi, M. M. A., Poon, J., & Chung, V. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101878 · Cited by 192
“They can predict next words based on the former ones in the language model [76].”
A survey of word embeddings based on deep learning
Wang, S., Zhou, W., & Jiang, C. (2020) · Computing · doi:10.1007/s00607-019-00768-7 · Cited by 193
“…and the relationship between target word and its context words through neural networks and contain fruitful semantic information in the embeddings, which can be obtained by training language models [19, 20, 22–27], or by building specifically neural networks for generating word embeddings [28–36].”
BackgroundWu, Y.-C., Yin, F., & Liu, C.-L. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.12.026 · Cited by 193
Code Completion with Neural Attention and Pointer Networks
Li, J., Wang, Y., Lyu, M. R., & King, I. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/578 · Cited by 112
“Neural Language Modeling Deep learning techniques such as RNNs have achieved the state-of-the-art results in the language mod ling task [Mikolov et al., 2010; Zaremba et al., 2014].”
BackgroundMethodDeep Recursive Network Embedding with Regular Equivalence
Tu, K., Cui, P., Wang, X., Yu, P. S., & Zhu, W. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220068 · Cited by 179
“For any computable function, there exists a finite recurrent neural network (RNN) [24] that can compute it.”
BackgroundReal-Time Crash Risk Prediction using Long Short-Term Memory Recurrent Neural Network
Yuan, J., Abdel-Aty, M., Gong, Y., & Cai, Q. (2019) · Transportation Research Record: Journal of the Transportation Research Board · doi:10.1177/0361198119840611 · Cited by 191
“Therefore, RNNs have been widely employed in many fields to conduct sequential data analysis and prediction, including language model (23), speech recognition (24), machine translation (25), and so forth.”
MethodRobust MVDR beamforming using time-frequency masks for online/offline ASR in noise
Higuchi, T., Ito, N., Yoshioka, T., & Nakatani, T. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472671 · Cited by 184
“In our experiments, we performed speaker independent decoding by using a deep convolutional neural network (CNN) acoustic model [18, 19] and a class-based recurrent neural network language model (RNN-LM) [20, 21].”
MethodHierarchical Recurrent Attention Network for Response Generation
Xing, C., Wu, Y., Wu, W., Huang, Y., & Zhou, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11965 · Cited by 184
Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
Venieris, S. I., Kouris, A., & Bouganis, C.-S. (2019) · ACM Computing Surveys · doi:10.1145/3186332 · Cited by 189
“heir mapping to optimised hardware. Beyond the spatial pattern recognition abilities of CNNs, Recurrent Neural Networks (RNNs) enhance NNs with the ability to learn data sequences by retaining memory [60]. While this broadens the application field of neural networks, additional recurrent connections between layers int…”
BackgroundA Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P., & Sun, L. (2025) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 182
“To solve this problem, recurrent neural networks ( RNNs ) [152] were later introduced for language modeling tasks, allowing for modeling relatively long dependency.”
Generative chemistry: drug discovery with deep learning generative models
Bian, Y., & Xie, X.-Q. (2021) · Journal of Molecular Modeling · doi:10.1007/s00894-021-04674-8 · Cited by 171
Classification, Denoising, and Deinterleaving of Pulse Streams With Recurrent Neural Networks
Liu, Z.-M., & Yu, P. S. (2019) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2018.2874139 · Cited by 185
“They process sequential samples one by one to extract information contained in streams [26], [27], or predict upcoming samples based on previous ones [28].”
BackgroundMethodSentiment analysis using deep learning approaches: an overview
Habimana, O., Li, Y., Li, R., Gu, X., & Yu, G. (2020) · Science China Information Sciences · doi:10.1007/s11432-018-9941-6 · Cited by 186
Signal Modulation Classification Based on the Transformer Network
Cai, J., Gan, F., Cao, X., & Liu, W. (2022) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2022.3176640 · Cited by 183
“The TRN can be seen as an extension of the RNN by removing sequential computation with improved performance [38], [39], [40].”
MethodLoad Balancing in Data Center Networks: A Survey
Zhang, J., Yu, F. R., Wang, S., Huang, T., Liu, Z., & Liu, Y. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2816042 · Cited by 184
An introduction to deep learning in medical physics: advantages, potential, and challenges
Shen, C., Nguyen, D., Zhou, Z., Jiang, S. B., Dong, B., & Jia, X. (2020) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ab6f51 · Cited by 185
“…of the most popular version of RNN, other recurrent networks have been devised as well, including bi-directional RNN (Schuster and Paliwal, 1997; Mikolov et al., 2010; Sak et al., 2014; Zaremba et al., 2014), gated recurrent unit (GRU) (Cho et al., 2014), neural turing machines (Graves et al.,…”
BackgroundCurrent trends in the development of intelligent unmanned autonomous systems
Zhang, T., Li, Q., Zhang, C.-S., Liang, H.-W., Li, P., Wang, T.-M., Li, S., Zhu, Y.-L., & Wu, C. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601650 · Cited by 184
“, 2014) and language (Mikolov et al., 2010; Vinyals et al., 2015), models with recurrent structures have brought significant improvements to state-of-the-art performance.”
BackgroundSemisupervised sequence modeling for elastic impedance inversion
Alfarraj, M., & AlRegib, G. (2019) · Interpretation · doi:10.1190/int-2018-0250.1 · Cited by 175
“sequence samples which allows them to capture long temporal dependencies in sequential data. RNNs have been utilized to solve many problems in language modeling and natural language processing (NLP)(Mikolov et al., 2010), speech and audio processing (Graves et al., 2013), video processing (Ma et al., 2017), petrophysi…”
BackgroundMethodBig code != big vocabulary: open-vocabulary models for source code
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., & Janes, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380342 · Cited by 182
“Current NLMs are based on architectures such as recurrent neural networks (RNN) [63], long short-term memory (LSTM) [45], or Transformer [85] that model long range dependencies: a study of LSTM NLMs showed that they use context as large as 250 words [56], much longer than n-grams.”
Recurrent neural network and LSTM models for lexical utterance classification
Ravuri, S., & Stolcke, A. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-42 · Cited by 175
“Recent work in Recurrent Neural Network Language modeling [11] suggests that temporal modeling of an entire sentence through a series of hidden units can outperform models based on the Markov assumption.”
BackgroundAdvances of deep learning applications in ground-penetrating radar: A survey
Tong, Z., Gao, J., & Yuan, D. (2020) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2020.120371 · Cited by 180
Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information
Yuan, Z., Zhang, W., Tian, C., Rong, X., Zhang, Z., Wang, H., Fu, K., & Sun, X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3163706 · Cited by 178
“The words in the text T are embedded into the word vector and fed into the RNN [43], and then the representation of the text t is obtained through the multilayer perceptron MLP.”
Contextual LSTM (CLSTM) models for Large scale NLP tasks
Ghosh, S., Vinyals, O., Strope, B. P., Roy, S., Dean, T., & Heck, L. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06291 · Cited by 178
“RNN-based language models (RNN-LMs) were proposed by Mikolov et al. [32], and in particular the variant using LSTMs was introduced by Sundermeyer et al. [38].”
BackgroundMethodWord Embeddings: A SurveyPreprint
Almeida, F. D. M., & Xexeo, G. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09069 · Cited by 418
“In (Mikolov et al. (2010)), the idea of using Recurrent Neural Networks (RNNs) to train lan- pendix A for more information.”
BackgroundMethodConvolutional recurrent neural networks: Learning spatial dependencies for image representation
Zuo, Z., Shuai, B., Wang, G., Liu, X., Wang, X., Wang, B., & Chen, Y. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301268 · Cited by 174
“RNNs [8, 9, 17] are neural networks for modeling sequential dependencies.”
BackgroundAccelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC
Nurvitadhi, E., Sim, J., Sheffield, D., Mishra, A., Krishnan, S., & Marr, D. (2016) · 2016 26th International Conference on Field Programmable Logic and Applications (FPL) · doi:10.1109/fpl.2016.7577314 · Cited by 175
Deep learning approaches for automatic detection of sleep apnea events from an electrocardiogram
Erdenebayar, U., Kim, Y. J., Park, J.-U., Joo, E. Y., & Lee, K.-J. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105001 · Cited by 178
Artificial Intelligence Enabled Internet of Things: Network Architecture and Spectrum Access
Song, H., Bai, J., Yi, Y., Wu, J., & Liu, L. (2020) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2019.2954643 · Cited by 179
“on both the current sentences and previous sentences [33].”
BackgroundMultiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation
Serban, I., Klinger, T., Tesauro, G., Talamadupula, K., Zhou, B., Bengio, Y., & Courville, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10984 · Cited by 177
“We define the 20 K most frequent words as the vocabulary and the word embedding dimensionality to size 300 for all models, with the exception of the RNNLM and HRED on Twitter, where we use embedding dimensionality of size 400 .”
MethodInfluentialMulti-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents
Liu, P., Qiu, X., Chen, X., Wu, S., & Huang, X. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1280 · Cited by 174
Extractive Summarization using Continuous Vector Space Models
Kågebäck, M., Mogren, O., Tahmasebi, N., & Dubhashi, D. (2014) · Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC) · doi:10.3115/v1/w14-1504 · Cited by 177
Evaluating the Underlying Gender Bias in Contextualized Word Embeddings
Basta, C., Costa-jussà, M. R., & Casas, N. (2019) · Proceedings of the First Workshop on Gender Bias in Natural Language Processing · doi:10.18653/v1/w19-3805 · Cited by 189
“In this work we focus on the particular NLP area of word embeddings (Mikolov et al., 2010), which represent words in a numerical vector space.”
BackgroundDeep and Embedded Learning Approach for Traffic Flow Prediction in Urban Informatics
Zheng, Z., Yang, Y., Liu, J., Dai, H.-N., & Zhang, Y. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2909904 · Cited by 175
“teristics during the training and predicting phase [29]–[31].”
BackgroundSkin disease diagnosis with deep learning: a review
Li, H., Pan, Y., Zhao, J., & Zhang, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.096 · Cited by 178
Deep Neural Network Language Models
Arısoy, E., Sainath, T. N., Kingsbury, B. E. D., & Ramabhadran, B. (2012) · Cited by 174
RMDL: Random Multimodel Deep Learning for Classification
Kowsari, K., Heidarysafa, M., Brown, D. E., Meimandi, K. J., & Barnes, L. E. (2018) · International Conference on Information System and Data Mining (ICISDM) · doi:10.1145/3206098.3206111 · Cited by 212
“RNN have been successfully used for natural language processing [37].”
MethodBi-LSTM-CRF Sequence Labeling for Keyphrase Extraction from Scholarly Documents
Alzaidy, R., Caragea, C., & Giles, C. L. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313642 · Cited by 167
“Because LSTMs are capable of preserving information over previous inputs of a sequence into the current input state, they have been a natural choice for applications involving temporal and sequence data such as speech recognition, language modeling and translation [14, 35].”
BackgroundDense Captioning with Joint Inference and Visual Context
Yang, L., Tang, K., Yang, J., & Li, L.-J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.214 · Cited by 167
“Therefore, we can borrow successful recipes from the object detection literature in designing our dense captioning algorithm.”
MethodLearning Multi-View Representation With LSTM for 3-D Shape Recognition and Retrieval
Ma, C., Guo, Y., Yang, J., & An, W. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2875512 · Cited by 173
“, language modeling [18], [19], speech recognition [20], machine translation [21], image captioning [22], and image generation [23].”
BackgroundPredicting online shopping behaviour from clickstream data using deep learning
Koehn, D., Lessmann, S., & Schaal, M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113342 · Cited by 171
Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities
White, M., Tufano, M., Martinez, M., Monperrus, M., & Poshyvanyk, D. (2019) · 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668043 · Cited by 163
“ch term in the file-level corpus [63], and recurrent neural networks in particular can serve as effective architectures for language models of both natural language corpora and source code [11], [15], [64]–[68]. These models learn from the order of terms in a corpus, imputing embeddings to the terms in such a way that…”
BackgroundImproving the accuracy of global forecasting models using time series data augmentation
Bandara, K., Hewamalage, H., Liu, Y.-H., Kang, Y., & Bergmeir, C. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108148 · Cited by 169
Adversarial examples for models of code
Yefet, N., Alon, U., & Yahav, E. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428230 · Cited by 161
Sun, Q., & Ge, Z. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3010331 · Cited by 171
“Asa kind of typical data-driven approach, deep learning (DL) methods, developed from artificial neural-network techniques (ANN), have dramatically improved the state of the art in many fields, such as speech recognition, visual object recognition, object detection, and so on [9]–[11].”
BackgroundGmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330723 · Cited by 169
“In recent years with the rapid development in deep learning techniques, a multitude of neural network-based language models have been proposed, significantly improving state-of-theart performance [3, 26, 36, 37, 40, 49, 51].”
BackgroundLack of selectivity for syntax relative to word meanings throughout the language network
Fedorenko, E., Blank, I. A., Siegelman, M., & Mineroff, Z. (2020) · Cognition · doi:10.1016/j.cognition.2020.104348 · Cited by 170
Min, X., Yu, B., & Wang, F. (2019) · Scientific Reports · doi:10.1038/s41598-019-39071-y · Cited by 170
“15 adapted the TopicRNN approach 16 , which combines probabilistic topic modeling 17 and Recurrent Neural Network (RNN) 18 to better capture long-term dependencies in sequences, to predict the readmission risk of heart failure patients.”
MethodBarack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1598 · Cited by 168
“RNN language models (Mikolov et al., 2010) parameterize this distribution using a recurrent structure:”
MethodMulti-domain Neural Network Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gašić, M., Mrkšić, N., Rojas-Barahona, L. M., Su, P.-H., Vandyke, D., & Young, S. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1015 · Cited by 167
Deep learning for Arabic NLP: A survey
Al-Ayyoub, M., Nuseir, A., Alsmearat, K., Jararweh, Y., & Gupta, B. (2018) · Journal of Computational Science · doi:10.1016/j.jocs.2017.11.011 · Cited by 167
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
Tang, J., & Wang, K. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220021 · Cited by 167
“us parameters are shown to have better effectiveness and good representation power [21]. Recently, with the great impact of neural networks on computer vision [17, 22] and natural language processing [19, 25], a new branch of IR using neural networks has shown strong performances. As neural networks have incredible po…”
BackgroundA Systematic Literature Review of Intelligent Data Analysis Methods for Smart Sport Training
Rajšp, A., & Fister, I. (2020) · Applied Sciences · doi:10.3390/app10093013 · Cited by 166
“• Deep learning [32]: Recurrent Neural Networks (RNN) [33], Long Short-Term Memory (LSTM) [34], Convolutional Neural Networks [35] (CNN).”
BackgroundModeling Interestingness with Deep Neural Networks
Gao, J., Pantel, P., Gamon, M., He, X., & Deng, L. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1002 · Cited by 165
“One potentially effective model for such a purpose is based on the architecture of recurrent neural networks (e.g., Mikolov et al. 2010; Chen and Deng, 2014), which can be incorporated into the deep semantic model proposed in this paper.”
Health Status Assessment and Failure Prediction for Hard Drives with Recurrent Neural Networks
Xu, C., Wang, G., Liu, X., Guo, D., & Liu, T.-Y. (2016) · IEEE Transactions on Computers · doi:10.1109/tc.2016.2538237 · Cited by 165
“Recurrent Neural Networks (RNN) have been proven an effective tool to model temporal dependency in various applications, such as language models [12], [13], speech recognition [14], machine translation [15], and so on.”
MethodDeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks
Pezzotti, N., Hollt, T., Van Gemert, J., Lelieveldt, B. P. F., Eisemann, E., & Vilanova, A. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744358 · Cited by 165
“We want to extend DeepEyes by integrating different deep-learning libraries, such as TensorFlow [1] or Theano [42], and to the analysis of different and more exotic network architectures, such as Recurrent Neural Networks [30] and Deep Residual Networks [14].”
MethodHadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 103
A Recursive Recurrent Neural Network for Statistical Machine Translation
Liu, S., Yang, N., Li, M., & Zhou, M. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1140 · Cited by 160
Deep convolutional framework for abnormal behavior detection in a smart surveillance system
Ko, K.-E., & Sim, K.-B. (2018) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2017.10.001 · Cited by 163
“Thus, they can be applied to tasks such as speech recognition (Mikolov et al., 2010).”
BackgroundDeep-learning-based inverse design model for intelligent discovery of organic molecules
Kim, K., Kang, S., Yoo, J., Kwon, Y., Nam, Y., Lee, D., Kim, I., Choi, Y.-S., Jung, Y., Kim, S., et al. (2018) · npj Computational Materials · doi:10.1038/s41524-018-0128-1 · Cited by 163
“To learn the SMILES generation rule, the RNN is constructed in the form of a language model.(34) The first substring y1 is predicted given a start token and z, and the output yt at each time step t becomes the next input to predict the next output yt+ 1 until an end token is predicted with the constraint of ensuring t…”
Method
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.