Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 101–200 of 1,058
Dual Learning for Machine Translation
Xia, Y., He, D., Qin, T., Wang, L., Yu, N., Liu, T., & Ma, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00179 · Cited by 636
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04770 · Cited by 407
“, 1993) using the standard train/test/validation split by (Mikolov et al., 2010).”
Method(2014) · International Conference on Learning Representations · Cited by 634
A Survey of Multi-View Representation Learning
Li, Y., Yang, M., & Zhang, Z. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2872063 · Cited by 524
“Besides, multi-modal RNNs have been widely applied in image captioning [21, 22, 197], video captioning [23, 198, 199], visual question answering [200], and information retrieval [201].”
InfluentialDiez-Olivan, A., Del Ser, J., Galar, D., & Sierra, B. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.10.005 · Cited by 633
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629
“For the remainder of this section, we use as our case study a character-level language model [5, 38]: given a sequence of text data, a language model predicts the next token (character, in this case) that will occur.”
BackgroundMethodOne billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627
Self-citationA systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Deep learning code fragments for code clone detection
White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2025 · Cited by 596
Fault diagnosis of rolling bearings with recurrent neural network-based autoencoders
Liu, H., Zhou, J., Zheng, Y., Jiang, W., & Zhang, Y. (2018) · ISA Transactions · doi:10.1016/j.isatra.2018.04.005 · Cited by 606
Dual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3383184 · Cited by 589
“Recurrent Neural Networks (RNN) are another common choice in natural language processing [50, 73].”
BackgroundSpatial–Temporal Recurrent Neural Network for Emotion Recognition
Zhang, T., Zheng, W., Cui, Z., Zong, Y., & Li, Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2788081 · Cited by 595
“Recently, RNNs have achieved great success in processing sequential data, such as natural language processing [20], [21], action recognition [25], speech recognition [26], and so on.”
BackgroundMultimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. (2014) · International Conference on Machine Learning · Cited by 789
Recent Advances in Recurrent Neural Networks
Salehinejad, H., Sankar, S., Barfett, J., Colak, E., & Valaee, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01078 · Cited by 562
“The feedback loops are recurrent cycles over time or sequence (we call it time throughout this paper) [8], as shown in Figure 1.”
BackgroundA Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579
“eration by representing the whole dialogue history (including the current message ) with continuous representations or embeddings of words and phrases. The response is generated as RNN language model [52], the same as the decoder in [12]. [68] used hierarchical models, first capturing the meaning of individual utteranc…”
MethodContext dependent recurrent neural network language model
Mikolov, T., & Zweig, G. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424228 · Cited by 560
“For reduction of computational complexity, we used a factorization of the output layer using 100 classes, as described in [2].”
BackgroundMethodSelf-citationDeep learning enabled inverse design in nanophotonics
So, S., Badloe, T., Noh, J., Rho, J., & Bravo‐Abad, J. (2020) · Nanophotonics · doi:10.1515/nanoph-2019-0474 · Cited by 564
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Mehta, S., Rastegari, M., Shapiro, L., & Hajishirzi, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00941 · Cited by 551
“Dataset: The performance of our unit, the EESP, is evaluated on the Penn Treebank (PTB) dataset [30] as prepared by [35].”
MethodTransformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv · doi:10.48550/arxiv.1901.02860
“However, different from truncated BPTT, our method caches a sequence of hidden states instead of the last one, and should be applied together with the relative positional encoding technique described in Section 3.3.”
BackgroundMethodInfluentialGu, X., Zhang, H., & Kim, S. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180167 · Cited by 547
“We now introduce a widely-used deep neural network, the Recurrent Neural Networks (RNN) [49, 59] for the embedding of sequential data such as natural language sentences.”
MethodFinding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
Ling, W., Dyer, C., Black, A. W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., & Luis, T. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1176 · Cited by 541
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544
“…autoregressive language models were built on the backbones of n-grams (Bickel et al., 2005; Pauls and Klein, 2011) and recurrent neural networks (Mikolov et al., 2010), and have been applied to various NLG tasks such as summarization (Nallapati et al., 2017) and dialogue generation (Chen et…”
Deep learning in the construction industry: A review of present status and future innovations
Akinosho, T. D., Oyedele, L. O., Bilal, M., Ajayi, A. O., Delgado, M. D., Akinade, O. O., & Ahmed, A. A. (2020) · Journal of Building Engineering · doi:10.1016/j.jobe.2020.101827 · Cited by 537
A review of natural language processing techniques for opinion mining systems
Sun, S., Luo, C., & Chen, J. (2017) · Information Fusion · doi:10.1016/j.inffus.2016.10.004 · Cited by 542
“[135, 136] presented several modifications of the original RNN language model.”
MethodApplication of deep learning algorithms in geotechnical engineering: a short critical review
Zhang, W., Li, H., Li, Y., Liu, H., Chen, Y., & Ding, X. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09967-1 · Cited by 538
“…in different domains, such as, health and medicine (Lisboa 2002), business and management (Wong et al. 1997), natural language processing(NLP) (Mikolov et al. 2010), image processing (Ayyıldız and Çetinkaya 2017; Egmont-Petersen et al. 2002; He et al. 2015), geo-sciences and remote sensing…”
BackgroundParallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations
Hidasi, B., Quadrana, M., Karatzoglou, A., & Tikk, D. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959167 · Cited by 523
“distributed bag-of-words [18] and Language Modeling with RNNs [17].”
MethodQi, X., Chen, G., Li, Y., Cheng, X., & Li, C. (2019) · Engineering · doi:10.1016/j.eng.2019.04.012 · Cited by 529
“③ The recurrent neural network (RNN) [29] plays a key role in dealing with temporal dynamics, since it builds connections between the nodes in one layer.”
BackgroundInfluentialSiren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
“…autoregressive language models were built on the backbones of n-grams (Bickel et al., 2005; Pauls and Klein, 2011) and recurrent neural networks (Mikolov et al., 2010), and have been applied to various NLG tasks such as summarization (Nallapati et al., 2017) and dialogue generation (Chen et…”
Bilingual Word Embeddings for Phrase-Based Machine Translation
Zou, W. Y., Socher, R., Cer, D., & Manning, C. D. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1141 · Cited by 518
“Mnih and Hinton ( 2008), Morin and Bengio ( 2005) proposed efficient hierarchical continuous-space models. To systematically compare embeddings, Turian et al. (2010) evaluated improvements they bring to state-of-theart NLP benchmarks.”
BackgroundChallenges in Data-to-Document Generation
Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524
Strategies for training large scale neural network language models
Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163930 · Cited by 509
“The architecture is called the Recurrent neural network based language model (RNN LM), and we have described it in [7] and [5].”
MethodSelf-citationFrom Feedforward to Recurrent LSTM Neural Networks for Language Modeling
Sundermeyer, M., Ney, H., & Schluter, R. (2015) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2015.2400218 · Cited by 507
“Recurrent neural network LMs as introduced in [9] follow a very similar topology compared to feedforward neural networks.”
BackgroundMind's eye: A recurrent visual representation for image caption generation
Chen, X., & Zitnick, C. L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298856 · Cited by 498
“Our recurrent neural network model structure builds on the prior models proposed by [29, 32].”
BackgroundMethodInfluentialFine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings
Liu, P., Joty, S., & Meng, H. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1168 · Cited by 489
Zhu, Q., Liu, Z., & Yan, J. (2021) · Computational Mechanics · doi:10.1007/s00466-020-01952-9 · Cited by 497
“Some typical neural network architectures are fully connected neural network (FC-NN) [61], convolutional neural network (CNN) [62], and recurrent neural network (RNN) [63], which have been successfully used in a variety of machine learning applications, such as system identification and control, signal classification,…”
MethodDeep Learning for Sensor-based Human Activity Recognition
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2022) · ACM Computing Surveys · doi:10.1145/3447744 · Cited by 690
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Pedrycz, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 487
What to Do Next: Modeling User Behaviors by Time-LSTM
Zhu, Y., Li, H., Liao, Y., Wang, B., Guan, Z., Liu, H., & Cai, D. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/504 · Cited by 468
“To achieve this, we adapt the dynamic updated model in [Mikolov et al. , 2010] to our task as follows.”
MethodFVQA: Fact-Based Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2754246 · Cited by 479
Gu, X., Zhang, H., Zhang, D., & Kim, S. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950334 · Cited by 485
Automatic speech recognition for under-resourced languages: A survey
Besacier, L., Barnard, E., Karpov, A., & Schultz, T. (2014) · Speech Communication · doi:10.1016/j.specom.2013.07.008 · Cited by 482
“Artificial Neural Networks (ANN) including single hidden layer NN and multiple hidden layers NN (Deep Neural Networks DNN or Deep Belief Networks DBN) are also used for ASR subtasks such as acoustic modeling [Mohamed, 2012; Seide, 2011] and language modeling [Arisoy, 2012; Mikolov, 2010].”
MethodThe Microsoft 2017 Conversational Speech Recognition System
Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., & Stolcke, A. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461870 · Cited by 472
“The combination of deep networks and efficient training methods with older neural modeling concepts [2, 3, 4, 5, 6, 7, 8] have produced steady advances in both acoustic modeling [9, 10, 11, 12, 13, 14, 15] and language modeling [16, 17, 18, 19].”
MethodFast and Robust Neural Network Joint Models for Statistical Machine Translation
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., & Makhoul, J. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1129 · Cited by 481
“Ar-En Ch-En BLEU BLEU Baseline (w/o RNNLM) 40 .”
MethodInfluentialAnalysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., & Baroni, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1108 · Cited by 465
“More recently, RNNs have achieved impressive results in large-scale tasks such as language modeling for speech recognition and machine translation, and are by now standard tools for sequential natural language tasks (e.g., Mikolov et al., 2010; Graves, 2012; Wu et al., 2016).”
BackgroundA Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion
Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Grue Simonsen, J., & Nie, J.-Y. (2015) · Proceedings of the 24th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2806416.2806493 · Cited by 472
Multi-layer Representation Learning for Medical Concepts
Choi, E., Bahadori, M. T., Searles, E., Coffey, C., Thompson, M., Bost, J., Tejedor-Sojo, J., & Sun, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939823 · Cited by 469
“Keywords Representation Learning; Medical Concepts; Healthcare Analytics; Neural Networks”
BackgroundSpatio-Temporal Data Mining: A Survey of Problems and Methods
Atluri, G., Karpatne, A., & Kumar, V. (2019) · ACM Computing Surveys · doi:10.1145/3161602 · Cited by 461
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
“To solve this problem, recurrent neural networks (RNNs) [23] were later introduced for language modeling tasks , allowing for modeling relatively long dependency.”
BackgroundAchieving Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F. T. B., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05256 · Cited by 451
“Surprisingly, this is the case for both acoustic modeling [28, 29, 30, 31, 32, 33] and language modeling [34, 35, 36, 37].”
BackgroundMalware classification with recurrent networks
Pascanu, R., Stokes, J. W., Sanossian, H., Marinescu, M., & Thomas, A. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178304 · Cited by 423
“Recently, researchers have demonstrated excellent results using recurrent neural networks (RNNs) on language modeling [6], online handwritten recognition and generation [7], and speech recognition [8].”
BackgroundMethodRolling element bearing fault diagnosis using convolutional neural network and vibration image
Hoang, D.-T., & Kang, H.-J. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2018.03.002 · Cited by 450
Knowledge-Embedded Routing Network for Scene Graph Generation
Chen, T., Yu, W., Chen, R., & Lin, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00632 · Cited by 432
“Xu et al. [30] introduced an end-to-end model that learned to iteratively refine relationship and object prediction via message passing based on the RNNs [21].”
BackgroundTopic Aware Neural Response Generation
Xing, C., Wu, W., Wu, Y., Liu, J., Huang, Y., Zhou, M., & Ma, W.-Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10981 · Cited by 438
“Therefore, we follow the existing work and employ the following metrics: Perplexity: following (Vinyals and Le 2015) and (Mikolov et al. 2010), we employ perplexity as an evaluation metric.”
MethodBandara, K., Bergmeir, C., & Smyl, S. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.112896 · Cited by 435
“As a result, RNN architectures are heavily used in domains such as Natural Language Processing (Mikolov et al., 2010), machine translation (Sutskever et al., 2014), and speech recognition (Graves et al., 2013), and are also gaining popularity in time series research (Fei and Yeung, 2015; Paw lowski and Kurach, 2015; L…”
MethodInfluentialDeep Learning for Extreme Multi-label Text Classification
Liu, J., Chang, W.-C., Wu, Y., & Yang, Y. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080834 · Cited by 676
“ose areas include various tasks in natural language understanding [37], language modeling [33], machine translation[38], and more.”
BackgroundBouschery, S. G., Blazevic, V., & Piller, F. T. (2023) · Journal of Product Innovation Management · doi:10.1111/jpim.12656 · Cited by 444
Learning word embeddings efficiently with noise-contrastive estimation
Mnih, A., & Kavukcuoglu, K. (2013) · neural information processing systems · Cited by 651
PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference
Ankit, A., Hajj, I. E., Chalamalasetti, S. R., Ndu, G., Foltin, M., Williams, R. S., Faraboschi, P., Hwu, W.-M. W., Strachan, J. P., Roy, K., et al. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304049 · Cited by 424
“Recurrent Neural Networks (RNNs) [75] used for sequence processing compute weighted-sums on input and previous state.”
MethodSpeech emotion recognition using convolutional and Recurrent Neural Networks
Lim, W., Jang, D., & Lee, T. (2016) · 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA) · doi:10.1109/apsipa.2016.7820699 · Cited by 405
“Therefore, these sequential data analysis approaches are widely used not only for language modeling but also for machine translation [20-24].”
MethodMariani, M., & Dwivedi, Y. K. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2024.114542 · Cited by 409
“…(e.g., Bengio et al., 2000; Brown et al., 2020; Deng & Lin, 2022; Devlin et al., 2018; Graves and Jaitly, 2014; Good-fellow et al., 2014, 2020; Mikolov et al., 2010; Macdonald, 1954; Rad-ford et al., 2018; Schank & Abelson, 1975; Vaswani et al., 2017; Weizenbaum, 1966), but there are only a…”
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 370
Artificial intelligence and internet of things in small and medium-sized enterprises: A survey
Hansen, E. B., & Bøgh, S. (2021) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2020.08.009 · Cited by 407
Learning Fashion Compatibility with Bidirectional LSTMs
Han, X., Wu, Z., Jiang, Y.-G., & Davis, L. S. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123394 · Cited by 325
“Following [16] that utilizes softmax output to predict the next word in a sentence, we append a softmax layer on top of h t to calculate the probability of the next fashion item conditioned on previously seen items: ˝ ∈X where X contains all images (in multiple outfits) from the current batch.”
MethodTexygen: A Benchmarking Platform for Text Generation Models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210080 · Cited by 408
“…piece s t = [ x 0 , x 1 , ..., x t − 1 ] and the next token to be sampled from the model x t ∼ π θ ( x | s t ) , a vanilla MLE language model [11] adopts an explicit likelihood-based modeling of language, with the form: (cid:213) (cid:213) where x iterates over training data sentences and t…”
BackgroundMethodEntangled Transformer for Image Captioning
Li, G., Zhu, L., Liu, P., & Yang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00902 · Cited by 375
“Never-theless, because of the recurrent nature, RNNs [11, 27, 34] have difficulties in memorizing the inputs many steps ago, especially the initial visual input.”
BackgroundProfessor Forcing: A New Algorithm for Training Recurrent Networks
Lamb, A., Goyal, A., Zhang, Y., Zhang, S., Courville, A. C., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09038 · Cited by 290
“…networks (RNNs) have become to be the generative models of choice for sequential data (Graves, 2012) with impressive results in language modeling (Mikolov, 2010; Mikolov and Zweig, 2012), speech recognition (Bahdanau et al. , 2015; Chorowski et al. , 2015), Machine Translation (Cho et al. ,…”
BackgroundDeep learning in bioinformatics: Introduction, application, and perspective in the big data era
Li, Y., Huang, C., Ding, L., Li, Z., Pan, Y., & Gao, X. (2019) · Methods · doi:10.1016/j.ymeth.2019.04.008 · Cited by 401
TransVG: End-to-End Visual Grounding with Transformers
Deng, J., Yang, Z., Chen, T., Zhou, W., & Li, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00179 · Cited by 372
“Compared to the recurrent units in RNNs [18, 32, 44], the attention mechanism exhibits better performance in processing long sequences.”
BackgroundDeterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Lee, J., Mansimov, E., & Cho, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1149 · Cited by 386
“Post-Editing for Machine Translation Novak et al. (2016) proposed a convolutional neural network that iteratively predicts and applies token substitutions given a translation from a phasebased translation system.”
MethodProbabilistic forecasting with temporal convolutional neural network
Chen, Y., Kang, Y., Chen, Y., & Wang, Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.011 · Cited by 393
AI in Health: State of the Art, Challenges, and Future Directions
Wang, F., & Preininger, A. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677908 · Cited by 398
“Schwab et al. [50] proposed to tackle the same problem with RNN techniques.”
MethodInfluentialA deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems
Tang, M., Liu, Y., & Durlofsky, L. J. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.109456 · Cited by 394
Hybrid-augmented intelligence: collaboration and cognition
Zheng, N.-N., Liu, Z.-Y., Ren, P.-J., Ma, Y.-Q., Chen, S.-T., Yu, S.-Y., Xue, J.-R., Chen, B.-D., & Wang, F.-Y. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700053 · Cited by 387
“…on hybrid-augmented intelligence, more attention must be paid to other methods of learning and reasoning, such as deep learning based reinforcement learning (Mnih et al., 2015), recurrent neuron network based methods (Mikolov et al., 2010), and differentiable neuron computers (Graves et al., 2016).”
BackgroundMethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
“Our work is inspired by the long line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et…”
MethodHe, J., Zhao, L., Yang, H., Zhang, M., & Li, W. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2934760 · Cited by 383
“First, from the spectral aspect, a pixel (spectral vector) in HSI is analogous to a word vector [33] in natural language processing (NLP) because both are distributional representations and both convey some “meaning.”
BackgroundBrain-inspired computing with memristors: Challenges in devices, circuits, and systems
Zhang, Y., Wang, Z., Zhu, J., Yang, Y., Rao, M., Song, W., Zhuo, Y., Zhang, X., Cui, M., Shen, L., et al. (2020) · Applied Physics Reviews · doi:10.1063/1.5124027 · Cited by 384
De-identification of patient notes with recurrent neural networks
Dernoncourt, F., Lee, J. Y., Uzuner, O., & Szolovits, P. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw156 · Cited by 378
“As a result, methods based on ANNs have shown promising results for various tasks in natu-ral language processing, such as language modeling (Mikolov et al., 2010), text classification (Socher et al., 2013; Kim, 2014; Blunsom et al., 2014; Lee and Dernoncourt, 2016), question answering (We-ston et…”
MethodOpinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 371
Learn&Fuzz: Machine learning for input fuzzing
Godefroid, P., Peleg, H., & Singh, R. (2017) · 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115618 · Cited by 344
Training recurrent neural networks
Hinton, G. E., & Sutskever, I. (2013) · Cited by 383
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1473 · Cited by 357
A Survey of Learning Causality with Data: Problems and Methods
Guo, R., Cheng, L., Li, J., Hahn, P. R., & Liu, H. (2021) · ACM Computing Surveys · doi:10.1145/3397269 · Cited by 362
“neural networks is to learn representations for features. To study the causal effect of forming a group on receiving a loan in a microfinance platform, GloVe [116] and Recurrent Neural Networks (RNN) [103] embed text features into a low-dimensional space. Then outcomes are inferred by fitting a functionf (h,t)on factu…”
BackgroundPhased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
Neil, D., Pfeiffer, M., & Liu, S. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.1610.09513 · Cited by 319
“Applications include speech recognition [13], natural language processing [1, 20], and attention-based models for structured prediction [5, 29].”
BackgroundWhen large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367
“Language models perform the probabilistic modeling for the generation of natural language[26,27],i.e.,presentedwithonespecificcontext,thelanguagemodelsmakepredictions forthewordswhicharetobegeneratedforthefuturesteps.”
Zhu, Z., Peng, G., Chen, Y., & Gao, H. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.09.050 · Cited by 357
An analysis of environment, microphone and data simulation mismatches in robust speech recognition
Vincent, E., Watanabe, S., Nugraha, A. A., Barker, J., & Marxer, R. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2016.11.005 · Cited by 351
“Concerning other parts of the decoder, Hori et al. (2015) reported consistent improvements on real and simulated data by replacing the default 3-gram language model used in the baseline by a 5-gram language model with Kneser-Ney (KN) smoothing (Kneser and Ney, 1995), rescoring the lattice using a recurrent neural netw…”
InfluentialImage Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction
Noh, H., Seo, P. H., & Han, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.11 · Cited by 345
“Although encoders based on recurrent neural networks (RNNs) such as LSTM [11] and GRU [4] demonstrate impressive performance on sentence embedding [19, 25], their benefits in the ImageQA task are marginal in comparison to bag-of-words model [23].”
MethodExplain Images with Multimodal Recurrent Neural Networks
Mao, J., Xu, W., Yang, Y., Wang, J., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.1090 · Cited by 341
“We briefly introduce the simple Recurrent Neural Network (RNN) or Elman network [5] that is widely used for many natural language processing tasks, such as speech recognition [25, 26].”
BackgroundMethodMockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054458 · Cited by 337
“Autoregressive Predictive Coding (APC) [6] uses autoregressive models to encode temporal information of past acoustic sequence; the model predicts future frames like an RNN-based language model [11], optimized with reconstruction loss.”
MethodSequential Click Prediction for Sponsored Search with Recurrent Neural Networks
Zhang, Y., Dai, H., Xu, C., Feng, J., Wang, T., Bian, J., Wang, B., & Liu, T.-Y. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8917 · Cited by 337
“For example, RNN language model (Mikolov et al. 2010; Mikolov et al. 2011a; Mikolov et al. 2011b) successfully leverages long-span sequential information among the massive language corpus, which results in better performance than traditional neural networks language model (Bengio et al. 2006).”
BackgroundA survey on deep learning in medicine: Why, how and when?
Piccialli, F., Somma, V. D., Giampaolo, F., Cuomo, S., & Fortino, G. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.09.006 · Cited by 348
Li, X., & Wu, X. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178826 · Cited by 336
“In language modeling, RNNs were used as generative models over word sequences, and remarkable improvements were achieved [8] over the standard n-gram models.”
MethodCompute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363
“Hence we argue that 2010 is the starting date most consistent with the evidence, because: (a) the use of GPUs to train large ML models was already common at the time, (b) there were at least a few Deep Neural Networks that achieved highly competitive performance (notably [26, 27, 31]), and (c) this timeline is consist…”
MethodResultSahoo, B. B., Jha, R., Singh, A., & Kumar, D. (2019) · Acta Geophysica · doi:10.1007/s11600-019-00330-1 · Cited by 339
Zhu, X., Wen, S., Camtepe, S., & Xiang, Y. (2022) · ACM Computing Surveys · doi:10.1145/3512345 · Cited by 334
“Recurrent neural network (RNN) [127] is a preferable deep learning model for fuzzers to generate structured inputs [74, 79, 111, 112].”
BackgroundDeep Reinforcement Learning-Based Image Captioning with Embedding Reward
Ren, Z., Wang, X., Zhang, N., Lv, X., & Li, L.-J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.128 · Cited by 331
Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks
Zhang, Y., Yu, X., Cui, Z., Wu, S., Wen, Z., & Wang, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.31 · Cited by 317
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.