Recurrent neural network based language model
The DOI matches its Crossref record; 13 of the 14 references it cites were found in a registry or an index.
Abstract
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022; Rannen-Triki et al., 2024; Karami et al., 2025b;a), where model weights are updated at test time via gradient descent steps on a prediction loss.”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Mikolov et al., 2010
In the citing paper
“D. C. Cire¸ san et al. (2010b, 2011) made substantial improvements over the previous state-of-the-art results on MNIST, whilst Mikolov et al. (2010) surpassed all competition at the time on the Wall Street Journal task, an NLP task.”
From Compute Trends Across Three Eras of Machine Learning (2022) · cites this paper as Mikolov et al. (2010)
Citations
Cites
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · cited 1×
“In our work, we have used an architecture that is usually called a simple recurrent neural network or Elman network [7].”
MethodLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2005) · Cited by 823
A guide to recurrent neural networks and backpropagation
Mikael Bodén, A. (2001) · Cited by 246
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J.-L. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220601 · Cited by 117 · cited 3×
“Schwenk [4] describes several possible approaches that can be used for further performance improvements.”
BackgroundA Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
The AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Wan, V., Karafiát, M., Vepa, J., & Lincoln, M. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366923 · Cited by 113
Self-citationThe 2005 AMI System for the Transcription of Speech in Meetings
Hain, T., Burget, L., Dines, J., Garau, G., Karafiát, M., Lincoln, M., McCowan, I. A., Moore, D. C., Wan, V., Ordelman, R., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_38 · Cited by 70 · cited 2×
“To show that it is possible to obtain meaningful improvements in state of the art system, we experimented with lattices generated by AMI system used for NIST RT05 evaluation [13].”
MethodSelf-citationFast Text Compression with Neural Networks
Mahoney, M. V. (2000) · Cited by 44
Cited by
Items 701–800 of 1,058
Chen, J., Yan, S., & Wong, K. (2018) · Neural Computing and Applications · doi:10.1007/s00521-018-3442-0 · Cited by 104
“Mikolov et al. [29] introduce recurrent network to language model.”
BackgroundMulti-Behavioral Sequential Prediction with Recurrent Log-Bilinear Model
Liu, Q., Wu, S., & Wang, L. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2661760 · Cited by 87
“F”
MethodRobsut Wrod Reocginiton via Semi-Character Recurrent Neural Network
Sakaguchi, K., Duh, K., Post, M., & Van Durme, B. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10970 · Cited by 88
“We use an RNN because it has shown to be state-of-the-art language modeling (Mikolov et al. 2010) and it is also flexible to realize the findings from the Cmabrigde Uinervtisy effect.”
MethodSoftware defect prediction via LSTM
Deng, J., Lu, L., & Qiu, S. (2020) · IET Software · doi:10.1049/iet-sen.2019.0149 · Cited by 104
“Recurrent neural networks (RNNs) [13] have been wildly applied in natural language process (NLP) like language modelling [14], machine translation [15], speech recognition [16], and video captioning [17].”
BackgroundMulti-Task Travel Route Planning With a Flexible Deep Learning Framework
Huang, F., Xu, J. J., & Weng, J. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2987645 · Cited by 102
Neural Network Music Genre Classification
Pelchat, N., & Gelowitz, C. M. (2020) · Canadian Journal of Electrical and Computer Engineering · doi:10.1109/cjece.2020.2970144 · Cited by 112
“This means the output from a previous iteration may influence the current output [6].”
BackgroundCheng, S. L., Chen, J., Anastasiou, C., Angeli, P., Matar, O. K., Guo, Y., Pain, C. C., & Arcucci, R. (2022) · Journal of Scientific Computing · doi:10.1007/s10915-022-02059-4 · Cited by 101
“Addressing temporal sequences as directed graphs, RNNs manage to handle complex dynamical systems because of their ability of capturing historical dependencies through feedback loops [42].”
BackgroundEnsemble Multifeatured Deep Learning Models and Applications: A Survey
Abimannan, S., El-Alfy, E.-S. M., Chang, Y., Hussain, S., Shukla, S., & Satheesh, D. (2023) · IEEE Access · doi:10.1109/access.2023.3320042 · Cited by 119
“…Recurrent Neural Networks (RNNs) [138], and Long Short-Term Memory (LSTM) networks [139], each trained on different features like word embeddings [140], bag-of-words representations, or part-of-speech tags, ensemble models can capture more nuanced patterns and deliver improved classification…”
BackgroundZhang, C., Xu, Q., Yu, Y., Zhou, G., Zeng, K., Chang, F., & Ding, K. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102883 · Cited by 124
Few-shot learning for short text classification
Yan, L., Zheng, Y., & Cao, J. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-5772-4 · Cited by 116
“Moreover, we also compare our approach with three types of deep learning methods, the Simple RNN [23], LSTM [32] and DSC [40] which is based on autoencoders.”
MethodInfluentialEfficient Vector Representation for Documents through Corruption
Chen, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02377 · Cited by 84
“…from the simplest BoW and its term-frequency based variants (Salton and Buckley, 1988), language model based methods (Croft and Lafferty, 2013; Mikolov et al., 2010; Kim et al., 2015), topic models (Deerwester et al., 1990; Blei et al., 2003), Denoising Autoencoders and its variants (Vincent…”
BackgroundMethodInfluentialA Feasibility of Respiration Prediction Based on Deep Bi-LSTM for Real-Time Tumor Tracking
Wang, R., Liang, X., Zhu, X., & Xie, Y. (2018) · IEEE Access · doi:10.1109/access.2018.2869780 · Cited by 96
“Recurrent neural network(RNN) is specifically suitable for processing time series data and has been widely applied in many fields [17], [26], [27].”
BackgroundA deep learning framework for predicting cyber attacks rates
Fang, X., Xu, M., Xu, S., & Zhao, P. (2019) · EURASIP Journal on Information Security · doi:10.1186/s13635-019-0090-6 · Cited by 98
A Sentiment Analysis Method of Capsule Network Based on BiLSTM
Dong, Y., Fu, Y., Wang, L., Chen, Y., Dong, Y., & Li, J. (2020) · IEEE Access · doi:10.1109/access.2020.2973711 · Cited by 97
“Inrecent years, RNN has been applied in many fields, such as language model [13], speech recognition [14] and so on.”
MethodLearning Content-Weighted Deep Image Compression
Li, M., Zuo, W., Gu, S., You, J., & Zhang, D. D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983926 · Cited by 107
Coherent Dialogue with Attention-Based Language Models
Mei, H., Bansal, M., & Walter, M. R. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10961 · Cited by 92
“Our work explores the ability of recurrent neural network language models (Bengio et al. 2003; Mikolov 2010) to interpret and generate natural language conversations while still maintaining a relatively simple architecture.”
BackgroundLookNN: Neural network with no multiplication
Samragh, M., Imani, M., Koushanfar, F., & Rosing, T. (2017) · Design, Automation, and Test in Europe · doi:10.23919/date.2017.7927280 · Cited by 98
“Neural networks (NNs) are machine learning models that have shown promising accuracy in many tasks including but not limited to computer vision [1], voice recognition [2], [3], natural language processing [4], and health care [5].”
BackgroundA survey on deep neural network-based image captioning
Liu, X., Xu, Q., & Wang, N. (2018) · The Visual Computer · doi:10.1007/s00371-018-1566-y · Cited by 95
“Fortunately, new and effective LMs have appeared with the development of neural networks, such as recurrent neural networks (RNN) [37–39], bidirectional RNN [40,41], long– short-term memory (LSTM) networks [42] and its variant gated recurrent units [43].”
BackgroundRecurrent Memory Networks for Language Modeling
Tran, K., Bisazza, A., & Monz, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1036 · Cited by 84
“Recurrent Neural Networks (RNNs) (Elman, 1990; Mikolov et al., 2010) are remarkably powerful models for sequential data.”
Information Maximizing Visual Question Generation
Krishna, R., Bernstein, M., & Fei-Fei, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00211 · Cited by 84
“…task of transforming visual scenes to language, questions [47, 44], answers [64, 2] or captions [56, 30], has widely adopted an image-to-sequence architecture that encodes the image through a convolutional neural network (CNN) [33] and then decodes the language with a recurrent neural network [42].”
CNN-Based Polarimetric Decomposition Feature Selection for PolSAR Image Classification
Yang, C., Hou, B., Ren, B., Hu, Y., & Jiao, L. (2019) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2922978 · Cited by 90
“The common deep learning models contain convolutional neural network (CNN) [20], deep belief network (DBN) [30], and recurrent neural network (RNN) [31].”
BackgroundMultilingual End-to-End Speech Translation
Inaguma, H., Duh, K., Kawahara, T., & Watanabe, S. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003832 · Cited by 92
“This is intuitive since the E2E-ST model skipped the ASR decoder, RNNLM in the source language, and MT encoder parts.”
MethodCodeFill: multi-token code completion by jointly learning from structure and naming sequences
Izadi, M., Gismondi, R., & Gousios, G. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510172 · Cited by 86
“Recently, Neural LMs have gained popularity due to their superior performance and generalization capabilities [14, 35].”
BackgroundLuo, X., Zhou, W., Wang, W., Zhu, Y., & Deng, J. (2017) · IEEE Access · doi:10.1109/access.2017.2785229 · Cited by 92
“Generally, deep learning methods require a large amount of manually labeled data, which spend much additional time and cost [10].”
Spatio-Temporal Data Mining: A Survey of Problems and MethodsPreprint
Atluri, G., Karpatne, A., & Kumar, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04710 · Cited by 92
Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation
Su, J., Tan, Z., Xiong, D., Ji, R., Shi, X., & Liu, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10968 · Cited by 94
“f word order. Therefore, many researchers resort to order-sensitive models, falling into one of two classes: (1) sequence models and (2) topological models. The most typical sequence models are RNNs (Mikolov et al., 2010) with LSTM (Hochreiter and Schmidhuber, 1997; Sutskever et al., 2014; Le and Zuidema, 2015a; Zhu e…”
BackgroundMethodPartial Discharge Recognition with a Multi-Resolution Convolutional Neural Network
Li, G., Wang, X., Li, X., Yang, A., & Rong, M. (2018) · Sensors · doi:10.3390/s18103512 · Cited by 89
“Recurrent neural network (RNNs) [42] is the most popular choice for sequence mining tasks such as speech recognition, natural language translation, and video recogniti n.”
BackgroundDeep-learning-based burned area mapping using the synergy of Sentinel-1&2 data
Zhang, Q., Ge, L., Zhang, R., Metternicht, G. I., Du, Z., Kuang, J., & Xu, M. G. (2021) · Remote Sensing of Environment · doi:10.1016/j.rse.2021.112575 · Cited by 97
Efficient lattice rescoring using recurrent neural network language models
Liu, X., Wang, Y.-Q., Chen, X., Gales, M., & Woodland, P. C. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854535 · Cited by 91
Recurrent neural network language modeling for code switching conversational speech
Adel, H., Vu, N. T., Kraus, F., Schlippe, T., Li, H., & Schultz, T. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639306 · Cited by 89
“[2] introduced a refined form of neural networks for the task of language modeling.”
BackgroundCo-learning of Word Representations and Morpheme Representations
Qiu, S., Cui, Q., Bian, J., Gao, B., & Liu, T. (2014) · Cited by 85
Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation
Schwenk, H., Rousseau, A., & Attik, M. (2012) · HAL · Cited by 90
Novak, J. R., Minematsu, N., & Hirose, K. (2015) · Natural Language Engineering · doi:10.1017/s1351324915000315 · Cited by 87
“Recurrent Neural Network Language Models have recently enjoyed a resurgence in popularity in the context of Automatic Speech Recognition applications (Mikolov et al. 2010).”
BackgroundDeep Learning in Lane Marking Detection: A Survey
Zhang, Y., Lu, Z., Zhang, X., Xue, J., & Liao, Q. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3070111 · Cited by 115
Social media user behavior analysis applied to the fashion and apparel industry in the big data era
Xue, Z., Li, Q., & Zeng, X. (2023) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2023.103299 · Cited by 110
Wei, D., Wang, B., Lin, G., Liu, D., Dong, Z. Y., Liu, H., & Liu, Y. (2017) · Energies · doi:10.3390/en10030406 · Cited by 98
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 83
Zhao, J., Bao, W., Zhang, F., Zhu, S., Liu, Y., Lu, H., Shen, M., & Ye, Z. (2018) · Aquaculture · doi:10.1016/j.aquaculture.2018.04.064 · Cited by 91
Are All Languages Equally Hard to Language-Model?
Cotterell, R., Mielke, S. J., Eisner, J., & Roark, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2085 · Cited by 80
“A large part of this is due to the use of recurrent neural networks (Mikolov et al., 2010), which can generalize about Figure 1 : The primary findings of our paper are evinced in these plots.”
BackgroundInfluentialImportance of Search and Evaluation Strategies in Neural Dialogue Modeling
Kulikov, I., Miller, A., Cho, K., & Weston, J. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8609 · Cited by 83
“Each conditional distribution is modelled by a neural network, and popular choices include recurrent neural networks (Mikolov et al., 2010; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014), convolutional networks (Dauphin et al.”
BackgroundSun, G., Zhang, Y., Weiss, R. J., Cao, Y. J., Zen, H., Rosenberg, A., Ramabhadran, B., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053436 · Cited by 119
“This is similar to training a neural language model [19].”
Xu, S., & Noh, H. Y. (2020) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2020.107374 · Cited by 97
Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression
Li, M., Ma, K., You, J., Zhang, D. D., & Zuo, W. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2985225 · Cited by 104
“In image processing, PixelRNN [10] and PixelCNN [11] are among the first attempts to exploit long-range pixel dependencies for image generation.”
BackgroundInfluentialA Deep Reinforcement Learning Framework for Spectrum Management in Dynamic Spectrum Access
Song, H., Liu, L., Ashdown, J., & Yi, Y. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3052691 · Cited by 94
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 83
Artificial neural network for predicting nuclear power plant dynamic behaviors
El-Sefy, M., Yosri, A., El-Dakhakhni, W. W., Nagasaki, S., & Wiebe, L. D. A. (2021) · Nuclear Engineering and Technology · doi:10.1016/j.net.2021.05.003 · Cited by 96
Recurrent neural network based recommendation for time heterogeneous feedback
Wu, C., Wang, J., Liu, J., & Liu, W. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.06.028 · Cited by 84
Wang, Z., Cao, L., Zhang, Z., Gong, X., Sun, Y., & Wang, H. (2018) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.4413 · Cited by 83
You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
Schuster, R., Song, C., Tromer, E., & Shmatikov, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02220 · Cited by 61
“Modern language models are based on recurrent neural-network architectures [38] such as LSTMs [56] and, more recently, Transformers [17, 47].”
BackgroundNormalizing tweets with edit scripts and recurrent neural embeddings
Chrupała, G. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2111 · Cited by 81
“More recently SRNs were used as language models for speech recognition and shown to outperform classical n-gram language models (Mikolov et al., 2010; Mikolov, 2012b).”
MethodMOOCRC: A Highly Accurate Resource Recommendation Model for Use in MOOC Environments
Zhang, H., Huang, T., Lv, Z., Liu, S., & Yang, H. Q. (2018) · Mobile Networks and Applications · doi:10.1007/s11036-018-1131-y · Cited by 87
“…models such as deep belief networks (DBNs) [13 – 15], convolutional neural networks (CNNs) [16], stacked denoising autoencoders (SDAs) [17], and recurrent neural network (RNNs) [18] have achieved remarkable results in image processing, natural language semantics, speech, and text classification.”
BackgroundCSAN: Contextual Self-Attention Network for User Sequential Recommendation
Huang, X., Qian, S., Fang, Q., Sang, J., & Xu, C. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240609 · Cited by 108
“Recently, Recurrent Neural Network (RNN) approaches have achieved much success in sequence modeling [22].”
BackgroundMethodEmerging memory technologies for neuromorphic computing
Kim, C.-H., Lim, S., Woo, S. Y., Kang, W.-M., Seo, Y.-T., Lee, S., Lee, S., Kwon, D., Oh, S., Noh, Y., et al. (2018) · Nanotechnology · doi:10.1088/1361-6528/aae975 · Cited by 91
Compositional Distributional Semantics with Long Short Term Memory
Le, P., & Zuidema, W. H. (2015) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s15-1002 · Cited by 101
Automatic extraction of named entities of cyber threats using a deep Bi-LSTM-CRF network
Kim, G., Lee, C., Jo, J., & Lim, H. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01122-6 · Cited by 87
“We briefly describe sequence tagging models, known as LSTM [18], based on an RNN model achieving state-of-the-art performance across different research fields [9, 27].”
MethodRepresentation Learning for Natural Language Processing
Liu, Z., Lin, Y., & Sun, M. (2020) · Springer eBooks · doi:10.1007/978-981-15-5573-2 · Cited by 140
Model Transfer for Tagging Low-resource Languages using a Bilingual Dictionary
Fang, M., & Cohn, T. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2093 · Cited by 81
Multimodal machine translation through visuals and speech
Sulubacak, U., Caglayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2019) · Machine Translation · doi:10.1007/s10590-020-09250-0 · Cited by 105
“One of the first NMT studies by Kalchbrenner and Blunsom (2013) combined recurrent language modelling (Mikolov et al. 2010) and convolutional neural networks (CNN) to improve the performance of SMT systems through rescoring.”
BackgroundMethodMulti-Aspect Aware Session-Based Recommendation for Intelligent Transportation Services
Zhang⋆, Y., Li, Y., Wang, R., Hossain, M. S., & Lu, H. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2990214 · Cited by 81
Uncertainty-Aware Curriculum Learning for Neural Machine Translation
Zhou, Y., Yang, B., Wong, D. F., Wan, Y., & Chao, L. S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.620 · Cited by 133
“R NN LM (Mikolov et al., 2010), and a multilingual neural LM that trained on billions of external sentences –”
MethodRich Visual Knowledge-Based Augmentation Network for Visual Question Answering
Zhang, L., Liu, S., Liu, D., Zeng, P., Li, X., Song, J., & Gao, L. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3017530 · Cited by 92
Hierarchical deep multi-modal network for medical visual question answering
Gupta, D., Suman, S., & Ekbal, A. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113993 · Cited by 88
ReTransformer: ReRAM-based processing-in-memory architecture for transformer acceleration
Yang, X., Yan, B., Li, H., & Chen, Y. (2020) · doi:10.1145/3400302.3415640 · Cited by 195
“Today, the resounding success of deep learning has advanced NLP research by introducing different architectures of deep neural network (DNN) models, such as LSTM [13], RNN [21], GRU [5].”
BackgroundPham, T., Bui, T., Mai, L., & Nguyen, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.98 · Cited by 79
“Language models (Mikolov et al., 2010) with long short-term memory (LSTM) units (Hochreiter and Schmidhuber, 1997) were able to recover the original word order of a sentence from randomly-shuffled words even without any explicit syntactic information (Schmaltz et al.”
BackgroundA multi-rate sampling data fusion method for fault diagnosis and its industrial applications
Huang, K., Wu, S., Li, Y., Yang, C., & Gui, W. (2021) · Journal of Process Control · doi:10.1016/j.jprocont.2021.06.003 · Cited by 82
Convolutional Neural Network-Aided DP-64 QAM Coherent Optical Communication Systems
Li, C., Wang, Y., Wang, J., Yao, H., Liu, X., Gao, R., Yang, L., Xu, H., Zhang, Q., Ma, P., et al. (2022) · Journal of Lightwave Technology · doi:10.1109/jlt.2022.3146839 · Cited by 81
AlexNet architecture based convolutional neural network for toxic comments classification
Singh, I., Goyal, G., & Chandel, A. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.06.007 · Cited by 101
“This trend has been fueled by the effectiveness of word embeddings (Mikolov et al., 2010) and deep learning methods (Aken et al., 2018).”
Lu, Y., Wang, W. W., Hu, X., Xu, P., Zhou, S., & Cai, M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3173944 · Cited by 99
“Recurrent Neural Networks (RNNs) are specialized for handling sequential dependency which makes them the default choices to resolve the time-series problems [35], [36].”
BackgroundYang, C., Shi, X., Jie, L., & Han, J. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219821 · Cited by 99
“focused on image processing, we mainly review deep learning models on sequentialdata,becausewemodeluseractivitiesasmulti-dimensional time series. Current research on deep learning has agreed that RNN [33] is the best model for sequential data. Its network design with loops allows information to persist and has been wi…”
BackgroundMethodReinforcement Learning Based Text Style Transfer without Parallel Training Corpus
Gong, H., Bhat, S., Wu, L., Xiong, J., & Hwu, W. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1320 · Cited by 106
“The language model we use is a two-layer recurrent neural network pre-trained on the corpus in the target style so as to maximize the likelihood of the target sentences (Mikolov et al., 2010; Jozefowicz et al., 2016).”
MethodTop-down Tree Long Short-Term Memory Networks
Zhang, X., Lu, L., & Lapata, M. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1035 · Cited by 112
“We used an initial learning rate of 1.0 for all experiments and when there was no significant improvement in loglikelihood on the validation set, the learning rate was divided by 2 per epoch until convergence (Mikolov et al., 2010).”
MethodInfluentialWord Embedding Composition for Data Imbalances in Sentiment and Emotion Classification
Xu, R., Chen, T., Xia, Y., Lu, Q., Liu, B., & Wang, X. (2015) · Cognitive Computation · doi:10.1007/s12559-015-9319-y · Cited by 88
“proposed a recurrent neural network-based language model (RNNLM) [31] in which the context of a word is represented by neurons with recurrent connections such that there is no limit on the context window.”
BackgroundBidirectional recurrent neural network language models for automatic speech recognition
Arısoy, E., Sethy, A., Ramabhadran, B., & Chen, S. F. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7179007 · Cited by 100
“Recurrent neural networks have been shown to yield superior performance in language modeling over a wide range of domains [1, 2, 3].”
BackgroundDeep Sentiment Representation Based on CNN and LSTM
Huang, Q., Chen, R., Zheng, X., & Dong, Z. (2017) · International Conference on Green Informatics (ICGI) · doi:10.1109/icgi.2017.45 · Cited by 92
“As another popular network, RNNs can deal with sequence data and learn long-term dependence [9].”
BackgroundEnabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework
Pardos, Z. A., Tang, S., Davis, D., & Le, C. V. (2017) · Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale · doi:10.1145/3051457.3051471 · Cited by 78
“dimensions is added to convert input indices to a continuous multi dimensional space, a technique commonly used in language modeling [23].”
MethodPetrophysical property estimation from seismic data using recurrent neural networks
Alfarraj, M., & AlRegib, G. (2018) · Second International Meeting for Applied Geoscience & Energy · doi:10.1190/segam2018-2995752.1 · Cited by 97
Multiple Spatio-temporal Feature Learning for Video-based Emotion Recognition in the Wild
Lu, C., Zheng, W., Li, C., Tang, C., Liu, S., Yan, S., & Zong, Y. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3264992 · Cited by 88
Towards Neural Mixture Recommender for Long Range Dependent User Sequences
Tang, J., Belletti, F., Jain, S., Chen, M., Beutel, A., Xu, C., & Chi, E. H. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313650 · Cited by 101
“Deep neural networks have gained tremendous success in the fields of Computer Vision [28, 31] and Natural Language Processing [2, 36].”
BackgroundShi, X., Wang, T., Wang, L., Liu, H., & Yan, N. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023190 · Cited by 101
“…strong modeling power to learn sequential information, as neurons send feedback signals to the other neurons in the same hidden layer (i.e., the input of the hidden layer considers the output from the preceding time steps), which provides RNN with the memory from the states in the history [17].”
BackgroundRadar Target Characterization and Deep Learning in Radar Automatic Target Recognition: A Review
Jiang, W. Q., Wang, Y., Li, Y., Lin, Y., & Shen, W. (2023) · Remote Sensing · doi:10.3390/rs15153742 · Cited by 93
A Survey of Industrial AIoT: Opportunities, Challenges, and Directions
Awaisi, K. S., Ye, Q., & Sampalli, S. (2024) · IEEE Access · doi:10.1109/access.2024.3426279 · Cited by 109
Convolutive Bottleneck Network features for LVCSR
Vesely, K., Karafiát, M., & Grézl, F. (2011) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2011.6163903 · Cited by 80
“First, it is possible to use more advanced language model, for example 4-gram or Recurrent Neural Network LM [10].”
MethodSelf-citationChen, X., Liu, X., Qian, Y., Gales, M. J. F., & Woodland, P. C. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472829 · Cited by 77
Image Caption Generation with Part of Speech Guidance
He, X., Shi, B., Bai, X., Xia, G.-S., Zhang, Z., & Dong, W. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.10.018 · Cited by 79
Insider Threat Detection with Long Short-Term Memory
Lu, J., & Wong, R. K. (2019) · Proceedings of the Australasian Computer Science Week Multiconference · doi:10.1145/3290688.3290692 · Cited by 75
“Our proposed system uses a Recurrent Neural Network (RNN) as the core model [22].”
MethodPower Consumption Predicting and Anomaly Detection Based on Long Short-Term Memory Neural Network
Wang, X., Zhao, T., Liu, H., & He, R. (2019) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda.2019.8725704 · Cited by 85
“[11] explored the effectiveness of LSTM in power demand forecasting area and found that LSTM outperforms both Gradient Boosting Tree (GBT) and Support Vector Regression (SVR) [12].”
BackgroundAttention Recurrent Autoencoder Hybrid Model for Early Fault Diagnosis of Rotating Machinery
Kong, X., Li, X., Zhou, Q., Hu, Z., & Shi, C. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3051948 · Cited by 94
“The RNN model was proposed by Mikolov et al. [33].”
MethodNeuroPose: 3D Hand Pose Tracking using EMG Wearables
Liu, Y., Zhang, S., & Gowda, M. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449890 · Cited by 76
“In particular, NeuroPose explores architectures in Recurrent Neural Networks (RNN)[58], Encoder-Decoder[21], ResNets[40] in achieving a high accuracy.”
Deep scaffold hopping with multimodal transformer neural networks
Zheng, S., Lei, Z., Ai, H., Chen, H., Deng, D., & Yang, Y. (2021) · Journal of Cheminformatics · doi:10.1186/s13321-021-00565-5 · Cited by 95
Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition
Zheng, X., Zhang, C., & Woodland, P. C. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688232 · Cited by 125
Telepresence Social Robotics towards Co-Presence: A Review
Almeida, L., Menezes, P. J., & Dias, J. M. M. (2022) · Applied Sciences · doi:10.3390/app12115557 · Cited by 98
Detecting predatory conversations in social media by deep convolutional neural networks
Ebrahimi, M., Suen, C. Y., & Ormandjieva, O. (2016) · Digital Investigation · doi:10.1016/j.diin.2016.07.001 · Cited by 78
Dasgupta, T., Naskar, A., Dey, L., & Saha, R. (2018) · Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications · doi:10.18653/v1/w18-3713 · Cited by 75
“Both recurrent neural networks (Williams and Zipser, 1989; Mikolov et al., 2010) and convolution neural networks (LeCun et al., 1998; Kim, 2014) have been used to automatically score input essays.”
BackgroundMethodFake News Detection with Deep Diffusive Network Model.Preprint
Zhang, J., Cui, L., Fu, Y., & Gouza, F. B. (2018) · arXiv (Cornell University) · Cited by 89
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Augmenting Neural Response Generation with Context-Aware Topical Attention
Dziri, N., Kamalloo, E., Mathewson, K., & Zaiane, O. (2019) · Proceedings of the First Workshop on NLP for Conversational AI · doi:10.18653/v1/w19-4103 · Cited by 75
“…success of deep neural networks in natural language processing tasks such as machine translation (Sutskever et al., 2014) and language modeling (Mikolov et al., 2010), there has been growing research interest in building data-driven dialogue systems that are capable of conducting conversations…”
BackgroundMa, X., Zeng, J., Peng, L., Fortino, G., & Zhang⋆, Y. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.10.041 · Cited by 84
Predicting the helpfulness score of online reviews using convolutional neural network
Saumya, S., Singh, J. P., & Dwivedi, Y. K. (2019) · Soft Computing · doi:10.1007/s00500-019-03851-5 · Cited by 87
“This paper shows that substantial enhancements can be attained by learning continuous sentence representation using a CNN model.”
BackgroundCombining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
Wang, Z., Chai, J., & Xia, S. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2938520 · Cited by 98
“We address the challenge by using a LSTM architecture [2], which has recently demonstrated impressive performance on tasks as varied as speech recognition [3], [4], language translation [5], [6], [7], and image generation [8].”
MethodFrom static to dynamic word representations: a survey
Wang, Y., Hou, Y., Che, W., & Liu, T. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01069-8 · Cited by 102
“A subset of such down-stream tasks includes Language Modeling (LM) [7, 73], Chunking [20, 110], Part-of-Speech Tagging [20], Named Entity Recognition (NER) [20, 86], Sentiment Analysis (SA) [97, 109], Semantic Role Labeling (SRL) [20], Dependency Parsing [13], Machine Translation (MT) [16], Nature…”
BackgroundMethod
Metadata
- Authors
- Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S.
- Affiliations
- Brno University of Technology (CZ)
- Johns Hopkins University (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 1045-1048
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W179875071
- Cited by
- 5,390
- References
- 14
- Citations by year
- 2026: 66 · 2025: 160 · 2024: 279 · 2023: 404 · 2022: 418 · 2021: 526 · 2020: 741 · 2019: 780 · 2018: 649 · 2017: 504 · 2016: 343 · 2015: 254 · 2014: 147 · 2013: 66 · 2012: 35
- FWCI
- 30.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- recurrent neural networks, language models, perplexity reduction, speech recognition, word error rate, n-gram comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.