The DOI matches its DataCite record; 28 of the 31 references it cites were found in a registry or an index.
Regularizing and Optimizing LSTM Language Models
Open accessAbstract
Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For a baseline comparison, the AWD-LSTM [48] model, fine-tuned on the entire IMDB corpus [47], gave us a score of only 0.23.”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,705
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058
Self-citationMoses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735
(2013) · Cited by 3,740
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. M. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,573
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 3,174
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Research works printed but not linked: 16
- Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In International Conference on Machine Learning , pp. 1120-1128, 2016.
- Ba, J., Kiros, J., and Hinton, G. E. Layer normalization. CoRR , abs/1607.06450, 2016. · arXiv
- Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. C. Recurrent batch normalization. CoRR , abs/1603.09025, 2016. · arXiv
- Journal of Machine Learning Research , 12(Jul):21212159, 2011.
- Gal, Y. and Ghahramani, Z. A theoretically grounded application of dropout in recurrent neural networks. In NIPS , 2016.
- Hinton, G. E. Connectionist learning procedures. Artificial intelligence , 40(1-3):185-234, 1989.
- Jing, L., Shen, Y., Dubˇ cek, T., Peurifoy, J., Skirlo, S., Tegmark, M., and Soljaˇ ci´ c, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN. arXiv preprint arXiv:1612.05231 , 2016. · arXiv
- Keskar, N. and Saon, G. A nonmonotone learning rate strategy for sgd training of deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on , pp. 4974-4978. IEEE, 2015.
- Kim, Y., Jernite, Y ., Sontag, D., and Rush, A. M. Characteraware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence , 2016.
- Melis, G., Dyer, C., and Blunsom, P. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint arXiv:1707.05589 , 2017. · arXiv
- Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. Recurrent neural network based language model. In INTERSPEECH , 2010.
- Panageas, I. and Piliouras, G. Gradient descent converges to minimizers: The case of non-isolated critical points. CoRR, abs/1605.00405 , 2016. · arXiv
- Polyak, B. and Juditsky, A. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838-855, 1992.
- Semeniuta, S., Severyn, A., and Barth, E. Recurrent dropout without memory loss. In COLING , 2016.
- Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26-31, 2012.
- Wan, L., Zeiler, M., Zhang, S., LeCun, Y, and Fergus, R. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML-13) , pp. 1058-1066, 2013.
Cited by
Sorted by citations
Items 201–300 of 695
Depression and Anxiety Prediction Using Deep Language Models and Transfer Learning
Rutowski, T., Shriberg, E. E., Harati, A., Lü, Y., Chlebek, P., & Oliveira, R. (2020) · International Conference on Behavioral, Economic, and Socio-Cultural Computing · doi:10.1109/besc51023.2020.9348290 · Cited by 24
Transfer learning for small molecule retention predictions
Osipenko, S. V., Botashev, K., Николаев, Е. Н., & Kostyukevich, Y. (2021) · Journal of Chromatography A · doi:10.1016/j.chroma.2021.462119 · Cited by 26
Pre-train, Interact, Fine-tune: a novel interaction representation for text classification
Zheng, J., Cai, F., Chen, H., & de Rijke, M. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102215 · Cited by 25
Stolen Probability: A Structural Weakness of Neural Language Models
Demeter, D., Kimmel, G. J., & Downey, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.198 · Cited by 24
Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
Sun, X., Zhang, Z., Ren, X., Luo, R., & Li, L. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17385 · Cited by 29
Optimizing Deep Neural Networks Through Neuroevolution With Stochastic Gradient Descent
Zhang, H., Hao, K., Gao, L., Wei, B., & Tang, X. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3146327 · Cited by 25
Ragheb, W., Azé, J., Bringay, S., & Servajean, M. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3078898 · Cited by 22
DRRNets: Dynamic Recurrent Routing via Low-Rank Regularization in Recurrent Neural Networks
Shan, D., Luo, Y., Zhang, X., & Zhang, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105818 · Cited by 28
Bashar, M. A., Nayak, R., Luong, K., & Balasubramaniam, T. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00780-w · Cited by 24
Filice, R. W. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.010 · Cited by 22
Nguyen, M., Ngo, G. H., & Chen, N. F. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2955246 · Cited by 23
Cohen, T., & Pakhomov, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.176 · Cited by 28
Automatic Incident Triage in Radiation Oncology Incident Learning System
Syed, K., Sleeman, W. C., Hagan, M. P., Palta, J. R., Kapoor, R., & Ghosh, P. (2020) · Healthcare · doi:10.3390/healthcare8030272 · Cited by 22
A real-time hostile activities analyses and detection system
Dadkhah, S., Shoeleh, F., Yadollahi, M. M., Zhang, X., & Ghorbani, A. A. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107175 · Cited by 23
Recurrent neural network pruning using dynamical systems and iterative fine-tuning
Chatzikonstantinou, C., Konstantinidis, D., Dimitropoulos, K., & Daras, P. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.001 · Cited by 23
Modeling Vocabulary for Big Code Machine LearningPreprint
Babii, H., Janes, A., & Robbes, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01873 · Cited by 24
DeLighT: Very Deep and Light-weight TransformerPreprint
Mehta, S., Ghazvininejad, M., Iyer, S. V., Zettlemoyer, L., & Hajishirzi, H. (2020) · arXiv (Cornell University) · Cited by 21
Quantifying Long Range Dependence in Language and User Behavior to improve RNNs
Belletti, F., Chen, M., & Ed, H. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330944 · Cited by 21
Natural Language Word Prediction Model Based on Multi-Window Convolution and Residual Network
Yang, J., Wang, H., & Guo, K. (2020) · IEEE Access · doi:10.1109/access.2020.3031200 · Cited by 21
Liu, S., Mocanu, D. C., Pei, Y., & Pechenizkiy, M. (2021) · TU/e Research Portal · doi:10.48550/arxiv.2101.09048 · Cited by 21
Vision-Based Topological Mapping and Navigation With Self-Organizing Neural Networks
Hu, Y., Subagdja, B., Tan, A., & Yin, Q. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3084212 · Cited by 21
Sentiment Analysis in Code-Mixed Telugu-English Text with Unsupervised Data Normalization
Varma, K. S. S., Sathineni, P., & Mamidi, R. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_086 · Cited by 21
A Survey on Dropout Methods and Experimental Verification in Recommendation
Li, Y., Ma, W., Chen, C., Zhang, M., Liu, Y., Ma, S., & Yang, Y. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3187013 · Cited by 30
On Training Recurrent Networks with Truncated Backpropagation Through time in Speech Recognition
Tang, H., & Glass, J. R. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639517 · Cited by 20
Major–Minor Long Short-Term Memory for Word-Level Language Model
Shuang, K., Li, R., Gu, M., Loo, J., & Su, S. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2947563 · Cited by 22
You May Not Need AttentionPreprint
Press, O., & Smith, N. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13409 · Cited by 27
Automated Business Process Discovery from Unstructured Natural-Language Documents
Chambers, A. J., Stringfellow, A. M., Luo, B. B., Underwood, S. J., Allard, T., Johnston, I. A., Brockman, S., Shing, L., Wollaber, A., & VanDam, C. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-66498-5_18 · Cited by 20
Yu, S., Song, J., Kim, H., Lee, S., Ryu, W.-J., & Yoon, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.3 · Cited by 21
Liu, Z., Xu, Y., Winata, G. I., & Fung, P. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5327 · Cited by 23
USMPep: universal sequence models for major histocompatibility complex binding affinity prediction
Vielhaben, J., Wenzel, M. A., Samek, W., & Strodthoff, N. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03631-1 · Cited by 20
Neural Architecture Search as Sparse Supernet
Wu, Y. X., Liu, A., Huang, Z., Zhang, S., & Gool, L. V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17243 · Cited by 20
Localization of Fake News Detection via Multitask Transfer LearningPreprint
Cruz, J. C. B., Tan, J. A., & Cheng, C. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09295 · Cited by 21
Improved Analysis of Clipping Algorithms for Non-convex OptimizationPreprint
Zhang, B., Jin, J., Fang, C., & Wang, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02519 · Cited by 20
Advancing Sequence-to-Sequence Based Speech Recognition
Tüske, Z., Audhkhasi, K., & Saon, G. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3018 · Cited by 20
HR-STAN: High-Resolution Spatio-Temporal Attention Network for 3D Human Motion Prediction
Medjaouri, O., & Desai, K. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00286 · Cited by 19
Bonde, A., Lorenzen, S. S., Brixen, G., Troelsen, A., & Sillesen, M. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1249835 · Cited by 19
Time Adaptive Recurrent Neural Network
Kag, A., & Saligrama, V. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01490 · Cited by 19
Noisin: Unbiased Regularization for Recurrent Neural NetworksPreprint
Dieng, A. B., Ranganath, R., Altosaar, J., & Blei, D. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01500 · Cited by 23
Model-based Asynchronous Hyperparameter and Neural Architecture SearchPreprint
Klein, A., Tiao, L. C., Lienart, T., Archambeau, C., & Seeger, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10865 · Cited by 40
Proceedings of The Third Workshop on Representation Learning for NLP
(2018) · doi:10.18653/v1/w18-30 · Cited by 20
He, Y., Zhang, Q., Wang, S., Chen, Z., Cui, Z. Y., Guo, Z.-H., & Huang, D.-S. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3165592 · Cited by 18
Pyramidal Recurrent Unit for Language Modeling
Mehta, S., Koncel-Kedziorski, R., Rastegari, M., & Hajishirzi, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1491 · Cited by 18
Thai Named Entity Recognition Using Bi-LSTM-CRF with Word and Character Representation
Thattinaphanich, S., & Prom–on, S. (2019) · doi:10.1109/incit.2019.8912091 · Cited by 18
DATA: Differentiable ArchiTecture Approximation With Distribution Guided Sampling
Zhang, X., Chang, J., Guo, Y., Meng, G., Xiang, S., Lin, Z., & Pan, C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3020315 · Cited by 19
Effective Batching for Recurrent Neural Network Grammars
Noji, H., & Oseki, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.380 · Cited by 19
Wide-Ranging Review Manipulation Attacks
Kaghazgaran, P., Alfifi, M., & Caverlee, J. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3358034 · Cited by 21
GNN-LM: Language Modeling based on Global Contexts via GNNPreprint
Meng, Y., Zong, S., Li, X., Sun, X., Zhang, T., Wu, F. Q., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08743 · Cited by 30
Training Recurrent Neural Networks via Forward Propagation Through Time
Kag, A., & Saligrama, V. (2021) · Cited by 17
Transformative reconstruction of missing acoustic well logs using multi-head self-attention BiRNNs
Fan, X., Meng, F. J., Deng, J., Semnani, A., Zhao, P., & Zhang, Q. (2024) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2024.213513 · Cited by 18
Relevance-Promoting Language Model for Short-Text Conversation
Li, X., Li, P., Bi, W., Liu, X., & Lam, W. P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6340 · Cited by 16
Understanding language-elicited EEG data by predicting it from a fine-tuned language model
Schwartz, D., & Mitchell, T. M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1005 · Cited by 17
Aggregated Momentum: Stability Through Passive DampingPreprint
Lucas, J., Sun, S., Zemel, R. S., & Grosse, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.00325 · Cited by 45
Language modelling for biological sequences – curated datasets and baselinesPreprint
Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.09.983585 · Cited by 19
Zhang, L., Zhu, Y., Banda, O. A. V., Du, L., Gan, L., & Li, X. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.118628 · Cited by 16
Towards Zero-shot Language Modeling
Ponti, E. M., Vulić, I., Cotterell, R., Reichart, R., & Korhonen, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1288 · Cited by 16
Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach
Du, W., Lin, Z., Shen, Y., O’Donnell, T. J., Bengio, Y., & Zhang, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.591 · Cited by 18
Sundar, A., Ramakrishnan, A., Balaji, A., & Thenmozhi, D. (2021) · SN Computer Science · doi:10.1007/s42979-021-00943-8 · Cited by 18
A transfer learning approach for reaction discovery in small data situations using generative model
Singh, S., & Sunoj, R. B. (2022) · iScience · doi:10.1016/j.isci.2022.104661 · Cited by 15
Leveraging Similar Users for Personalized Language Modeling with Limited Data
WELCH, C. A., Gu, C., Kummerfeld, J. K., Pérez‐Rosas, V., & Mihalcea, R. F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.122 · Cited by 17
Modeling Local Dependence in Natural Language with Multi-Channel Recurrent Neural Networks
Xu, C., Huang, W., Wang, H., Wang, G., & Liu, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015525 · Cited by 15
Reading selectively via Binary Input Gated Recurrent Unit
Li, Z., Wang, P., Lu, H., & Cheng, J. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/705 · Cited by 15
Towards a Unified End-to-End Approach for Fully Unsupervised Cross-Lingual Sentiment Analysis
Feng, Y., & Wan, X. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1097 · Cited by 15
AutoShrink: A Topology-Aware NAS for Discovering Efficient Neural Architecture
Zhang, T., Cheng, H.-P., Li, Z., Yan, F., Huang, C., Li, H. H., & Chen, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6163 · Cited by 14
CapsNet-SSP: multilane capsule network for predicting human saliva-secretory proteins
Du, W., Sun, Y., Li, G., Cao, H., Pang, R., & Li, Y. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03579-2 · Cited by 16
Hierarchical and Self-Attended Sequence Autoencoder
Chien, J., & Wang, C.-W. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3068187 · Cited by 15
PaLM: A Hybrid Parser and Language Model
Peng, H., Schwartz, R., & Smith, N. A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1376 · Cited by 16
A novel anomaly detection method for gas turbines using weight agnostic neural network search
Zhong, S., Liu, D., Lin, L., Zhao, M., Fu, X., & Guo, F. (2020) · Asia-Pacific International Symposium on Advanced Reliability and Maintenance Modeling (APARM) · doi:10.1109/aparm49247.2020.9209368 · Cited by 15
SDM: Sequential Deep Matching Model for Online Large-scale Recommender SystemPreprint
Lv, F., Jin, T., Yu, C., Sun, F., Lin, Q., Yang, K., & Ng, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00385 · Cited by 15
On Convergence and Generalization of Dropout TrainingPreprint
Mianjy, P., & Arora, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12711 · Cited by 15
Improving Healthcare Outcomes with Learning Models for Machine Commonsense Reasoning Systems
Saeedi, S., Fong, A. C. M., Gupta, A. K., & Carr, S. (2022) · IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE) · doi:10.1109/rasse54974.2022.10019733 · Cited by 14
Transfer Learning for Health-related Twitter Data
Dirkson, A. R., & Verberne, S. (2019) · Social Media Mining for Health Research and Applications Workshop and Shared Tasks (SMM4H) · doi:10.18653/v1/w19-3212 · Cited by 15
Understanding Gradient Clipping In Incremental Gradient Methods.
Jiang, Q., Wu, Y., Zhuang, B., Wang, S., & Xiao, J. Z. (2021) · International Conference on Artificial Intelligence and Statistics · Cited by 14
Revisiting and Improving Retrieval-Augmented Deep Assertion Generation
Sun, W., Li, H., Yan, M. L., Lei, Y., & Zhang, H. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00090 · Cited by 19
Zhou, Z., Chen, Y., Liu, X., Zhang, X., & Zhang, H. (2024) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2023.2298296 · Cited by 14
Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences
Jacob, A. P., Lin, Z., Sordoni, A., & Bengio, Y. (2018) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w18-3020 · Cited by 15
Noji, H., & Takamura, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.309 · Cited by 18
Identifying Twitter users who repost unreliable news sources with linguistic information
Mu, Y., & Αλέτρας, Ν. (2020) · PeerJ Computer Science · doi:10.7717/peerj-cs.325 · Cited by 14
When Behavior Analysis Meets Social Network Alignment
Zhang, Z., Ren, F., Zhang, J., Su, S., Yang, Y., Wei, Q., Sun, L., Zhu, G., & Guo, C. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3197985 · Cited by 14
Domain-Independent Dominance of Adaptive Methods
Savarese, P., McAllester, D., Babu, S., & Maire, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01602 · Cited by 18
Learning Architectures from an Extended Search Space for Language Modeling
Li, Y., Hu, C., Zhang, Y., Xu, N., Jiang, Y., Xiao, T., Zhu, J., Liu, T., & Li, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.592 · Cited by 14
Automated Speech Generation from UN General Assembly Statements: Mapping Risks in AI Generated TextsPreprint
Aylett-Bullock, J., & Luengo-Oroz, M. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01946 · Cited by 14
Extrapolation for Large-batch Training in Deep LearningPreprint
Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05720 · Cited by 14
Nondeterminism and Instability in Neural Network OptimizationPreprint
Summers, C., & Dinneen, M. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04514 · Cited by 21
AutoDropout: Learning Dropout Patterns to Regularize Deep Networks
Pham, H., & Le, Q. V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17127 · Cited by 15
Data Mining Based Marketing Decision Support System Using Hybrid Machine Learning Algorithm
Kumar, T. S. (2020) · Journal of Artificial Intelligence and Capsule Networks · doi:10.36548/jaicn.2020.3.006 · Cited by 19
Sarcasm Over Time and Across Platforms: Does the Way We Express Sarcasm Change?
Bouazizi, M., & Ohtsuki, T. O. (2022) · IEEE Access · doi:10.1109/access.2022.3174862 · Cited by 16
Text clustering based on pre-trained models and autoencoders
Xu, Q., Gu, H., & Ji, S. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2023.1334436 · Cited by 16
Restricted Recurrent Neural Networks
Diao, E., Ding, J., & Tarokh, V. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006257 · Cited by 14
Shahidi, H., Li, M., & Lin, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.355 · Cited by 13
Dissecting Contextual Word Embeddings: Architecture and RepresentationPreprint
Peters, M. E., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08949 · Cited by 90
Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI DatasetPreprint
Callender, L., Hawthorne, C., & Engel, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00188 · Cited by 13
A Hybridized Approach for Enhanced Fake Review Detection
Xu, S., Cuan, H., Yin, Z., & Yin, C. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3411635 · Cited by 15
Enhancing Claims Handling Processes with Insurance Based Language Models
Dimri, A., Yerramilli, S., Lee, P., Afra, S., & Jakubowski, A. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00284 · Cited by 12
Mu, B., Guo, P., Guo, D., Zhou, P., Chen, W., & Xie, L. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447563 · Cited by 12
How to represent a word and predict it, too: Improving tied architectures for language modelling
Gulordava, K., Aina, L., & Boleda, G. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1323 · Cited by 14
English Broadcast News Speech Recognition by Humans and Machines
Thomas, S., Suzuki, M., Huang, Y., Kurata, G., Tüske, Z., Saon, G., Kingsbury, B. E. D., Picheny, M., Dibert, T., Kaiser-Schatzlein, A., et al. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683211 · Cited by 12
Ragheb, W., Azé, J., Bringay, S., & Servajean, M. (2019) · HAL · Cited by 12
End-to-End Negation Resolution as Graph Parsing
Kurtz, R., Oepen, S., & Kuhlmann, M. (2020) · International Workshop/Conference on Parsing Technologies · doi:10.18653/v1/2020.iwpt-1.3 · Cited by 15
Transfer Learning Code Vectorizer based Machine Learning Models for Software Defect Prediction
Singh, R., Singh, J., Gill, M. S., Malhotra, R., & Garima, G. (2020) · International Conference on Computational Performance Evaluation (ComPE) · doi:10.1109/compe49325.2020.9200076 · Cited by 12
Non-Autoregressive vs Autoregressive Neural Networks for System Identification
Weber, D. O. M., & Gühmann, C. (2021) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2021.11.252 · Cited by 12
Metadata
- Authors
- Merity, S., Keskar, N. S., & Socher, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1708.02182
- OpenAlex
- W2743945814
- Cited by
- 697
- References
- 31
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 34 · 2023: 46 · 2022: 74 · 2021: 126 · 2020: 168 · 2019: 150 · 2018: 80 · 2017: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- LSTM language models, DropConnect, perplexity reduction, Penn Treebank, WikiText-2
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
