Regularizing and Optimizing LSTM Language Models
Open accessAbstract
Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The AWD-LSTM model fine-tuned on the full IMDB corpus had a usefulness score of 0.23.
“For a baseline comparison, the AWD-LSTM [48] model, fine-tuned on the entire IMDB corpus [47], gave us a score of only 0.23.”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,051 · see citing works
Self-citationMoses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · see citing works
(2013) · Cited by 3,740 · see citing works
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. M. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,233 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · see citing works
Learning invariance from transformation sequences
Földiák, P. (1991) · Neural Computation · doi:10.1162/neco.1991.3.2.194 · Cited by 661 · cited 1× · see citing works
“TAR falls under the broad category of slowness regularizers (Hinton, 1989; Földiák, 1991; Luciw & Schmidhuber, 2012; Jonschkowski & Brock, 2015) which penalize the model from producing large changes in the hidden state.”
BackgroundUsing the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 642 · see citing works
The Marginal Value of Adaptive Gradient Methods in Machine LearningPreprint
Wilson, A., Roelofs, R., Stern, M., Srebro, N., & Recht, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08292 · Cited by 600 · see citing works
Axiom: A Householder-Parameterized Pure Unitary RNN for Long-Range Sequence ModelingPreprint
Arjovsky, M., Shah, A., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06464 · Cited by 510 · see citing works
Quasi-Recurrent Neural NetworksPreprint
Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01576 · Cited by 481 · see citing works
Self-citationTrain faster, generalize better: Stability of stochastic gradient descentPreprint
Hardt, M., Recht, B., & Singer, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01240 · Cited by 358 · see citing works
Stochastic Gradient Descent as Approximate Bayesian Inference
Mandt, S., Hoffman, M. D., & Blei, D. M. (2017) · Journal of Machine Learning Research · doi:10.5555/3122009.3208015 · Cited by 337 · see citing works
Improving Neural Language Models with a Continuous CachePreprint
Grave, E., Joulin, A., & Usunier, N. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1612.04426 · Cited by 316 · see citing works
Tying Word Vectors and Word Classifiers: A Loss Framework for Language ModelingPreprint
Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 314 · see citing works
Self-citationRecurrent Highway NetworksPreprint
Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 243 · see citing works
Full-Capacity Unitary Recurrent Neural NetworksPreprint
Wisdom, S., Powers, T., Hershey, J. R., Roux, J. L., & Atlas, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00035 · Cited by 242 · see citing works
Zoneout: Regularizing RNNs by Randomly Preserving Hidden ActivationsPreprint
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A. C., & Pal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01305 · Cited by 214 · see citing works
Learning state representations with robotic priors
Jonschkowski, R., & Brock, O. (2015) · Autonomous Robots · doi:10.1007/s10514-015-9459-7 · Cited by 158 · see citing works
Stochastic Gradient Descent as Approximate Bayesian InferencePreprint
Mandt, S., Hoffman, M. D., & Blei, D. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04289 · Cited by 105 · see citing works
Query-Reduction Networks for Question AnsweringPreprint
Seo, M., Min, S., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04582 · Cited by 77 · see citing works
Unbiasing Truncated Backpropagation Through TimePreprint
Tallec, C., & Ollivier, Y. (2017) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1705.08209 · Cited by 47 · see citing works
Strongly-Typed Recurrent Neural NetworksPreprint
Balduzzi, D., & Ghifary, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02218 · Cited by 46 · see citing works
Revisiting Activation Regularization for Language RNNsPreprint
Merity, S., McCann, B., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01009 · Cited by 36 · see citing works
Self-citationLuciw, M., & Schmidhuber, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33266-1_35 · Cited by 21 · see citing works
Flagged first, then by citations
2 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 16
- Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In International Conference on Machine Learning , pp. 1120-1128, 2016.
- Ba, J., Kiros, J., and Hinton, G. E. Layer normalization. CoRR , abs/1607.06450, 2016. · arXiv
- Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. C. Recurrent batch normalization. CoRR , abs/1603.09025, 2016. · arXiv
- Journal of Machine Learning Research , 12(Jul):21212159, 2011.
- Gal, Y. and Ghahramani, Z. A theoretically grounded application of dropout in recurrent neural networks. In NIPS , 2016.
- Hinton, G. E. Connectionist learning procedures. Artificial intelligence , 40(1-3):185-234, 1989.
- Jing, L., Shen, Y., Dubˇ cek, T., Peurifoy, J., Skirlo, S., Tegmark, M., and Soljaˇ ci´ c, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN. arXiv preprint arXiv:1612.05231 , 2016. · arXiv
- Keskar, N. and Saon, G. A nonmonotone learning rate strategy for sgd training of deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on , pp. 4974-4978. IEEE, 2015.
- Kim, Y., Jernite, Y ., Sontag, D., and Rush, A. M. Characteraware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence , 2016.
- Melis, G., Dyer, C., and Blunsom, P. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint arXiv:1707.05589 , 2017. · arXiv
- Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. Recurrent neural network based language model. In INTERSPEECH , 2010.
- Panageas, I. and Piliouras, G. Gradient descent converges to minimizers: The case of non-isolated critical points. CoRR, abs/1605.00405 , 2016. · arXiv
- Polyak, B. and Juditsky, A. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838-855, 1992.
- Semeniuta, S., Severyn, A., and Barth, E. Recurrent dropout without memory loss. In COLING , 2016.
- Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26-31, 2012.
- Wan, L., Zeiler, M., Zhang, S., LeCun, Y, and Fergus, R. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML-13) , pp. 1058-1066, 2013.
Cited by
Sorted by citations
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · see citing works
Efficient Neural Architecture Search via Parameter SharingPreprint
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 1,082 · see citing works
Fastai: A Layered API for Deep Learning
Howard, J., & Gugger, S. (2020) · Information · doi:10.3390/info11020108 · Cited by 992 · see citing works
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362 · see citing works
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
Self-citationSearching for a Robust Neural Architecture in Four GPU Hours
Dong, X., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00186 · Cited by 658 · see citing works
Born Again Neural NetworksPreprint
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04770 · Cited by 798 · see citing works
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network
Mehta, S., Rastegari, M., Shapiro, L., & Hajishirzi, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00941 · Cited by 551 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
Parallel spatio-temporal attention-based TCN for multivariate time series prediction
Fan, J., Zhang, K., Huang, Y.-P., Zhu, Y., & Chen, B.-P. (2021) · Neural computing & applications (Print) · doi:10.1007/s00521-021-05958-z · Cited by 397 · see citing works
Dissecting Contextual Word Embeddings: Architecture and Representation
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1179 · Cited by 434 · see citing works
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.217 · Cited by 401 · see citing works
Lookahead Optimizer: k steps forward, 1 step backPreprint
Zhang, M. R., Lucas, J., Hinton, G., & Ba, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08610 · Cited by 470 · see citing works
Fine-tuned Language Models for Text Classification.Preprint
Howard, J., & Ruder, S. (2018) · arXiv (Cornell University) · Cited by 357 · see citing works
Plan-and-Write: Towards Better Automatic Storytelling
Yao, L., Peng, N., Weischedel, R., Knight, K. K., Zhao, D., & Yan, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017378 · Cited by 328 · see citing works
Speech Emotion Classification Using Attention-Based LSTM
Xie, Y., Liang, R., Liang, Z., Huang, C.-W., Zou, C., & Schuller, B. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2925934 · Cited by 268 · see citing works
Sharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · see citing works
Breaking the Softmax Bottleneck: A High-Rank RNN Language ModelPreprint
Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03953 · Cited by 291 · see citing works
Domain Adaptation: Challenges, Methods, Datasets, and Applications
Singhal, P., Walambe, R. A., Ramanna, S., & Kotecha, K. V. (2023) · IEEE Access · doi:10.1109/access.2023.3237025 · Cited by 253 · see citing works
A Gated Recurrent Unit Approach to Bitcoin Price Prediction
Dutta, A., Kumar, S., & Basu, M. (2020) · Journal of risk and financial management · doi:10.3390/jrfm13020023 · Cited by 230 · see citing works
Neural Voice Cloning with a Few SamplesPreprint
Arık, S. Ö., Chen, J., Peng, K., Ping, W., & Zhou, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06006 · Cited by 229 · see citing works
Ordered Neurons: Integrating Tree Structures into Recurrent Neural NetworksPreprint
Shen, Y., Tan, S., Sordoni, A., & Courville, A. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09536 · Cited by 222 · see citing works
When deep learning met code search
Cambronero, J., Li, H., Kim, S., Sen, K., & Chandra, S. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3340458 · Cited by 215 · see citing works
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-BatchesPreprint
Wen, Y., Vicol, P. A., Ba, J. L., Tran, D., & Grosse, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04386 · Cited by 208 · see citing works
Classification of Hand Movements From EEG Using a Deep Attention-Based LSTM Network
Zhang, G., Davoodnia, V., Sepas‐Moghaddam, A., Zhang, Y., & Etemad, A. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2956998 · Cited by 201 · see citing works
Evaluating the Search Phase of Neural Architecture SearchPreprint
Yu, K., Sciuto, C., Jaggi, M., Musat, C., & Salzmann, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08142 · Cited by 219 · see citing works
Fantastic Generalization Measures and Where to Find ThemPreprint
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02178 · Cited by 194 · see citing works
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1598 · Cited by 168 · see citing works
Sun, Z., Di, L., & Fang, H. (2018) · International Journal of Remote Sensing · doi:10.1080/01431161.2018.1516313 · Cited by 180 · see citing works
A Novel Deep Learning Model for the Detection and Identification of Rolling Element-Bearing Faults
Shenfield, A., & Howarth, M. (2020) · Italian National Conference on Sensors · doi:10.3390/s20185112 · Cited by 173 · see citing works
Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation
Liu, Y., & Wang, D. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2941148 · Cited by 172 · see citing works
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196 · see citing works
Data Evaluation and Enhancement for Quality Improvement of Machine Learning
Chen, H., Chen, J., & Ding, J. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3070863 · Cited by 166 · see citing works
Improving Abstraction in Text Summarization
Kryściński, W., Paulus, R., Xiong, C., & Socher, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1207 · Cited by 163 · see citing works
Self-citationFinBERT: Financial Sentiment Analysis with Pre-trained Language Models
Araci, D. (2019) · arXiv · Cited by 602 · see citing works
Bayer, M., Kaufhold, M., Buchhold, B., Keller, M., Dallmeyer, J., & Reuter, C. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01553-3 · Cited by 150 · see citing works
Shahriar, S. (2022) · Displays · doi:10.1016/j.displa.2022.102237 · Cited by 147 · see citing works
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep LearningPreprint
Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S. C. H., & Weinan, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05627 · Cited by 146 · see citing works
Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT
Li, X., & Fourches, D. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-00430-x · Cited by 145 · see citing works
Politeness Transfer: A Tag and Generate Approach
Madaan, A., Setlur, A., Parekh, T., Póczos, B., Neubig, G., Yang, Y., Salakhutdinov, R., Black, A. W., & Prabhumoye, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.169 · Cited by 143 · see citing works
Multi-zone indoor temperature prediction with LSTM-based sequence to sequence model
Fang, Z., Crimier, N., Scanu, L., Midelet, A., Alyafi, A. A., & DELINCHANT, B. (2021) · Energy and Buildings · doi:10.1016/j.enbuild.2021.111053 · Cited by 141 · see citing works
Survey of Dropout Methods for Deep Neural NetworksPreprint
Labach, A., Salehinejad, H., & Valaee, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.13310 · Cited by 137 · see citing works
Augment your batch: Improving generalization through instance repetition
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., & Soudry, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00815 · Cited by 166 · see citing works
SMILES Pair Encoding: A Data-Driven Substructure Tokenization Algorithm for Deep Learning
Li, X., & Fourches, D. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01127 · Cited by 133 · see citing works
A Survey of Deep Learning for Scientific DiscoveryPreprint
Raghu, M., & Schmidt, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11755 · Cited by 174 · see citing works
A Review of Text Corpus-Based Tourism Big Data Mining
Li, Q., Li, S., Zhang, S., Hu, J., & Hu, J. (2019) · Applied Sciences · doi:10.3390/app9163300 · Cited by 135 · see citing works
A Multi-Patient Data-Driven Approach to Blood Glucose Prediction
Aliberti, A., Pupillo, I., Terna, S., Macii, E., Cataldo, S. D., Patti, E., & Acquaviva, A. (2019) · IEEE Access · doi:10.1109/access.2019.2919184 · Cited by 130 · see citing works
On Network Design Spaces for Visual Recognition
Radosavovic, I., Johnson, J., Xie, S., Lo, W.-Y., & Dollar, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00197 · Cited by 127 · see citing works
An Analysis of Neural Language Modeling at Multiple ScalesPreprint
Merity, S., Keskar, N. S., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08240 · Cited by 125 · see citing works
Self-citationWeighted Transformer Network for Machine TranslationPreprint
Ahmed, K., Keskar, N., & Socher, R. (2017) · arXiv.org · doi:10.48550/arxiv.1711.02132 · Cited by 143 · see citing works
Self-citationDecentralized Deep Learning with Arbitrary Communication CompressionPreprint
Koloskova, A., Lin, T., Stich, S. U., & Jaggi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09356 · Cited by 118 · see citing works
FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural NetworkPreprint
Kusupati, A., Singh, M., Bhatia, K., Kumar, A., Jain, P., & Varma, M. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1901.02358 · Cited by 208 · see citing works
DeepLMS: a deep learning predictive model for supporting online learning in the Covid-19 era
Dias, S. B., Hadjileontiadou, S. J., Diniz, J., & Hadjileontiadis, L. J. (2020) · Scientific Reports · doi:10.1038/s41598-020-76740-9 · Cited by 115 · see citing works
Evaluating The Search Phase of Neural Architecture Search
Yu, K., Sciuto, C., Jaggi, M., Musat, C., & Salzmann, M. (2020) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 136 · see citing works
A multihead LSTM technique for prognostic prediction of soil moisture
Datta, P., & Faroughi, S. A. (2023) · Geoderma · doi:10.1016/j.geoderma.2023.116452 · Cited by 112 · see citing works
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · see citing works
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing
Klyuchnikov, N., Trofimov, I., Artemova, E., Salnikov, M., Fedorov, M. V., Filippov, A., & Burnaev, E. V. (2022) · IEEE Access · doi:10.1109/access.2022.3169897 · Cited by 117 · see citing works
Deep Feature-Based Text Clustering and its Explanation
Guan, R., Zhang, H., Liang, Y., Giunchiglia, F., Huang, L., & Feng, X. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3028943 · Cited by 107 · see citing works
Optimizing Millions of Hyperparameters by Implicit DifferentiationPreprint
Lorraine, J., Vicol, P. A., & Duvenaud, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02590 · Cited by 107 · see citing works
Unsupervised Recurrent Neural Network Grammars
Kim, Y., Rush, A. M., Yu, L., Kuncoro, A., Dyer, C., & Melis, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1114 · Cited by 102 · see citing works
A comparative study of effective approaches for Arabic sentiment analysis
Farha, I. A., & Magdy, W. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102438 · Cited by 106 · see citing works
Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding
Abdelnabi, S., & Fritz, M. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00083 · Cited by 105 · see citing works
Language Models with TransformersPreprint
Wang, C., Li, M., & Smola, A. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09408 · Cited by 130 · see citing works
Massively Parallel Hyperparameter Tuning
Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., & Talwalkar, A. (2018) · Cited by 99 · see citing works
Kochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00267-x · Cited by 112 · see citing works
Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition
Jiang, Y., Hu, C., Xiao, T., Zhang, C., & Zhu, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1367 · Cited by 99 · see citing works
PrivFT: Private and Fast Text Classification With Homomorphic Encryption
Badawi, A. A., Hoang, L., Mun, C. F., Laine, K., & Aung, K. M. M. (2020) · IEEE Access · doi:10.1109/access.2020.3045465 · Cited by 96 · see citing works
Research Progress of RNN Language Model
Xiao, J., & Zhou, Z. (2020) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica50127.2020.9182390 · Cited by 96 · see citing works
Smarter Contracts: Detecting Vulnerabilities in Smart Contracts with Deep Transfer Learning
Sendner, C., Chen, H., Fereidooni, H., Petzi, L., König, J., Stang, J., Dmitrienko, A., Sadeghi, A., & Koushanfar, F. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.23263 · Cited by 95 · see citing works
Deep learning for the PSIPRED Protein Analysis Workbench
Buchan, D. W. A., Moffat, L., Lau, A. M. C., Kandathil, S. M., & Jones, D. T. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae328 · Cited by 94 · see citing works
Monaural Speech Dereverberation Using Temporal Convolutional Networks With Self Attention
Zhao, Y., Wang, D., Xu, B., & Zhang, T. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2995273 · Cited by 92 · see citing works
Understanding and Improving Knowledge DistillationPreprint
Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Ed, H., & Jain, S. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03532 · Cited by 90 · see citing works
Straight to the Tree: Constituency Parsing with Neural Syntactic Distance
Shen, Y., Lin, Z., Jacob, A. P., Sordoni, A., Courville, A. C., & Bengio, Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1108 · Cited by 85 · see citing works
Tang, Q., Nie, F., Zhao, Q., & Chen, W. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac357 · Cited by 85 · see citing works
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 83 · see citing works
Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition
Zheng, X., Zhang, C., & Woodland, P. C. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688232 · Cited by 79 · see citing works
Hodorog, A., Petri, I., & Rezgui, Y. (2022) · Sustainable Cities and Society · doi:10.1016/j.scs.2022.104026 · Cited by 78 · see citing works
Fantastic Generalization Measures and Where to Find Them
Jiang, Y., Neyshabur, B., Krishnan, D., Mobahi, H., & Samy Bengio, A. (2020) · International Conference on Learning Representations · Cited by 75 · see citing works
A Dataset for Telling the Stories of Social Media Videos
Gella, S., Lewis, M., & Rohrbach, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1117 · Cited by 69 · see citing works
Named Entity Recognition With Parallel Recurrent Neural Networks
Žukov-Gregorič, A., Bachrach, Y., & Coope, S. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2012 · Cited by 69 · see citing works
Amer, F., Jung, Y., & Golparvar-Fard, M. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103929 · Cited by 68 · see citing works
hULMonA: The Universal Language Model in Arabic
Jundi, O. E., Antoun, W., Droubi, N. E., Hajj, H. M., El-Hajj, W., & Shaban, K. B. (2019) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/w19-4608 · Cited by 71 · see citing works
Recurrent neural network from adder’s perspective: Carry-lookahead RNN
Jiang, H., Qin, F., Cao, J., Peng, Y. K., & Shao, Y. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.08.032 · Cited by 67 · see citing works
Fix your classifier: the marginal value of training the last weight layerPreprint
Hoffer, E., Hubara, I., & Soudry, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04540 · Cited by 66 · see citing works
Training DNNs with Hybrid Block Floating PointPreprint
Oliveira, D. L. D., Lin, T., Jaggi, M., & Falsafi, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01526 · Cited by 68 · see citing works
The Implicit and Explicit Regularization Effects of DropoutPreprint
Wei, C., Kakade, S. M., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12915 · Cited by 63 · see citing works
Story Realization: Expanding Plot Events into Sentences
Ammanabrolu, P., Tien, E., Cheung, W., Luo, Z., Ma, W., Martin, L. J., & Riedl, M. O. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6232 · Cited by 16 · see citing works
Universal Language Model Fine-Tuning for Text Classification
Reddy, R., V (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.53887 · Cited by 60 · see citing works
Shah, Z., Surian, D., Dyda, A., Coiera, E. W., Mandl, K. D., & Dunn, A. G. (2019) · Journal of Medical Internet Research · doi:10.2196/14007 · Cited by 60 · see citing works
A Survey of Active Learning for Text Classification using Deep Neural NetworksPreprint
Schröder, C., & Niekler, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07267 · Cited by 97 · see citing works
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation
Bień, M., Gilski, M., Maciejewska, M., Taisner, W., Wisniewski, D., & Ławrynowicz, A. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.4 · Cited by 58 · see citing works
GamePad: A Learning Environment for Theorem ProvingPreprint
Huang, D., Dhariwal, P., Song, D., & Sutskever, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00608 · Cited by 58 · see citing works
Character-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works
Large Scale Legal Text Classification Using Transformer ModelsPreprint
Shaheen, Z., Wohlgenannt, G., & Filtz, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12871 · Cited by 99 · see citing works
Lessons on Parameter Sharing across Layers in Transformers
Takase, S., & Kiyono, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.5 · Cited by 68 · see citing works
Generating Empathetic Responses by Looking Ahead the User’s Sentiment
Shin, J., Xu, P., Madotto, A., & Fung, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054379 · Cited by 68 · see citing works
Learning Long-Term Temporal Features With Deep Neural Networks for Human Action Recognition
Yu, S., Xie, L., Liu, L., & Xia, D. (2019) · IEEE Access · doi:10.1109/access.2019.2962284 · Cited by 53 · see citing works
- Authors
- Merity, S., Keskar, N. S., & Socher, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1708.02182
- OpenAlex
- W2743945814
- Cited by
- 697
- References
- 31
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 34 · 2023: 46 · 2022: 74 · 2021: 126 · 2020: 168 · 2019: 150 · 2018: 80 · 2017: 3 · 2016: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- LSTM language models, DropConnect, perplexity reduction, Penn Treebank, WikiText-2
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
