The DOI matches its DataCite record; 28 of the 31 references it cites were found in a registry or an index.
Regularizing and Optimizing LSTM Language Models
Open accessAbstract
Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For a baseline comparison, the AWD-LSTM [48] model, fine-tuned on the entire IMDB corpus [47], gave us a score of only 0.23.”
From Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing (2023) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,705
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,058
Self-citationMoses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735
(2013) · Cited by 3,740
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. M. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,573
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 3,174
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Flagged first, then by citations
3 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 16
- Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In International Conference on Machine Learning , pp. 1120-1128, 2016.
- Ba, J., Kiros, J., and Hinton, G. E. Layer normalization. CoRR , abs/1607.06450, 2016. · arXiv
- Cooijmans, T., Ballas, N., Laurent, C., and Courville, A. C. Recurrent batch normalization. CoRR , abs/1603.09025, 2016. · arXiv
- Journal of Machine Learning Research , 12(Jul):21212159, 2011.
- Gal, Y. and Ghahramani, Z. A theoretically grounded application of dropout in recurrent neural networks. In NIPS , 2016.
- Hinton, G. E. Connectionist learning procedures. Artificial intelligence , 40(1-3):185-234, 1989.
- Jing, L., Shen, Y., Dubˇ cek, T., Peurifoy, J., Skirlo, S., Tegmark, M., and Soljaˇ ci´ c, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN. arXiv preprint arXiv:1612.05231 , 2016. · arXiv
- Keskar, N. and Saon, G. A nonmonotone learning rate strategy for sgd training of deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on , pp. 4974-4978. IEEE, 2015.
- Kim, Y., Jernite, Y ., Sontag, D., and Rush, A. M. Characteraware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence , 2016.
- Melis, G., Dyer, C., and Blunsom, P. On the State of the Art of Evaluation in Neural Language Models. arXiv preprint arXiv:1707.05589 , 2017. · arXiv
- Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. Recurrent neural network based language model. In INTERSPEECH , 2010.
- Panageas, I. and Piliouras, G. Gradient descent converges to minimizers: The case of non-isolated critical points. CoRR, abs/1605.00405 , 2016. · arXiv
- Polyak, B. and Juditsky, A. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838-855, 1992.
- Semeniuta, S., Severyn, A., and Barth, E. Recurrent dropout without memory loss. In COLING , 2016.
- Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26-31, 2012.
- Wan, L., Zeiler, M., Zhang, S., LeCun, Y, and Fergus, R. Regularization of neural networks using dropconnect. In Proceedings of the 30th international conference on machine learning (ICML-13) , pp. 1058-1066, 2013.
Cited by
Sorted by citations
Items 101–200 of 695
Mol2Context-vec: learning molecular representation from context awareness for drug discovery
Lv, Q., Chen, G., Zhao, L., Zhong, W., & Chen, C. Y. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab317 · Cited by 54
#YouToo? Detection of Personal Recollections of Sexual Harassment on Social Media
Chowdhury, A. G., Sawhney, R., Shah, R. R., & Mahata, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1241 · Cited by 52
Hunting for Insider Threats Using LSTM-Based Anomaly Detection
Villarreal-Vasquez, M., Modelo-Howard, G., Dube, S., & Bhargava, B. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3135639 · Cited by 52
A Survey on Transfer Learning in Natural Language ProcessingPreprint
Alyafeai, Z., Al-shaibani, M. S., & Ahmad, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04239 · Cited by 120
Gogineni, A. K., Swayamjyoti, S., Sahoo, D., Sahu, K. K., & Kishore, R. (2020) · IOP SciNotes · doi:10.1088/2633-1357/abcd29 · Cited by 54
Luo, Z., Wang, R., Sun, Y., Liu, J., Chen, Z., & Zhang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad534 · Cited by 51
Tshakwanda, P. M., Arzo, S. T., & Devetsikiotis, M. (2024) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2024.3398540 · Cited by 60
Sentiment Analysis on Mobile Phone Reviews Using Supervised Learning Techniques
Shaheen, M. (2019) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2019.07.04 · Cited by 50
Named Entity Recognition, Linking and Generation for Greek Legislation
Angelidis, I., Chalkidis, I., & Koubarakis, M. (2018) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-61499-935-5-1 · Cited by 52
Multimodal Document Image Classification
Jain, R., & Wigington, C. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00021 · Cited by 50
Improving Speech Recognition Using Consistent Predictions on Synthesized Speech
Wang, G., Rosenberg, A., Chen, Z., Zhang, Y., Ramabhadran, B., Wu, Y., & Moreno, P. J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053831 · Cited by 49
Dong, T., Yang, Q., Ebadi, N., Luo, X. R., & Rad, P. (2021) · Journal of Advanced Transportation · doi:10.1155/2021/5540046 · Cited by 58
A Double-Deep Spatio-Angular Learning Framework for Light Field-Based Face Recognition
Sepas‐Moghaddam, A., Haque, M. A., Correia, P. L., Nasrollahi, K., Moeslund, T. B., & Pereira, F. M. B. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2916669 · Cited by 48
Towards Non-Saturating Recurrent Units for Modelling Long-Term Dependencies
Chandar, S., Sankar, C., Vorontsov, E., Kahou, S. E., & Bengio, Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013280 · Cited by 57
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
Xu, Y., Liu, X., Shen, Y., Liu, J., & Gao, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1271 · Cited by 60
Shen, T., Quach, V., Barzilay, R., & Jaakkola, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.420 · Cited by 58
Evolution of transfer learning in natural language processingPreprint
Malte, A., & Ratadiya, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07370 · Cited by 81
Zhao, S., Li, F., Chen, X., Guan, X., Jiang, J., Huang, D., Qing, Y., Wang, S., Wang, P., Zhang, G., et al. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3094364 · Cited by 49
Efficient Sequence Learning with Group Recurrent Networks
Gao, F., Wu, L., Zhao, L., Qin, T., Cheng, X.-Q., & Liu, T.-Y. (2018) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n18-1073 · Cited by 47
Structured pruning of recurrent neural networks through neuron selection
Wen, L., Zhang, X., Bai, H., & Xu, Z. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.11.018 · Cited by 49
“BNN - BN = ?”: Training Binary Neural Networks without Batch Normalization
Chen, T., Zhang, Z., Ouyang, X., Liu, Z., Shen, Z., & Wang, Z. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00520 · Cited by 45
Augment your batch: better training with larger batchesPreprint
Hoffer, E., Ben‐Nun, T., Hubara, I., Giladi, N., Hoefler, T., & Soudry, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09335 · Cited by 45
Bhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584060 · Cited by 47
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
Welleck, S., Kulikov, I., Kim, J., Pang, R. Y., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.448 · Cited by 60
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
Papadimitriou, I., & Jurafsky, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.554 · Cited by 58
Cruz-Benito, J., Vishwakarma, S., Martín-Fernández, F., & Faro, I. (2021) · AI · doi:10.3390/ai2010001 · Cited by 50
Latent Relation Language Models
Hayashi, H., Hu, Z., Xiong, C., & Neubig, G. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6298 · Cited by 44
Compression of recurrent neural networks for efficient language modeling
Grachev, A. M., Ignatov, D. I., & Savchenko, A. V. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.03.057 · Cited by 43
Hybrid Deep Learning Models for Thai Sentiment Analysis
Pasupa, K., & Ayutthaya, T. S. N. (2021) · Cognitive Computation · doi:10.1007/s12559-020-09770-0 · Cited by 46
Low Resource Text Classification with ULMFit and BacktranslationPreprint
Shleifer, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09244 · Cited by 75
Enabling Cost-Effective, SLO-Aware Machine Learning Inference Serving on Public Cloud
Zhang, C., Yu, M., Wang, W., & Yan, F. (2020) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2020.3006751 · Cited by 41
Towards Binary-Valued Gates for Robust LSTM TrainingPreprint
Li, Z., He, D., Tian, F., Chen, W., Qin, T., Wang, L.-W., & Liu, T.-Y. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1806.02988 · Cited by 52
Beyond class-level privacy leakage: breaking record-level privacy in federated learning
Yuan, X., Ma, X., Zhang, L., Fang, Y., & Wu, D. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3089713 · Cited by 45
Late Reverberation Suppression Using Recurrent Neural Networks with Long Short-Term Memory
Zhao, Y., Wang, D., Xu, B., & Zhang, T. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462275 · Cited by 38
Neural Networks Versus Conventional Filters for Inertial-Sensor-based Attitude Estimation
Weber, D. O. M., Gühmann, C., & Seel, T. (2020) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion45008.2020.9190634 · Cited by 41
Aithal, S. G., Rao, A. B., & Singh, S. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02348-9 · Cited by 40
PepCVAE: Semi-Supervised Targeted Design of Antimicrobial Peptide SequencesPreprint
Das, P., Wadhawan, K., Chang, O., Sercu, T., dos Santos, C. N., Riemer, M., Chenthamarakshan, V., Padhi, I., & Mojsilović, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.07743 · Cited by 68
Natural Language Processing Methods for Language Modeling
Nemeskey, D. M. (2021) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2020.066 · Cited by 41
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment
Chowdhury, A. G., Sawhney, R., Mathur, P., Mahata, D., & Shah, R. R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/n19-3018 · Cited by 39
Language Modeling with Sparse Product of Sememe Experts
Gu, Y., Yan, J., Zhu, H., Liu, Z., Xie, R., Sun, M., Lin, F., & Lin, L. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1493 · Cited by 37
Medhat, N., Moussa, S. M., Badr, N. L., & Tolba, M. F. (2020) · IEEE Access · doi:10.1109/access.2020.3039931 · Cited by 37
Evaluating Saliency Methods for Neural Language Models
Ding, S., & Koehn, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.399 · Cited by 51
Lutz, O., Chen, H., Fereidooni, H., Sendner, C., Dmitrienko, A., Sadeghi, A., & Koushanfar, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12607 · Cited by 76
Proceedings of the Fourth Arabic Natural Language Processing Workshop
El-Hajj, W., Belguith, L. H., Bougares, F., Magdy, W., Zitouni, I., Tomeh, N., & El Hajj, M. (2019) · doi:10.18653/v1/w19-46 · Cited by 37
Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition
Liu, A., Du, J., & Stoyanov, V. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1117 · Cited by 44
Building Language Models for Text with Named Entities
Parvez, M. R., Chakraborty, S., Ray, B., & Chang, K.-W. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1221 · Cited by 38
Ge, F., Zhang, Y., Xu, J., Arif, M., Song, J., & Yu, D. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab530 · Cited by 37
An Adaptive and Momental Bound Method for Stochastic LearningPreprint
Ding, J., Ren, X., Luo, R., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12249 · Cited by 36
INTRUSION DETECTION IN COMPUTER NETWORKS USING LATENT SPACE REPRESENTATION AND MACHINE LEARNING
Hamolia, V., Melnyk, V. A., Zhezhnych, P., & Shilinh, A. (2020) · International Journal of Computing · doi:10.47839/ijc.19.3.1893 · Cited by 37
Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings
Kumar, V., Bhotia, T. S., Kumar, V., & Chakraborty, T. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00327 · Cited by 39
Pretraining model for biological sequence data
Song, B., Li, Z., Lin, X., Wang, J., Wang, T., & Fu, X. (2021) · Briefings in Functional Genomics · doi:10.1093/bfgp/elab025 · Cited by 37
Rethinking Perturbations in Encoder-Decoders for Fast Training
Takase, S., & Kiyono, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.460 · Cited by 39
From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text
Tarunesh, I., Kumar, S., & Jyothi, P. (2021) · doi:10.18653/v1/2021.acl-long.245 · Cited by 36
The COVID-19 fake news detection in Thai social texts
Mookdarsanit, P., & Mookdarsanit, L. (2021) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v10i2.2745 · Cited by 38
Identifying wildlife observations on twitter
Edwards, T., Jones, C. B., & Corcoran, P. (2021) · Ecological Informatics · doi:10.1016/j.ecoinf.2021.101500 · Cited by 34
RAPPPID: towards generalizable protein interaction prediction with AWD-LSTM twin networks
Szymborski, J., & Emad, A. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac429 · Cited by 34
IIITT@LT-EDI-EACL2021-Hope Speech Detection: There is always Hope in TransformersPreprint
Puranik, K., Hande, A., Priyadharshini, R., Thavareesan, S., & Chakravarthi, B. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09066 · Cited by 40
Lin, K. Y., Quan, X., Jin, C., Shi, Z., & Yang, J. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.885627 · Cited by 34
Direct Output Connection for a High-Rank Language Model
Takase, S., Suzuki, J., & Nagata, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1489 · Cited by 41
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
Wang, X. E., Wu, J., Zhang, D., Su, Y., & Wang, W. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018965 · Cited by 35
Improving Low-Resource Named Entity Recognition using Joint Sentence and Token Labeling
Kruengkrai, C., Nguyen, T. H., Aljunied, S. M., & Bing, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.523 · Cited by 36
ADH-PPI: An attention-based deep hybrid model for protein-protein interaction prediction
Asim, M. N., Ibrahim, M. A., Malik, M. I., Dengel, A. R., & Ahmed, S. (2022) · iScience · doi:10.1016/j.isci.2022.105169 · Cited by 32
The STC System for the CHiME 2018 Challenge
Medennikov, I., Sorokin, I., Romanenko, A., Popov, D., Khokhlov, Y., Prisyach, T., Malkovskiy, N., Bataev, V., Astapov, S., Korenevsky, M. L., et al. (2018) · International Workshop on Speech Processing in Everyday Environments (CHiME) · doi:10.21437/chime.2018-1 · Cited by 32
Word Senses as Clusters of Meaning Modulations: A Computational Model of Polysemy
Li, J., & Joanisse, M. F. (2021) · Cognitive Science · doi:10.1111/cogs.12955 · Cited by 37
Efficient Algorithms for Device Placement of DNN Graph OperatorsPreprint
Tarnawski, J., Phanishayee, A., Devanur, N. R., Mahajan, D., & Paravecino, F. N. (2020) · Neural Information Processing Systems · doi:10.48550/arxiv.2006.16423 · Cited by 87
Data augmentation in a hybrid approach for aspect-based sentiment analysis
Liesting, T., Frăsincar, F., & Truşcǎ, M. M. (2021) · ACM Symposium on Applied Computing · doi:10.1145/3412841.3441958 · Cited by 31
Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals
Sirpal, P., Damseh, R., Peng, K., Nguyen, D. K., & Lesage, F. (2021) · Neuroinformatics · doi:10.1007/s12021-021-09538-3 · Cited by 31
Manas: multi-agent neural architecture search
Lopes, V., Carlucci, F. M., Esperança, P. M., Singh, M., Yang, A., Gabillon, V., Xu, H., Chen, Z., & Wang, J. (2023) · Machine Learning · doi:10.1007/s10994-023-06379-w · Cited by 38
Transfer Learning for British Sign Language ModellingPreprint
Mocialov, B., Turner, G. H., & Hastie, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02144 · Cited by 31
Randomness In Neural Network Training: Characterizing The Impact of ToolingPreprint
Zhuang, D., Zhang, X., Song, S. L., & Hooker, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11872 · Cited by 53
BEND: Benchmarking DNA Language Models on biologically meaningful tasksPreprint
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.12570 · Cited by 54
Probing for Referential Information in Language Models
Sorodoc, I.-T., Gulordava, K., & Boleda, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.384 · Cited by 37
Weight Dropout for Preventing Neural Networks from Overfitting
Sanjar, K., Rehman, A., Paul, A., & Kim, J. (2020) · International Conference on Orange Technology (ICOT) · doi:10.1109/icot51877.2020.9468799 · Cited by 30
Neto, E. J. D. M., & Clementino, M. B. M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272287 · Cited by 38
Deep Reinforcement Learning For Modeling Chit-Chat Dialog With Discrete Attributes
Sankar, C., & Ravi, S. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5901 · Cited by 29
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model
Mielke, S. J., & Eisner, J. M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016843 · Cited by 5
User Identity Linkage via Co-Attentive Neural Network From Heterogeneous Mobility Data
Feng, J., Li, Y., Yang, Z., Zhang, M., Wang, H., Cao, H., & Jin, D. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2989732 · Cited by 29
Evaluating Language Model Finetuning Techniques for Low-resource LanguagesPreprint
Cruz, J. C. B., & Cheng, C. K. (2019) · arXiv (Cornell University) · doi:10.13140/rg.2.2.23028.40322 · Cited by 50
Addressing Some Limitations of Transformers with Feedback MemoryPreprint
Fan, A., Lavril, T., Grave, É., Joulin, A., & Sukhbaatar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09402 · Cited by 90
Complaint Identification in Social Media with Transformer Networks
Jin, M., & Αλέτρας, Ν. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.157 · Cited by 32
A Tutorial on Deep Latent Variable Models of Natural LanguagePreprint
Kim, Y., Wiseman, S., & Rush, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06834 · Cited by 41
Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient ClippingPreprint
Gorbunov, E., Danilova, M., & Gasnikov, A. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.10785 · Cited by 41
Leveraging Small Software Engineering Data Sets with Pre-Trained Neural Networks
Robbes, R., & Janes, A. (2019) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2019.00016 · Cited by 27
Guo, X., Zhou, B., Pigg, D., Spottiswoode, B. S., Casey, M., Liu, C., & Dvornek, N. C. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102524 · Cited by 28
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTKPreprint
Li, Y., Ma, T., & Zhang, H. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04596 · Cited by 27
Interpretable multi-timescale models for predicting fMRI responses to continuous natural speechPreprint
Jain, S., Mahto, S., Turek, J. S., Vo, V. A., LeBel, A., & Huth, A. G. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.10.02.324392 · Cited by 30
Gradients are Not All You NeedPreprint
Metz, L., Freeman, C. D., Schoenholz, S. S., & Kachman, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05803 · Cited by 122
Exploring Self-Identified Counseling Expertise in Online Support Forums
Lahnala, A., Zhao, Y., WELCH, C. A., Kummerfeld, J. K., An, L. C., Resnicow, K., Mihalcea, R. F., & Pérez‐Rosas, V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.392 · Cited by 30
Language Informed Modeling of Code-Switched Text
Chandu, K. R., Manzini, T., Singh, S., & Black, A. W. (2018) · Workshop on Computational Approaches to Code Switching · doi:10.18653/v1/w18-3211 · Cited by 26
Xue, Y., Su, W., Wang, H., Yang, D., & Jiang, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2937975 · Cited by 26
Gonen, H., & Goldberg, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1427 · Cited by 26
Character n-Gram Embeddings to Improve RNN Language Models
Takase, S., Suzuki, J., & Nagata, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015074 · Cited by 25
Al-Shwaheen, T. I., Hau, Y. W., Ass'Ad, N., & AbuAlSamen, M. M. (2020) · IEEE Access · doi:10.1109/access.2020.3047186 · Cited by 28
An Evaluation on Information Composition in Dementia Detection Based on Speech
Zheng, C., Bouazizi, M., & Ohtsuki, T. O. (2022) · IEEE Access · doi:10.1109/access.2022.3203068 · Cited by 26
Tuarob, S., Satravisut, M., Sangtunchai, P., Nunthavanich, S., & Noraset, T. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103381 · Cited by 29
Xiao, J. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2036 · Cited by 26
CPT: Efficient Deep Neural Network Training via Cyclic PrecisionPreprint
Fu, Y., Guo, H., Li, M., Yang, X., Ding, Y., Chandra, V., & Lin, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09868 · Cited by 37
Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization
Shi, Y., Hwang, M.-Y., Lei, X., & Sheng, H. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683533 · Cited by 24
Metadata
- Authors
- Merity, S., Keskar, N. S., & Socher, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1708.02182
- OpenAlex
- W2743945814
- Cited by
- 697
- References
- 31
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 34 · 2023: 46 · 2022: 74 · 2021: 126 · 2020: 168 · 2019: 150 · 2018: 80 · 2017: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- LSTM language models, DropConnect, perplexity reduction, Penn Treebank, WikiText-2
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
