CoChat
Sign Up
PreprintCited by 41Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

The DOI matches its DataCite record.

Direct Output Connection for a High-Rank Language Model

Open access
Takase, Sho, Suzuki, Jun, Nagata, Masaaki·2018·arXiv
DOI 10.48550/arxiv.1808.10143arXiv 1808.10143
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper proposes a state-of-the-art recurrent neural network (RNN) language model that combines probability distributions computed not only from a final RNN layer but also from middle layers. Our proposed method raises the expressive power of a language model based on the matrix factorization interpretation of language modeling introduced by Yang et al. (2018). The proposed method improves the current state-of-the-art language model and achieves the best score on the Penn Treebank and WikiText-2, which are the standard benchmark datasets. Moreover, we indicate our proposed method contributes to two application tasks: machine translation and headline generation. Our code is publicly available at: https://github.com/nttcslab-nlp/doc_lm.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1808.10143

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 44

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic lan- guage model. Journal of Machine Learning Re- search, 3:1137-1155.
  2. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In 1st Meeting of the North American Chap- ter of the Association for Computational Linguistics (NAACL 2000), pages 132-139.
  3. Stanley F. Chen and Joshua Goodman. 1996. An em- pirical study of smoothing techniques for language modeling. In Proceedings of the 34th Annual Meet- ing on Association for Computational Linguistics (ACL 1996), pages 310-318.
  4. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing (EMNLP 2016), pages 2331-2336.
  5. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2016), pages 199-209.
  6. Jeffrey L Elman. 1990. Finding Structure in Time. Cognitive science, 14(2):179-211.
  7. Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Im- proving neural parsing by disentangling model com- bination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (ACL 2017), pages 161-166.
  8. Yarin Gal and Zoubin Ghahramani. 2016. A Theoreti- cally Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Informa- tion Processing Systems 29 (NIPS 2016).
  9. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017. Improving Neural Language Models with a Continuous Cache. In Proceedings of the 5th Inter- national Conference on Learning Representations (ICLR 2017).
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735-1780.
  11. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017).
  12. Nikita Kitaev and Dan Klein. 2018. Constituency pars- ing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 2676- 2686.
  13. Shun Kiyono, Sho Takase, Jun Suzuki, Naoaki Okazaki, Kentaro Inui, and Masaaki Nagata. 2017. Source-side prediction for neural headline genera- tion. CoRR.
  14. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In In Proceedings of the IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP 1995), pages 181-184.
  15. Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. 2017. Dynamic evaluation of neural sequence models. CoRR.
  16. Thang Luong, Hieu Pham, and Christopher D. Man- ning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1412- 1421.
  17. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a Large Anno- tated Corpus of English: The Penn Treebank. Com- putational Linguistics, 19(2):313-330.
  18. Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. Proceedings of the 6th International Con- ference on Learning Representations (ICLR 2018).
  19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and Optimizing LSTM Language Models. In Proceedings of the 6th Inter- national Conference on Learning Representations (ICLR 2018).
  20. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mix- ture Models. In Proceedings of the 5th Inter- national Conference on Learning Representations (ICLR 2017).
  21. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceed- ings of the 11th Annual Conference of the Interna- tional Speech Communication Association (INTER- SPEECH 2010), pages 1045-1048.
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed Representa- tions of Words and Phrases and their Compositional- ity. In Advances in Neural Information Processing Systems 26 (NIPS 2013), pages 3111-3119.
  23. Andriy Mnih and Koray Kavukcuoglu. 2013. Learn- ing Word Embeddings Efficiently with Noise- Contrastive Estimation. In Advances in Neural Information Processing Systems 26 (NIPS 2013), pages 2265-2273.
  24. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Pro- ceedings of the Joint Workshop on Automatic Knowl- edge Base Construction and Web-scale Knowledge Extraction, pages 95-100.
  25. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018), pages 2227-2237.
  26. Boris T Polyak and Anatoli B Juditsky. 1992. Ac- celeration of Stochastic Approximation by Averag- ing. SIAM Journal on Control and Optimization, 30(4):838-855.
  27. Ofir Press and Lior Wolf. 2017. Using the Output Em- bedding to Improve Language Models. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics (EACL 2017), pages 157-163.
  28. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 379- 389.
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (ACL 2016), pages 86-96.
  30. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017).
  31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Re- search, 15(1):1929-1958.
  32. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. In Pro- ceedings of the Deep Learning Workshop in ICML 15.
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Net- works. In Advances in Neural Information Process- ing Systems 27 (NIPS 2014), pages 3104-3112.
  34. Jun Suzuki, Sho Takase, Hidetaka Kamigaito, Makoto Morishita, and Masaaki Nagata. 2018. An empirical study of building a strong baseline for constituency parsing. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (ACL 2018), pages 612-618.
  35. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Ser- manet, Scott Reed, Dragomir Anguelov, Dumitru Er- han, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceed- ings of the 28th IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR 2015), pages 1-9.
  36. Sho Takase, Jun Suzuki, and Masaaki Nagata. 2017. Input-to-output gate to improve rnn language mod- els. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJC- NLP 2017), pages 43-48.
  37. Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. 2013. Regularization of Neural Net- works using DropConnect. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013), pages 1058-1066.
  38. Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Se- mantically Conditioned LSTM-based Natural Lan- guage Generation for Spoken Dialogue Systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1711-1721.
  39. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018).
  40. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. In Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014).
  41. Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (ACL 2017), pages 1095-1104.
  42. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2017. Recurrent Highway Networks. Proceedings of the 34th Inter- national Conference on Machine Learning (ICML 2017), pages 4189-4198.
  43. Konrad Zolna, Devansh Arpit, Dendi Suhubdy, and Yoshua Bengio. 2018. Fraternal dropout. In Pro- ceedings of the 6th International Conference on Learning Representations (ICLR 2018).
  44. Barret Zoph and Quoc V. Le. 2017. Neural Archi- tecture Search with Reinforcement Learning. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017).

Cited by

Sorted by citations

See all 41

Metadata

Authors
Takase, S., Suzuki, J., & Nagata, M.
Affiliations
Tohoku University (JP)
Corresponding author
Nagata, Masaaki
ORCID
Suzuki, Jun
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
4599-4609
DOI
10.48550/arxiv.1808.10143
arXiv
1808.10143
OpenAlex
W2888799392
Cited by
41
References
0
Citations by year
2026: 1 · 2025: 2 · 2024: 1 · 2023: 3 · 2022: 3 · 2021: 5 · 2020: 8 · 2019: 16 · 2018: 2
FWCI
2.75
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
recurrent neural network language models, Penn Treebank, WikiText-2, machine translation, headline generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review