CoChat
Sign Up
PreprintCited by 161Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.

Language Models with Transformers

Open access
Wang, Chenguang, Li, Mu, Smola, Alexander J.·2019·arXiv
DOI 10.48550/arxiv.1904.09408arXiv 1904.09408
Open this paper in CoChat — free accountOpen PDF

Abstract

The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks using pre-trained language models on large-scale corpora. Surprisingly, these Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to efficiently incorporate the word-level sequential context crucial to language modeling. In this paper, we explore effective Transformer architectures for language model, including adding additional LSTM layers to better capture the sequential context while still keeping the computation efficient. We propose Coordinate Architecture Search (CAS) to find an effective architecture through iterative refinement of the model. Experimental results on the PTB, WikiText-2, and WikiText-103 show that CAS achieves perplexities between 20.42 and 34.11 on all problems, i.e. on average an improvement of 12.0 perplexity units compared to state-of-the-art LSTMs. The source code is publicly available.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1904.09408

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Changing to a sub-word vocabulary can reduce perplexity substantially in an otherwise unchanged architecture, for example by about 30% in a pre-GPT-2 LSTM.

    “Changing to a sub-word vocabulary can reduce perplexity substantially in an otherwise unchanged architecture, for example by ∼ 30% in a pre-GPT-2 LSTM (C. Wang, M. Li, and Smola 2019).”

    From Algorithmic progress in language models (2024) · cites this paper as C. Wang, M. Li, and Smola 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language mod- eling. arXiv preprint arXiv:1809.10853. · arXiv
  2. Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. 2017. Accelerating neural ar- chitecture search using performance prediction. arXiv preprint arXiv:1705.10823. · arXiv
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vin- cent, and Christian Jauvin. 2003. A neu- ral probabilistic language model. JMLR, 3(Feb):1137-1155.
  4. Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. 2017. Smash: one-shot model architecture search through hypernet- works. arXiv preprint arXiv:1708.05344. · arXiv
  5. Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. 2018. Efficient architecture search by network transformation. AAAI.
  6. Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 2015. Net2net: Accelerating learning via knowledge transfer. CoRR.
  7. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. · arXiv
  8. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2017. Simple and efficient architec- ture search for convolutional neural networks. CoRR.
  9. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. CoRR.
  10. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language mod- eling. arXiv preprint arXiv:1611.01462. · arXiv
  11. Haifeng Jin, Qingquan Song, and Xia Hu. 2018. Efficient neural architecture search with network morphism. arXiv preprint arXiv:1806.10282. · arXiv
  12. Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. 2018. Dynamic evaluation of neural sequence models. In ICML, pages 2771- 2780.
  13. Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi. 2018. Simple recurrent units for highly parallelizable recurrence. In Proceed- ings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 4470-4481.
  14. Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2017a. Progres- sive neural architecture search. arXiv preprint arXiv:1712.00559. · arXiv
  15. Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. 2017b. Hierarchical representations for ef- ficient architecture search. arXiv preprint arXiv:1711.00436. · arXiv
  16. Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. DARTS: differentiable architecture search. CoRR.
  17. Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589. · arXiv
  18. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR.
  19. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In ICML, pages 641-648.
  20. Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. 2018. Efficient neu- ral architecture search via parameter sharing. In ICML, pages 4092-4101.
  21. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2018. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548. · arXiv
  22. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. In NIPS, pages 5998- 6008.
  24. Alex Wang, Amapreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and anal- ysis platform for natural language understand- ing. arXiv preprint arXiv:1804.07461. · arXiv
  25. Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. 2016. Network morphism. In ICML, pages 564-572.
  26. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. · arXiv
  27. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2017. Breaking the softmax bottleneck: A high-rank RNN lan- guage model. CoRR.
  28. Barret Zoph and Quoc V. Le. 2016. Neural ar- chitecture search with reinforcement learning. CoRR.
  29. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In CVPR, pages 8697-8710.

Cited by

Sorted by citations

Metadata

Authors
Wang, C., Li, M., & Smola, A. J.
Affiliations
Amazon (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1904.09408
arXiv
1904.09408
OpenAlex
W2936497627
Cited by
161
References
5
Citations by year
2026: 10 · 2025: 23 · 2024: 31 · 2023: 24 · 2022: 17 · 2021: 29 · 2020: 22 · 2019: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
transformer models, language modeling, self-attention, positional encoding, pre-trained language models, perplexity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review