CoChat

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Open access
Dai, Zihang, Yang, Zhilin, Yang, Yiming, Carbonell, Jaime, Le, Quoc, Salakhutdinov, Ruslan · 2019 · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

The DOI matches its Crossref record; 65 of the 67 references it cites were found in a registry or an index.

Abstract

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence.It consists of a segment-level recurrence mechanism and a novel positional encoding scheme.Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem.As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation.Notably, we improve the state-ofthe-art results of bpc/perplexity to 0.99 on en-wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens.Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch 1 .

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p19-1285

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Illustration of the vanilla model with a segment length 4.
Figure 1 Illustration of the vanilla model with a segment length 4.CC-BY · From the PDF
Figure 2 Illustration of the Transformer-XL model with a segment length 4.
Figure 2 Illustration of the Transformer-XL model with a segment length 4.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

12 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Attention Is All You Need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 7×

    “On the other hand, the direct connections between long-distance word pairs baked in attention mechanisms might ease optimization and enable the learning of long-term dependency (Bahdanau et al., 2014; Vaswani et al., 2017).”
    BackgroundMethodInfluential
  • Long Short-Term Memory

    Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×

    “Recurrent neural networks (RNNs), in particular Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997), have been a standard solution to language modeling and obtained strong results on multiple benchmarks.”
    Background
  • Exploiting Generative AI to Scale up Intelligent Tutoring Systems

    Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×

    “Second, though it is possible to use padding to respect the sentence or other semantic boundaries, in practice it has been standard practice to simply chunk long text into fixed-length segments due to improved efficiency (Peters et al., 2018; Devlin et al., 2018; Al-Rfou et al., 2018).”
    BackgroundInfluential
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×

    “Second, though it is possible to use padding to respect the sentence or other semantic boundaries, in practice it has been standard practice to simply chunk long text into fixed-length segments due to improved efficiency (Peters et al., 2018; Devlin et al., 2018; Al-Rfou et al., 2018).”
    BackgroundInfluential
  • Neural Machine Translation by Jointly Learning to Align and Translate

    Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×

    “On the other hand, the direct connections between long-distance word pairs baked in attention mechanisms might ease optimization and enable the learning of long-term dependency (Bahdanau et al., 2014; Vaswani et al., 2017).”
    Background
  • Deep Contextualized Word Representations

    Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 4×

    “Second, though it is possible to use padding to respect the sentence or other semantic boundaries, in practice it has been standard practice to simply chunk long text into fixed-length segments due to improved efficiency (Peters et al., 2018; Devlin et al., 2018; Al-Rfou et al., 2018).”
    BackgroundInfluential
  • Recurrent neural network based language model

    Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 5×

    “However, different from truncated BPTT, our method caches a sequence of hidden states instead of the last one, and should be applied together with the relative positional encoding technique described in Section 3.3.”
    BackgroundMethodInfluential
  • Neural Architecture Search with Reinforcement Learning

    Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974

    Self-citation
  • A Neural Probabilistic Language Model

    Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479

  • DARTS: Differentiable architecture search

    Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248

    Self-citation
  • Generating Sequences With Recurrent Neural Networks

    Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · cited 2×

    “, 2001), and the introduction of gating in LSTMs and the gradient clipping technique (Graves, 2013) might not be sufficient to fully address this issue.”
    BackgroundMethod
  • Self-Attention with Relative Position Representations

    Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · cited 3×

    “Moreover, our relative positional encodings is also superior to Shaw et al. (2018) on short sequences.”
    Background
  • Untitled work

    (2018) · International Conference on Machine Learning · Cited by 1,465

    Self-citation
  • Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies

    (2009) · A Field Guide to Dynamical Recurrent Networks · doi:10.1109/9780470544037.ch14 · Cited by 1,246

  • A theoretically grounded application of dropout in recurrent neural networks

    Gal, Y., & Ghahramani, Z. (2016) · Cambridge University Engineering Department Publications Database · Cited by 1,089

  • Language Modeling with Gated Convolutional Networks

    Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08083 · Cited by 1,080 · cited 1×

    “…architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and…”
    Background
  • Neural Turing Machines

    Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 1×

    “Thus, we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m nτ ∈ R M × d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014; Weston et al., 2014).”
    Background
  • One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013) · arXiv · doi:10.48550/arxiv.1312.3005 · Cited by 1,051 · cited 5×

    “We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103 (Merity et al., 2016), enwiki8 (LLC, 2009), text8 (LLC, 2009), One Billion Word (Chelba et al., 2013), and Penn Treebank (Mikolov & Zweig,…”
    BackgroundInfluential
  • Semi-supervised Sequence Learning

    Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 2×

    “Language modeling is among the important problems that require modeling long-term dependency, with successful applications such as unsupervised pretraining (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018).”
    BackgroundSelf-citation
  • Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

    Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972

  • Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933

    Self-citation
  • Exploring the Limits of Language Modeling

    Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 823 · cited 1×

    “…al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et al., 2018).”
    Background
  • A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

    Gal, Y., & Ghahramani, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.05287 · Cited by 766 · cited 2×

    “, 2018), improving regularization and optimization algorithms (Gal and Ghahramani, 2016) , speeding up the Softmax computation (Grave et al.”
    Background
  • Untitled work

    (2017) · International Conference on Machine Learning · Cited by 758

  • Regularizing and Optimizing LSTM Language Models

    Merity, S., Keskar, N. S., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.02182 · Cited by 697 · cited 2×

    “…al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al.,…”
    BackgroundMethod
  • One billion word benchmark for measuring progress in statistical language modeling

    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627 · cited 5×

    “We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103 (Merity et al., 2016), enwiki8 (LLC, 2009), text8 (LLC, 2009), One Billion Word (Chelba et al., 2013), and Penn Treebank (Mikolov & Zweig,…”
    BackgroundInfluential
  • Abstract Functional Language Logic: A Competitive Mixture of Experts Architecture for Paradox-Free Reasoning and Adaptive Intelligence

    Torres H., J. P. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.17754506 · Cited by 580

  • Context dependent recurrent neural network language model

    Mikolov, T., & Zweig, G. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424228 · Cited by 560 · cited 6×

    “We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103 (Merity et al., 2016), enwiki8 (LLC, 2009), text8 (LLC, 2009), One Billion Word (Chelba et al., 2013), and Penn Treebank (Mikolov & Zweig,…”
    BackgroundInfluential
  • Efficient Neural Architecture Search via Parameter Sharing

    Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 517

    Self-citation
  • A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

    Le, Q. V., Jaitly, N., & Hinton, G. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00941 · Cited by 513 · cited 2×

    “…since the ubiquitous adaption of LSTM, many efforts have been spent on relieving the vanishing gradient problem, including better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal…”
    BackgroundSelf-citation
  • Pointer Sentinel Mixture Models

    Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07843 · Cited by 427 · cited 10×

    “Our best model in this settings use attention lengths of 1,600 and 3,800 on WikiText-103 and enwiki8 respectively.”
    BackgroundMethodInfluential
  • Recurrent Batch NormalizationPreprint

    Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.09025 · Cited by 424

  • Neural Machine Translation in Linear Time

    Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · cited 1×

    “…Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016);…”
    Background
  • Hierarchical Multiscale Recurrent Neural Networks

    Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386 · cited 2×

    “In the last few years, the field of language modeling has witnessed many significant advances, including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et…”
    Background
  • Sharp nearby, fuzzy far away: How neural language models use context

    Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · cited 1×

    “…but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al.,…”
    Background
  • Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

    Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · arXiv · doi:10.48550/arxiv.1805.04623 · Cited by 302 · cited 1×

    “…but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al.,…”
    Background
  • Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

    Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 288 · cited 1×

    “…Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005;…”
    Background
  • Efficient softmax approximation for GPUs

    Grave, É., Joulin, A., Cissé, M., & Jeǵou, H. (2017) · International Conference on Machine Learning · Cited by 219

  • TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

    Dieng, A. B., Wang, C., Gao, J., & Paisley, J. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01702 · Cited by 175 · cited 2×

    “, 2015; Wang and Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
    Background
  • Efficient softmax approximation for GPUs

    Grave, É., Joulin, A., Cissé, M., Grangier, D., & Jeǵou, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04309 · Cited by 132 · cited 2×

    “We adopted adaptive softmax and input representations (Baevski and Auli, 2018; Grave et al., 2016a).”
    BackgroundMethod
  • An Analysis of Neural Language Modeling at Multiple Scales

    Merity, S., Keskar, N. S., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08240 · Cited by 123 · cited 2×

    “The “Transformer-XL (151M)” setting uses a standard parameter budget as previous work Merity et al. (2018), where we observe a similar effect when increasing the attention length during evaluation.”
    Method
  • HyperNetworks

    Ha, D., Dai, A., & Le, Q. V. (2016) · arXiv · doi:10.48550/arxiv.1609.09106 · Cited by 100

    Self-citation
  • Factorization tricks for LSTM networks

    Kuchaiev, O., & Ginsburg, B. (2017) · arXiv · doi:10.48550/arxiv.1703.10722 · Cited by 99 · cited 1×

    “…Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et…”
    Background
  • Learning Longer-term Dependencies in RNNs with Auxiliary Losses

    Trinh, T. H., Dai, A. M., Luong, M.-T., & Le, Q. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00144 · Cited by 84 · cited 2×

    “…many efforts have been spent on relieving the vanishing gradient problem, including better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization…”
    BackgroundSelf-citation
  • Improving Neural Language Models with a Continuous Cache

    Grave, É., Joulin, A., & Usunier, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04426 · Cited by 83 · cited 3×

    “We adopted adaptive softmax and input representations (Baevski & Auli, 2018; Grave et al., 2016a).”
    BackgroundMethod
  • Multiplicative LSTM for sequence modelling

    Krause, B., Lu, L., Murray, I., & Renals, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07959 · Cited by 78 · cited 2×

    “…including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al.,…”
    Background
  • Recurrent Highway Networks

    Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 74

  • An Improved Relative Self-Attention Mechanism for Transformer with Application to Music GenerationPreprint

    Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Hawthorne, C., Dai, A. M., Hoffman, M. D., & Eck, D. (2018) · arXiv (Cornell University) · Cited by 73

  • Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

    Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03953 · Cited by 62 · cited 2×

    “, 2016a) , and enriching the output distribution family (Yang et al., 2017).”
    BackgroundSelf-citation
  • Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding

    Ke, N. R., Goyal, A., Bilaniuk, O., Binas, J., Mozer, M. C., Pal, C., & Bengio, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03702 · Cited by 58

  • Fast-Slow Recurrent Neural Networks

    Mujika, A., Meier, F., & Steger, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08639 · Cited by 57

  • Character-Level Language Modeling with Deeper Self-Attention

    Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56

  • Document Context Language Models

    Ji, Y., Cohn, T., Kong, L., Dyer, C., & Eisenstein, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03962 · Cited by 55 · cited 2×

    “ually defined (Mikolov and Zweig, 2012; Ji et al., 2015; Wang and Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al.”
    Background
  • On Multiplicative Integration with Recurrent Neural Networks

    Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., & Salakhutdinov, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.06630 · Cited by 54 · cited 1×

    “…better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization Mikolov et al. (2014); Koutnik et al. (2014); Wu et al. (2016); Li et al. (2018).”
    BackgroundSelf-citation
  • Mesh-TensorFlow: Deep Learning for Supercomputers

    Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02084 · Cited by 50

  • Topic Compositional Neural Language Model

    Wang, W., Gan, Z., Wang, W., Shen, D., Huang, J., Ping, W., Satheesh, S., & Carin, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09783 · Cited by 49 · cited 2×

    “, 2015; Wang and Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
    Background
  • Adaptive Input Representations for Neural Language Modeling

    Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 48 · cited 4×

    “We adopted adaptive softmax and input representations (Baevski and Auli, 2018; Grave et al., 2016a).”
    MethodInfluential
  • Larger-Context Language Modelling with Recurrent Neural Network

    Wang, T., & Cho, K. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1125 · Cited by 47 · cited 1×

    “Existing works range from ones where context representations are manually defined (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
    Background
  • Larger-Context Language Modelling

    Wang, T., & Cho, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03729 · Cited by 38 · cited 1×

    “Existing works range from ones where context representations are manually defined (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
    Background
  • Fast Parametric Learning with Activation Memorization

    Rae, J. W., Dyer, C., Dayan, P., & Lillicrap, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10049 · Cited by 20

  • Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability EstimationPreprint

    Shazeer, N., Pelemans, J., & Chelba, C. I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1454 · Cited by 13

  • Pushing the bounds of dropoutPreprint

    Melis, G., Blundell, C., Kočiský, T., Hermann, K. M., Dyer, C., & Blunsom, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09208 · Cited by 10

  • Memory Networks

    Weston, J., Chopra, S., & Bordes, A. (2014) · arXiv · doi:10.48550/arxiv.1410.3916 · cited 1×

    “Thus, we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m nτ ∈ R M × d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014; Weston et al., 2014).”
    Background

Research works printed but not linked: 1

  1. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. arXiv preprint arXiv:1802.05365. · arXiv

Cited by

See all 3,169

Metadata

Authors
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R.
Affiliations
  • Carnegie Mellon University (US)
  • Google (United States) (US)
Venue
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Published
2019
Pages
2978-2988
ISSN
0736-587X (ISSN-L)
OpenAlex
W2964110616
Cited by
3,169
References
67
Citations by year
2026: 163 · 2025: 361 · 2024: 434 · 2023: 585 · 2022: 504 · 2021: 612 · 2020: 403 · 2019: 107
FWCI
155.55
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
Funders
  • National Science Foundation (1546329, 1763562)
  • U.S. Department of Energy
  • Nvidia
  • Office of Science
  • Office of Naval Research (N000141812861)
  • Advanced Scientific Computing Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
Transformer-XL, long-range dependencies, positional encoding, language modeling, temporal coherence, sequence generation, pre-trained language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.