CoChat
Sign Up
PreprintCited by 6,687Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Open access
Yang, Zhilin, Dai, Zihang, Yang, Yiming, Carbonell, Jaime, Salakhutdinov, Ruslan, Le, Quoc V.·2019·arXiv
DOI 10.48550/arxiv.1906.08237arXiv 1906.08237
Open this paper in CoChat — free accountOpen PDF

Abstract

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1906.08237

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Publicly-provided models like BERT, GPT, and XLNet may fail in some domain-specific tasks due to little knowledge connection between specific and open domain.

    “Even though they can refresh the state-of-the-art of GLUE (Wang et al. 2018) benchmark by learning from open-domain corpora, they may fail in some domain-specific tasks, due to little knowledge connection between specific and open domain.”

    From K-BERT: Enabling Language Representation with Knowledge Graph (2020) · cites this paper as Yang et al. 2019

    Section 3.1 (p. 6) describes pretraining XLNet on broad text corpora, and the Introduction (p. 2) reports strong performance on GLUE tasks. The paper does not say that BERT, GPT, or XLNet may fail on domain-specific tasks because of a weak knowledge connection between open and specific domains; the gaps it discusses concern modeling objectives, not domain knowledge. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • XL-Net extends BERT by predicting masked tokens auto-regressively in a permuted order.

    “XL-Net (Yang et al., 2019) extends BERT by predicting masked tokens auto-regressively in a permuted order.”

    From BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020) · cites this paper as Yang et al., 2019

    The Introduction (p. 1) confirms that XLNet uses an autoregressive objective over permutations of the factorization order. But it explicitly says XLNet does not rely on data corruption, so describing its prediction targets as masked tokens incorrectly imports BERT’s masking method. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Recent work has shown that downstream performance can dramatically improve when pre-training is scaled to large batch sizes and corpora.

    “Recent work has shown that downstream performance can dramatically improve when pre-training is scaled to large batch sizes (Yang et al., 2019;Liu et al., 2019) and corpora.”

    From BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020) · cites this paper as Yang et al., 2019

    Section 3.1 (p. 6) says XLNet-Large was pretrained on expanded datasets with a batch size of 8192, and Section 3.3 (p. 7) reports strong downstream results. But the paper does not establish that increasing batch size caused those improvements; its fair comparison with BERT holds training data and hyperparameters constant (Section 3.2, p. 7). (LLM, conf=0.81)

    Experimental — accuracy not yet measured

  • Scaling up produces improved performance compared to more carefully-engineered methods in transfer learning for NLP.

    “Recent results suggest that this may hold true for transfer learning in NLP (Liu et al., 2019c; Radford et al., 2019; Yang et al., 2019; Lan et al., 2019), i.e. it has repeatedly been shown that scaling up produces improved performance compared to more carefully-engineered methods.”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Yang et al., 2019

    Section 3.1 (p. 6) says XLNet scaled up training, and Section 3.3 (p. 7) reports that it generally outperformed BERT and RoBERTa. But the paper also attributes gains to XLNet’s novel pretraining objective and architectural changes (Introduction, p. 2); it does not show that scaling up, rather than those engineered changes, produced better performance than more carefully engineered methods. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The baseline model uses a relatively small amount of pre-training compared to BERT, XLNet, and RoBERTa.

    “Our baseline model uses a relatively small amount of pre-training ( 1 / 4 as much as BERT (Devlin et al., 2018), 1 / 16 as much as XLNet (Yang et al., 2019), 1 / 64 as much as RoBERTa (Liu et al., 2019c), etc.).”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Yang et al., 2019

    Section 3.1 (p. 6) reports that XLNet-Large was pretrained for 500,000 steps with a batch size of 8,192, supporting the cited figure as an XLNet pretraining benchmark. The paper does not describe the citing authors’ baseline, so it cannot establish that the baseline used 1/16 as much pretraining as XLNet. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 41 · Cited by 6,687

Research works printed but not linked: 29

  1. [2][2] Anonymous. Bam! born-again multi-task networks for natural language understanding. anonymous preprint under review, 2018. -
  2. [3][3] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853 , 2018. - · arXiv
  3. [4][4] Yoshua Bengio and Samy Bengio. Modeling high-dimensional discrete data with multi-layer neural networks. In Advances in Neural Information Processing Systems , pages 400-406, 2000. -
  4. [7][7] Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems , pages 3079-3087, 2015. -
  5. [8][8] Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the eleventh ACM international conference on web search and data mining , pages 126-134. ACM, 2018. -
  6. [9][9] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 , 2019. - · arXiv
  7. [11][11] William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the\_. arXiv preprint arXiv:1801.07736 , 2018. - · arXiv
  8. [12][12] Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning , pages 881-889, 2015. -
  9. [13][13] Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management , pages 55-64. ACM, 2016. -
  10. [14][14] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 , 2018. - · arXiv
  11. [15][15] Rie Johnson and Tong Zhang. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 562-570, 2017. -
  12. [16][16] Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290 , 2019. - · arXiv
  13. [17][17] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 , 2018. - · arXiv
  14. [19][19] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. - · arXiv
  15. [20][20] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504 , 2019. - · arXiv
  16. [22][22] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems , pages 6294-6305, 2017. -
  17. [23][23] Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semisupervised text classification. arXiv preprint arXiv:1605.07725 , 2016. - · arXiv
  18. [24][24] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759 , 2016. - · arXiv
  19. [25][25] Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. Improving question answering with external knowledge. arXiv preprint arXiv:1902.00993 , 2019. - · arXiv
  20. [26][26] Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword fifth edition, linguistic data consortium. Technical report, Technical Report. Linguistic Data Consortium, Philadelphia, Tech. Rep. , 2011. -
  21. [29][29] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 , 2018. - · arXiv
  22. [32][32] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research , 17(1):71847220, 2016. -
  23. [33][33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017. -
  24. [34][34] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR. -
  25. [35][35] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation. arXiv preprint arXiv:1904.12848 , 2019. - · arXiv
  26. [36][36] Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval , pages 55-64. ACM, 2017. -
  27. [37][37] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank rnn language model. arXiv preprint arXiv:1711.03953 , 2017. - · arXiv
  28. [38][38] Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. Dual comatching network for multi-choice reading comprehension. arXiv preprint arXiv:1901.09381 , 2019. - · arXiv
  29. [39][39] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems , pages 649-657, 2015. -

Authors
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V.
Affiliations
  • Carnegie Mellon University (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1906.08237
arXiv
1906.08237
OpenAlex
W2950813464
Cited by
6,687
References
41
Citations by year
2026: 21 · 2025: 393 · 2024: 752 · 2023: 1133 · 2022: 1019 · 2021: 1686 · 2020: 1395 · 2019: 282 · 2018: 6
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • TIB Data Manager
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
XLNet, Transformer-XL, language comprehension, question answering, natural language inference, sentiment analysis, document ranking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review