CoChat
Sign Up
PreprintCited by 215Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Cloze-driven Pretraining of Self-attention Networks

Open access
Baevski, Alexei, Edunov, Sergey, Liu, Yinhan, Zettlemoyer, Luke, Auli, Michael·2019·arXiv
DOI 10.48550/arxiv.1903.07785arXiv 1903.07785
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experiments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with the concurrently introduced BERT model. We also present a detailed analysis of a number of factors that contribute to effective pretraining, including data domain and size, model capacity, and variations on the cloze objective.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1903.07785

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some recent notable exceptions have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia).

    “Some recent notable exceptions (Baevski et al., 2019; Liu et al., 2019c; Yang et al., 2019) have compared pre-training on a new large (often Common Crawl-sourced) data set to using a smaller preexisting data set (often Wikipedia).”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Baevski et al., 2019

  • Baevski et al. (2019) similarly observed that truncating the pre-training data set size can degrade downstream task performance.

    “Baevski et al. (2019) similarly observed that truncating the pre-training data set size can degrade downstream task performance.”

    From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Baevski et al. (2019)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. Alexei Baevski and Michael Auli. 2018. Adaptive in- put representations for neural language modeling. arXiv, abs/1809.10853. · arXiv
  2. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In Proc. of TAC.
  3. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Find- ings of the 2018 conference on machine translation (WMT18). In Proc. of WMT.
  4. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2018. Semeval- 2017 task 1: Semantic textual similarity -multilin- gual and cross-lingual focused evaluation. In Proc. of SemEval.
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognizing textual entailment challenge. Machine learning challenges, evaluat- ing predictive uncertainty, visual object classifica- tion, and recognizing textual entailment, pages 177- 190.
  6. Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. arXiv, abs/1511.01432. · arXiv
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. CoRR, abs/1810.04805. · arXiv
  8. William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proc. of IWP.
  9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The pascal recognizing textual entailment challenge. Proc. of the ACL- PASCAL workshop on textual entailment and para- phrasing.
  10. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Ar- mand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proc. of LREC.
  11. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for gpus. In Proc. of ICML.
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. In Proc. of CVPR.
  13. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv, abs/1611.01462. · arXiv
  14. Yoon Kim, Yacine Jernite, David Sontag, and Alexan- der M Rush. 2016. Character-aware neural language models. In AAAI, pages 2741-2749.
  15. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proc. of ICLR.
  16. Nikita Kitaev and Dan Klein. 2018. Constituency pars- ing with a self-attentive encoder. In Proc. of ACL.
  17. Ilya Loshchilov and Frank Hutter. 2016. SGDR: stochastic gradient descent with restarts. arXiv, abs/1608.03983. · arXiv
  18. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Con- textualized word vectors. In Proc. of NIPS.
  19. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context em- bedding with bidirectional lstm. In Proc. of CoNLL.
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations.
  21. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine trans- lation. In Proc. of WMT.
  22. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proc. of ICML.
  23. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proc. of ACL.
  24. Jason Phang, Thibault Fevry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv, abs/1811.01088. · arXiv
  25. Ofir Press and Lior Wolf. 2017. Using the output em- bedding to improve language models. In Proc. of EACL.
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ ques- tions for machine comprehension of text. arXiv, abs/1606.05250. · arXiv
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL.
  28. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree- bank. In Proc. of EMNLP.
  29. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. 2013. On the importance of ini- tialization and momentum in deep learning. In Proc. of ICML.
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS.
  31. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. arXiv, abs/1804.07461. · arXiv
  32. Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL.
  33. Kelly W. Zhang and Samuel R. Bowman. 2018. Lan- guage modeling teaches you more syntax than trans- lation does: Lessons learned through auxiliary task analysis. arXiv, abs/1809.10040. · arXiv
  34. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv, abs/1506.06724. · arXiv

Cited by

Sorted by citations

Authors
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., & Auli, M.
Affiliations
Meta (Israel) (IL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
5359-5368
DOI
10.48550/arxiv.1903.07785
arXiv
1903.07785
OpenAlex
W2920812691
Cited by
215
References
0
Citations by year
2026: 2 · 2025: 8 · 2024: 15 · 2023: 18 · 2022: 36 · 2021: 50 · 2020: 67 · 2019: 19
FWCI
16.45
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
cloze task, self-attention network, pre-training, language model pretraining, language modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review