CoChat
Sign Up
Cited by 108Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 64 of the 69 references it cites were found in a registry or an index, and 1 are not checked yet.

When Do You Need Billions of Words of Pretraining Data?

Open access
Zhang, Yian, Warstadt, Alex, Li, Xiaocheng, Bowman, Samuel R.·2021·Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
DOI 10.18653/v1/2021.acl-long.90
Open this paper in CoChat — free accountOpen PDF

Abstract

Yian Zhang, Alex Warstadt, Xiaocheng Li, Samuel R. Bowman. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.acl-long.90

Cites

64 of 65 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overall learning curves for the five evaluation methods. For each method, we compute overall performance for each RoBERTa model tested as the macro average over sub-task's performance after normalization. We fit an exponential curve which we scale to have an initial value of 0 and an asymptote at 1. Classifier and MDL probing mainly test models' encoding of linguistic features; BLiMP tests model's understanding of linguistic phenomena; LAMA tests factual knowledge; SuperGLUE is a suite of conventional NLU tasks.
Figure 1 Overall learning curves for the five evaluation methods. For each method, we compute overall performance for each RoBERTa model tested as the macro average over sub-task's performance after normalization. We fit an exponential curve which we scale to have an initial value of 0 and an asymptote at 1. Classifier and MDL probing mainly test models' encoding of linguistic features; BLiMP tests model's understanding of linguistic phenomena; LAMA tests factual knowledge; SuperGLUE is a suite of conventional NLU tasks.CC-BY · From the PDF
Figure 2 Classifier probing results for each task in the edge probing suite. Results are adjusted with min-max normalization for readability (see the Appendix for a non-normalized version). In each subplot we also plot the overall edge-probing performance, which we calculate for each MiniBERTa as its average F1 score on the 10 edgeprobing tasks (after normalization). For context, we also plot BERTLARGE performance for each task as reported by Tenney et al. (2019a).
Figure 2 Classifier probing results for each task in the edge probing suite. Results are adjusted with min-max normalization for readability (see the Appendix for a non-normalized version). In each subplot we also plot the overall edge-probing performance, which we calculate for each MiniBERTa as its average F1 score on the 10 edgeprobing tasks (after normalization). For context, we also plot BERTLARGE performance for each task as reported by Tenney et al. (2019a).CC-BY · From the PDF
Figure 3 Edge Probing results for each group of tasks adjusted using min-max normalization. Syntactic tasks are Part-of-Speech, Dependencies, and Constituents. The commonsense task is Winograd coref. Semantic tasks are all remaining tasks.
Figure 3 Edge Probing results for each group of tasks adjusted using min-max normalization. Syntactic tasks are Part-of-Speech, Dependencies, and Constituents. The commonsense task is Winograd coref. Semantic tasks are all remaining tasks.CC-BY · From the PDF
Figure 4 MDL results for each edge probing task. We do not plot a exponential curve for the Winograd coref. results because we could not find an adequate fit.
Figure 4 MDL results for each edge probing task. We do not plot a exponential curve for the Winograd coref. results because we could not find an adequate fit.CC-BY · From the PDF
Figure 5 BLiMP results by category. BLiMP has 67 constituent datasets covering 12 linguistic phenomena. For each task the objective is to predict the more grammatically acceptable sentence of a minimal pair in an unsupervised setting. For context, we also plot human accuracy numbers from Warstadt et al. (2020a) and RoBERTaLARGE performance from Salazar et al. (2020).
Figure 5 BLiMP results by category. BLiMP has 67 constituent datasets covering 12 linguistic phenomena. For each task the objective is to predict the more grammatically acceptable sentence of a minimal pair in an unsupervised setting. For context, we also plot human accuracy numbers from Warstadt et al. (2020a) and RoBERTaLARGE performance from Salazar et al. (2020).CC-BY · From the PDF
Figure 6 LAMA results. The metric for all tasks is mean precision at 1, i.e. the proportion of examples where the model assigns the highest probability to the ground truth token. For context, we also plot RoBERTaLARGE results.
Figure 6 LAMA results. The metric for all tasks is mean precision at 1, i.e. the proportion of examples where the model assigns the highest probability to the ground truth token. For context, we also plot RoBERTaLARGE results.CC-BY · From the PDF
Figure 7 SuperGLUE results. The metric for BoolQ, COPA, WiC, RTE is accuracy, and for CB it is the average of accuracy and F1 score. Results are adjusted with min-max normalization for readability (see the Appendix for a non-normalized version). For context, we plot RoBERTaLARGE performance reported at https://github.com/ pytorch/fairseq/tree/master/examples/roberta .
Figure 7 SuperGLUE results. The metric for BoolQ, COPA, WiC, RTE is accuracy, and for CB it is the average of accuracy and F1 score. Results are adjusted with min-max normalization for readability (see the Appendix for a non-normalized version). For context, we plot RoBERTaLARGE performance reported at https://github.com/ pytorch/fairseq/tree/master/examples/roberta .CC-BY · From the PDF
Figure 8 Our absolute edge probing dev set results (not normalized) compared to BERTLARGE test set results from Tenney et al. (2019b).
Figure 8 Our absolute edge probing dev set results (not normalized) compared to BERTLARGE test set results from Tenney et al. (2019b).CC-BY · From the PDF
Figure 9 Our absolute SuperGLUE results (not normalized) compared to RoBERTaLARGE results from Liu et al. (2019).
Figure 9 Our absolute SuperGLUE results (not normalized) compared to RoBERTaLARGE results from Liu et al. (2019).CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 7

  1. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognis- ing Textual Entailment.
  2. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment chal- lenge. In Textual Analysis Conference (TAC).
  3. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC-2018).
  4. Charles Lovering, Rohan Jha, Tal Linzen, and El- lie Pavlick. 2021. Predicting inductive biases of fine-tuned models. In International Conference on Learning Representations.
  5. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
  6. Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alterna- tives: An evaluation of commonsense causal reason- ing. In 2011 AAAI Spring Symposium Series.
  7. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI

Cited by

Sorted by citations

See all 108

Related

See all

Metadata

Authors
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R.
Affiliations
New York University (US)
ORCID
Venue
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2021
Pages
1112-1125
DOI
10.18653/v1/2021.acl-long.90
OpenAlex
W3176198948
Cited by
85
References
69
Citations by year
2026: 3 · 2025: 7 · 2024: 9 · 2023: 40 · 2022: 16 · 2021: 10
FWCI
8.66
Type
conference paper
Language
en
Open access
Licence
cc-by
Funders
National Science Foundation (1850208)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
pre-training, language models, training data requirements

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review