CoChat
Sign Up
PreprintCited by 6Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Do Fine-tuned Commonsense Language Models Really Generalize?

Open access
Kejriwal, Mayank, Shen, Ke·2020·arXiv
DOI 10.48550/arxiv.2011.09159arXiv 2011.09159
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, transformer-based methods such as RoBERTa and GPT-3 have led to significant experimental advances in natural language processing tasks such as question answering and commonsense reasoning. The latter is typically evaluated through multiple benchmarks framed as multiple-choice instances of the former. According to influential leaderboards hosted by the Allen Institute (evaluating state-of-the-art performance on commonsense reasoning benchmarks), models based on such transformer methods are approaching human-like performance and have average accuracy well over 80% on many benchmarks. Since these are commonsense benchmarks, a model that generalizes on commonsense reasoning should not experience much performance loss across multiple commonsense benchmarks. In this paper, we study the generalization issue in detail by designing and conducting a rigorous scientific study. Using five common benchmarks, multiple controls and statistical analysis, we find clear evidence that fine-tuned commonsense language models still do not generalize well, even with moderate changes to the experimental setup, and may, in fact, be susceptible to dataset bias. We also perform selective studies, including qualitative and consistency analyses, to gain deeper insight into the problem.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2011.09159

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Attention-level prompting consistently improves the baseline by a large margin in all tested missing-modality scenarios.

    “First, we find that our attention-level prompting consistently improves the baseline by a large margin in all the scenarios, showing that”

    From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [13]

  • Finetuning the entire ViLT model requires updating 113M parameters and achieves 46.45 F1-Macro on MM-IMDb for the missing-both case.

    “In comparison with finetuning the entire model which needs to update 113M parameters of the ViLT, resulting in 46.45 F1-Macro score on MM-IMDb for the missing-both case”

    From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [13]

  • The proposed method keeps pre-trained ViLT frozen and trains only 221K prompt parameters (0.2% of the entire model) while achieving 42.66 F1-Macro on MM-IMDb for the missing-both case.

    “our proposed method keeps the pre-trained ViLT frozen and only needs to train the missing-aware prompts of 221K parameters (i.e., merely 0.2% compared to the entire model), while still achieving competitive performance of 42.66 in F1-Macro (see Table 1).”

    From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [13]

  • Finetuning a transformer on relatively small-scale target datasets can result in restricted generalizability and stability.

    “finetuning a transformer on relatively small-scale target datasets can result in restricted generalizability [9, 10] and stability [24]”

    From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [10]

Cites

Flagged first, then by citations

1 of the 50 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Kejriwal, M., & Shen, K.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2011.09159
arXiv
2011.09159
OpenAlex
W3098008462
Cited by
6
References
50
Citations by year
2023: 3 · 2021: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
commonsense reasoning, generalization, fine-tuning, data bias, benchmark evaluation, multiple-choice question answering, consistency analysis, transformer models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review