CoChat
Sign Up
PreprintCited by 758Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index.

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Open access
Min, Sewon, Lyu, Xinxi, Holtzman, Ari, Artetxe, Mikel, Lewis, Mike, Hajishirzi, Hannaneh, Zettlemoyer, Luke·2022·arXiv
DOI 10.48550/arxiv.2202.12837arXiv 2202.12837
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.12837

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 57 references

Research works printed but not linked: 4

  1. Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fer- nandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP.
  2. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop.
  3. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. ArXiv.
  4. Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019b. A survey of zero-shot learning: Set- tings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST).

Cited by

Sorted by citations

Items 701–758 of 758

Related

See all

Metadata

Authors
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L.
Affiliations
  • University of Washington (US)
  • Meta (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
11048-11064
DOI
10.48550/arxiv.2202.12837
arXiv
2202.12837
OpenAlex
W4385567149
Cited by
752
References
57
Citations by year
2026: 135 · 2025: 227 · 2024: 179 · 2023: 191 · 2022: 20
FWCI
85.30
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2044660)
  • Office of Naval Research (N00014-18, N00014-18-1-2826, ONR-N00014, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
in-context learning, demonstrations, large language models, label noise, input distribution, label space, GPT-3, classification tasks, multiple-choice tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review