In-Context Learning with Long-Context Models: An In-Depth Exploration
Open accessAbstract
As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of in-context learning (ICL) at this extreme scale on multiple datasets and models. We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations. We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can exceed long-context ICL performance with additional data. We use the ICL setting to study several properties of both in-context learning and long-context models. We show that long-context ICL is less sensitive to random input shuffling than short-context ICL, that grouping of same-label examples negatively impacts performance, and that the performance boosts do not arise from cumulative gain from encoding many examples together. We conclude that long-context ICL can be an effective tool, and may not require long-context for encoding the demonstration set at all.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bertsch et al. concurrently show benefits of scaling up in-context learning to many demonstrations on several classification datasets with up to 151 labels, albeit using smaller context windows of up to 80k tokens.
“Bertsch et al. [6] also concurrently shows benefits of scaling up in-context learning to many demonstrations on several classification datasets with up to 151 labels, albeit also using smaller context windows of up to 80k tokens (using Llama2-80k [13]).”
From Many-Shot In-Context Learning (2024) · cites this paper as [6]
Bertsch et al. concurrently show that many-shot ICL generally outperforms parameter-efficient fine-tuning (LoRA) on classification tasks.
“Aligned with our findings, Bertsch et al. [6] concurrently show that many-shot ICL generally outperforms parameter-efficient fine-tuning (LoRA) on classification tasks.”
From Many-Shot In-Context Learning (2024) · cites this paper as [6]
Concurrent works and our preliminary results with GPT-4-Turbo, Gemini 1.5 Flash, and Claude-3-Opus indicate that other LLMs can also benefit from many-shot ICL.
“That said, concurrent works [2, 6] as well as our preliminary results with GPT-4-Turbo, Gemini 1.5 Flash, and Claude-3-Opus (§4.5) indicate that other LLMs can also benefit from many-shot ICL.”
From Many-Shot In-Context Learning (2024) · cites this paper as [6]
Bertsch et al. argue that the benefits of many in-context demonstrations arise from having access to more similar examples.
“Using a clever experiment with blocked attention, Bertsch et al. [6] also argue that the benefits of many in-context demonstrations arise from having access to more similar examples.”
From Many-Shot In-Context Learning (2024) · cites this paper as [6]
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 7
- Authors
- Bertsch, A., Ivgi, M., Xiao, E., Alon, U., Berant, J., Gormley, M. R., & Neubig, G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2405.00200
- OpenAlex
- W4396601147
- Cited by
- 7
- References
- 0
- Citations by year
- 2025: 3 · 2024: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by-nc-nd
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2140739)
- European Commission
- Yükseköğretim Kurulu
- H2020 European Research Council (802800)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- in-context learning, long-context models, example retrieval, fine-tuning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
