Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 55 of the 60 references it cites were found in a registry or an index; 9 of the 26 sources its claims cite were checked for support.

In-Context Retrieval-Augmented Language Models

Open access
Ram, Ori, Levine, Yoav, Dalmedigos, Itay, Muhlgay, Dor, Shashua, Amnon, Leyton-Brown, Kevin, Shoham, Yoav · 2023 · Transactions of the Association for Computational Linguistics

Abstract

Abstract Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.1

Overview

CoChat Summary

TL;DR

In-Context RALM prepends retrieved documents to a frozen LM’s input without architecture changes or training, and off-the-shelf retrievers yield large perplexity gains across model sizes and corpora. Tailoring document ranking via zero-shot or trained rerankers further improves performance, including on open-domain QA.

Key findings

  • With BM25 retrieval (`s = 4`, `ℓ = 32`), In-Context RALM improved LM perplexity enough to match a 2–3× larger model; e.g., 6.7B OPT matched 66B OPT, and 345M GPT-2 + BM25 outperformed 762M GPT-2.
  • BM25 outperformed dense retrievers (Contriever, Spider, BERT mean pooling); optimal query length was `ℓ = 32` for BM25 vs. `ℓ = 64` for dense retrievers, and more frequent retrieval (smaller stride) improved performance.
  • Predictive reranking improved WikiText-103 word perplexity for GPT-2 110M from 29.6 to 26.8 and GPT-2 1.5B from 16.6 to 15.4; zero-shot reranking also improved over taking the top BM25 document.
  • On ODQA with two DPR documents, LLaMA-13B improved NQ exact match from 12.0% to 31.0% and TriviaQA from 54.8% to 60.1%; LLaMA-33B improved NQ from 13.7% to 32.3% and TriviaQA from 58.3% to 62.7%.

Methods

  • Evaluated on five LM datasets (WikiText-103, RealNews, ArXiv, Stack Exchange, FreeLaw) and two ODQA datasets (Natural Questions, TriviaQA), using GPT-2 (110M–1.5B), GPT-Neo/GPT-J (1.3B–6B), OPT (125M–66B), and LLaMA (7B–33B) models with max sequence length 1,024.
  • Retrieval corpora: Wikipedia (Dec. 20, 2018) for WikiText-103/ODQA with 120 WikiText dev/test articles removed; training data for other datasets; non-overlapping 100-word passages truncated at 256 tokens. Retrievers: BM25, BERT mean pooling, Contriever, Spider; DPR for ODQA.
  • Main analyses measured perplexity on LM datasets and exact match on ODQA; reranking used top-16 BM25 documents, with zero-shot LM reranking and a RoBERTa-base predictive reranker trained for 10,000 steps (peak LR 1e-5, batch 32, 300,000 WikiText-103 examples).

Limitations

  • Stated: the paper considers only prepending a single external document; adding more documents could drive further gains. It retrieves every fixed interval of `s` tokens, and future work could retrieve more sparsely (e.g., when a specialized model predicts retrieval is needed) for latency/cost gains.
  • Stated: predictive reranking still leaves room compared with top-16 BM25 oracle results; oracle could improve with `k > 16` or stronger RALM-dedicated retrievers.
  • Evident: main LM experiments use one document and max sequence length 1,024 despite some models supporting 2,048; the predictive reranker assumes availability of training data from the target corpus and was trained on WikiText-103.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00605

Cites

55 of 60 references verified3 citation errors · 2 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 An example of In-Context RALM : We simply prepend the retrieved document before the input prefix.
Figure 1 An example of In-Context RALM : We simply prepend the retrieved document before the input prefix.CC-BY · From the PDF
Figure 2 Our framework, dubbed In-Context RALM , provides large language modeling gains on the test set of WikiText-103, without modifying the LM . Adapting the use of a BM25 retriever (Robertson and Zaragoza, 2009) to the LM task ( § 5) yields significant gains, and choosing the grounding documents via our new class of Predictive Rerankers ( § 6) provides a further boost. See Table 1 for the full results on five diverse corpora.
Figure 2 Our framework, dubbed In-Context RALM , provides large language modeling gains on the test set of WikiText-103, without modifying the LM . Adapting the use of a BM25 retriever (Robertson and Zaragoza, 2009) to the LM task ( § 5) yields significant gains, and choosing the grounding documents via our new class of Predictive Rerankers ( § 6) provides a further boost. See Table 1 for the full results on five diverse corpora.CC-BY · From the PDF
Figure 3 The performance of four off-the-shelf retrievers used for In-Context RALM on the development set of WikiText-103. All RALMs are run with s = 4 ( i.e. , retrieval is applied every four tokens). For each RALM, we report the result of the best query length ℓ (see Figures 6, 9, 10).
Figure 3 The performance of four off-the-shelf retrievers used for In-Context RALM on the development set of WikiText-103. All RALMs are run with s = 4 ( i.e. , retrieval is applied every four tokens). For each RALM, we report the result of the best query length ℓ (see Figures 6, 9, 10).CC-BY · From the PDF
Figure 4 Results of OPT models (Zhang et al., 2022) on the test set of WikiText-103 (word-level perplexity) and the development set of RealNews (token-level perplexity). In-Context RALM models use a BM25 retriever with s = 4 ( i.e. , the retriever is called every four tokens) and ℓ = 32 ( i.e. , the retriever query is comprised of the last 32 tokens of the prefix). In-Context RALM with an off-the-shelf retriever improved the performance of a 6.7B parameter OPT model to match that of a 66B parameter OPT model.
Figure 4 Results of OPT models (Zhang et al., 2022) on the test set of WikiText-103 (word-level perplexity) and the development set of RealNews (token-level perplexity). In-Context RALM models use a BM25 retriever with s = 4 ( i.e. , the retriever is called every four tokens) and ℓ = 32 ( i.e. , the retriever query is comprised of the last 32 tokens of the prefix). In-Context RALM with an off-the-shelf retriever improved the performance of a 6.7B parameter OPT model to match that of a 66B parameter OPT model.CC-BY · From the PDF
Figure 5 An analysis of perplexity as a function of s , the retrieval stride , i.e. , the number of tokens between consecutive retrieval operations, on the development set of WikiText-103. Throughout the paper, we use s = 4 to balance perplexity and runtime.
Figure 5 An analysis of perplexity as a function of s , the retrieval stride , i.e. , the number of tokens between consecutive retrieval operations, on the development set of WikiText-103. Throughout the paper, we use s = 4 to balance perplexity and runtime.CC-BY · From the PDF
Figure 6 An analysis of perplexity as a function of the number of tokens in the query ℓ for BM25 on the development set of WikiText-103. In the appendix, we show similar trade-offs for dense retrievers within WikiText-103. Throughout the paper, we use a query length of ℓ = 32 tokens.
Figure 6 An analysis of perplexity as a function of the number of tokens in the query ℓ for BM25 on the development set of WikiText-103. In the appendix, we show similar trade-offs for dense retrievers within WikiText-103. Throughout the paper, we use a query length of ℓ = 32 tokens.CC-BY · From the PDF
Figure 7 Potential for gains from reranking. Perplexity improvement (on the development set of WikiText-103) from an oracle that takes the best of the top-16 documents retrieved by BM25 rather than the first.
Figure 7 Potential for gains from reranking. Perplexity improvement (on the development set of WikiText-103) from an oracle that takes the best of the top-16 documents retrieved by BM25 rather than the first.CC-BY · From the PDF
Figure 8 Zero-shot performance of In-Context RALM on the development set of Natural Questions and TriviaQA, when varying the number of documents (retrieved by DPR) shown in-context.
Figure 8 Zero-shot performance of In-Context RALM on the development set of Natural Questions and TriviaQA, when varying the number of documents (retrieved by DPR) shown in-context.CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF

Verification

Claims the paper makes

5 Valid · 4 Not valid · 9 Not checked · 43 Own results and interpretations

42 more claims

Create a free account to see all 61 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 9
  • The quote only partly states this claim. 3
  • It may state two separate claims as one. 4

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

9 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 55 references

Research works printed but not linked: 1

  1. Levine, 2022, Huge frozen language models as readers for open-domain question answering

Cited by

See all 451

Metadata

Authors
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y.
Affiliations
  • AI21 Labs, Israel. orir@ai21.com
  • AI21 Labs, Israel. yoavl@ai21.com
  • AI21 Labs, Israel. itayd@ai21.com
  • AI21 Labs, Israel. dorm@ai21.com
  • AI21 Labs, Israel. amnons@ai21.com
  • AI21 Labs, Israel. kevinlb@ai21.com
  • AI21 Labs, Israel. yoavs@ai21.com
Corresponding author
  • Ram, Ori
  • Levine, Yoav
  • Dalmedigos, Itay
  • Muhlgay, Dor
  • Shashua, Amnon
  • Shoham, Yoav
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2023
Volume
11
Pages
1316-1331
ISSN
2307-387X (ISSN-L)
OpenAlex
W4388778348
Cited by
449
References
55
Citations by year
2026: 160 · 2025: 189 · 2024: 80 · 2023: 20
FWCI
42.21
Type
journal article
Language
en
Open access
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
retrieval-augmented language models, document retrieval, document ranking, factual accuracy, source attribution, pre-trained language models, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: In-Context Retrieval-Augmented Language Models. https://app.cochat.ai/papers/10.1162/tacl_a_00605
The paper
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023). In-Context Retrieval-Augmented Language Models. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00605

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.