REALM: Retrieval-Augmented Language Model Pre-Training
Open accessAbstract
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditional retrieval-augmented language models only read from human-written corpora.
“While traditional retrieval-augmented language models (Guu et al., 2020; Lewis et al., 2020; Borgeaud et al., 2022) only read from human-written corpora”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Guu et al., 2020
The Introduction (p. 1) describes REALM retrieving documents from a textual corpus such as Wikipedia, and Implementation Details (p. 6) specifies that its knowledge corpus was derived from English Wikipedia. This supports REALM as an example that reads from a human-written corpus, but the paper does not establish that traditional retrieval-augmented language models only read from such corpora. (LLM, conf=0.86)
Experimental — accuracy not yet measuredRAG enhances LLM performance by incorporating external evidence, improving factual grounding and reducing hallucination in knowledge-intensive tasks.
“RAG enhances LLM performance by incorporating external evidence, improving factual grounding and reducing hallucination in knowledge-intensive tasks [4, 11].”
From TaSR-RAG: Taxonomy-guided Structured Reasoning for Retrieval-Augmented Generation (2026) · cites this paper as [4]
The Introduction (pp. 1–2) says REALM retrieves external documents to inform predictions and reports improved accuracy on knowledge-intensive open-domain question answering. It does not report a reduction in hallucinations or evaluate factual grounding as a separate outcome; the evidence is for REALM specifically, not all RAG systems. (LLM, conf=0.86)
Experimental — accuracy not yet measuredRetrieval-Augmented Generation (RAG) mitigates the hallucination of large language models (LLMs).
“Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs).”
From Benchmarking Large Language Models in Retrieval-Augmented Generation (2024) · cites this paper as Guu et al. 2020
The Abstract (p. 1) reports that REALM retrieves external documents and improves open-domain question-answering accuracy. That supports the promise of retrieval for more accurate answers, but the paper studies retrieval-augmented language-model pre-training, not Retrieval-Augmented Generation, and does not assess hallucination reduction. (LLM, conf=0.82)
Experimental — accuracy not yet measuredLarge language models have the ability to retrieve factual knowledge as explicit knowledge bases, presenting an opportunity to construct more comprehensive knowledge graphs within recommender systems.
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [61]
The Introduction (p. 1) says REALM retrieves documents from a textual corpus such as Wikipedia to inform predictions, supporting the factual-knowledge retrieval claim. It does not describe retrieving knowledge as an explicit knowledge base; the Future Work section (p. 9) treats structured knowledge as a possible extension. The recommender-system application is the citing authors’ proposed implication, not a finding attributed to REALM. (LLM, conf=0.84)
Experimental — accuracy not yet measuredGMAI can retrieve relevant context from existing databases, in the form of articles, images or entire previous cases.
“Furthermore, building on recent retrieval-based approaches, GMAI can retrieve relevant context from existing databases, in the form of articles, images or entire previous cases 19,20 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [20]
The Introduction (p. 1) says REALM retrieves relevant documents from a corpus such as Wikipedia, supporting the article-retrieval part. Future Work (p. 9) proposes image retrieval as a possible extension, not an existing capability. The paper does not establish that GMAI retrieves images or entire previous cases. (LLM, conf=0.88)
Experimental — accuracy not yet measured
REALM integrates retrieval into language-model pretraining, while RETRO conditions language models on retrieved chunks from very large corpora.
“REALM integrates retrieval into language-model pretraining, while RETRO conditions language models on retrieved chunks from very large corpora [6, 4].”
From Oracle Agent Memory as an Enterprise Memory Substrate for Long-Horizon AI Agents (2026) · cites this paper as [6]
Retrieval-augmented language models improve token predictions by retrieving information from a data store.
“Retrieval-augmented language models improve token predictions by retrieving information from a data store.”
From Adapting Language Models to Compress Contexts (2023) · cites this paper as Guu et al., 2020
REALM demonstrated significant performance improvements in open-domain question-answering tasks and introduced the first unsupervised pre-training method for a knowledge retriever using masked language modeling and backpropagation.
“This work demonstrated significant performance improvements in open-domain question-answering tasks, introducing the first unsupervised pre-training method for a knowledge retriever using masked language modeling and backpropagation.”
From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Guu et al., 2020
Retrieval-Augmented Generation decreases factual errors in knowledge-intensive tasks.
“Retrieval-Augmented Generation (RAG) methods (Figure 1 left; Lewis et al. 2020; Guu et al. 2020) augment the input of LLMs with relevant retrieved passages, reducing factual errors in knowledge-intensive tasks (Ram et al., 2023; Asai et al., 2023a).”
From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Guu et al. 2020
In-context learning methods now routinely invoke tools, leading to LM pipelines that use retrieval models, multimodal foundation models, and more traditional tools like APIs and calculators.
“In-context learning methods now routinely invoke tools, leading to LM pipelines that use retrieval models (Chen et al., 2017; Lewis et al., 2020; Guu et al., 2020; Lazaridou et al., 2022; Izacard et al., 2022), multimodal foundation models, and more traditional tools like APIs (Nakano et al., 2021) and calculators.”
From DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023) · cites this paper as Guu et al., 2020
Even retrieval-augmented models still do not guarantee attribution.
“Even retrievalaugmented models (Guu et al., 2020;Lewis et al., 2020), which retrieve relevant documents and then condition on them to generate text, still do not guarantee attribution.”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Guu et al., 2020
RaLMSpec reduces the service overhead of iterative RaLMs with speculative retrieval and batched verification while ensuring consistency of model outputs.
“RaLMSpec reduces the service overhead of iterative RaLMs with speculative retrieval and batched verification while ensuring the consistency of model outputs [12].”
From Exploring Optimal Prefetching Sizes in RaLMSpec to Enhance Retrieval-Augmented Generation Efficiency (2025) · cites this paper as [12]
The Introduction (pp. 1–2) describes REALM’s learned retriever and cached, asynchronously updated retrieval computations, not RaLMSpec. Searches for RaLMSpec, speculative retrieval, batched verification, and output consistency found no passage in our copy addressing the attributed claim. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 38 · Cited by 574
Research works printed but not linked: 24
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014. · arXiv
- Berant, J., Chou, A., Frostig, R., and Liang, P. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1533-1544, 2013.
- Clark, C. and Gardner, M. Simple and effective multiparagraph reading comprehension. In Annual Meeting of the Association for Computational Linguistics , 2017.
- Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In Advances in neural information processing systems , pp. 3079-3087, 2015.
- Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. ArXiv , abs/1410.5401, 2014. · arXiv
- Hashimoto, T. B., Guu, K., Oren, Y., and Liang, P. S. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems , pp. 10052-10062, 2018.
- Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In Advances in neural information processing systems , pp. 3294-3302, 2015.
- Lee, K., Salant, S., Kwiatkowski, T., Parikh, A., Das, D., and Berant, J. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436 , 2016. · arXiv
- Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems , pp. 3111-3119, 2013b.
- Miller, A., Fisch, A., Dodge, J., Karimi, A.-H., Bordes, A., and Weston, J. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126 , 2016. · arXiv
- Min, S., Chen, D., Hajishirzi, H., and Zettlemoyer, L. A discrete hard em approach for weakly supervised question answering. arXiv preprint arXiv:1909.04849 , 2019a. · arXiv
- Min, S., Chen, D., Zettlemoyer, L., and Hajishirzi, H. Knowledge guided text retrieval and reading for open domain question answering. arXiv preprint arXiv:1911.03868 , 2019b. · arXiv
- Petroni, F., Rockt¨ aschel, T., Lewis, P., Bakhtin, A., Wu, Y ., Miller, A. H., and Riedel, S. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 , 2019. · arXiv
- Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pp. 23832392, 2016.
- Rajpurkar, P., Jia, R., and Liang, P. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 , 2018. · arXiv
- Ram, P. and Gray, A. G. Maximum inner-product search using cone trees. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining , pp. 931-939, 2012.
- Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:TBD , 2020.
- Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333-389, 2009.
- Sang, E. T. K. and De Meulder, F. Introduction to the conll2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 , pp. 142-147, 2003.
- Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations , 2016.
- Shen, F., Liu, W., Zhang, S., Yang, Y., and Tao Shen, H. Learning binary codes for maximum inner product search. In Proceedings of the IEEE International Conference on Computer Vision , pp. 4148-4156, 2015.
- Shrivastava, A. and Li, P. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In Advances in Neural Information Processing Systems , pp. 2321-2329, 2014.
- Sukhbaatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. In Advances in neural information processing systems , 2015.
- Weston, J., Chopra, S., and Bordes, A. Memory networks. arXiv preprint arXiv:1410.3916 , 2014. · arXiv
- Authors
- Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W.
- Affiliations
- Google (United States) (US)
- ORCID
- Lee, Kenton
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.08909
- OpenAlex
- W3007672467
- Cited by
- 574
- References
- 38
- Citations by year
- 2026: 18 · 2025: 45 · 2024: 51 · 2023: 90 · 2022: 129 · 2021: 165 · 2020: 76
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- retrieval-augmented language models, language model pretraining, open-domain question answering, masked language modeling, interpretability, modularity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
