Improving language models by retrieving from trillions of tokens
Open accessAbstract
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GMAI can retrieve relevant context from existing databases, in the form of articles, images or entire previous cases.
“Furthermore, building on recent retrieval-based approaches, GMAI can retrieve relevant context from existing databases, in the form of articles, images or entire previous cases 19,20 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [19]
The Introduction (p. 1) describes Retro retrieving similar text from a large database, and the training-dataset section (p. 3) includes Wikipedia articles among its text sources. The paper does not establish that GMAI can retrieve images or entire previous medical cases; it supports the retrieval-based approach, but not those extensions. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLonger contexts enable moving forward to smart context composition techniques, including fill-in-the-middle and retrieval-augmented generation methods.
“Longer contexts enable us to move forward to”
From Context Composing for Full Line Code Completion (2024) · cites this paper as [2]
The Introduction (p. 1) and Method (p. 2) describe a retrieval-enhanced language model that incorporates retrieved text chunks, supporting the retrieval part of the claim. The paper does not say that longer contexts enable such methods or discuss fill-in-the-middle as a context-composition technique. (LLM, conf=0.85)
Experimental — accuracy not yet measuredModels augmented by retrieving information from large databases outperform larger models trained on larger datasets, effectively leveraging available information and providing added benefits such as interpretability.
“Along these lines, recent studies demonstrated that models augmented by retrieving information from large databases outperform larger models trained on larger datasets, effectively leveraging available information and also providing added benefits such as interpretability 124,125”
From Multimodal biomedical AI (2022) · cites this paper as [125]
The Conclusion (p. 15) says Retro outperforms previous models trained on large-scale datasets, and §3.2 (p. 10) says retrieved neighbours make its outputs more interpretable. However, the performance comparison is dataset-dependent: §4.1 (p. 11) reports that Retro outperforms the much larger Jurassic-1 on a majority of Pile test sets, not that it outperforms larger models generally. (LLM, conf=0.86)
Experimental — accuracy not yet measuredREALM integrates retrieval into language-model pretraining, while RETRO conditions language models on retrieved chunks from very large corpora.
“REALM integrates retrieval into language-model pretraining, while RETRO conditions language models on retrieved chunks from very large corpora [6, 4].”
From Oracle Agent Memory as an Enterprise Memory Substrate for Long-Horizon AI Agents (2026) · cites this paper as [4]
RAG, REALM, and RETRO retrieve from large corpora.
“RAG [7], REALM [10], and RETRO [11] retrieve from large corpora.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [11]
RETRO-like methods integrate a retrieval system into the Transformer model.
“Another promising direction for neural networks-based code completion systems is the RETRO-like methods [2] that integrate a retrieval system into the Transformer model.”
From Context Composing for Full Line Code Completion (2024) · cites this paper as [2]
Retrieving supporting evidence for LLM inference at a fixed time interval is a naive yet effective method.
“One naive yet effective method is to retrieve supporting evidence for LLM inference with a fixed time interval, such as every fixed number of generated tokens[107, 108] or every sentence [109]”
From Foundations of GenIR (2025) · cites this paper as [108]
Users can insert custom local modules after some existing blocks, which could allow use-cases like retrieval-augmented generation.
“Finally, users can insert custom local modules after some of the existing blocks, which could allow use-cases like retrieval-augmented generation (Borgeaud et al., 2021;Lewis et al., 2020).”
From Petals: Collaborative Inference and Fine-tuning of Large Models (2023) · cites this paper as Borgeaud et al., 2021
Reranking can be performed using rule-based methods that depend on predefined metrics like Diversity, Relevance, and MRR, or model-based approaches like Encoder-Decoder models from the BERT series (e.g., SpanBERT), specialized reranking models such as Cohere rerank or bge-raranker-large, and general large language models like GPT.
“Reranking can be performed using rule-based methods that depend on predefined metrics like Diversity, Relevance, and MRR, or model-based approaches like Encoder-Decoder models from the BERT series (e.g., SpanBERT), specialized reranking models such as Cohere rerank or bge-raranker-large, and general large language models like GPT [12], [99].”
From Retrieval-Augmented Generation for Large Language Models: A Survey (2023) · cites this paper as [12]
Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
This paper’s reference list could not be split into single entries.
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“One area is retrieval models that aim to drastically reduce model sizes by embedding large amounts of text the model can have access to later (Guu et al., 2020; Borgeaud et al., 2021).”
BackgroundFoundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
“Furthermore, building on recent retrieval-based approaches, GMAI can retrieve relevant context from existing databases, in the form of articles, images or entire previous cases 19,20 .”
MethodAcosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
“RETRO [Borgeaud et al. , 2022] leverages retrieval augmentation for large-scale pre-training from scratch, achieving a reduction in model parameters while surpassing standard GPT models in terms of perplexity.”
BackgroundMethodTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“Technical approaches to reducing risks of environmental harm include segmenting LMs into less large LMs that search and retrieve information from a distinct data corpus [25, 88, 97, 102, 115].”
BackgroundInfluentialSelf-citationProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
In-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449 · see citing works
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.272 · Cited by 353 · see citing works
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · arXiv · doi:10.48550/arxiv.2212.10509 · Cited by 349 · see citing works
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · see citing works
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340 · see citing works
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
“39 ,”
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · see citing works
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303 · see citing works
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07016 · Cited by 214
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211 · see citing works
“In order to alleviate this issue, retrieval-augmented LMs have been proposed by allowing the model to explicitly access external data [94], [313]–[316].”
BackgroundMethodPredictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · see citing works
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2025) · Science Advances · doi:10.1126/sciadv.adt3813 · Cited by 213 · see citing works
“Some have suggested to use retrieval-augmented LLMs that provide verifiable facts from trusted sources (Lewis et al., 2020; Borgeaud et al., 2022; Ahmed et al., 2023) or let the user provide all relevant facts to the LLM in order to protect scientific literature from accumulating subtle…”
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
“This approach effectively increases the number of data tokens seen during training (by a factor of ∼ 10 in Borgeaud et al. (2021)).”
Self-citationRetrieval-augmented generation for educational application: A systematic survey
Li, Z., Wang, Z., Wang, W., Hung, K., Xie, H., & Wang, F. L. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100417 · Cited by 151 · see citing works
Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review
Neha, F., Bhati, D., & Shukla, D. K. (2025) · AI · doi:10.3390/ai6090226 · Cited by 117 · see citing works
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81 · see citing works
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81 · see citing works
Calibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2024) · Proceedings of the 56th Annual ACM Symposium on Theory of Computing · doi:10.1145/3618260.3649777 · Cited by 91 · see citing works
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73 · see citing works
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“For example, retrieval-augmented methods in NLP (Lewis et al., 2020; Borgeaud et al., 2022; Chen et al., 2017) can be viewed as retrieving from a semantic memory of unstructured text (e.g., Wikipedia).”
BackgroundMethodHagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Han, B., Susnjak, T., & Mathrani, A. (2024) · Applied Sciences · doi:10.3390/app14199103 · Cited by 68 · see citing works
A review of LLMs and their applications in the architecture, engineering and construction industry
Kampelopoulos, D., Tsanousa, A., Vrochidis, S., & Kompatsiaris, I. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11241-7 · Cited by 69 · see citing works
MemGPT: Towards LLMs as Operating Systems
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2310.08560 · Cited by 64 · see citing works
“…generated response against the ‘gold response’ using ROUGE-L scores (Lin, 2004) and an ‘LLM judge’, which is instructed to evaluate whether or not the generated response is consistent with the gold response (GPT-4 has been shown to have high agreement with human evaluators (Zheng et al., 2023)).”
MethodCRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
“[90] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.”
BackgroundTraining language models with memory augmentation
Zhong, Z., Lei, T., & Chen, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.382 · Cited by 60 · see citing works
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.232 · Cited by 59 · see citing works
AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?
Lohn, A., & Musser, M. (2022) · Center for Security and Emerging Technology · doi:10.51593/2021ca009 · Cited by 33 · see citing works
Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.890 · Cited by 31 · see citing works
Gupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanan, V., Kumaraguru, P., & Sheth, A. (2022) · Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology · doi:10.18653/v1/2022.clpsych-1.12 · Cited by 27 · see citing works
A-Mem: Agentic Memory for LLM Agents
Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., & Zhang, Y. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-0593 · Cited by 51 · see citing works
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2023) · arXiv · doi:10.48550/arxiv.2309.03883 · Cited by 29 · see citing works
“3) Not grounding on external knowledge: Our method relies on the model’s internal knowledge without using external retrieval modules (Izacard et al., 2022; Borgeaud et al., 2022; Ram et al., 2023).”
MethodDSI++: Updating Transformer Memory with New Documents
Mehta, S., Gupta, J., Tay, Y., Dehghani, M., Tran, V., Rao, J., Najork, M., Strubell, E., & Metzler, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.510 · Cited by 21 · see citing works
Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions
Cao, S., Zhang, J., Shi, J., Lv, X., Yao, Z., Tian, Q., Hou, L., & Li, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.835 · Cited by 17 · see citing works
Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In
Yu, Z., Xiong, C., Yu, S., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.17331 · Cited by 17 · see citing works
Query Rewriting for Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2305.14283 · Cited by 14 · see citing works
EvoR: Evolving Retrieval for Code Generation
Su, H., Jiang, S., Lai, Y., Wu, H., Shi, B., Liu, C., Liu, Q., & Yu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.143 · Cited by 12 · see citing works
“The retrieval-augmented generation (RAG) paradigm has raised significant attention due to its efficiency in adapting large language models (LLMs) without training (Guu et al., 2020; Karpukhin et al., 2020; Izacard et al., 2023; Borgeaud et al., 2022; Asai et al., 2023).”
Deanthropomorphising NLP: Can a language model be conscious?
Shardlow, M., & Przybyła, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307521 · Cited by 16 · see citing works
Beamaggr: Beam aggregation reasoning over multi-source knowledge for multi-hop question answering
Chu, Z., Chen, J., Chen, Q., Wang, H., Zhu, K., Du, X., Yu, W., Liu, M., & Qin, B. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.67 · Cited by 11 · see citing works
ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8 · see citing works
“This method involves integrating LLMs with extensive databases to enhance their reasoning capabilities [6].”
MethodCoelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48 · see citing works
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
“Other works retrieve individual input tokens or text segments and add them to the LM input (Guu et al., 2020; Borgeaud et al., 2022).”
BackgroundAgentic Context Engineering: Evolving Contexts for Self-Improving Language Models
Zhang, Q., Hu, C., Upasani, S., Ma, B., Hong, F., Kamanuru, V., Rainton, J., Wu, C., Ji, M., Li, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.04618 · Cited by 3 · see citing works
“Contexts are inter-pretable and explainable for users and developers (Wei et al., 2022; Wang et al., 2022), allow rapid integration of new knowledge at runtime (Lewis et al., 2020; Borgeaud et al., 2022), and can be shared across models or modules in a compound system (Khot et al., 2022).”
Wang, B., Gong, X., Wang, Y., Xu, R., & Huang, H. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3497185 · Cited by 3 · see citing works
Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations
Tilwani, D., Saxena, Y., Mohammadi, S., Padia, A., Raff, E., Sheth, A., Parthasarathy, S., & Gaur, M. (2024) · arXiv · doi:10.48550/arxiv.2405.02228 · Cited by 2 · see citing works
“RETRO [17] retrieves text chunks during pretraining to ground generation in external knowledge, while retrieval-augmented generation (RAG) [9] combines dense retrieval with sequence-to-sequence models for knowledge-intensive tasks.”
Context Composing for Full Line Code Completion
Semenkin, A., Sokolov, Y., & Vu, E. (2024) · arXiv (Cornell University) · doi:10.1145/3643796.3648446 · Cited by 3 · see citing works
“Longer contexts enable us to move forward to “smart” context composition techniques, which include fill-in-the-middle [3] and retrieval-augmented generation [2] methods.”
BackgroundMethodDo as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1 · see citing works
Exploring Optimal Prefetching Sizes in RaLMSpec to Enhance Retrieval-Augmented Generation Efficiency
Liu, S. (2025) · Highlights in Science, Engineering and Technology · doi:10.54097/hhff9g78
“Furthermore, Borgeaud et al. investigated large-scale retrieval in massive datasets and highlighted caching benefits, underscoring the lack of exploration on how prefetching size affects retrieval speed and cache efficiency [10].”
Oracle Agent Memory as an Enterprise Memory Substrate for Long-Horizon AI Agents
Alake, R., Bernardis, C., Cayet, P., Engel, L., Hilloulin, D., Hong, S., Hosler, A., Kavantzas, N., Kossyk, I., Le, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.13157
“REALM integrates retrieval into language-model pretraining, while RETRO conditions language models on retrieved chunks from very large corpora [6, 4].”
Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
Du, P. (2026) · arXiv · Cited by 1 · see citing works
“RAG [Lewis et al., 2020] married a pre-trained generator with a dense document retriever, and RETRO [Borgeaud et al., 2022] showed that pulling from a trillion -token corpus at inference time could match much larger models at a fraction of the parameter count.”
Gopal, A. M., Pirbadian, A., Carlson, K. D., Lewis, M. A., & Tapson, J. (2026) · arXiv
“RAG [12], Fusion-in-Decoder [13], RETRO [14], Atlas [15], and REPLUG [16] all condition generation on retrieved text but require in-context ingestion at inference. xRAG [17] compresses documents to reduce token cost but still ingests a compressed representation per query.”
When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents
Hu, Q., Long, Q., & Wang, W. (2026) · arXiv · doi:10.48550/arxiv.2604.27003
“A growing body of work equips LLM agents with external memory, from retrieval-augmented language models (Borgeaud et al., 2022; Wang et al., 2023b) to agent systems that store and reuse experience across interactions (Zhong et al., 2024; Shinn et al., 2023; Wang et al., 2023a; Packer et al., 2023;…”
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
“One naive yet effective method is to retrieve supporting evidence for LLM inference with a fixed time interval, such as every fixed number of generated tokens[107, 108] or every sentence [109].”
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008
“…retrieval augmented generation (RAG) is an effective framework that retrieves relevant information from external knowledge sources to aid and ground language models’ generation (Lewis et al., 2020; Khandelwal et al., 2020; Borgeaud et al., 2021; Izacard and Grave, 2021; Izacard et al., 2022b).”
Continual Learning, Not Training: Online Adaptation For Agents
Jaglan, A., & Barnes, J. (2025) · arXiv · doi:10.48550/arxiv.2511.01093
“…2023; Agrawal et al., 2025), (3) retrieval-augmented systems that perform lookup rather than skill synthesis (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022; Asai et al., 2024; Lin et al., 2024), and (4) agent memory mechanisms that passively store experiences without extracting…”
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · arXiv · Cited by 59 · see citing works
Selective Forgetting: A Graph-Based Memory Framework for Long-Term LLM Agents
Rusu, T., Khanzadeh, S., & Alalfi, M. (2026) · arXiv · doi:10.48550/arxiv.2608.28978
“Dense retrieval Karpukhin et al. [2020] and jointly pre-trained retrieval-reading models such as REALM Guu et al. [2020] and RETRO Borgeaud et al. [2022] improved retrieval quality on scale, while retrieval has been shown to reduce hallucinations in dialogue Shuster et al. [2021].”
OmniRouter: Budget and Performance Controllable Multi-LLM Routing
Mei, K., Xu, W., Guo, M., Lin, S., & Zhang, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.20576 · Cited by 2 · see citing works
The AI Hippocampus: How Far are We From Human Memory?
Jia, Z., Li, J., Kang, Y., Wang, Y., Wu, T., Wang, Q., Wang, X., Zhang, S., Shen, J., Li, Q., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.09113
Kummara, H. (2026) · Cambridge Open Engage (preprint) · doi:10.33774/coe-2026-xpkvb
Synapse: Empowering LLM Agents with Episodic-Semantic Memory via Spreading Activation
Jiang, H., Chen, J., Pan, Y., Chen, L., You, W., Zhou, Y., Zhang, R., Abate, Y., & Liu, T. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1108
Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering
Chu, Z., Fan, H., Chen, J., Wang, Q., Yang, M., Liang, J., Wang, Z., Li, H., Tang, G., Liu, M., et al. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.123
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Suzgun, M., Yuksekgonul, M., Bianchi, F., Jurafsky, D., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.07952
GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction
Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2023) · doi:10.1101/2023.10.10.561776 · Cited by 118 · see citing works
“In contrast, our present work focuses on the explicit modeling of related sequences (known as retrieval augmentation in natural language processing [39]).”
- Authors
- Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., Casas, D. D. L., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., & Sifre, L.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2112.04426
- OpenAlex
- W4226082499
- Cited by
- 382
- References
- 1
- Citations by year
- 2026: 7 · 2025: 86 · 2024: 113 · 2023: 141 · 2022: 35
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- retrieval-augmented language models, retro style, cross-attention, declarative memory, parameter-efficient fine-tuning, retrofitting, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
