CoChat
Sign Up
Cited by 827Topic ModelingInformation Retrieval and Search BehaviorMultimodal Machine Learning Applications

A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

Open access
Fan, Wenqi, Ding, Yujuan, Ning, Liangbo, Wang, Shijie, Li, Hengyun, Yin, Dawei, Chua, Tat-Seng, Li, Qing·2024·Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
DOI 10.1145/3637528.3671470
Open this paper in CoChat — free accountOpen PDF

Abstract

As one of the most advanced techniques in AI, Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge, providing huge convenience for numerous tasks. Particularly in the era of AI-Generated Content (AIGC), the powerful capacity of retrieval in providing additional knowledge enables RAG to assist existing generative AI in producing high-quality outputs. Recently, Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation, while still facing inherent limitations such as hallucinations and out-of-date internal knowledge. Given the powerful abilities of RAG in providing the latest and helpful auxiliary information, Retrieval-Augmented Large Language Models (RA-LLMs) have emerged to harness external and authoritative knowledge bases, rather than solely relying on the model's internal knowledge, to augment the quality of the generated content of LLMs. In this survey, we comprehensively review existing research studies in RA-LLMs, covering three primary technical perspectives: Furthermore, to deliver deeper insights, we discuss current limitations and several promising directions for future research. Updated information about this survey can be found at: https://advanced-recommender-systems.github.io/RAG-Meets-LLMs/

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3637528.3671470

Cites

102 of 102 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modular RAG [28] and related surveys [24, 35, 86, 126] focus on conceptual overviews and lack implementation or evaluation support for GraphRAG.

    “However, Modular RAG [28] and related surveys [24, 35, 86, 126] focus on conceptual overviews and lack implementation or evaluation support for GraphRAG.”

    From LEGO-GraphRAG: Modularizing Graph-Based Retrieval-Augmented Generation for Design Space Exploration (2025) · cites this paper as [24]

    The Introduction and Conclusion describe this paper as a survey of RAG architectures, training strategies, and applications, supporting its identification as an overview. The cited passages do not establish the broader claim that the surveys lack implementation or evaluation support for GraphRAG; searches for GraphRAG and graph-specific implementation or evaluation found no passage addressing that point. No page numbers were provided in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The lack of a unified categorization for systematically analyzing GraphRAG techniques hinders effective research summarization and identification of promising candidates.

    “The lack of a unified categorization for systematically analyzing these techniques-despite prior efforts such as Modular RAG [28] and other surveys [24, 35, 86, 126] on LLMs with graphs-hinders effective research summarization and the identification of promising candidates for further study.”

    From LEGO-GraphRAG: Modularizing Graph-Based Retrieval-Augmented Generation for Design Space Exploration (2025) · cites this paper as [24]

    The Conclusion says rapid advances in retrieval-augmented LLMs create a need for a comprehensive, systematic overview, and the Introduction describes the survey's organization by architecture, training, and application. Neither passage establishes a lack of unified categorization specifically for GraphRAG techniques or says that this lack hinders research summarization and identification of promising candidates. The paper surveys retrieval-augmented LLMs broadly, rather than LLMs with graphs specifically. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • RAG combines parametric knowledge stored in model parameters with non-parametric information retrieved from external sources, enabling access to current, domain-specific knowledge while maintaining parameter efficiency.

    “RAG combines parametric knowledge stored in model parameters with non-parametric information retrieved from external sources, enabling access to current, domain-specific knowledge while maintaining parameter efficiency [597, 315, 257].”

    From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [257]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. [5]5. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In ICLR.
  2. [6]6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bog- dan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In ICML. 2206-2240.
  3. [7]7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS.
  4. [8]8. Stefan Buttcher, Charles LA Clarke, and Gordon V Cormack. 2016. Information retrieval: Implementing and evaluating search engines. Mit Press.
  5. [16]16. Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation. In EMNLP. 12318-12337.
  6. [17]17. Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2024. Lift yourself up: Retrieval-augmented text generation with self-memory. In NeurIPS.
  7. [20]20. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. arXiv:2401.01301 (2024). · arXiv
  8. [21]21. Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William W. Cohen. 2022. Mention Memory: incorporating textual knowledge into Trans- formers through entity mention attention. In ICLR.
  9. [24]24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1). 4171-4186.
  10. [30]30. Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. In ICLR.
  11. [37]37. Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. 2023. Recommender systems in the era of large language models (llms). arXiv:2307.02046 (2023). · arXiv
  12. [38]38. Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as Experts: Sparse Memory Access with Entity Supervision. In EMNLP. 4937-4951.
  13. [42]42. Michael R. Glass, Gaetano Rossiello, Md. Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, Rerank, Generate. In NAACL-HLT. 2701-2715.
  14. [43]43. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017. Improving Neural Language Models with a Continuous Cache. In ICLR.
  15. [44]44. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In ICML. 3929-3938.
  16. [47]47. Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2023. Reveal: Retrieval- augmented visual-language pre-training with multi-source multimodal knowl- edge memory. In CVPR. 23369-23379.
  17. [49]49. Gautier Izacard and Edouard Grave. 2021. Distilling Knowledge from Reader to Retriever for Question Answering. In ICLR.
  18. [53]53. Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and Multi-cultural Figurative Language Understanding. In ACL.
  19. [56]56. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In ICLR.
  20. [58]58. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In SIGIR. 39-48.
  21. [59]59. Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. 2023. Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models. In EMNLP.
  22. [62]62. Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2022. Copy is All You Need. In ICLR.
  23. [63]63. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In ICML. 19274-19286.
  24. [64]64. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. In NeurIPS.
  25. [65]65. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. 7871-7880.
  26. [66]66. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS. 9459-9474.
  27. [67]67. Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. 2024. SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models. In NeurIPS.
  28. [70]70. Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yu Gu, Zhiyuan Liu, and Ge Yu. 2023. Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data. In ACL.
  29. [71]71. Xiaoqian Li, Ercong Nie, and Sheng Liang. 2023. From Classification to Gen- eration: Insights into Crosslingual Retrieval Augmented ICL. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
  30. [73]73. Zonglin Li, Ruiqi Guo, and Sanjiv Kumar. 2022. Decoupled context processing for context augmented language modeling. In NeurIPS. 21698-21710.
  31. [75]75. Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. RA-DIT: Retrieval-Augmented Dual Instruction Tuning. In ICLR.
  32. [76]76. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Does Gender Matter? Towards Fairness in Dialogue Systems. In ACL.
  33. [80]80. Alejandro Lozano, Scott L Fleming, Chia-Chun Chiang, and Nigam Shah. 2023. Clinfo. ai: An open-source retrieval-augmented large language model system for answering medical questions using scientific literature. In PACIFIC SYMPOSIUM ON BIOCOMPUTING 2024. 8-23.
  34. [81]81. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning. In ICLR.
  35. [84]84. Chang Ma, Haiteng Zhao, Lin Zheng, Jiayi Xin, Qintong Li, Lijun Wu, Zhi- hong Deng, Yang Lu, Qi Liu, and Lingpeng Kong. 2023. Retrieved Sequence Augmentation for Protein Representation Learning. bioRxiv (2023), 2023-02.
  36. [88]88. Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Ha- jishirzi, and Luke Zettlemoyer. 2023. Nonparametric Masked Language Model- ing. In ACL (Findings). 2097-2118.
  37. [90]90. Neil O'Hare, Paloma De Juan, Rossano Schifanella, Yunlong He, Dawei Yin, and Yi Chang. 2016. Leveraging user interaction signals for web image search. In SIGIR. 559-568.
  38. [91]91. Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval Augmented Code Generation and Summa- rization. In EMNLP (Findings). 2719-2734.
  39. [93]93. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In ICLR.
  40. [95]95. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML. 8748-8763.
  41. [96]96. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  42. [98]98. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented lan- guage models. Trans. Assoc. Comput. Linguist. 11 (2023), 1316-1331.
  43. [100]100. Parikshit Ram and Alexander G Gray. 2012. Maximum inner-product search using cone trees. In KDD. 931-939.
  44. [101]101. Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, Vol. 242. Citeseer, 29-48.
  45. [103]103. Benjamin Z. Reichman and Larry Heck. 2024. Retrieval-Augmented Genera- tion: Is Dense Passage Retrieval Retrieving? https://arxiv.org/html/2402.11035v1 (2024).
  46. [105]105. Yubing Ren, Yanan Cao, Ping Guo, Fang Fang, Wei Ma, and Zheng Lin. 2023. Retrieve-and-sample: Document-level event argument extraction via hybrid retrieval augmentation. In ACL. 293-306.
  47. [109]109. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. In NeurIPS.
  48. [113]113. Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2022. XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Se- mantic Parsing. In EMNLP (Findings). 5248-5259.
  49. [122]122. Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2024. Spectr: Fast speculative decoding via optimal transport. In NeurIPS.
  50. [128]128. Liang Wang, Nan Yang, and Furu Wei. 2024. Learning to Retrieve In-Context Examples for Large Language Models. In EACL. 1752-1767.
  51. [131]131. Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard G. Baraniuk, and Anima Anandkumar. 2023. Retrieval-based Controllable Molecule Generation. In ICLR.
  52. [133]133. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. In NeurIPS. 24824-24837.
  53. [135]135. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettle- moyer. 2020. Scalable Zero-shot Entity Linking with Dense Entity Retrieval. In EMNLP. 6397-6407.
  54. [136]136. Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. Memorizing Transformers. In ICLR.
  55. [137]137. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. RECOMP: Improving retrieval- augmented LMs with context compression and selective augmentation. In ICLR.
  56. [138]138. Jitao Xu, Josep-Maria Crego, and Jean Senellart. 2020. Boosting neural machine translation with similar translations. In ACL. 1570-1579.
  57. [139]139. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond Goldfish Memory: Long-Term Open-Domain Conversation. In ACL (1). 5180-5197.
  58. [141]141. Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In ICML. 39818-39833.
  59. [145]145. Dani Yogatama, Cyprien de Masson d'Autume, and Lingpeng Kong. 2021. Adap- tive semiparametric language models. TACL 9 (2021), 362-373.
  60. [147]147. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than Retrieve: Large Language Models are Strong Context Generators. In ICLR.
  61. [160]160. Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs. In ICLR.

Cited by

Sorted by citations

Related

Authors
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q.
Affiliations
  • Hong Kong Polytechnic University (HK)
  • Baidu (China) (CN)
  • National University of Singapore (SG)
ORCID
Venue
Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher
ACM
Published
2024-08-24
Pages
6491-6501
DOI
10.1145/3637528.3671470
OpenAlex
W4401857375
Semantic Scholar
eb9c4a07a336e8deefe7b399c550d3af0241238e
Cited by
827
References
102
Citations by year
2026: 417 · 2025: 370 · 2024: 40
FWCI
244.05
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • General Research Funds from the Hong Kong Research Grants Council (PolyU 15200021, 15207322, and 15200023)
  • internal research funds from The Hong Kong Polytechnic University (P0036200, P0042693, P0048625, P0048752, and P0051361)
  • Research Collaborative Project (P0041282)
  • National Natural Science Foundation of China (62102335)
  • SHTM Interdisciplinary Large Grant (P0043302)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
retrieval-augmented generation, large language models, external knowledge retrieval, hallucination mitigation, AI-generated content, knowledge base, generative AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review