Cited by 50Topic ModelingInformation Retrieval and Search BehaviorMultimodal Machine Learning Applications

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

Open access
Li, Zhuowan, Li, Cheng, Zhang, Mingyang, Mei, Qiaozhu, Bendersky, Michael·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track

Abstract

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts.However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly.We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both.We benchmark RAG and LC across various public datasets using three latest LLMs.Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance.However, RAG's significantly lower cost remains a distinct advantage.Based on this observation, we propose SELF-ROUTE, a simple yet effective method that routes queries to RAG or LC based on model self-reflection.SELF-ROUTE significantly reduces the computation cost while maintaining a comparable performance to LC.Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-industry.66

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RAG primarily relies on surface-level lexical similarity between snippets and often fails to capture code segments with deeper semantic or functional dependencies-particularly when such relationships are implicit, abstracted, or span multiple components.

    “While RAG effectively reduces context length, it primarily relies on surface-level lexical similarity between snippets. Consequently, it often fails to capture code segments with deeper semantic or functional dependencies-particularly when such relationships are implicit, abstracted, or span multiple components.”

    From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [28]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured
  • Hybrid approaches achieve cost-effective performance through intelligent query routing.

    “while hybrid approaches achieve cost-effective performance through intelligent query routing [17].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 42

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical re- port. arXiv preprint arXiv:2303.08774. · arXiv
  2. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088. · arXiv
  3. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to re- trieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511. · arXiv
  4. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508. · arXiv
  5. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. · arXiv
  6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
  7. Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610. · arXiv
  8. Lingjiao Chen, Matei Zaharia, and James Zou. 2023a. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. · arXiv
  9. Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, and Xipeng Qiu. 2024. Unified active retrieval for retrieval augmented generation. arXiv preprint arXiv:2406.12534. · arXiv
  10. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Co- han, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and an- swers anchored in research papers. arXiv preprint arXiv:2105.03011. · arXiv
  11. Mandy Guo, Joshua Ainslie, David C Uthus, Santi- ago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yin- fei Yang. 2022. Longt5: Efficient text-to-text trans- former for long sequences. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 724-736.
  12. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pa- supat, and Mingwei Chang. 2020. Retrieval aug- mented language model pre-training. In Interna- tional conference on machine learning, pages 3929- 3938. PMLR.
  13. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reason- ing steps. arXiv preprint arXiv:2011.01060. · arXiv
  14. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan- tanu Acharya, Dima Rekesh, Fei Jia, and Boris Gins- burg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654. · arXiv
  15. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301. · arXiv
  16. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning. arXiv preprint arXiv:2112.09118. · arXiv
  17. Gautier Izacard and Edouard Grave. 2020. Lever- aging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. · arXiv
  18. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large lan- guage models through question complexity. In Pro- ceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Papers), pages 7029-7043.
  19. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839. · arXiv
  20. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172. · arXiv
  21. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.
  22. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. In search of needles in a 10m haystack: Recur- rent memory finds what llms miss. arXiv preprint arXiv:2402.10790. · arXiv
  23. Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023. Lla- trieval: Llm-verified retrieval for verifiable genera- tion. arXiv preprint arXiv:2311.07838. · arXiv
  24. Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452. · arXiv
  25. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Asso- ciation for Computational Linguistics, 12:157-173.
  26. Yuanhua Lv and ChengXiang Zhai. 2009. Adaptive relevance feedback in information retrieval. In Pro- ceedings of the 18th ACM conference on Informa- tion and knowledge management, pages 255-264.
  27. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval- augmented large language models. arXiv preprint arXiv:2305.14283. · arXiv
  28. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conver- sational memory of llm agents. arXiv preprint arXiv:2402.17753. OpenAI. 2023. Gpt-3.5-turbo. https://platform. openai.com/docs/models/gpt-3-5-turbo. · arXiv
  29. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. · arXiv
  30. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Stan- dardized comparison over long language sequences. arXiv preprint arXiv:2201.03533. · arXiv
  31. Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301.12652. · arXiv
  32. Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802. · arXiv
  33. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Mul- tihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554.
  34. Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023. Self-knowledge guided retrieval augmentation for large language models. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 10303-10315.
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824- 24837.
  36. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subra- manian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025. · arXiv
  37. Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented gener- ation. arXiv preprint arXiv:2401.15884. · arXiv
  38. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. arXiv preprint arXiv:1809.09600. · arXiv
  39. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024. Lv-eval: A balanced long-context benchmark with 5 length lev- els up to 256k. arXiv preprint arXiv:2402.05136. · arXiv
  40. Chengxiang Zhai and John Lafferty. 2001. Model- based feedback in the language modeling approach to information retrieval. In Proceedings of the tenth international conference on Information and knowl- edge management, pages 403-410.
  41. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zi- hang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024. Infinity bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718. · arXiv
  42. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi- domain meeting summarization. arXiv preprint arXiv:2104.05938. · arXiv

Cited by

Sorted by citations

Authors
Li, Z., Li, C., Zhang, M., Mei, Q., & Bendersky, M.
Affiliations
University of Michigan (US)
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
Published
2024
Pages
881-893
OpenAlex
W4404783788
Cited by
49
References
0
Citations by year
2026: 28 · 2025: 21
FWCI
14.46
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
retrieval-augmented generation, long-context large language models, hybrid retrieval-augmented generation, query routing, computation cost reduction, long-context understanding, LLM benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review