Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 99

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

Lyu, Yuanjie, Li, Zhiyu, Niu, Simin, Xiong, Feiyu, Tang, Bo, Wang, Wenjin, Wu, Hao, Liu, Huanyong, Xu, Tong, Chen, Enhong·2025·ACM Transactions on Information Systems
DOI 10.1145/3701228
Open this paper in CoChat — free account

Abstract

Retrieval-augmented generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate “hallucinated” content. However, evaluating RAG systems is a challenge. Most benchmarks focus primarily on question-answering applications, neglecting other potential scenarios where RAG could be beneficial. Accordingly, in the experiments, these benchmarks often assess only the LLM components of the RAG pipeline or the retriever in knowledge-intensive scenarios, overlooking the impact of external knowledge base construction and the retrieval component on the entire RAG pipeline in non-knowledge-intensive scenarios. To address these issues, this article constructs a large-scale and more comprehensive benchmark and evaluates all the components of RAG systems in various RAG application scenarios. Specifically, we refer to the CRUD actions that describe interactions between users and knowledge bases and also categorize the range of RAG applications into four distinct types—create, read, update, and delete (CRUD). “Create” refers to scenarios requiring the generation of original, varied content. “Read” involves responding to intricate questions in knowledge-intensive situations. “Update” focuses on revising and rectifying inaccuracies or inconsistencies in pre-existing texts. “Delete” pertains to the task of summarizing extensive texts into more concise forms. For each of these CRUD categories, we have developed different datasets to evaluate the performance of RAG systems. We also analyze the effects of various components of the RAG system, such as the retriever, context length, knowledge base construction, and LLM. Finally, we provide useful insights for optimizing the RAG technology for different scenarios. The source code is available at GitHub: https://github.com/IAAR-Shanghai/CRUD_RAG .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3701228

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 47

  1. 10.1145/3331184.3331265 · DOI
  2. Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts (ACL ’23). 41–46.
  3. 10.1145/3539618.3591923 · DOI
  4. Sébastien Bubeck Varun Chandrasekaran Ronen Eldan Johannes Gehrke Eric Horvitz Ece Kamar Peter Lee Yin Tat Lee Yuanzhi Li Scott Lundberg et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712. Retrieved from https://arxiv.org/abs/2303.12712 · arXiv
  5. Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP ’20). 6251–6258.
  6. Shuyang Cao and Lu Wang. 2024. Verifiable generation with subsentence-level fine-grained citations. arXiv:2406.06125. · arXiv
  7. 10.1145/3583780.3614905 · DOI
  8. Xin Cheng Di Luo Xiuying Chen Lemao Liu Dongyan Zhao and Rui Yan. 2023. Lift Yourself Up: Retrieval-augmented text generation with self memory. arXiv:2305.02437. Retrieved from https://arxiv.org/abs/2305.02437 · arXiv
  9. Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2023. Promptagator: Few-shot dense retrieval from 8 examples. In Proceedings of the 11th International Conference on Learning Representations (ICLR ’23).
  10. Shahul Es Jithin James Luis Espinosa-Anke and Steven Schockaert. 2023. RAGAs: Automated evaluation of retrieval augmented generation. arXiv:2309.15217. Retrieved from https://arxiv.org/abs/2309.15217 · arXiv
  11. 10.1145/2766462.2767780 · DOI
  12. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL ’23), 1762–1777.
  13. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP ’23), 6465–6488.
  14. Hangfeng He Hongming Zhang and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv:2301.00303. Retrieved from https://arxiv.org/abs/2301.00303 · arXiv
  15. Gautier Izacard Patrick S. H. Lewis Maria Lomeli Lucas Hosseini Fabio Petroni Timo Schick Jane Dwivedi-Yu Armand Joulin Sebastian Riedel and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv:2208.03299. Retrieved from https://arxiv.org/abs/2208.03299 · arXiv
  16. Bin Ji, Huijun Liu, Mingzhe Du, and See-Kiong Ng. 2024. Chain-of-thought improves text generation with citations in large language models. In Proceedings of 38th AAAI Conference on Artificial Intelligence (AAAI ’24), 36th Conference on Innovative Applications of Artificial Intelligence (IAAI ’24), 14th Symposium on Educational Advances in Artificial Intelligence (EAAI ’14), 18345–18353.
  17. Haim Kilov. 1990. From semantic to object-oriented data modeling. In Proceedings of the 1st International Conference on Systems Integration (Systems Integration ’90). IEEE, 385–393.
  18. 10.5555/3495724.3496517 · doi:10.5555/3495724.3496517
  19. Weitao Li Junkai Li Weizhi Ma and Yang Liu. 2024. Citation-enhanced generation for LLM-based chatbots. arXiv:2402.16063. Retrieved from https://arxiv.org/abs/2402.16063 · arXiv
  20. Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. Are ChatGPT and GPT-4 general-purpose solvers for financial text analytics? A study on several typical tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, 408–422.
  21. Xun Liang Shichao Song Simin Niu Zhiyu Li Feiyu Xiong Bo Tang Zhaohui Wy Dawei He Peng Cheng Zhonghao Wang et al. 2023. UHGEval: Benchmarking the hallucination of Chinese large language models via unconstrained generation. arXiv:2311.15296. Retrieved from https://arxiv.org/abs/2311.15296 · arXiv
  22. Qi Liu, Gang Guo, Jiaxin Mao, Zhicheng Dou, Ji-Rong Wen, Hao Jiang, Xinyu Zhang, and Zhao Cao. 2024. An analysis on matching mechanisms and token pruning for late-interaction models. ACM Transactions on Information Systems 42, 5 (2024), 118:1–118:28.
  23. Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. arXiv:1905.13164. Retrieved from https://arxiv.org/abs/1905.13164 · DOI · Paper page
  24. 10.1145/3511808.3557319 · DOI
  25. Xinbei Ma Yeyun Gong Pengcheng He Hai Zhao and Nan Duan. 2023. Query rewriting for retrieval-augmented large language models. arXiv:2305.14283. Retrieved from https://arxiv.org/abs/2305.14283 · arXiv
  26. Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples!. In Findings of the Association for Computational Linguistics (EMNLP ’23), 10572–10601.
  27. Fabio Petroni Aleksandra Piktus Angela Fan Patrick Lewis Majid Yazdani Nicola De Cao James Thorne Yacine Jernite Vladimir Karpukhin Jean Maillard et al. 2020. KILT: A benchmark for knowledge intensive language tasks. arXiv:2009.02252. Retrieved from https://arxiv.org/abs/2009.02252 · DOI · Paper page
  28. 10.1145/3397271.3401110 · DOI
  29. Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT ’21), 1172–1183.
  30. Thomas Scialom Paul-Alexis Dray Patrick Gallinari Sylvain Lamprier Benjamin Piwowarski Jacopo Staiano and Alex Wang. 2021. QuestEval: Summarization asks for fact-based evaluation. arXiv:2103.12693. Retrieved from https://arxiv.org/abs/2103.12693 · DOI · Paper page
  31. Xinyue Shen Zeyuan Chen Michael Backes and Yang Zhang. 2023. In ChatGPT we trust? Measuring and characterizing the reliability of ChatGPT. arXiv:2304.08979. Retrieved from https://arxiv.org/abs/2304.08979 · arXiv
  32. Weijia Shi Sewon Min Michihiro Yasunaga Minjoon Seo Rich James Mike Lewis Luke Zettlemoyer and Wen-tau Yih. 2023. REPLUG: Retrieval-augmented black-box language models. arXiv:2301.12652. Retrieved from https://arxiv.org/abs/2301.12652 · arXiv
  33. Ciprian-Octavian Truica, Florin Radulescu, Alexandru Boicea, and Ion Bucur. 2015. Performance evaluation for CRUD operations in asynchronously replicated document oriented database. In Proceedings of the 20th International Conference on Control Systems and Computer Science. IEEE, 191–196.
  34. 10.1007/s11704-023-2689-5 · DOI
  35. Yilin Wen Zifeng Wang and Jimeng Sun. 2023. MindMap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv:2308.09729. Retrieved from https://arxiv.org/abs/2308.09729 · arXiv
  36. Derong Xu Wei Chen Wenjun Peng Chao Zhang Tong Xu Xiangyu Zhao Xian Wu Yefeng Zheng and Enhong Chen. 2023. Large language models for generative information extraction: A survey. Frontiers of Computer Science. Retrieved from https://journal.hep.com.cn/fcs/EN/10.1007/s11704-024-40555-y · DOI · Paper page
  37. Fangyuan Xu Weijia Shi and Eunsol Choi. 2023. RECOMP: Improving retrieval-augmented LMs with compression and selective augmentation. arXiv:2310.04408. Retrieved from https://arxiv.org/abs/2310.04408 · arXiv
  38. Yilong Xu Jinhua Gao Xiaoming Yu Baolong Bi Huawei Shen and Xueqi Cheng. 2024. ALiiCE: Evaluating positional fine-grained citation generation. arXiv:2406.13375. Retrieved from https://arxiv.org/abs/2406.13375 · arXiv
  39. Haoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang, Ning Cheng, Ming Li, and Jing Xiao. 2023. PRCA: Fitting black-box large language models for retrieval question answering via pluggable reward-driven contextual adapter. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP ’23). Association for Computational Linguistics, 5364–5375. · DOI · Paper page
  40. Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential reasoning learning for language representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP ’20), 7170–7186.
  41. 10.1145/3397271.3401323 · DOI
  42. Wenhao Yu Hongming Zhang Xiaoman Pan Kaixin Ma Hongwei Wang and Dong Yu. 2023. Chain-of-Note: Enhancing robustness in retrieval-augmented language models. arXiv:2311.09210. Retrieved from https://arxiv.org/abs/2311.09210 · arXiv
  43. 10.1145/984321.984322 · DOI · Paper page
  44. Chao Zhang, Shiwei Wu, Haoxin Zhang, Tong Xu, Yan Gao, Yao Hu, and Enhong Chen. 2024. NoteLLM: A retrievable large language model for note recommendation. In Companion Proceedings of the ACM on Web Conference 2024, 170–179. · DOI
  45. Peitian Zhang Shitao Xiao Zheng Liu Zhicheng Dou and Jian-Yun Nie. 2023. Retrieve anything to augment large language models. arXiv:2310.07554. Retrieved from https://arxiv.org/abs/2310.07554 · arXiv
  46. Weijia Zhang Mohammad Aliannejadi Yifei Yuan Jiahuan Pei Jia-Hong Huang and Evangelos Kanoulas. 2024. Towards fine-grained citation evaluation in generated text: A comparative analysis of faithfulness metrics. arXiv:2406.15264. Retrieved from https://arxiv.org/abs/2406.15264 · arXiv
  47. 10.1145/3624918.3625329 · DOI

Cited by

Metadata

Authors
Lyu, Y., Li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., & Chen, E.
Affiliations
  • University of Science and Technology of China, Hefei, China
  • Institute for Advanced Algorithms Research (Shanghai), Shanghai, China
  • Renmin University of China, Beijing, China
  • 360 AI Research Institute, Beijing, China
ORCID
Venue
ACM Transactions on Information Systems
Publisher
Association for Computing Machinery (ACM)
Published
2025-01-21
Volume
43
Issue
2
Pages
1-32
ISSN
  • 1046-8188
  • 1558-2868
DOI
10.1145/3701228
Cited by
99
References
15
Language
en
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Funders
  • National Science and Technology Major Project (2023ZD0121104)
  • National Natural Science Foundation of China (62222213, 62072423)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.