Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 173

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.

Evaluation of Retrieval-Augmented Generation: A Survey

Open access
Yu, Hao, Gan, Aoran, Zhang, Kai, Tong, Shiwei, Liu, Qi, Liu, Zhaofeng · 2025 · Communications in Computer and Information Science

Abstract

Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-981-96-1024-2_8

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 30 references

Research works printed but not linked: 41

  1. Balaguer, A., et al.: RAG vs fine-tuning: pipelines, tradeoffs, and a case study on agriculture. Technical report (2024). arXiv:2401.08406 [cs] type: article · arXiv
  2. Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M.: Seven failure points when engineering a retrieval augmented generation system (2024). https://doi.org/10.48550/ARXIV.2401.05856 · DOI
  3. Deng, Y., Zhang, W., Chen, Z., Gu, Q.: Rephrase and respond: let large language models ask better questions for themselves (2023). https://doi.org/10.48550/ARXIV.2311.04205 · DOI · Paper page
  4. Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of Wikipedia: knowledge-powered conversational agents. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)
  5. Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D.: MRQA 2019 shared task: evaluating generalization in reading comprehension. In: Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D. (eds.) Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1–13. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-5801. https://aclanthology.org/D19-5801 · DOI · Paper page
  6. Gienapp, L., et al.: Evaluating generative ad hoc information retrieval. Technical report (2023). arXiv:2311.04694 [cs] type: article · arXiv
  7. Gottschalk, S., Demidova, E.: Eventkg: a multilingual event-centric temporal knowledge graph (2018). https://doi.org/10.48550/ARXIV.1804.04526 · arXiv · doi:10.48550/arxiv.1804.04526
  8. Huang, J., Shao, H., Chang, K.C.C., Xiong, J., Hwu, W.: Understanding jargon: combining extraction and generation for definition modeling. In: Proceedings of EMNLP (2022) · DOI
  9. Huang, Y., Huang, J.: A survey on retrieval-augmented text generation for large language models (2024). https://doi.org/10.48550/ARXIV.2404.10981 · DOI
  10. Kamalloo, E., Thakur, N., Lassance, C., Ma, X., Yang, J.H., Lin, J.: Resources for brewing beir: reproducible reference models and an official leaderboard (2023) · DOI
  11. Khattab, O., Zaharia, M.: Colbert: efficient and effective passage search via contextualized late interaction over BERT (2020). https://doi.org/10.48550/ARXIV.2004.12832 · arXiv · doi:10.48550/arxiv.2004.12832 · Paper page
  12. Lahitani, A.R., Permanasari, A.E., Setiawan, N.A.: Cosine similarity to determine similarity measure: study case in online essay assessment. In: 2016 4th International Conference on Cyber and IT Service Management, pp. 1–6 (2016). https://doi.org/10.1109/CITSM.2016.7577578 · DOI · Paper page
  13. Lanchantin, J., Toshniwal, S., Weston, J., Szlam, A., Sukhbaatar, S.: Learning to reason and memorize with self-notes (2023). https://doi.org/10.48550/ARXIV.2305.00833 · DOI
  14. Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 2020, pp. 9459–9474. Curran Associates Inc., Red Hook (2020)
  15. Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81. Association for Computational Linguistics, Barcelona, Spain (2004). https://aclanthology.org/W04-1013
  16. Liu, Y., et al.: Recall: a benchmark for LLMs robustness against external counterfactual knowledge (2023). https://doi.org/10.48550/ARXIV.2311.08147 · DOI
  17. Lyu, Y., et al.: Crud-rag: a comprehensive Chinese benchmark for retrieval-augmented generation of large language models (2024). https://doi.org/10.48550/ARXIV.2401.17043 · DOI
  18. OpenAI, Achiam, J., et al.: GPT-4 Technical Report (2023). https://doi.org/10.48550/ARXIV.2303.08774 · DOI · Paper page
  19. Petroni, F., et al.: KILT: a benchmark for knowledge intensive language tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523–2544. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.200. https://aclanthology.org/2021.naacl-main.200 · DOI · Paper page
  20. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
  21. Ramos, J., et al.: Using TF-IDF to determine word relevance in document queries. In: Proceedings of the First Instructional Conference on Machine Learning, vol. 242, pp. 29–48. Citeseer (2003)
  22. Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: BM25 and beyond. Found. Trends® Inf. Retrieval 3(4), 333–389 (2009) · DOI · Paper page
  23. Rosset, C., et al.: Researchy questions: a dataset of multi-perspective, decompositional questions for LLM web agents (2024). https://doi.org/10.48550/ARXIV.2402.17896 · DOI · Paper page
  24. Sai, A.B., Mohankumar, A.K., Khapra, M.M.: A survey of evaluation metrics used for NLG systems. ACM Comput. Surv. (CSUR) 55(2), 1–39 (2022) · DOI · Paper page
  25. Shahabi, C., Kolahdouzan, M.R., Sharifzadeh, M.: A road network embedding technique for k-nearest neighbor search in moving object databases. In: Proceedings of the 10th ACM International Symposium on Advances in Geographic Information Systems, pp. 94–100 (2002) · doi:10.1145/585147.585167
  26. Tang, Y., Yang, Y.: Multihop-rag: benchmarking retrieval-augmented generation for multi-hop queries (2024). https://doi.org/10.48550/ARXIV.2401.15391 · DOI · Paper page
  27. Vaswani, A., et al.: Attention is all you need (2017). https://doi.org/10.48550/ARXIV.1706.03762 · DOI · Paper page
  28. Wang, A., et al.: SuperGLUE: a stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537 (2019) · arXiv
  29. Wang, S., Khramtsova, E., Zhuang, S., Zuccon, G.: Feb4rag: evaluating federated search in the context of retrieval augmented generation (2024). https://doi.org/10.48550/ARXIV.2402.11891 · DOI
  30. Wang, S., et al.: Domainrag: a Chinese benchmark for evaluating domain-specific retrieval-augmented generation (2024). https://doi.org/10.48550/ARXIV.2406.05654 · DOI · Paper page
  31. Xu, Z., et al.: Let LLMs take on the latest challenges! a Chinese dynamic question answering benchmark (2024). https://doi.org/10.48550/ARXIV.2402.19248 · DOI
  32. Yao, J.Y., Ning, K.P., Liu, Z.H., Ning, M.N., Yuan, L.: LLM lies: hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023) · arXiv
  33. Yu, X., Cheng, H., Liu, X., Roth, D., Gao, J.: ReEval: automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1333–1351. Association for Computational Linguistics, Mexico City, Mexico (2024). https://aclanthology.org/2024.findings-naacl.85 · DOI
  34. Zhang, K., et al.: EATN: an efficient adaptive transfer network for aspect-level sentiment analysis. IEEE Trans. Knowl. Data Eng. 35(1), 377–389 (2021)
  35. Zhang, K., Zhang, H., Liu, Q., Zhao, H., Zhu, H., Chen, E.: Interactive attention transfer network for cross-domain sentiment classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5773–5780 (2019) · DOI
  36. Zhang, K., et al.: Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. arXiv preprint arXiv:2203.16369 (2022) · DOI
  37. Zhang, Q., et al.: A survey for efficient open domain question answering. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14447–14465. Association for Computational Linguistics, Toronto, Canada (2023). https://doi.org/10.18653/v1/2023.acl-long.808. https://aclanthology.org/2023.acl-long.808 · DOI
  38. Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., Van Durme, B.: Record: bridging the gap between human and machine commonsense reading comprehension (2018). https://doi.org/10.48550/ARXIV.1810.12885 · arXiv · doi:10.48550/arxiv.1810.12885 · Paper page
  39. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with BERT. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020. OpenReview.net (2020). https://openreview.net/forum?id=SkeHuCVFDr
  40. Zhang, Y., Khalifa, M., Logeswaran, L., Lee, M., Lee, H., Wang, L.: Merging generated and retrieved knowledge for open-domain QA. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4710–4728. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.emnlp-main.286. https://aclanthology.org/2023.emnlp-main.286 · DOI
  41. Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., Chua, T.S.: Retrieving and reading: a comprehensive survey on open-domain question answering. Technical report (2021). arXiv:2101.00774 [cs] type: article · arXiv

Cited by

See all 173

Metadata

Authors
Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z.
Venue
Communications in Computer and Information Science
Published
2025
Pages
102-120
ISSN
  • 1865-0929 (ISSN-L)
  • 1865-0937
OpenAlex
W4406771321
Cited by
173
References
30
FWCI
168.43
Type
conference paper
Language
en
Open access
green
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation of Retrieval-Augmented Generation: A Survey. https://app.cochat.ai/papers/10.1007/978-981-96-1024-2_8
The paper
Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2025). Evaluation of Retrieval-Augmented Generation: A Survey. Communications in Computer and Information Science. https://doi.org/10.1007/978-981-96-1024-2_8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.