The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Evaluation of Retrieval-Augmented Generation: A Survey
Open accessAbstract
Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 1×
“In the inferencing step, Large Language Models (LLMs) interpret the prompted input to generate accurate and in-depth responses that align with the query’s intent and integrate the extracted information [30,8].”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“Searches in a dynamic database may lead to misleading results or contradicting the facts.”
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jégou, H. (2017) · arXiv · doi:10.48550/arxiv.1702.08734 · Cited by 4,140
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · cited 1×
“By retrieving relevant information from external sources, RAG significantly reduces the incidence of hallucinations [20] or factually incorrect generative outputs, thereby improving the content’s reliability and richness.”
Background
Research works printed but not linked: 41
- Balaguer, A., et al.: RAG vs fine-tuning: pipelines, tradeoffs, and a case study on agriculture. Technical report (2024). arXiv:2401.08406 [cs] type: article · arXiv
- Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M.: Seven failure points when engineering a retrieval augmented generation system (2024). https://doi.org/10.48550/ARXIV.2401.05856 · DOI
- Deng, Y., Zhang, W., Chen, Z., Gu, Q.: Rephrase and respond: let large language models ask better questions for themselves (2023). https://doi.org/10.48550/ARXIV.2311.04205 · DOI · Paper page
- Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of Wikipedia: knowledge-powered conversational agents. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)
- Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D.: MRQA 2019 shared task: evaluating generalization in reading comprehension. In: Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D. (eds.) Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1–13. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-5801. https://aclanthology.org/D19-5801 · DOI · Paper page
- Gienapp, L., et al.: Evaluating generative ad hoc information retrieval. Technical report (2023). arXiv:2311.04694 [cs] type: article · arXiv
- Gottschalk, S., Demidova, E.: Eventkg: a multilingual event-centric temporal knowledge graph (2018). https://doi.org/10.48550/ARXIV.1804.04526 · arXiv · doi:10.48550/arxiv.1804.04526
- Huang, J., Shao, H., Chang, K.C.C., Xiong, J., Hwu, W.: Understanding jargon: combining extraction and generation for definition modeling. In: Proceedings of EMNLP (2022) · DOI
- Huang, Y., Huang, J.: A survey on retrieval-augmented text generation for large language models (2024). https://doi.org/10.48550/ARXIV.2404.10981 · DOI
- Kamalloo, E., Thakur, N., Lassance, C., Ma, X., Yang, J.H., Lin, J.: Resources for brewing beir: reproducible reference models and an official leaderboard (2023) · DOI
- Khattab, O., Zaharia, M.: Colbert: efficient and effective passage search via contextualized late interaction over BERT (2020). https://doi.org/10.48550/ARXIV.2004.12832 · arXiv · doi:10.48550/arxiv.2004.12832 · Paper page
- Lahitani, A.R., Permanasari, A.E., Setiawan, N.A.: Cosine similarity to determine similarity measure: study case in online essay assessment. In: 2016 4th International Conference on Cyber and IT Service Management, pp. 1–6 (2016). https://doi.org/10.1109/CITSM.2016.7577578 · DOI · Paper page
- Lanchantin, J., Toshniwal, S., Weston, J., Szlam, A., Sukhbaatar, S.: Learning to reason and memorize with self-notes (2023). https://doi.org/10.48550/ARXIV.2305.00833 · DOI
- Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 2020, pp. 9459–9474. Curran Associates Inc., Red Hook (2020)
- Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81. Association for Computational Linguistics, Barcelona, Spain (2004). https://aclanthology.org/W04-1013
- Liu, Y., et al.: Recall: a benchmark for LLMs robustness against external counterfactual knowledge (2023). https://doi.org/10.48550/ARXIV.2311.08147 · DOI
- Lyu, Y., et al.: Crud-rag: a comprehensive Chinese benchmark for retrieval-augmented generation of large language models (2024). https://doi.org/10.48550/ARXIV.2401.17043 · DOI
- OpenAI, Achiam, J., et al.: GPT-4 Technical Report (2023). https://doi.org/10.48550/ARXIV.2303.08774 · DOI · Paper page
- Petroni, F., et al.: KILT: a benchmark for knowledge intensive language tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523–2544. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.200. https://aclanthology.org/2021.naacl-main.200 · DOI · Paper page
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
- Ramos, J., et al.: Using TF-IDF to determine word relevance in document queries. In: Proceedings of the First Instructional Conference on Machine Learning, vol. 242, pp. 29–48. Citeseer (2003)
- Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: BM25 and beyond. Found. Trends® Inf. Retrieval 3(4), 333–389 (2009) · DOI · Paper page
- Rosset, C., et al.: Researchy questions: a dataset of multi-perspective, decompositional questions for LLM web agents (2024). https://doi.org/10.48550/ARXIV.2402.17896 · DOI · Paper page
- Sai, A.B., Mohankumar, A.K., Khapra, M.M.: A survey of evaluation metrics used for NLG systems. ACM Comput. Surv. (CSUR) 55(2), 1–39 (2022) · DOI · Paper page
- Shahabi, C., Kolahdouzan, M.R., Sharifzadeh, M.: A road network embedding technique for k-nearest neighbor search in moving object databases. In: Proceedings of the 10th ACM International Symposium on Advances in Geographic Information Systems, pp. 94–100 (2002) · doi:10.1145/585147.585167
- Tang, Y., Yang, Y.: Multihop-rag: benchmarking retrieval-augmented generation for multi-hop queries (2024). https://doi.org/10.48550/ARXIV.2401.15391 · DOI · Paper page
- Vaswani, A., et al.: Attention is all you need (2017). https://doi.org/10.48550/ARXIV.1706.03762 · DOI · Paper page
- Wang, A., et al.: SuperGLUE: a stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537 (2019) · arXiv
- Wang, S., Khramtsova, E., Zhuang, S., Zuccon, G.: Feb4rag: evaluating federated search in the context of retrieval augmented generation (2024). https://doi.org/10.48550/ARXIV.2402.11891 · DOI
- Wang, S., et al.: Domainrag: a Chinese benchmark for evaluating domain-specific retrieval-augmented generation (2024). https://doi.org/10.48550/ARXIV.2406.05654 · DOI · Paper page
- Xu, Z., et al.: Let LLMs take on the latest challenges! a Chinese dynamic question answering benchmark (2024). https://doi.org/10.48550/ARXIV.2402.19248 · DOI
- Yao, J.Y., Ning, K.P., Liu, Z.H., Ning, M.N., Yuan, L.: LLM lies: hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023) · arXiv
- Yu, X., Cheng, H., Liu, X., Roth, D., Gao, J.: ReEval: automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1333–1351. Association for Computational Linguistics, Mexico City, Mexico (2024). https://aclanthology.org/2024.findings-naacl.85 · DOI
- Zhang, K., et al.: EATN: an efficient adaptive transfer network for aspect-level sentiment analysis. IEEE Trans. Knowl. Data Eng. 35(1), 377–389 (2021)
- Zhang, K., Zhang, H., Liu, Q., Zhao, H., Zhu, H., Chen, E.: Interactive attention transfer network for cross-domain sentiment classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5773–5780 (2019) · DOI
- Zhang, K., et al.: Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. arXiv preprint arXiv:2203.16369 (2022) · DOI
- Zhang, Q., et al.: A survey for efficient open domain question answering. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14447–14465. Association for Computational Linguistics, Toronto, Canada (2023). https://doi.org/10.18653/v1/2023.acl-long.808. https://aclanthology.org/2023.acl-long.808 · DOI
- Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., Van Durme, B.: Record: bridging the gap between human and machine commonsense reading comprehension (2018). https://doi.org/10.48550/ARXIV.1810.12885 · arXiv · doi:10.48550/arxiv.1810.12885 · Paper page
- Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with BERT. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020. OpenReview.net (2020). https://openreview.net/forum?id=SkeHuCVFDr
- Zhang, Y., Khalifa, M., Logeswaran, L., Lee, M., Lee, H., Wang, L.: Merging generated and retrieved knowledge for open-domain QA. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4710–4728. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.emnlp-main.286. https://aclanthology.org/2023.emnlp-main.286 · DOI
- Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., Chua, T.S.: Retrieving and reading: a comprehensive survey on open-domain question answering. Technical report (2021). arXiv:2101.00774 [cs] type: article · arXiv
Cited by
A survey on knowledge-oriented retrieval-augmented generation
Cheng, M., Luo, Y., Ouyang, J., Liu, Q., Liu, H., Li, L., Yu, S., Zhang, B., Cao, J., Ma, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.10677 · Cited by 140
Self-citationEvaluation of large language models for discovery of gene set function
Hu, M., Alkhairy, S., Lee, I., Pillich, R. T., Fong, D., Smith, K., Bachelder, R., Ideker, T., & Pratt, D. (2025) · Nature Methods · doi:10.1038/s41592-024-02525-x · Cited by 104
“A first major direction would be to further boost LLM accuracy and interpretability, for which recent techniques such as fine-tuning 31 and retrieval-augmented generation 62 are showing considerable promise.”
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3
Metadata
- Authors
- Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z.
- Venue
- Communications in Computer and Information Science
- Publisher
- Springer Nature Singapore
- Published
- 2025
- Pages
- 102-120
- ISSN
- 1865-0929 (ISSN-L)
- 1865-0937
- OpenAlex
- W4406771321
- Cited by
- 173
- References
- 30
- FWCI
- 168.43
- Type
- conference paper
- Language
- en
- Open access
- green
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation of Retrieval-Augmented Generation: A Survey. https://app.cochat.ai/papers/10.1007/978-981-96-1024-2_8
- The paper
- Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2025). Evaluation of Retrieval-Augmented Generation: A Survey. Communications in Computer and Information Science. https://doi.org/10.1007/978-981-96-1024-2_8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.