The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Open accessAbstract
Large Language Models (LLMs) have significantly enhanced the capabilities of information access systems, especially with retrieval-augmented generation (RAG). Nevertheless, the evaluation of RAG systems remains a barrier to continued progress, a challenge we tackle in this work by proposing an automatic evaluation framework that is validated against human annotations. We believe that the nugget evaluation methodology provides a solid foundation for evaluating RAG systems. This approach, originally developed for the TREC Question Answering (QA) Track in 2003, evaluates systems based on atomic facts that should be present in good answers. Our efforts focus on "refactoring" this methodology, where we describe the AutoNuggetizer framework that specifically applies LLMs to both automatically create nuggets and automatically assign nuggets to system answers. In the context of the TREC 2024 RAG Track, we calibrate a fully automatic approach against strategies where nuggets are created manually or semi-manually by human assessors and then assigned manually to system answers. Based on results from a community-wide evaluation, we observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants. The agreement is stronger when individual framework components such as nugget assignment are automated independently. This suggests that our evaluation framework provides tradeoffs between effort and quality that can be used to guide the development of future RAG systems. However, further research is necessary to refine our approach, particularly in establishing robust per-topic agreement to diagnose system failures effectively.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pradeep et al. (2025) implement an automated LLM-based framework using the original vital/okay labels, called AutoNuggetizer, for nugget creation, ranking, assignment, and scoring.
“Recently, Pradeep et al. (2025) implement an automated LLM-based framework using the original vital/okay labels, called AutoNuggetizer, for nugget creation, ranking, assignment, and scoring (Pradeep et al., 2024).”
From All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations (2025) · cites this paper as Pradeep et al., 2025
Similar methods are used in evaluating recall, where references are used to generate nuggets-pieces of information vital for answering a given query.
“Similar methods are used in evaluating recall, where references are used to generate nuggets-pieces of information vital for answering a given query.”
From All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations (2025) · cites this paper as Pradeep et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
Retrieval evaluation with incomplete information
Buckley, C., & Voorhees, E. M. (2004) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1008992.1009000 · Cited by 742 · see citing works
How reliable are the results of large-scale information retrieval experiments?
Zobel, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291014 · Cited by 579 · see citing works
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
Evaluating evaluation measure stability
Buckley, C., & Voorhees, E. M. (2000) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/345508.345543 · Cited by 399 · see citing works
Information retrieval system evaluation
Sanderson, M., & Zobel, J. (2005) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1076034.1076064 · Cited by 369 · see citing works
Variations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291017 · Cited by 303 · see citing works
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
CRAG - Comprehensive RAG Benchmark
Yang, X., Sun, K., Xin, H., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2406.04744 · Cited by 158
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.20 · Cited by 148 · see citing works
Information Retrieval Evaluation
Harman, D. K. (2011) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-02276-0 · Cited by 103 · see citing works
Will Pyramids Built of Nuggets Topple Over?
Lin, J., & Demner-Fushman, D. (2006) · Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics - · doi:10.3115/1220835.1220884 · Cited by 77 · see citing works
Self-citationAutomatically evaluating answers to definition questions
Lin, J., & Demner‐Fushman, D. (2005) · doi:10.3115/1220575.1220692 · Cited by 59 · see citing works
Self-citationEvaluating answers to definition questions
Voorhees, E. M. (2003) · doi:10.3115/1073483.1073520 · Cited by 49 · see citing works
IR system evaluation using nugget-based test collections
Pavlu, V., Rajput, S., Golbus, P. B., & Aslam, J. A. (2012) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2124295.2124343 · Cited by 46 · see citing works
Methods for automatically evaluating answers to complex questions
Lin, J., & Demner‐Fushman, D. (2006) · Information Retrieval · doi:10.1007/s10791-006-9003-7 · Cited by 43 · see citing works
Self-citationCodeRAG-Bench: Can Retrieval Augment Code Generation?
Wang, Z. Z., Asai, A., Yu, X. V., Xu, F. F., Xie, Y., Neubig, G., & Fried, D. (2024) · arXiv · doi:10.48550/arxiv.2406.14497 · Cited by 34 · see citing works
Pencils Down! Automatic Rubric-based Evaluation of Retrieve/Generate Systems
Farzi, N., & Dietz, L. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672511 · Cited by 28 · see citing works
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
Pradeep, R., Thakur, N., Sharifymoghaddam, S., Zhang, E., Nguyen, R., Campos, D., Craswell, N., & Lin, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_9 · Cited by 24 · see citing works
Self-citationRAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
Han, R., Zhang, Y., Qi, P., Xu, Y., Wang, J., Liu, L., Wang, W. Y., Min, B., & Castelli, V. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.249 · Cited by 23 · see citing works
A Comparison of Methods for Evaluating Generative IRPreprint
Arabzadeh, N., & Clarke, C. L. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.04044 · Cited by 22 · see citing works
Lin, J., & Zhang, P. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277799 · Cited by 17 · see citing works
Self-citationUMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16 · see citing works
Self-citationGenerative Information Retrieval Evaluation
Alaofi, M., Arabzadeh, N., Clarke, C. L. A., & Sanderson, M. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_6 · Cited by 15 · see citing works
A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial LookPreprint
Upadhyay, S., Pradeep, R., Thakur, N., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.08275 · Cited by 9 · see citing works
Self-citationResearchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents
Rosset, C., Chung, H.-L., Qin, G., Chau, E. C., Feng, Z., Awadallah, A., Neville, J., & Rao, N. (2024) · arXiv · doi:10.48550/arxiv.2402.17896 · Cited by 6 · see citing works
Thakur, N., Bonifacio, L., Zhang, C., et al. (2024) · doi:10.18653/v1/2024.findings-emnlp.730 · Cited by 6 · see citing works
Self-citationEvaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You NeedPreprint
Wang, Y., Hernandez, A. G., Kyslyi, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.18064 · Cited by 6 · see citing works
Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2411.09607 · Cited by 6 · see citing works
Self-citationEvaluating RAG-Fusion with RAGElo: an Automated Elo-based FrameworkPreprint
Rackauckas, Z., Câmara, A., & Zavrel, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.14783 · Cited by 5 · see citing works
RAGProbe: An Automated Approach for Evaluating RAG ApplicationsPreprint
Sivasothy, S., Barnett, S., Kurniawan, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.19019 · Cited by 4 · see citing works
SFR-RAG: Towards Contextually Faithful LLMsPreprint
Nguyen, X.-P., Pandit, S., Purushwalkam, S., Xu, A., Chen, H. X., Ming, Y., Ke, Z., Savarese, S., Xong, C., & Joty, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.09916 · Cited by 2 · see citing works
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1 · see citing works
Self-citationMRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal ModelsPreprint
Hu, W., Gu, J.-C., Dou, Z.-Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08182 · Cited by 1 · see citing works
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Thakur, N., Lin, J., Havens, S., Carbin, M., Khattab, O., & Drozdov, A. (2025) · arXiv · doi:10.48550/arxiv.2504.13128
Self-citationMIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation SystemsPreprint
Thakur, N., Kazi, S., Luo, G., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.13716
Self-citationCypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era
Feng, Y., Papicchio, S., & Rahman, S. (2024) · arXiv · doi:10.48550/arxiv.2412.18702
Evaluating Retrieval Quality in Retrieval-Augmented Generation
Salemi, A., & Zamani, H. (2024) · arXiv · doi:10.48550/arxiv.2404.13781
The eleventh text REtrieval conference, TREC 2002
Voorhees, E. M., & Buckland, L. P. (2003) · doi:10.6028/nist.sp.500-251
Pradeep, R., Lee, D., Mousavi, A., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track · doi:10.18653/v1/2024.emnlp-industry.89
Self-citation
Flagged first, then by citations
Research works printed but not linked: 10
- [3][3] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, et al. 2022. Improving Language Models by Retrieving from Trillions of Tokens. In Proceedings of the 39th International Conference on Machine Learning (ICML 2022) . Baltimore, Maryland, 2206-2240. -
- [7][7] Hoa Trang Dang and Jimmy Lin. 2007. Different Structures for Evaluating Answers to Complex Questions: Pyramids Won't Topple, and Neither Will Human Assessors. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007) . Prague, Czech Republic, 768-775. -
- [8][8] Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations . St. Julians, Malta, 150-158. -
- [12][12] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020) . 3929-3938. -
- [17][17] Xingyu Ji, Aditya Parameswaran, and Madelon Hulsebos. 2024. TARGET: Benchmarking Table Retrieval for Generative Tasks. In Proceedings of the NeurIPS 2024 Third Table Representation Learning Workshop . -
- [18][18] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) . Addis Ababa, Ethiopia. -
- [20][20] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33 . 9459-9474. -
- [25][25] James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler. 2024. On the Evaluation of MachineGenerated Reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024) . Washington, D.C., 1904-1915. -
- [27][27] Ani Nenkova and Rebecca Passonneau. 2004. Evaluating Content Selection in Summarization: The Pyramid Method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 . Boston, Massachusetts, 145-152. -
- [49][49] Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A Critical Evaluation of Evaluations for Long-form Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Toronto, Canada, 3225-3245. -
Cited by
Sorted by citations
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C. (2025) · arXiv · doi:10.48550/arxiv.2508.20033
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
Self-citation
- Authors
- Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., & Lin, J.
- Affiliations
- University of Waterloo (CA)
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.15068
- OpenAlex
- W6910572094
- Cited by
- 3
- References
- 42
- Citations by year
- 2026: 2 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ministry of Science and ICT, South Korea
- National Institute of Standards and Technology
- Natural Sciences and Engineering Research Council of Canada
- Institute for Information and Communications Technology Promotion
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- retrieval-augmented generation, RAG evaluation, fact extraction, large language models, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
