The DOI matches its Crossref record; 17 of the 20 references it cites were found in a registry or an index.
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Open accessAbstract
Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Standard RAG evaluation frameworks (RAGAS faithfulness [Es et al., 2024], FActScore [Min et al., 2023], ARES [Saad-Falcon et al., 2024], FaithDial [Dziri et al., 2022]) all pass deceptive grounding responses by design because every claim is sourced from a real document.”
From Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (2026) · cites this paper as Saad-Falcon et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“RAG evaluation frameworks such as RAGAS [Es et al., 2024] and ARES [Saad-Falcon et al., 2024] assess faithfulness at the answer level, treating the full context-answer pair as a unit rather than scoring individual attribution-citation pairs”
From Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution (2026) · cites this paper as Saad-Falcon et al., 2024
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“no existing framework implements it.”
From Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (2026) · cites this paper as Saad-Falcon et al., 2024
Experimental — accuracy not yet measuredIn the citing paper
“Consequently, RAG systems are evaluated based on dimensions such as context relevance (that is, whether the retrieved context is relevant to the given prompt), answer faithfulness (that is, whether the response generated by the LLM is properly grounded in the retrieved context), and answer relevance (that is, whether the response is relevant to the user question) [27,100].”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [100]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 11
- Mubashara Akhtar, Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos, editors. 2023. Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER). Association for Computational Linguistics, Dubrovnik, Croatia.
- Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
- Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 252-262.
- Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Relevance-guided supervision for openqa with colbert. Transactions of the association for computa- tional linguistics, 9:929-944.
- Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. · arXiv
- Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650-1669, Dubrovnik, Croatia. Association for Computational Linguistics.
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
- David P Sander and Laura Dietz. 2021. Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want. In DESIRES, pages 136-146.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Cited by
Sorted by citations
A comprehensive survey of loss functions and metrics in deep learning
Terven, J. R., Esparza, D. M. C., Romero-González, J.-A., Ramírez-Pedraza, A., & Chávez‐Urbiola, E. A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11198-7 · Cited by 685
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2026) · Data Science and Engineering · doi:10.1007/s41019-025-00335-5 · Cited by 871
Lyu, Y., li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., & Chen, E. (2024) · ACM Transactions on Information Systems · doi:10.1145/3701228 · Cited by 199
When Search Engine Services Meet Large Language Models: Visions and Challenges
Xiong, H., Bian, J., Li, Y., Li, X., Du, M., Wang, S., Yin, D., & Helal, S. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3451185 · Cited by 111
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47
Metadata
- Authors
- Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 338-354
- OpenAlex
- W4401042808
- Cited by
- 148
- References
- 0
- Citations by year
- 2026: 84 · 2025: 48 · 2024: 16
- FWCI
- 43.61
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- retrieval-augmented generation, automated assessment, evaluation framework, large language models, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
