The DOI matches its Crossref record; 17 of the 20 references it cites were found in a registry or an index.
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Open accessAbstract
Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Standard RAG evaluation frameworks (RAGAS faithfulness [Es et al., 2024], FActScore [Min et al., 2023], ARES [Saad-Falcon et al., 2024], FaithDial [Dziri et al., 2022]) all pass deceptive grounding responses by design because every claim is sourced from a real document.”
From Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (2026) · cites this paper as Saad-Falcon et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“RAG evaluation frameworks such as RAGAS [Es et al., 2024] and ARES [Saad-Falcon et al., 2024] assess faithfulness at the answer level, treating the full context-answer pair as a unit rather than scoring individual attribution-citation pairs”
From Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution (2026) · cites this paper as Saad-Falcon et al., 2024
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“no existing framework implements it.”
From Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (2026) · cites this paper as Saad-Falcon et al., 2024
Experimental — accuracy not yet measuredIn the citing paper
“Consequently, RAG systems are evaluated based on dimensions such as context relevance (that is, whether the retrieved context is relevant to the given prompt), answer faithfulness (that is, whether the response generated by the LLM is properly grounded in the retrieved context), and answer relevance (that is, whether the response is relevant to the user question) [27,100].”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [100]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 11
- Mubashara Akhtar, Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos, editors. 2023. Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER). Association for Computational Linguistics, Dubrovnik, Croatia.
- Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
- Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
- Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 252-262.
- Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Relevance-guided supervision for openqa with colbert. Transactions of the association for computa- tional linguistics, 9:929-944.
- Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. · arXiv
- Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650-1669, Dubrovnik, Croatia. Association for Computational Linguistics.
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
- David P Sander and Laura Dietz. 2021. Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want. In DESIRES, pages 136-146.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Cited by
Sorted by citations
Items 101–157 of 157
Xu, Z., Wang, Z., Wang, H., Zhou, Y., & Sun, M. (2026) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit70033.2026.11558060
Data validation: SOC and LLM approaches
Dobrovodsky, P., Kail, E., Fleiner, R., Nyilas, G., Szarvák, A., & Póser, V. (2026) · IEEE International Conference and Workshop Óbuda on Electrical and Power Engineering (CANDO-EPE) · doi:10.1109/cando-epe71091.2026.11569448
AIANO: Enhancing Information Retrieval with AI-Augmented Annotation
Khattab, S., Bauer, M., Heine, L., Rostalski, T., Kleesiek, J., & Friedrich, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1926-1_18
Metadata, Structure, or Strategy? A Decomposition of RAG Context Enrichment
Zerhoudi, S., Granitzer, M., & Mitrović, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37676-3_4 · Cited by 1
Pookduang, P., & Chansanam, W. (2026) · Computers · doi:10.3390/computers15070427
A Comprehensive Framework for Secure Multi-Agent Retrieval-Augmented Generation Systems
Ghouil, C., Aloui, R., Khalfaoui, N., Mhatli, S., Aldalbahi, A. S., & Llamas‐Garro, I. (2026) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1109/iwcmc69287.2026.11579902
Martín, C. A., Torres, J., Aguilar, R. M., Alayón, S., & Bacallado, M. A. (2026) · Applied Sciences · doi:10.3390/app16136793
LGAgent: A Structured and Rule-Guided Multi-Agent Framework for Interpretable Legal ReasoningPreprint
Feng, X., Shu, F., Feng, M., Feng, Y., & Liu, H. F. (2026) · Research Square · doi:10.21203/rs.3.rs-9571992/v1
TERA-RAG: Temporal and Epistemic Reliability for Multi-Agent Debate
Truong, H. N., Nguyen, T., Nguyen, Q. V. H., Tran, T. P. T., Hoang, M. A., & Dang, T. K. (2026) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse68839.2026.11596782
Kullawattana, S., & Limprasert, W. (2026) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse68839.2026.11597084
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
Christakopoulou, E., Patel, V., Velaga, H., & Gaikwad, S. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808460
Dhole, K., & Agichtein, E. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809940 · Cited by 7
Operational AI Assistance in Slack-Based Troubleshooting: A Measurement Framework and Field Results
Bommareddy, R. K. R. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620929
Evaluation of RAG-based LLMs: A case study in the insurgency knowledge domain
Chatterjee, D., & Chatterjee, N. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261469010
Hu, Q., Ning, S., Bao, C., Chen, G., Wang, J., Yang, W. H., Chen, H., Jia, X., Zhou, W., & Zhang, G. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817479
Measuring Political Chatbots: A Multi-dimensional Framework for Trustworthiness
Dieing, T. I. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_7
Smart MRAG: Hybrid Retrieval-Augmented Multimodal Q&A Technique
Narvekar, A., Bajaj, R., Mukherjee, A., & Katoch, A. (2026) · Journal of Computer Information Systems · doi:10.1080/08874417.2026.2707153
AEQAG: An Association-Enhanced Question Answering Generation Framework and its Application
Li, H., Yin, R., He, R., Yang, J., Wang, S., & Gao, J. (2026) · International Conference on Artificial Intelligence and Electromechanical Automation (AIEA) · doi:10.1109/aiea70743.2026.11632308
D’Antonio, R., Xie, H., & Guo, X. (2026) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3820755.3821485
CAGE: An Evaluation Framework for Cache-Augmented Generation Models
do Carmo, L. M., Brandão, W. C., & de Freitas, H. C. (2026) · Anais do Workshop em Desempenho de Sistemas Computacionais e de Comunicação (WPerformance) · doi:10.5753/wperformance.2026.23603
Benchmarking Normative AI Assistants Under Inconsistent Evidence with Paraconsistent Trace Semantics
Ulizko, M., Chernikov, A. V., Tomilov, I., Gusarova, N., & Vatian, A. S. (2026) · AI · doi:10.3390/ai7080321
Gangapuram, R. R., & Andreopoulos, W. (2026) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice70481.2026.00015
Li, M., Wang, H., Sun, Z., & Zhou, H. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10090291
Groß, J., Zentgraf, E., & Heidrich, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36587-3_25
García-López, I. M., & Guillén-Yparrea, N. (2026) · Computers · doi:10.3390/computers15090564
FitRAG: Self-Improving LLM-Based Services through Feedback-Driven Adapter Lifecycle Management
Syah, R. A., Haryanto, C. Y., Lomempow, E., Malik, K., & Putra, I. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00022
Verifier-Grounded Runtime Mediation for LLM Agent Procedural Compliance
Yabusaki, H. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00014
Kim, B., & Yu, Y. (2026) · Nuclear Engineering and Technology · doi:10.1016/j.net.2026.104687
Yang, T., Cao, Y., Ma, Y., Liang, Y., Liu, T., Qin, C., & Liao, X. (2026) · Energies · doi:10.3390/en19174207
Wei, F., Xie, T., Wang, Y., Wang, W., & Chen, L. (2026) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia71496.2026.11684660
Zaitsev, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39150-6_14
Beyond Textual Accuracy: Evaluating RAG-Based Assistant for Novice Operators in Industrial Robotics
Botticella, E., Wu, Z., Aliev, K., & Bruno, G. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-38618-2_7
HERAG: A Holistic Approach for Evaluating Organizational RAG Applications
Kleinekathöfer, F., Dick, F., Ahrens, A., Oltmanns, T., Meeuw, S., Keßler, R., & Gómez, J. M. (2026) · Progress in IS · doi:10.1007/978-3-032-14099-9_23
What Matters in RAG for Small Language Models? A Large-Scale Sensitivity Study
Frontera, G., Ferreira, L. N., & Dias, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39898-7_13
Yan, W., Song, Z., Liu, S., Ershun, P., & Fu, Q. (2026) · Sustainability · doi:10.3390/su181910102
Yuan, Y., Huang, J., Cai, Y., Wei, X., Huang, M., Chen, Y., & Liu, J. (2026) · Applied Sciences · doi:10.3390/app16199874
Zheng, L., Xie, X., Hou, B., & Wan, R. (2026) · International Conference on Computer Science and Blockchain (CCSB) · doi:10.1109/ccsb71176.2026.11716472
Bhoomika, S. S., Achutha, S., N, A. B., Shet, B. M., & Hegde, A. N. (2026) · International Conference on Ambient Intelligence, Knowledge Informatics and Industrial Electronics (AIKIIE) · doi:10.1109/aikiie69997.2026.11708454
Hatim, S. M., Nizam, A. A. K., Shamsudin, N., Zamani, N. A. M., & Ahmad, Z. (2026) · International Conference on Artificial Intelligence and Data Sciences (AiDAS) · doi:10.1109/aidas71914.2026.11712729
Pena, L., Peña, J.-M., & Leal, R. (2026) · Applied Sciences · doi:10.3390/app16199966
Oliveira, Í. S., de Oliveira, F. F. R., & Santos, I. S. (2026) · Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST) · doi:10.5753/sast.2026.26881
From Retrieval to Evidence: A Task-Adaptive Two-Layer Knowledge Architecture for LLM Agents
Volkov, A., Pustovoy, T., Istomin, D., & Otbetkina, T. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100322
Beyond Output Faithfulness: Claim-Centered Relational Auditability for Source-Grounded Generative AI
Wang, J., & Liu, P. (2026) · Information · doi:10.3390/info17100992
EvidenceCell-DL: Counterfactual Evidence Gates for Metadata Repair in Digital Libraries
Constantin, D., & Bălcău, C. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846054
Ather, H., Haddou, M., & Molladavoudi, S. (2026) · Journal of Official Statistics · doi:10.1177/0282423x261489965
Hallucination Detection Middleware using RAG Based Model
Lakshmanan, A., Nithyanandh, S., & Mohanapriya, D. (2026) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit2612556
Zhu, J., Wu, H., & Cao, Y. (2026) · Electronics · doi:10.3390/electronics15204759
YanXiaoZhi: An On-Premises RAG-Based Intelligent Service System for Graduate Affairs
Hu, S., Ni, Y., Duan, Y., & Tang, M. (2026) · International Conference of Data and Information Systems (DIS) · doi:10.1109/dis72268.2026.11725765
Bomfim, L. D. (2026) · Nexus Science Review · doi:10.66104/zkmx8c79
de Sousa, M. H. F., & Soares, A. C. B. (2026) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2026.28059
Kinac, A., & Ulu, E. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727975
Marigliano, R., & Carley, K. M. (2026) · Electronics · doi:10.3390/electronics15214962
Salas, D. G. P., Varela, P. M. R., & Egas, P. F. B. (2026) · Mathematical and Computational Applications · doi:10.3390/mca31060235
Olsen, C. (2026) · Proceedings of the Python in Science Conferences · doi:10.25080/khdg8264
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Pradeep, R., Thakur, N., Upadhyay, S., Campos, D., Craswell, N., & Lin, J. (2025) · arXiv · doi:10.48550/arxiv.2504.15068 · Cited by 3
Metadata
- Authors
- Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 338-354
- OpenAlex
- W4401042808
- Cited by
- 148
- References
- 0
- Citations by year
- 2026: 84 · 2025: 48 · 2024: 16
- FWCI
- 43.61
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- retrieval-augmented generation, automated assessment, evaluation framework, large language models, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
