FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
Open accessAbstract
Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, Amin Ahmad. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hallucinations remain prevalent in recent LLMs, with rates of roughly 40-50% on benchmarks such as FAITHBENCH.
“Hallucinations remain prevalent even in recent LLMs, with rates of roughly 40-50% on benchmarks such as FAITHBENCH (Bao et al., 2025) indicating persistent factual unreliability.”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Bao et al., 2025
Table 1 reports unwanted-hallucination rates of 40–48% for four of the ten modern LLMs, supporting the claim that hallucinations remain prevalent. The other six models have higher rates, and §3.2 cautions that these results apply only to FaithBench’s challenging summarization samples, not necessarily to samples generally. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 22
- Forrest Bao, Miaoran Li, Rogger Luo, and Ofer Mendelevitch. 2024. HHEM-2.1-Open. · DOI
- Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. 2023. Felm: benchmarking factuality evaluation of large language models. In Proceedings of the 37th Interna- tional Conference on Neural Information Processing Systems, pages 44502-44523.
- Xiang Chen, Duanzheng Song, Honghao Gui, Chenxi Wang, Ningyu Zhang, Yong Jiang, Fei Huang, Chengfei Lyu, Dan Zhang, and Huajun Chen. 2024. Factchd: Benchmarking fact-conflicting hallucina- tion detection. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intel- ligence, IJCAI-24, pages 6216-6224. International Joint Conferences on Artificial Intelligence Organi- zation. Main Track. · DOI
- Kunquan Deng, Zeyu Huang, Chen Li, Chenghua Lin, Min Gao, and Wenge Rong. 2024. Pfme: A mod- ular approach for fine-grained hallucination detec- tion and editing of large language models. Preprint, arXiv:2407.00488. · arXiv
- Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evalu- ation of retrieval augmented generation. In Proceed- ings of the 18th Conference of the European Chap- ter of the Association for Computational Linguistics: System Demonstrations, pages 150-158, St. Julians, Malta. Association for Computational Linguistics.
- Alexander R Fabbri, Wojciech Kryściński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summariza- tion evaluation. arXiv preprint arXiv:2007.12626. · arXiv
- Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, et al. 2025. The facts grounding leaderboard: Benchmarking llms' ability to ground responses to long-form input. arXiv preprint arXiv:2501.03200. · arXiv
- Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. 2024. ANAH: Analyt- ical annotation of hallucinations in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 8135-8158, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459- 9474.
- Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. 2024. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation. Preprint, arXiv:2406.07070. · arXiv
- Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization. Preprint, arXiv:2303.15621. · arXiv
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan Thomas Mcdonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line.
- Erwan Le Merrer and Gilles Tredan. 2024. Llms hallu- cinate graphs too: a structural perspective. Preprint, arXiv:2409.00159. · arXiv
- Timothee Mickus, Elaine Zosa, Raul Vazquez, Teemu Vahtola, Jörg Tiedemann, Vincent Segonne, Alessan- dro Raganato, and Marianna Apidianaki. 2024. SemEval-2024 task 6: SHROOM, a shared-task on hallucinations and related observable overgeneration mistakes. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1979-1993, Mexico City, Mexico. Association for Computational Linguistics. · DOI
- Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. Fine-grained halluci- nation detection and editing for language models. Preprint, arXiv:2401.06855. · arXiv
- Sanjana Ramprasad, Elisa Ferracane, and Zachary Lip- ton. 2024. Analyzing LLM behavior in dialogue sum- marization: Unveiling circumstantial hallucination trends. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 12549-12561, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Hannah Sansford, Nicholas Richardson, Hermina Maretic, and Juba Saada. 2024. Grapheval: A knowledge-graph based llm hallucination evaluation framework. In KiL'24: Workshop on Knowledge- infused Learning co-located with 30th ACM KDD Conference. · DOI
- Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. Minicheck: Efficient fact-checking of llms on ground- ing documents. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing. Association for Computational Linguis- tics.
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL-HLT. TruLens. 2024. Moving to trulens v1: Reliable and modular logging and evaluation.
- Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328-11348, Toronto, Canada. Association for Computational Linguistics.
- Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khy- athi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, and Yejin Choi. 2024. Wildhallucinations: Evaluating long-form factuality in llms with real- world entity queries. Preprint, arXiv:2407.17468. Source: The incident occurred on the A9 north of Berriedale in Caithness at about 14:00. Several other people received minor injuries in the crash. The road was closed and diversions were put in place via the A897 Helmsdale to Melvich road. Police Scotland said inquiries into the incident were ongoing. Summary: The passage describes an incident that occurred on the A9 north of Berriedale in Caithness at around 14:00. This incident resulted in a crash, and several people received minor injuries. The road was closed, and diversions were set up via the A897 Helmsdale to Melvich road. Police Scotland is currently conducting ongoing inquiries into the incident. Explanation: The source describes a past event, but "is currently conducting" in the summary implies present action, causing confusion. Human Label: Questionable ------------------- Source: Sheryl Lee. In 2016, she appeared in Cafe Society, and also completed the Showtime revival of Twin Peaks (2017), reprising her role of Laura Palmer. Sheryl Lee Ralph. In 2016 she played Madame Morrible in Wicked on Broadway. Summary: Sheryl Lee appeared in the film Cafe Society in 2016 and reprised her role as Laura Palmer in the Show- time revival of Twin Peaks in 2017. · arXiv
- Sheryl Lee Ralph played Madame Morrible in the Broadway production of Wicked in 2016. Explanation: The source indicates that 2017 was the release year for the Showtime revival of Twin Peaks, while Sheryl Lee completed her role in 2016. The summary misrepresents this timeline, causing a distortion of the original information. Human Label: Unwanted.Intrinsic ------------------- Source: As of 22 February 2020 , 77,984 cases have been confirmed , including in all provinces of China and more than two dozen other countries . LLM Summary: As of 22 February 2020, there were 77,984 confirmed cases of the disease, present in all Chinese provinces and over two dozen other countries. Explanation: The source does not specify that the cases are related to a disease, and this information cannot be inferred with certainty. Human Label: Unwanted.Extrinsic ------------------- Source: Albert Lawrence Brooks (born Albert Lawrence Einstein; July 22, 1947) is an American actor, filmmaker, author and comedian. Finding Dory. Directed by Andrew Stanton with co-direction by Angus MacLane, the screenplay was written by Stanton and Victoria Strouse. Summary: The passage provides information about Albert Lawrence Brooks, an American actor, filmmaker, author, and comedian. It also mentions the film "Finding Dory," which was directed by Andrew Stanton with co-direction by Angus MacLane. The screenplay for the film was written by Stanton and Victoria Strouse. Explanation: The source does not explicitly state that "Finding Dory" is a film, but this can be reasonably inferred from the context or verified with general knowledge. Human Label: Benign
Cited by
Sorted by citations
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9 · see citing works
LENS: Layers of Evaluation of Hallucination in GenAI Systems
Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189150 · Cited by 16 · see citing works
Mind the Link: Discourse Link-Aware Hallucination Detection in Summarization
Lee, D., Jung, H., & Choi, Y. S. (2025) · Applied Sciences · doi:10.3390/app151910506 · Cited by 1 · see citing works
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Seethakantha, S. K., Thai, D. N., Gudi, V. P., Tiwari, S., Matar, R., Mitra, A., Zhao, W., McCallum, A., & Salloum, W. (2026) · arXiv · doi:10.48550/arxiv.2605.28910
TCK-QA: Benchmarking Knowledge Graph-Augmented LLMs for Question Answering on the Turkish Penal CodePreprint
Ay, B., & Okuyan, H. (2026) · Research Square · doi:10.21203/rs.3.rs-9804227/v1
Wang, C., Li, C., Wang, P., Zhang, L., Liu, S., 祁吉, Hu, J., Cai, Y., Zhao, H., & Pu, R. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104997
Policy-Guided RAG: Enforcing Verbatim and Controlled Synthesis
Asadi, M. N., Tavakoli, L., & Bilgrami, M. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808490
Numerical Hallucinations in Retrieval-Augmented Generation: Detection and Analysis
Roy, S. S. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809882
A Meta-study of Hallucination Detection Across Provenance, Typology, and Granularities
Belikova, J., & Rykov, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41546-2_17
- Authors
- Bao, F. S., Li, M., Qu, R., Luo, G., Wan, E., Tang, Y., Fan, W., Tamber, M. S., Kazi, S., Sourabh, V., Qi, M., Tu, R., Xu, C., Gonzales, M., Mendelevitch, O., & Ahmad, A.
- Affiliations
- Iowa State University (US)
- University of Southern California (US)
- University of Waterloo (CA)
- University of Wisconsin–Madison (US)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
- Published
- 2025
- Pages
- 448-461
- OpenAlex
- W4411119711
- Cited by
- 8
- References
- 0
- Citations by year
- 2026: 5 · 2025: 1 · 2024: 2
- FWCI
- 5.72
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- hallucinations, summarization, large language models, benchmark, factual accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
