Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
Open accessAbstract
Improvements in large language models have led to increasing optimism that they can serve as reliable evaluators of natural language generation outputs. In this paper, we challenge this optimism by thoroughly re-evaluating five state-of-the-art factuality metrics on a collection of 11 datasets for summarization, retrieval-augmented generation, and question answering. We find that these evaluators are inconsistent with each other and often misestimate system-level performance, both of which can lead to a variety of pitfalls. We further show that these metrics exhibit biases against highly paraphrased outputs and outputs that draw upon faraway parts of the source documents. We urge users of these factuality metrics to proceed with caution and manually validate the reliability of these metrics in their domain of interest before proceeding.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Godbole and Jia (2025) re-evaluated these evaluation metrics, and find pitfalls in current factuality metrics.
“Godbole and Jia (2025) re-evaluated these evaluation metrics, and find pitfalls in current factuality metrics.”
From All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations (2025) · cites this paper as Godbole and Jia, 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · arXiv · doi:10.48550/arxiv.2301.13298 · Cited by 3 · cited 1×
“There is a potential confounder in our analysis that the human annotations may not be accurate or have significant room for ambiguity(Krishna et al., 2023; Subbiah et al., 2024; Li et al., 2024).”
Flagged first, then by citations
Research works printed but not linked: 14
- Anthony Chen, Panupong Pasupat, Sameer Singh, Hon- grae Lee, and Kelvin Guu. 2023a. Purr: Effi- ciently editing language model hallucinations by denoising language model corruptions. Preprint, arXiv:2305.14908. · arXiv
- Hung-Ting Chen, Fangyuan Xu, Shane Arora, and Eu- nsol Choi. 2023b. Understanding retrieval augmen- tation for long-form question answering. Preprint, arXiv:2310.12150. · arXiv
- Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024. Complex claim verification with evidence retrieved in the wild. In Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3569-3587, Mexico City, Mexico. As- sociation for Computational Linguistics. · DOI
- Juri Opitz. 2024. Schroedinger's threshold: When the AUC doesn't predict accuracy. In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evalu- ation (LREC-COLING 2024), pages 14400-14406, Torino, Italia. ELRA and ICCL.
- Sanjana Ramprasad and Byron C. Wallace. 2024. Do automatic factuality metrics measure factuality? a critical evaluation. Preprint, arXiv:2411.16638. · arXiv
- Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural lan- guage generation models. Computational Linguistics, 49(4):777-840. · DOI · Paper page
- Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics. · DOI · Paper page
- Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021. Perturbation CheckLists for evaluating NLG evaluation metrics. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7219-7234, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI
- Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. Minicheck: Efficient fact-checking of llms on ground- ing documents. Preprint, arXiv:2404.10774. · arXiv
- Liyan Tang, Philippe Laban, and Greg Durrett. 2024b. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8818-8847, Miami, Florida, USA. Association for Computational Linguistics. · DOI · Paper page
- Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024c. TofuEval: Evaluating hallucinations of LLMs on topic-focused dialogue summarization. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4455-4480, Mexico City, Mexico. Association for Computational Linguistics. · DOI · Paper page
- Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foun- dational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17086-17105, Miami, Florida, USA. Association for Computational Lin- guistics. · DOI
- David Wan, Jesse Vig, Mohit Bansal, and Shafiq Joty. 2024. On positional bias of faithfulness for long- form summarization. Preprint, arXiv:2410.23609. · arXiv
- Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. Automatic evaluation of attri- bution by large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4615-4635, Singapore. Association for Computational Linguistics. Flan-PaLM-540B 67.4 (0.0) 59.7 (-7.7) 61.0 (-6.3) 58.5 (-8.8) 58.5 (-8.8) 61.2 (-6.1) 57.8 (-9.6) Flan-UL2-20B 79.4 (0.0) 68.8 (-10.6) 71.4 (-8.0) 68.7 (-10.7) 68.7 (-10.7) 67.8 (-11.6) 70.0 (-9.4) GPT-3 76.2 (0.0) 63.0 (-13.2) 68.6 (-7.6) 65.6 (-10.6) 65.6 (-10.6) 68.3 (-7.9) 72.5 (-3.7) Headroom 67.4 (0.0) 59.7 (-7.7) 61.0 (-6.3) 58.5 (-8.8) 58.5 (-8.8) 61.2 (-6.1) 57.8 (-9.6) Table 17: Reveal: Predicted instance-level error rates for systems. Quantification bias in parentheses. label GPT-4-turbo GPT-3.5-turbo Bespoke-7B Bespoke-7B (cs=500) MiniCheck-FT5 MiniCheck-Roberta System Name Flan-PaLM-540B 74.7 (0.0) 72.1 (-2.6) 74.7 (0.0) 69.5 (-5.2) 69.5 (-5.2) 69.5 (-5.2) 69.5 (-5.2) Flan-UL2-20B 84.2 (0.0) 76.6 (-7.6) 79.5 (-4.7) 76.6 (-7.6) 76.6 (-7.6) 77.2 (-7.0) 80.1 (-4.1) GPT-3 78.9 (0.0) 66.3 (-12.6) 74.2 (-4.7) 68.4 (-10.5) 68.4 (-10.5) 72.1 (-6.8) 77.4 (-1.6) Headroom 74.7 (0.0) 66.3 (-8.4) 74.2 (-0.5) 68.4 (-6.3) 68.4 (-6.3) 69.5 (-5.2) 69.5 (-5.2) Table 18: Reveal: Predicted summary-level error rates for systems. Quantification bias in parentheses. label GPT-4-turbo GPT-3.5-turbo Bespoke-7B Bespoke-7B (cs=500) MiniCheck-FT5 MiniCheck-Roberta System bing_chat 9.4 6.1 (-3.3) 10.7 (1.2) 10.7 (1.2) 9.8 (0.4) 11.5 (2.0) 13.1 (3.7) neeva 27.3 23.4 (-3.9) 31.2 (3.9) 26.3 (-1.0) 28.6 (1.3) 29.3 (2.0) 34.5 (7.2) perplexity 30.7 19.6 (-11.2) 29.4 (-1.4) 26.6 (-4.1) 26.8 (-3.9) 28.0 (-2.7) 37.2 (6.5) you 31.3 34.3 (3.0) 32.8 (1.5) 28.4 (-3.0) 25.4 (-6.0) 29.9 (-1.5) 47.8 (16.4) Headroom 9.4 (0.0) 6.1 (-3.3) 10.7 (1.2) 10.7 (1.2) 9.8 (0.4) 11.5 (2.0) 13.1 (3.7) · DOI
Cited by
Sorted by citations
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
“Godbole and Jia (2025) re-evaluated these evaluation metrics, and find pitfalls in current factuality metrics.”
- Authors
- Godbole, A., & Jia, R.
- Affiliations
- University of Southern California (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Pages
- 22889-22912
- arXiv
- 2501.14883
- OpenAlex
- W4412887776
- Cited by
- 0
- References
- 1
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2403436)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- factuality evaluation, large language models, natural language generation, summarization, retrieval-augmented generation, question answering, factual accuracy, assessment reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
