Assessing The Factual Accuracy of Generated Text
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
We propose a model-based metric to estimate the factual accuracy of generated text that is complementary to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy). We introduce and release a new large-scale dataset based on Wikipedia and Wikidata to train relation classifiers and end-to-end fact extraction models. The end-to-end models are shown to be able to extract complete sets of facts from datasets with full pages of text. We then analyse multiple models that estimate factual accuracy on a Wikipedia text summarization task, and show their efficacy compared to ROUGE and other model-free variants by conducting a human evaluation study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Thus, relation-based metrics [99] focus on the overlap of relation tuples and introduce a metric that computes the overlap of relation tuples extracted using trained end-to-end fact extraction models.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [99]
The cited source says what the claim attributes to it.
In the citing paper
“In schemabased systems (Goodrich et al., 2019), it is easy to treat F s and F t as atomic facts with constant named entities and predicates, thus computing the F1 score as described above is straightforward.”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Goodrich et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Existing evaluation metrics fail to adequately assess factual consistency and lack interpretability, making diagnosing and mitigating errors difficult.”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Goodrich et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Recently, there has been wide empirical success in text summarization [15, 21, 27], machine translation [1, 36, 39], dialogue response generation [12, 28, 29], and other text generation tasks.”
BackgroundMethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“For evaluation, these models generally rely on metrics like ROUGE (RecallOriented Understudy for Gisting Evaluation) [13], BLEU (Bilingual Evaluation Understudy) [23] and perplexity [3] that measure locally constrained n-gram overlap.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Recently, there has been wide empirical success in text summarization [15, 21, 27], machine translation [1, 36, 39], dialogue response generation [12, 28, 29], and other text generation tasks.”
Background(2004) · Cited by 9,718
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 2×
“ext. We additionally perform detailed analysis of the fact extraction models. Typical fact extraction pipelines are a multistage process consisting of part-of-speech tagging, named entity recognition [5, 7, 9] that produces entities {ei}and then relation classification that predicts a relation rk for every pair of ent…”
MethodInfluentialWikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342 · cited 2×
“l number of facts in the article. 3 DATASET We create a dataset for fact extraction using distant supervision that is based entirely on the English Wikipedia corpus and the Wikidata knowledge baseWKB [37]. Our distant supervisor is very similar to the one proposed by Mintz et al. [17]. Although the inputs and labels f…”
BackgroundMethodIncorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008 · cited 2×
“Entities can be identified with named-entity recognition (NER) systems like Chiu and Nichols [5], Finkel et al. [7], Lample et al. [9] that take in 𝑋 and produce the set { 𝑒 } .”
Background
Cited by
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
“[55] propose a fact extraction model to extract relation tuples (subject, relation, object).”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.03629 · Cited by 4,981
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“Thus, Goodrich et al. (2019) focus on the overlap of relation tuples and introduce a metric that computes the overlap of relation tuples extracted using trained end-to-end fact extraction models.”
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P., & Sun, L. (2025) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 182
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
“Goodrich et al. (2019) proposed a modelbased automatic metric to assess the faithfulness of Wikipedia summaries; they trained an end-to-end model to extract a complete set of OpenIE-style (Banko et al., 2007) facts from both the source text and the generated summary.”
MethodResultEvaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.04226 · Cited by 463
“Specifically, [221] proposed a Transformer-based end-to-end fact extraction model, which enables the structured prediction of relation tuples for factualness assessment.”
MethodEvaluating Large Language Models: A Comprehensive SurveyPreprint
Guo, Z., Jin, R., Chuang, L. I. U., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19736 · Cited by 478
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356
“Goodrich et al. (2019) use information extraction models to measure factual overlap, but facts are restricted to pre-defined schemas.”
MethodDurmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306
Safeguarding large language models: a survey
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11389-2 · Cited by 321
“This can be achieved by the fact-based metrics that are based on entity [33], relation [34], and knowledge F1 metric [35].”
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00453 · Cited by 238
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“Thus, Goodrich et al. (2019) focus on the overlap of relation tuples and introduce a metric that computes the overlap of relation tuples extracted using trained end-to-end fact extraction models.”
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Pagnoni, A., Balachandran, V., & Tsvetkov, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.383 · Cited by 197
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Zhang, Y., Merck, D., Tsai, E., Manning, C. D., & Langlotz, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.458 · Cited by 171
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180
“For example, factual-centred metrics [19, 20, 40, 57] and benchmarks [34, 35, 65] have been proposed to measure and reduce hallucination on specific datasets.”
BackgroundKabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160
“Previous studies show that LLM s can acquire factually incorrect knowledge during training and propagate the incorrect knowledge to generated content [9, 33, 35, 39, 56].”
BackgroundAn Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics
Koh, H. Y., Ju, J., Liu, M., & Pan, S. (2023) · ACM Computing Surveys · doi:10.1145/3545176 · Cited by 112
“Similarly, other proposed models implement factual consistency classiiers by incorporating structured knowledge such as OpenIE triples [39] or dependency arc [40].”
BackgroundReasoning Over Semantic-Level Graph for Fact Checking
Zhong, W., Xu, J., Tang, D., Xu, Z., Duan, N., Zhou, M., Wang, J., & Yin, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.549 · Cited by 151
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3332837 · Cited by 145
“This underlines the need for culturally sensitive and adaptable GPT models and highlights that the evaluation of GPT hallucinations cannot rely solely on algorithmic methods (e.g., [16], [25], [26], [39], [40], [41], [42], [43]).”
BackgroundMethodLogical Natural Language Generation from Open-Domain Tables
Chen, W., Chen, J., Su, Y., Chen, Z., & Wang, W. Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.708 · Cited by 118
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132
Security and Privacy on Generative Data in AIGC: A Survey
Wang, T., Zhang, Y., Qi, S., Zhao, R., Xia, Z., & Weng, J. (2024) · ACM Computing Surveys · doi:10.1145/3703626 · Cited by 156
“On the one hand, adversarial attacks can attack the generative model to prevent it from learning the non-compliant content in the real data, which enhances the compliance of generative data.”
Enhancing Factual Consistency of Abstractive Summarization
Zhu, C., Hinthorn, W., Xu, R., Zeng, Q., Zeng, M., Huang, X., & Jiang, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.58 · Cited by 127
“Several approaches have been proposed to evaluate a summary’s factual consistency (Kry´sci´nski et al., 2019a; Goodrich et al., 2019; Maynez et al., 2020).”
BackgroundMulti-Fact Correction in Abstractive Text Summarization
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J. C. K., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.749 · Cited by 98
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113
“Evaluation approaches that address this gap were recently proposed for tasks like ∗ Work done during an internship at Google Research. machine translation and abstractive summarization (Sellam et al., 2020; Xu et al., 2020; Goodrich et al., 2019).”
BackgroundMethodInfluentialReducing Quantity Hallucinations in Abstractive Summarization
Zhao, Z., Cohen, S. B., & Webber, B. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.203 · Cited by 99
Evaluating Factuality in Generation with Dependency-level Entailment
Goyal, T., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.322 · Cited by 93
Semnani, S. J., Yao, V., Zhang, H., & Lam, M. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.157 · Cited by 146
“Approaches like Qian et al. (2023) and TripleScore (Goodrich et al., 2019) only consider simple entity-relations between retrieved documents and system outputs, while we generalize that notion to all forms of facts.”
BackgroundLi, W., Wu, W., Chen, M., Liu, J., Xiao, X., & Wu, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.05227 · Cited by 99
“To resolve this problem, Goodrich et al. (2019) change to use relation extraction tools with fixed schema, which helps extracted triples easier to compare.”
BackgroundMethodInfluentialA Comprehensive Survey of Abstractive Text Summarization Based on Deep Learning
Zhang, M., Zhou, G., Yu, W., Huang, N., & Liu, W. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/7132226 · Cited by 102
Out-of-Distribution Detection and Selective Generation for Conditional Language ModelsPreprint
Ren, J., Luo, J., Zhao, Y., Krishna, K. S., Saleh, M., Lakshminarayanan, B., & Liu, P. J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15558 · Cited by 148
“Common errors from text generation models include disfluencies (Holtzman et al., 2020) and factual inaccuracies (Goodrich et al., 2019; Maynez et al., 2020).”
BackgroundSelf-citationGO FIGURE: A Meta Evaluation of Factuality in Summarization
Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.42 · Cited by 75
“Recent efforts by NLP researchers have drawn attention to the issue of factual errors and hallucinations in the output of neural summarization models (Cao et al., 2018; Massarelli et al., 2019; Zhao et al., 2020; Falke et al., 2019b; Goodrich et al., 2019; Celikyilmaz et al., 2020).”
BackgroundNeural Deepfake Detection with Factual Structure of Text
Zhong, W., Tang, D., Xu, Z., Wang, R., Duan, N., Zhou, M., Wang, J., & Yin, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.193 · Cited by 46
CoCon: A Self-Supervised Approach for Controlled Text GenerationPreprint
Chan, A., Ong, Y.-S., Pung, B. T. W., Zhang, A., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03535 · Cited by 87
“Finer control over the content of LMs’ outputs can pave the way towards generated texts that are factual [11] or free from inappropriate content [30].”
BackgroundEvaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
Dušek, O., & Kasner, Z. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.19 · Cited by 48
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
Huang, Y., Feng, X., Feng, X., & Qin, B. (2021) · arXiv · doi:10.48550/arxiv.2104.14839 · Cited by 62
“In fixed schema extraction, a relation is predicted Metric Category Summarization Dataset Code Goodrich et al. [2019] Triple-based Wikipedia Falke et al. [2019] Textual-Entailment-based CNN/DM Mishra et al. [2020] Textual-Entailment-based CNN/DM QAGS [Wang et al. , 2020] QA-based CNN/DM, XSum FEQA [Durmus et al. , 2020…”
InfluentialFactual consistency evaluation of summarization in the Era of large language models
Luo, Z., Xie, Q., & Ananiadou, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124456 · Cited by 80
Improving Abstractive Summarization of Legal Rulings through Textual Entailment
Feijo, D. D. V., & Moreira, V. P. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-021-09305-4 · Cited by 59
“Goodrich et al. (2019) proposed using triplets subject-relation-object from a Part-Of-Speech (POS) tagger to check facts between the generated summary and the source document.”
MethodAn Empirical Study of Clinical Note Generation from Doctor-Patient Encounters
Ben Abacha, A., Yim, W.-W., Fan, Y., & Lin, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.168 · Cited by 54
“For instance, commonly used evaluation metrics such as ROUGE do not assess whether summaries are factually consistent with source documents, include critical errors, or lack important information (Goodrich et al., 2019).”
MethodFactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Ribeiro, L., Liu, M., Gurevych, I., Dreyer, M., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.236 · Cited by 98
LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network
Zhong, W., Tang, D., Feng, Z., Duan, N., Zhou, M., Gong, M., Shou, L., Jiang, D., Wang, J., & Yin, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.539 · Cited by 48
Program Enhanced Fact Verification with Verbalization and Graph Attention Network
Yang, X., Nie, F., Feng, Y., Liu, Q., Chen, Z., & Zhu, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.628 · Cited by 61
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation
Xie, Y., Sun, F., Deng, Y., Li, Y., & Ding, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.10 · Cited by 41
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations
Clinciu, M., Eshghi, A., & Hastie, H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.202 · Cited by 85
Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges
Becker, J., Wahle, J. P., Gipp, B., & Ruas, T. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.22258 · Cited by 43
LLMs and Stack Overflow discussions: Reliability, impact, and challenges
Silva, L. D., Samhi, J., & Khomh, F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112541 · Cited by 45
Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation
Mao, Y., Ren, X., Ji, H., & Han, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12723 · Cited by 28
“As ROUGE (Lin, 2004) does not correlate well with factual consistency (Falke et al., 2019), model-based metrics learned by weak supervision (Goodrich et al., 2019; Kryscinski et al., 2019) are proposed to measure factual consistency explicitly.”
BackgroundUpadhyay, R., & Viviani, M. (2025) · Discover Computing · doi:10.1007/s10791-025-09505-5 · Cited by 49
“These systems, in fact, must not only ensure the retrieval of content that aligns with the intended topic, i.e., topical relevance , but also rigorously uphold the principles of reliable and trustworthy information, i.e., factual accuracy [3].”
A Review of Faithfulness Metrics for Hallucination Assessment in Large Language Models
Malin, B., Kalganova, T., & Boulgouris, N. V. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3579203 · Cited by 47
“Though, this can be mitigated somewhat through defined tuple output schema [11].”
InfluentialFAR-ASS: Fact-aware reinforced abstractive sentence summarization
Zhang, M., Zhou, G., Yu, W., & Liu, W. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2020.102478 · Cited by 38
Qian, H., Zhu, Y., Dou, Z., Gu, H., Zhang, X., Liu, Z., Lai, R., Cao, Z., Nie, J., & Wen, J.-R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.04358 · Cited by 53
“QAGS (Wang et al., 2020) and Triple Score (Goodrich et al., 2019) are two metrics for evaluating factualness.”
BackgroundBalachandran, V., Hajishirzi, H., Cohen, W., & Tsvetkov, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.667 · Cited by 30
Utama, P., Bambrick, J., Moosavi, N., & Gurevych, I. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.199 · Cited by 24
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33
Kouris, P., Alexandridis, G., & Stafylopatis, A. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00950-5 · Cited by 38
“Since factual consistency concerns the domain of automatic TS [14, 87, 88, 96], Tables 6 and 7 report the factual consistency ("Evaluation metrics") of the experiments on Gigaword and CNN/DailyMail datasets, respectively.”
BackgroundMethodResultInfluentialMeasuring and Modifying Factual Knowledge in Large Language Models
Pezeshkpour, P. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00122 · Cited by 52
“Furthermore, recent studies (Goodrich et al., 2019; Shuster et al., 2021; Ji et al., 2023) utilize factual knowledge to detect and mitigate hallucination in the generated output of LLMs.”
BackgroundTODSum: Task-Oriented Dialogue Summarization with State TrackingPreprint
Zhao, L., Zheng, F., He, K., Zeng, W., Lei, Y., Jiang, H., Wu, W., Xu, W., Guo, J. Q., & Meng, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12680 · Cited by 39
“They are divided into 4 types: triple-based (Goodrich et al. 2019; Zhang et al. 2020b), textual entailment-based (Falke et al. 2019), QA-based (Wang, Cho, and Lewis 2020; Dur-mus, He, and Diab 2020), and pre-trained classifiers-based (Kryscinski et al. 2020).”
Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection
Ramírez, G., Birch, A., & Titov, I. (2024) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.48550/arxiv.2405.02134 · Cited by 39
“We use NaturalQuestions (Kwiatkowski et al., 2019), that contains real questions from human users; we use Wikifact (Petroni et al., 2019; Goodrich et al., 2019), which consists of a knowledge base completion problem to test factual knowledge; we use bAbI (Weston et al., 2016), that tests language…”
MethodChallenges in Domain-Specific Abstractive Summarization and How to Overcome Them
Afzal, A., Vladika, J., Braun, D., & Matthes, F. (2023) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0011744500003393 · Cited by 34
“The IE-based metrics compare the sets of extracted triples (subject, relation, object) and named entities from both the source document and generated summary to detect hallucination (Goodrich et al., 2019).”
MethodQuestion-aware transformer models for consumer health question summarization
Yadav, S., Gupta, D., Abacha, A. B., & Demner‐Fushman, D. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104040 · Cited by 39
Multi-Document Summarization with Determinantal Point Process Attention
Perez-Beltrachini, L., & Lapata, M. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12522 · Cited by 35
“For the WikiCatSum dataset, we apply the factual accuracy metric defined in Goodrich, Rao, Liu, and Saleh (2019) that compares subject-relation-object triples extracted from the reference ( T set) against those extracted from the generated summary ( G set).”
MethodSaleh, M. E., Wazery, Y. M., & Ali, A. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124153 · Cited by 31
Self-citationFrom Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E. N., Diamantaras, K., & Bratsas, C. (2025) · AI · doi:10.3390/ai6100260 · Cited by 35
Fan, A., & Gardent, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.586 · Cited by 38
A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
Adams, G., Zucker, J. E., & Elhadad, N. (2023) · PubMed · doi:10.48550/arxiv.2303.03948 · Cited by 38
“, 2022), fact-based, reference-free overlap (Goodrich et al., 2019), and those which directly learn from human judgments (Ribeiro et al.”
BackgroundResultTowards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
Liu, T., Zheng, X., Chang, B., & Sui, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17583 · Cited by 21
How Far are We from Robust Long Abstractive Summarization?
Koh, H. Y., Ju, J., Zhang, H., Liu, M. Z., & Pan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.172 · Cited by 47
Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking
Akhtar, M., Schlichtkrull, M., & Vlachos, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.647 · Cited by 30
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Jia, Q., Ren, S., Liu, Y., & Zhu, K. Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.679 · Cited by 36
Integration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 25
CAM: A Large Language Model-based Creative Analogy Mining Framework
Bhavya, B., Xiong, J., & Zhai, C. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3587431 · Cited by 32
“for abstractive summarization [22]), it is unclear how to directly use them for assessing correctness of novel analogies.”
BackgroundTable-based Fact Verification with Self-adaptive Mixture of Experts
Zhou, Y., Liu, X., Zhou, K., & Wu, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.13 · Cited by 18
Reliability Check: An Analysis of GPT-3’s Response to Sensitive Topics and Prompt Wording
Khatun, A., & Brown, D. G. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.8 · Cited by 31
Fact-based Content Weighting for Evaluating Abstractive Summarisation
Xu, X., Dušek, O., Li, J., Rieser, V., & Konstas, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.455 · Cited by 23
“Goodrich et al. (2019) come close to our approach by using entity and relation extraction, but their approach is limited to texts that lend themselves to be represented by RDF triples.”
MethodSadeghi, M., Pöttgen, D., Ebel, P., & Vogelsang, A. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659573 · Cited by 25
“Given the current progress in AI, we expect even more persuasive explanations in the future that may even increase the explanation bias [6, 10, 24, 26, 27, 33, 34].”
BackgroundExploring Decomposition for Table-based Fact Verification
Yang, X., & Zhu, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.90 · Cited by 26
Automated Summarization of Stack Overflow Posts
Kou, B., Chen, M., & Zhang, T.-Y. (2023) · International Conference on Software Engineering · doi:10.1109/icse48619.2023.00158 · Cited by 20
Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization
Dreyer, M., Liu, M., Nan, F., Atluri, S., & Ravi, S. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.156 · Cited by 38
“The research community is developing automatic factuality metrics (Wang et al., 2020; Kryscinski et al., 2020; Goodrich et al., 2019; Goyal and Durrett, 2020; Ribeiro et al., 2022) and methods that attempt to increase factuality (Fan et al., 2018; Scialom et al., 2019; Zhang et al., 2019; Falke et…”
BackgroundCO2Sum:Contrastive Learning for Factual-Consistent Abstractive SummarizationPreprint
Liu, W., Wu, H., Mu, W., Li, Z., Chen, T., & Nie, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01147 · Cited by 20
A Survey of Automatic Hallucination Evaluation on Natural Language GenerationPreprint
Qi, S., Gui, L., He, Y., & Yuan, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.12041 · Cited by 25
“…Evidence SF&WF (§3.3) RefChecker (Hu et al., 2024b), HDM-2 (Paudel et al., 2025) Comparison and Judge (§4) Lexicon-based(§4.1) Exact Match Fact acc (Goodrich et al., 2019), Maskeval (Liu et al., 2022) QG-QA Answer Match FEQA (Durmus et al., 2020), QAGS (Wang et al., 2020) , QuestEval (Scialom et…”
InfluentialA Graph-to-Sequence Learning Framework for Summarizing Opinionated Texts
Wei, P., Zhao, J., & Mao, W. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3071667 · Cited by 21
“models [52], [53], and we leave the improvement of faithfulness in opinionated text summarization to future work.”
BackgroundManakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.4 · Cited by 14
Understanding the Impact of Using Generative AI Tools in a Database Course
Osorio, V. R., Bernuy, A. Z., Simion, B., & Liut, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701785 · Cited by 19
“Previous studies show that GenAI tools can acquire incorrect knowledge during training, hence, generating incorrect content [3, 12], which can be hard to recognize, especially for students who may lack expertise on the topic that they are learning [8, 11].”
Truth or Error? Towards systematic analysis of factual errors in abstractive summaries
Lux, K.-M., Sappelli, M., & Larson, M. (2020) · Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2020.eval4nlp-1.1 · Cited by 14
“Additionally, Goodrich et al. (2019) compare several models such as relation extraction, binary classification and end-to-end models (E2E) for estimating factual accuracy on a Wikipedia text summarization task.”
BackgroundMethodAlambo, A., Banerjee, T., Thirunarayan, K., & Cajita, M. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871798 · Cited by 17
“As extensively discussed in the literature [4], [27], we also argue that lexical measures (i.e., ROUGE) do not fully quantify the factual accuracy of a generated summary while a metric that measures entity-level overlap between a ground truth summary (impression) and a model-generated summary…”
BackgroundEntity-Driven Fact-Aware Abstractive Summarization of Biomedical Literature
Alambo, A., Banerjee, T., Thirunarayan, K., & Raymer, M. L. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956656 · Cited by 17
“show that injecting named entities and entity-aware facts mined from biomedical knowledge bases into abstractive summarization models can boost their performance in terms of entity-level factual consistency [19], [35]–[37], N-gram novelty [30], [38], and semantic equivalence measured using BERTScore [39].”
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2021) · arXiv · doi:10.48550/arxiv.2104.08704 · Cited by 12
“, 2019), information extraction (Zhang et al., 2020; Goodrich et al., 2019) or question answering (Scialom et al.”
BackgroundResource for Error Analysis in Text Simplification: New Taxonomy and Test Collection
Vendeville, B., Магдановна, Е. Л., & Loor, P. D. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730304 · Cited by 15
“Current efforts in evaluating errors in open-ended generation primarily focus on truthfulness with respect to the source document or world knowledge [8, 11].”
InfluentialLanguage Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 11
Give the Truth: Incorporate Semantic Slot into Abstractive Dialogue Summarization
Zhao, L., Zeng, W., Xu, W., & Guo, J. H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.209 · Cited by 13
“They range from using fact triples (Goodrich et al., 2019; Zhang et al., 2020), textual entailment predictions (Falke et al., 2019), adversarially pre-trained classifiers (Kryscinski et al., 2020), to question answering (QA) systems (Wang et al., 2020; Durmus et al., 2020).”
BackgroundA Transformer based approach for Abstractive Text Summarization of Radiology Reports
Balouch, B. A. K., & Hussain, F. (2023) · International Conference on Applied Engineering and Natural Sciences · doi:10.59287/icaens.1042 · Cited by 10
ALS-MRS: Incorporating aspect-level sentiment for abstractive multi-review summarization
Zhao, Q., Niu, J., & Liu, X. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109942 · Cited by 11
Token-Level Fact Correction in Abstractive Summarization
Shin, J., Park, S.-B., & Song, H.-J. (2023) · IEEE Access · doi:10.1109/access.2022.3233854 · Cited by 10
“According to the previous studies [4], [5], [6], [7], about 30% of summaries generated by abstractive summarization models contain factual errors.”
WebRED: Effective Pretraining And Finetuning For Relation Extraction On The WebPreprint
Ormándi, R., Saleh, M., Winter, E., & Rao, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09681 · Cited by 12
“…applications in information extraction, creating or extending knowledge bases, automatically annotating structured information found in text and recently, in evaluating the factual consistency of abstractive text summarization (Goodrich et al., 2019; Kryściński et al., 2019; Zhang et al., 2019).”
BackgroundSelf-citation
Metadata
- Authors
- Goodrich, B., Rao, V., Liu, P. J., & Saleh, M.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2019-07-25
- Pages
- 166-175
- OpenAlex
- W2947681066
- Cited by
- 162
- References
- 49
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 17 · 2023: 28 · 2022: 34 · 2021: 30 · 2020: 24 · 2019: 1
- FWCI
- 8.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- factual accuracy, fact extraction, relation extraction, Wikipedia, Wikidata, text summarization, ROUGE, BLEU, large-scale datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.