Assessing The Factual Accuracy of Generated Text
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
We propose a model-based metric to estimate the factual accuracy of generated text that is complementary to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy). We introduce and release a new large-scale dataset based on Wikipedia and Wikidata to train relation classifiers and end-to-end fact extraction models. The end-to-end models are shown to be able to extract complete sets of facts from datasets with full pages of text. We then analyse multiple models that estimate factual accuracy on a Wikipedia text summarization task, and show their efficacy compared to ROUGE and other model-free variants by conducting a human evaluation study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Thus, relation-based metrics [99] focus on the overlap of relation tuples and introduce a metric that computes the overlap of relation tuples extracted using trained end-to-end fact extraction models.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [99]
The cited source says what the claim attributes to it.
In the citing paper
“In schemabased systems (Goodrich et al., 2019), it is easy to treat F s and F t as atomic facts with constant named entities and predicates, thus computing the F1 score as described above is straightforward.”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Goodrich et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Existing evaluation metrics fail to adequately assess factual consistency and lack interpretability, making diagnosing and mitigating errors difficult.”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Goodrich et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“Recently, there has been wide empirical success in text summarization [15, 21, 27], machine translation [1, 36, 39], dialogue response generation [12, 28, 29], and other text generation tasks.”
BackgroundMethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“For evaluation, these models generally rely on metrics like ROUGE (RecallOriented Understudy for Gisting Evaluation) [13], BLEU (Bilingual Evaluation Understudy) [23] and perplexity [3] that measure locally constrained n-gram overlap.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Recently, there has been wide empirical success in text summarization [15, 21, 27], machine translation [1, 36, 39], dialogue response generation [12, 28, 29], and other text generation tasks.”
Background(2004) · Cited by 9,718
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 2×
“ext. We additionally perform detailed analysis of the fact extraction models. Typical fact extraction pipelines are a multistage process consisting of part-of-speech tagging, named entity recognition [5, 7, 9] that produces entities {ei}and then relation classification that predicts a relation rk for every pair of ent…”
MethodInfluentialWikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342 · cited 2×
“l number of facts in the article. 3 DATASET We create a dataset for fact extraction using distant supervision that is based entirely on the English Wikipedia corpus and the Wikidata knowledge baseWKB [37]. Our distant supervisor is very similar to the one proposed by Mintz et al. [17]. Although the inputs and labels f…”
BackgroundMethodIncorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008 · cited 2×
“Entities can be identified with named-entity recognition (NER) systems like Chiu and Nichols [5], Finkel et al. [7], Lample et al. [9] that take in 𝑋 and produce the set { 𝑒 } .”
Background
Cited by
Items 101–186 of 186
Entity-Based Semantic Adequacy for Data-to-Text Generation
Faille, J., Gatt, A., & Gardent, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.132 · Cited by 16
“In text sum-marisation, Goodrich et al. (2019) compare relation tuples extracted from a ground-truth summary and a generated one using either a Named entity Recognition and a Relation Classifier or an end-to-end Transformer model to extract these tuples.”
MethodFine-grained Factual Consistency Assessment for Abstractive Summarization Models
Zhang, S., Niu, J., & Wei, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.9 · Cited by 12
“The approaches based on fact extraction (Goodrich et al., 2019; Wang et al., 2020; Durmus et al., 2020) evaluate summaries by comparing the critical facts extracted from the original text and the summary.”
BackgroundMethodInfluentialAnalyzing the Abstractiveness-Factuality Tradeoff With Nonlinear Abstractiveness Constraints.Preprint
Dreyer, M., Liu, M., Feng, N., Atluri, S., & Ravi, S. (2021) · arXiv (Cornell University) · Cited by 11
Idhafi, Z., Agustian, S., Yanto, F., & Harahap, N. S. (2023) · Jurnal CoSciTech (Computer Science and Information Technology) · doi:10.37859/coscitech.v4i3.6311 · Cited by 8
Leveraging speaker-aware structure and factual knowledge for faithful dialogue summarization
Zhao, L., Xu, W., Zhang, C., & Guo, J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108550 · Cited by 7
Probing Factually Grounded Content Transfer with Factual Ablation
West, P., Quirk, C., Galley, M., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.294 · Cited by 10
Towards improving coherence and diversity of slogan generation
Jin, Y., Bhatia, A., Wanvarie, D., & Le, P. T. V. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000474 · Cited by 14
Nandwani, Y., Kumar, V., Raghu, D., Joshi, S., & Lastras, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.639 · Cited by 13
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization
Zeng, Z., Chen, J., Xu, W., & Li, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.337 · Cited by 9
“(1) Employ information extraction tools to extract facts and leveraging it by building additional ob-jective (Cao et al., 2018; Goodrich et al., 2019; Zhang et al., 2020a; Zhu et al., 2021).”
MethodEchoGen: Generating Conclusions from Echocardiogram Notes
Tang, L., Kooragayalu, S., Wang, Y., Ding, Y., Durrett, G., Rousseau, J. F., & Peng, Y. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.35 · Cited by 8
“Goodrich et al. (2019) measure the factual consistency as the ratio of overlap between relation triplets under fixed schema extracted from the reference and the generated summary.”
BackgroundZhang, M., Zhou, G., Yu, W., Liu, W., Huang, N., & Yu, Z. (2022) · Neural Computing and Applications · doi:10.1007/s00521-021-06880-0 · Cited by 5
“Some researchers carried out investigations on the fact descriptions of summarization [20–22], and the results showed that about 30% of abstractive summaries suffer from factual errors.”
BackgroundMethodKnowledge Extraction From Texts Based on Wikidata
Shimorina, A., Heinecke, J., & Herledan, F. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.33 · Cited by 6
“We preprocessed 6 existing datasets to adapt them to sentence-based RE (see Table 2): FewRel (Han et al., 2018), T-REx (Elsahar et al., 2018), Do-cRED (Yao et al., 2019), WikiFact (Goodrich et al., 2019), Wiki20m (Han et al., 2020), and WebRED (Ormandi et al., 2021).”
BackgroundMethodInfluentialRERG: Reinforced evidence reasoning with graph neural network for table-based fact verification
Zhao, G., Yang, P., & Yu, Y. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04130-x · Cited by 5
“The proliferation of available online digital contents has been accompanied by an increase of misinformation, resulting in undesired influences among the masses [8, 18, 23].”
BackgroundIs In-hospital Meta-information Useful for Abstractive Discharge Summary Generation?
Ando, K., Komachi, M., Okumura, T., Horiguchi, H., & Matsumoto, Y. (2022) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai57707.2022.00034 · Cited by 5
“Some previous studies on faithfulness, which also analyze the output of summarization, have employed words or entities [36]–[38].”
MethodWhat Makes a Good Summary? Reconsidering the Focus of Automatic Summarization.Preprint
ter Hoeve, M., Kiseleva, J., & de Rijke, M. (2020) · arXiv (Cornell University) · Cited by 5
Expository Text Generation: Imitate, Retrieve, Paraphrase
Balepur, N., Huang, J., & Chang, K. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.729 · Cited by 4
Adapting LLMs for Satellite Communications: Methodology, Challenges, and Impact
Mozo, A., Gálvez, S., Christou, I. T., Vogiatzis, D., Navarro, T., & Valverde, F. L. (2025) · IEEE Access · doi:10.1109/access.2025.3605022 · Cited by 5
Trustworthiness of large language models: hallucinations
Brunello, N. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-33592-1.00007-3 · Cited by 5
Revisiting computer authorship: a longitudinal perspective
Henrickson, L., & Leppänen, L. (2026) · AI & Society · doi:10.1007/s00146-025-02783-z · Cited by 4
“These are issues that scholars have been considering through reflection (Pennock 2024), proposals for evaluative metrics (Goodrich et al. 2019), and empirical studies about human trust in AI systems (Glikson and Williams Woolley 2020).”
A Survey of Factual Consistency in Summarization from 2021 to 2023
Li, Y., Hao, X., & Li, L. (2023) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait59945.2023.10468958 · Cited by 5
“Previous works typically obtain factual consistency scores by comparing the relational triples between documents and summaries [32], [16].”
BackgroundAsynchronous Event Driven Brain Teaser Using Node.js
Subbulakshmi, N., Begum, S. A., Vignesh, R. V., & Chandru, R. (2024) · International Conference on Electronics and Sustainable Communication Systems (ICESC) · doi:10.1109/icesc60852.2024.10689905 · Cited by 4
“• Model -Specific Hallucinations: These hallucinations, which are directly linked to specific models or architectures, can manifest in Transformer-based models such as GPT as repetitive patterns, unnecessary outputs, or overuse of specific phrases [11], [3], [12].”
BackgroundPost-Editing Extractive Summaries by Definiteness Prediction
Kabbara, J., & Cheung, J. C. K. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.312 · Cited by 5
“Compared to full-fledged abstractive summarization, limited post-editing is less likely to lead to problems of factual correctness and consistency, which are a known issue of existing abstractive systems (Cao et al., 2018; Goodrich et al., 2019; Kryściński et al., 2019).”
BackgroundEvaluating Factual Consistency of Texts with Semantic Role Labeling
Fan, J., Aumiller, D., & Gertz, M. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.9 · Cited by 16
Liu, R., Zhang, Y., Yang, B., Shi, Q., & Tian, L. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103853 · Cited by 5
What Makes a Good and Useful Summary? Incorporating Users in Automatic Summarization Research
ter Hoeve, M., Kiseleva, J., & de Rijke, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.4 · Cited by 12
“Figure 1: Example of most current summarization techniques vs. summarization while incorporating the users in the process. and Passonneau, 2004; Paulus et al., 2018; Narayan et al., 2018b; Goodrich et al., 2019; Wang et al., 2020; Xie et al., 2021).”
MethodImproving named entity correctness of abstractive summarization by generative negative sampling
Chen, Z., & Lin, H. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101504 · Cited by 4
CoP: Factual Inconsistency Detection by Controlling the Preference
She, S., Geng, X., Huang, S., & Chen, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26589 · Cited by 5
Fact Checking Machine Generated Text with Dependency Trees
Estes, A., Vedula, N., Collins, M., Cecil, M., & Rokhlenko, O. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.46 · Cited by 5
“First, the input whose factuality is to be determined can be unstructured , as in abstrac-tive summarization (Teredesai et al., 2019; Goyal and Durrett, 2020) or the FEVER task (Thorne et al., 2018; Zhou et al., 2019; Zhang et al., 2020; Bekoulis et al., 2021; Vedula and Parthasarathy, 2021).”
BackgroundQaDialMoE: Question-answering Dialogue based Fact Verification with Mixture of Experts
Wang, L., Zhang, P., Lu, X., Zhang, L., Yan, C., & Zhang, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.229 · Cited by 6
“…of documents, is an important NLP area (Cohen et al., 2011) and has been explored to various applications such as detecting fake news, rumor, and deceptive opinions (Rashkin et al., 2017; Thorne et al., 2018; Goodrich et al., 2019; Vaibhav et al., 2019; Chen et al., 2020; Kryscinski et al., 2020).”
Information Retrieval in the Age of Generative AI: The RGB Model
Garetto, M., Cornacchia, A., Galante, F., Leonardi, E., Nordio, A., & Tarable, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730008 · Cited by 9
Kuculo, T., Abdollahi, S., & Gottschalk, S. (2025) · Semantic Web · doi:10.1177/22104968251363759 · Cited by 10
“…datasets without human annotations such as DBpedia-SEE and Wikidata-SEE for semantic EE or datasets for relation extraction (Elsahar et al., 2018; Goodrich et al., 2019; Han et al., 2020; Yao et al., 2019) overcomes the issues of training data dimensionality but always comes with questions…”
Using Semi-Supervised Learning and Wikipedia to Train an Event Argument Extraction System
Zajec, P., & Mladenić, D. (2022) · Informatica · doi:10.31449/inf.v46i1.3577 · Cited by 3
Niharika, P., & Thangaraj, S. J. J. (2022) · International Conference on Contemporary Computing and Informatics (IC3I) · doi:10.1109/ic3i56241.2022.10072802 · Cited by 3
“The discovery that a model trained to create Wikipedia articles also learned to translate names between languages was more inspiring to our effort (Ong et al. 2021; Goodrich et al. 2019) .”
BackgroundLi, S., & Xu, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191247 · Cited by 3
“For instance, Goodrich et al. [14] employ information extraction to extract triples from source documents and the generated summaries, and then they take the overlap of the triples as the measure of factual consistency.”
BackgroundXia, S.-J., Zhang, H., Jiang, Y., Chen, X., Chen, Y., Li, Z., Wan, Z., & Sun, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5731427 · Cited by 4
Cheng, Y., & Choi, I. (2025) · KNOWLEDGE ORGANIZATION · doi:10.31083/ko39046 · Cited by 2
MIRAGE: A Metrics lIbrary for Rating hAllucinations in Generated tExt
Vendeville, B., Магдановна, Е. Л., Loor, P. D., & Kamps, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761644 · Cited by 2
“FactAcc [10] takes a distinct approach by extracting relation tuples (subject, relation, object) from both source and generated text, then evaluating entailment between extracted facts.”
Detection of Hallucinations Based on the Internal States of Large Language Models
Aisin, T. R., & Shamardina, T. V. (2025) · Automatic Documentation and Mathematical Linguistics · doi:10.26907/1562-5419-2025-28-6-1282-1305 · Cited by 4
Faber, W., Bootsma, R. E., Huibers, T., van Dulmen, S., & Brinkkemper, S. (2024) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0012422300003657 · Cited by 2
MTAS: A Reference-Free Approach for Evaluating Abstractive Summarization Systems
Zhu, X., Jiang, M., Zhang, X., Nie, L., & Ding, Z. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660820 · Cited by 3
“It is further revealed that existing metrics ignore the factual nature of summaries and therefore cannot measure the factual consistency between summaries [14, 47].”
Amendola, M., Passarella, A., & Perego, R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3793531 · Cited by 2
“While they provide quick and often relevant answers, they are prone to hallucinations (confident but incorrect outputs) and can sometimes generate misleading or harmful content [12, 18, 30].”
Optimizing Factual Accuracy in Text Generation through Dynamic Knowledge SelectionPreprint
Qian, H., Dou, Z., Tan, J., Chen, H., Gu, H., Lai, R., Zhang, X., Cao, Z., & Wen, J.-R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.15711 · Cited by 7
“4 To evaluate the factuality, we use TripleScore [65] and FactScore [66].”
MethodOk, C., Lee, E., Ju, H., Lee, J., & Oh, D. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115961 · Cited by 1
Reasoning Capabilities and Invariability of Large Language Models
Raganato, A., Peñaloza, R., Viviani, M., & Pasi, G. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00025 · Cited by 2
“A recent example considering the Semantic Web is [21], but other evaluations have been made in the past, focusing also on the accuracy and veracity of the provided answers [22], [23].”
MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA
Srivastava, N., Vollmers, D., Speck, R., Karalis, N., Zahera, H. M., & Ngomo, A.-C. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41614-8_9 · Cited by 1
“Large language models (LLMs) have achieved strong performance across diverse language and knowledge-intensive tasks [19,53,54], but remain prone to hallucinations [18,21], factual inconsistency [15], and multi-hop reasoning failures [25].”
Analysis and Research of Abstractive Automatic Summarization Based on Sequential Facts
Liu, Y., Li, Y., & Li, L. (2021) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc54101.2021.9660463 · Cited by 1
FalCon: A Faithful Contrastive Framework for Response Generation in TableQA Systems
Fang, S., Chen, J., Shen, X., Chen, Y., & Xiao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-00129-1_13 · Cited by 2
“While advanced Natural Language Generation (NLG) techniques have been successful in producing fluent text, these methods still face great challenge caused by factual incorrectness [11,17].”
BackgroundImproving Neural Abstractive Summarization with Reliable Sentence Sampling
Chernyshev, D., & Dobrov, B. V. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-12285-9_16 · Cited by 1
Method for Evaluating Quality of Automatically Generated Product Descriptions
Fukumoto, K., Takeuchi, R., & Nadamoto, A. (2022) · International Symposium on Information and Communication Technology · doi:10.1145/3568562.3568583 · Cited by 1
“Goodrich et al.[6] describe a method for evaluating the accuracy of generated text contents.”
MethodClusterVote: Automatic Summarization Dataset Construction with Document Clusters
Chernyshev, D., & Dobrov, B. V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20980-2_10 · Cited by 1
Evaluation Axes for Automatically Generated Product Descriptions
Fukumoto, K., Takeuchi, R., Terada, H., Bato, M., & Nadamoto, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21047-1_42 · Cited by 1
Implementing Natural Language Generation through industry-specific chatbots
Kapoor, A., & Shetty, S. D. (2023) · Soft Computing Research Society eBooks · doi:10.56155/978-81-955020-2-8-6 · Cited by 1
Content Richness Evaluation Method For Abstractive Summarization
Zhang, X., Hu, D., Li, B., Qin, Y. X., & Li, L. (2023) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis59572.2023.10263021 · Cited by 1
Guo, B., Feng, C., Liu, F., Li, X., & Wang, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7894-6_11 · Cited by 2
Prog-TAPAS: Enabling Table and Program Representation Consistency for Fact Verification
Wang, S., Zhang, C., & Liu, T. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650908 · Cited by 1
Enhancing Generative Models for Scientific Text Simplification
Vendeville, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_39 · Cited by 1
Generating Coherent and Diverse Slogans with Sequence-to-Sequence Transformer.Preprint
Jin, Y., Bhatia, A., Wanvarie, D., & Le, P. T. V. (2021) · arXiv (Cornell University) · Cited by 1
AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
Aghaebrahimian, A. (2025) · arXiv · doi:10.48550/arxiv.2512.03634
“Decomposing text into constituents such as entities (Lee et al., 2022) or triples (subject, predicate, object) is exercised in several other works (Goodrich et al., 2019; Thorne et al., 2018; Min et al., 2023) as a direct and model-free alternative to validate factual consistency of generated texts.”
InfluentialZhang, H., Anjum, S., Fan, H., Zheng, W., Huang, Y., & Feng, Y. (2025) · arXiv · doi:10.48550/arxiv.2503.16541
“Entity-based hallucination metrics [13] and relation-based fact extraction models [4] provide more structured approaches.”
Few Shot Learning for Information VerificationPreprint
Khalid, U., & Beg, M. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10956
“This can also be combined with sequence to sequence generation and relation classification which can be trained to produce structured tuples [29].”
BackgroundEisenstadt, R., & Elhadad, M. (2021) · Workshop on Understanding Implicit and Underspecified Language (UnImplicit) · doi:10.18653/v1/2021.unimplicit-1.3
“Goodrich et al. (2019) attempts to measure factuality by extracting data from the generated text using OpenIE techniques and comparing it with the original data.”
MethodAccessing Diverse Web Knowledge with Natural Language Interface
chen, W. (2021) · eScholarship (California Digital Library)
Research and application of matching network
Jiang, C., Mo, J., & Pan, Z. (2022) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3579654.3579775
“3579775 widely applied in real life, such as QA [15], dialogue system [2], sentiment analysis [23], text mining [11], text summarization [10], etc.”
BackgroundChernyshev, D., & Dobrov, B. V. (2023) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661823030070
“The accuracy of the presented facts is the proportion of the triplets <subject–relation–object> extracted from the summary which are supported by the source text: is the main way to calculate the factual accuracy in summarization [7].”
BackgroundTell Me the Truth: A System to Measure the Trustworthiness of Large Language ModelsPreprint
Lipizzi, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4758300
Generating Factual Text via Entailment Recognition Task
Dai, J., Cheng, P., & Liu, J. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.051745
Learning Latent Variable for Logical Reasoning in Table-Based Fact Verification
Wu, J., & Hou, M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651044
Entity hallucination index for text summarization models
Katwe, P., Balabantaray, R. C., & Vittala, K. P. (2025) · Applied Computing and Informatics · doi:10.1108/aci-09-2024-0371 · Cited by 2
Ashour, S. (2025) · The political economy of the Middle East/The political economy of the Middle East · doi:10.1007/978-981-96-7963-8_4
Escudero-Arnanz, Ó., Valero-Méndez, M. E., Sánchez-Ramos, N., & Soguero-Ruíz, C. (2026) · JMIR AI · doi:10.2196/83206
“…assessed multiple dimensions, including content accuracy, defined as the degree of alignment with scientific evidence, contextual relevance, referring to the correct use of retrieved information, and factual consistency, which required the absence of unsupported claims or hallucinations [22].”
How Faithful Are Your Summaries? A Study of NLI-Based Verification in Portuguese
Paula, F. S. F., Westhelle, M., Corrêa, M. C. M., Bencke, L. R., & Moreira, V. P. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37834
“There were approaches based on information extraction that checked the presence of similar information pieces in summaries and source texts [Cao et al. 2018, Goodrich et al. 2019, Nan et al. 2021].”
Lipizzi, C. (2025) · doi:10.35199/dsm2025.16 · Cited by 1
Vendeville, B., Магдановна, Е. Л., & Loor, P. D. (2024) · HAL · doi:10.24348/coria.2024.position_31
Transforming Free-Form Sentences into Sequence of Unambiguous Sentences with Large Language Model
Yeole, N. K. (2024) · VTechWorks (Virginia Tech) · Cited by 2
Une revue sur les hallucinations des LLM
Metheniti, E., Bhar, S., & Asher, N. M. (2025) · HAL · Cited by 2
Pelgrom, N., & Grahn, H. (2025) · DiVA (Blekinge Institute of Technology)
Zhao, W. X., Zhou, K., Li, J., Tang, T., & Wen, J.-R. (2025) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-981-96-6259-3_12
Vo, H., Tran, N., Vo, K., Tran-Truong, P. T., & Ha, S. (2026) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c68850.2026.00020
“Traditional text similarity metrics such as ROUGE and BLEU exhibit fundamental limitations when applied to technical domains [22], [23] - a design decision may achieve high textual similarity while violating core architectural principles.”
Text Generation with and without Retrieval
Fan, H. (2021) · theses.fr (ABES) · doi:10.70675/87ca97d1z017cz478azac26z38a64f0a75e0
Data-Based Natural Language Generation : Evaluation and Explainability
Faille, J. (2023) · theses.fr (ABES) · doi:10.70675/3e73dd43z7aa6z4405za5cez27c78b408a3c
Contributions to evaluating and improving the faithfulness for text generation
Zhang, K. (2025) · theses.fr (ABES) · doi:10.70675/28453a20z78f2z4fa2za6edz700065e69e97
Fact-Aware Text Generation for Low-Resource Languages: A Systematic Survey with Emphasis on Marathi
Ghorpade, M., & Patankar, S. (2026) · IEEE International Conference on AI Engineering and Innovations (AIEI) · doi:10.1109/aiei69164.2026.11497956
“In dynamic knowledge selection strategies [22] and factuality evaluation literature [36] post-generation validation is put into focus.”
LLM-SocRec: Enhancing Graph-based Social Recommendation via Collaborative Large Language Models
Li, Z., Tan, Z., Li, W., & Mo, T. (2026) · Machine Learning · doi:10.1007/s10994-026-07063-5 · Cited by 1
“We can observe that most models show some decline in understanding general tasks after fine-tuning, but LLM-SocRec exhibits the smallest decrease and even shows no decline on the WikiFact dataset.”
Sakib, A. N., Chisty, M. M. O., & Zhang, Z. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812342
Consistent and efficient long document understanding
Zeng, Q. (2023) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 16
Uribe, G. A., Pardo, J. D. D., Sanchez, C. A. S., Florez, D., Haslbeck, F., McFayden, J. A., Rummler, A., & Maire, G. (2026) · Journal of Instrumentation · doi:10.1088/1748-0221/21/11/p11010
Metadata
- Authors
- Goodrich, B., Rao, V., Liu, P. J., & Saleh, M.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2019-07-25
- Pages
- 166-175
- OpenAlex
- W2947681066
- Cited by
- 162
- References
- 49
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 17 · 2023: 28 · 2022: 34 · 2021: 30 · 2020: 24 · 2019: 1
- FWCI
- 8.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- factual accuracy, fact extraction, relation extraction, Wikipedia, Wikidata, text summarization, ROUGE, BLEU, large-scale datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.