Annotating and Modeling Fine-grained Factuality in Summarization
Open accessAbstract
Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors.While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current techniques are succeeding and failing.We explore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level.Our observations are threefold.First, exhibited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSUM.Second, human-labeled data with fine-grained annotations provides a more effective training signal than sentence-level annotations or synthetic data.Finally, we show that our best factuality detection model enables training of more factual XSUM summarization models by allowing us to identify non-factual tokens in the training data. 1Reference Summary: An early-medieval gold pendant created from an imitation of a Byzantine coin that was found in a Norfolk field is a "rare find", a museum expert has said. Source Article Fragment:Discovered on land at North Elmham, near Dereham, the circa 600 AD coin was created by French rulers of the time to increase their available currency.[…] The pendant was declared treasure by the Norfolk coroner on Wednesday.An 18th century coin believed to be worth more than #1m has been discovered.A gold pendant created from a necklace was found in a field Entitycentric (Ent-C)The pendant was declared a treasure by the Norfolk coroner on Wednesday.The pendant was declared a treasure by the Ohio coroner on March.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most prior hallucination detection benchmarks concentrated on task-specific hallucinations.
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [102]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · see citing works
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · see citing works
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
Information Fusion in the Context of Multi-Document Summarization
Barzilay, R., McKeown, K., & Elhadad, M. (1999) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/1034678.1034760 · Cited by 485 · see citing works
Posterior Regularization for Structured Latent Variable Models
Ganchev, K., Graça, J., Gillenwater, J., & Taskar, B. (2010) · Cited by 443 · see citing works
Statistics-Based Summarization - Step One: Sentence Compression
Knight, K. K., & Marcu, D. (2000) · Cited by 392 · see citing works
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Cao, Z., Wei, F., Li, W., & Li, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11912 · Cited by 359 · see citing works
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356 · see citing works
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. X. (2020) · International Conference on Machine Learning · Cited by 325 · see citing works
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306 · see citing works
Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1213 · Cited by 265 · see citing works
Jointly Learning to Extract and Compress
Berg-Kirkpatrick, T., Gillick, D., & Klein, D. (2011) · Cited by 225 · see citing works
Guiding Semi-Supervision with Constraint-Driven Learning
Chang, M., Ratinov, L., & Roth, D. (2007) · Cited by 192 · see citing works
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Zhang, Y., Merck, D., Tsai, E., Manning, C. D., & Langlotz, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.458 · Cited by 171 · see citing works
Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints
Durrett, G., Berg-Kirkpatrick, T., & Klein, D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1188 · Cited by 150 · see citing works
Self-citationMultiple Aspect Summarization Using Integer Linear Programming
Woodsend, K., & Lapata, M. (2012) · Cited by 112 · see citing works
Multi-Fact Correction in Abstractive Text Summarization
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J. C. K., & Liu, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.749 · Cited by 105 · see citing works
Neural Syntactic Preordering for Controlled Paraphrase Generation
Goyal, T., & Durrett, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.22 · Cited by 94 · see citing works
Self-citationEvaluating Factuality in Generation with Dependency-level Entailment
Goyal, T., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.322 · Cited by 93 · see citing works
Self-citationNon-expert evaluation of summarization systems is risky
Gillick, D., & Liu, Y. (2010) · Cited by 71 · see citing works
Improved Natural Language Generation via Loss Truncation
Kang, D. D., & Hashimoto, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.66 · Cited by 71 · see citing works
Extractive vs. NLG-based abstractive summarization of evaluative text
Carenini, G., & Cheung, J. C. K. (2008) · doi:10.3115/1708322.1708330 · Cited by 69 · see citing works
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
Tian, R., Narayan, S., Sellam, T., & Parikh, A. P. (2019) · arXiv · doi:10.48550/arxiv.1910.08684 · Cited by 60 · see citing works
Controlling the Amount of Verbatim Copying in Abstractive Summarization
Song, K., Wang, B., Feng, Z., Liu, R., & Liu, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6420 · Cited by 48 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · arXiv · doi:10.48550/arxiv.1808.08745 · Cited by 39 · see citing works
Analyzing Sentence Fusion in Abstractive Summarization
Lebanoff, L., Muchovej, J., Dernoncourt, F., Kim, D. S., Kim, S., Chang, W. H., & Liu, F. (2019) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/d19-5413 · Cited by 38 · see citing works
Joint Parsing and Generation for Abstractive Summarization
Song, K., Lebanoff, L., Guo, Q., Qiu, X., Xue, X., Li, C., Yu, D., & Liu, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6419 · Cited by 29 · see citing works
On the Abstractiveness of Neural Document Summarization
Zhang, F., Yao, J.-G., & Yan, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1089 · Cited by 22 · see citing works
Generating Fact Checking Explanations
Atanasova, P. (2020) · Springer eBooks · doi:10.1007/978-3-031-51518-7_4 · Cited by 14 · see citing works
Truth or Error? Towards systematic analysis of factual errors in abstractive summaries
Lux, K.-M., Sappelli, M., & Larson, M. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.1 · Cited by 14 · see citing works
Factual Error Correction for Abstractive Summarization ModelsPreprint
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08712 · Cited by 12 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 9
- Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Asso- ciation for Computational Linguistics (TACL).
- Tanya Goyal and Greg Durrett. 2020a. Evaluating fac- tuality in generation with dependency-level entail- ment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3592-3603.
- Yuning Mao, Xiang Ren, Heng Ji, and Jiawei Han. 2020. Constrained Abstractive Summarization: Pre- serving Factual Consistency with Constrained Gen- eration. In arXiv preprint 2010.12723. · arXiv
- Ani Nenkova and Rebecca Passonneau. 2004. Evaluat- ing content selection in summarization: The pyra- mid method. In Proceedings of the Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
- Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neu- ral semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 6859-6866.
- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, R'emi Louf, Morgan Funtow- icz, and Jamie Brew. 2019. Huggingface's trans- formers: State-of-the-art natural language process- ing. ArXiv, abs/1910.03771. · arXiv
- Kristian Woodsend and Mirella Lapata. 2012. Multi- ple Aspect Summarization Using Integer Linear Pro- gramming. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Process- ing (EMNLP) and Computational Natural Language Learning (CoNLL).
- Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive sum- marization. In Findings of EMNLP.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
“…hallucinations, such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al., 2022), data-to-text(Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al.,…”
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“…hallucinations, such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021; Tang et al., 2022), data-to-text(Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al.,…”
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141 · see citing works
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.532 · Cited by 135 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection
Chen, S., Zhang, F., Sone, K., & Roth, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.475 · Cited by 83 · see citing works
Neural natural language processing for long texts: A survey on classification and summarization
Tsirmpas, D., Gkionis, I., Papadopoulos, G. T., & Mademlis, I. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108231 · Cited by 73 · see citing works
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 68 · see citing works
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Wan, D., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.74 · Cited by 58 · see citing works
Prompted Opinion Summarization with GPT-3.5
Bhaskar, A., Fabbri, A. R., & Durrett, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.591 · Cited by 59 · see citing works
Self-citationUnderstanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
Tang, L., Goyal, T., Fabbri, A. R., Laban, P., Xu, J., Yavuz, S., Kryściński, W., Rousseau, J. F., & Durrett, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.650 · Cited by 62 · see citing works
Self-citationLadhak, F., Durmus, E., He, H., Cardie, C., & McKeown, K. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.100 · Cited by 46 · see citing works
Evaluating Factuality in Text Simplification
Devaraj, A., Sheffield, W., Wallace, B., & Li, J. J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.506 · Cited by 40 · see citing works
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Ribeiro, L., Liu, M., Gurevych, I., Dreyer, M., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.236 · Cited by 39 · see citing works
Balachandran, V., Hajishirzi, H., Cohen, W. W., & Tsvetkov, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.667 · Cited by 32 · see citing works
Beyond the Chat: Executable and Verifiable Text-Editing with LLMs
Laban, P., Vig, J., Hearst, M., Xiong, C., & Wu, C.-S. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676419 · Cited by 31 · see citing works
Faithfulness-Aware Decoding Strategies for Abstractive Summarization
Wan, D., Liu, M., McKeown, K. R., Dreyer, M., & Bansal, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.210 · Cited by 35 · see citing works
Learning to Revise References for Faithful Summarization
Adams, G., Shing, H.-C., Sun, Q., Winestock, C., McKeown, K. R., & Elhadad, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.296 · Cited by 30 · see citing works
Multilingual Summarization with Factual Consistency Evaluation
Aharoni, R., Narayan, S., Maynez, J., Herzig, J., Clark, E., & Lapata, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.220 · Cited by 36 · see citing works
WiCE: Real-world entailment for claims in Wikipedia
Kamoi, R., Goyal, T., Rodriguez, J., & Durrett, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.470 · Cited by 26 · see citing works
Self-citationImproving the Faithfulness of Abstractive Summarization via Entity Coverage Control
Zhang, H., Yavuz, S., Kryściński, W., Hashimoto, K., & Zhou, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.40 · Cited by 28 · see citing works
Utama, P. A., Bambrick, J., Moosavi, N. S., & Gurevych, I. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.199 · Cited by 28 · see citing works
Training Dynamics for Text Summarization Models
Goyal, T., Xu, J., Li, J. J., & Durrett, G. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.163 · Cited by 24 · see citing works
Self-citationOpportunities and Risks of LLMs for Scalable Deliberation with Polis
Small, C. T., Vendrov, I., Durmus, E., Homaei, H., Barry, E., Cornebise, J., Suzman, T., Ganguli, D., & Megill, C. (2023) · arXiv · doi:10.48550/arxiv.2306.11932 · Cited by 20 · see citing works
“Prior work has proposed automated metrics to detect such errors (Maynez et al., 2020; Durmus et al., 2020; Kryscinski et al., 2020; Goyal and Durrett, 2021); however, they do not always correlate well with human judgements (Fabbri et al., 2021; Durmus et al., 2022).”
BackgroundToward Unifying Text Segmentation and Long Document Summarization
Cho, S., Song, K., Wang, X., Liu, F., & Yu, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.8 · Cited by 25 · see citing works
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Chen, Y., Liu, P., & Qiu, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.179 · Cited by 18 · see citing works
Li, S., & Xu, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103467 · Cited by 18 · see citing works
Zhang, S., Wan, D., & Bansal, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.120 · Cited by 24 · see citing works
How Far are We from Robust Long Abstractive Summarization?
Koh, H. Y., Ju, J., Zhang, H., Liu, M., & Pan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.172 · Cited by 18 · see citing works
Detecting and Mitigating Hallucinations in Multilingual Summarisation
Qiu, Y., Ziser, Y., Korhonen, A., Ponti, E. M., & Cohen, S. B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.551 · Cited by 21 · see citing works
DialSummEval: Revisiting Summarization Evaluation for Dialogues
Gao, M., & Wan, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.418 · Cited by 17 · see citing works
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
Joseph, S., Chen, L., Trienes, J., Göke, H., Coers, M., Xu, W., Wallace, B., & Li, J. J. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.459 · Cited by 17 · see citing works
Analyzing and Evaluating Faithfulness in Dialogue Summarization
Wang, B., Zhang, C., Zhang, Y., Chen, Y., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.325 · Cited by 17 · see citing works
Improving Factual Consistency in Summarization with Compression-Based Post-Editing
Fabbri, A. R., Choubey, P. K., Vig, J., Wu, C.-S., & Xiong, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.623 · Cited by 19 · see citing works
Dong, Y., Wieting, J., & Verga, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.76 · Cited by 16 · see citing works
Spurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16 · see citing works
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization
Choubey, P. K., Fabbri, A. R., Vig, J., Wu, C.-S., Liu, W., & Rajani, N. F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.685 · Cited by 23 · see citing works
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
Jain, S., Keshava, V., Mysore Sathyendra, S., Fernandes, P., Liu, P., Neubig, G., & Zhou, C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.537 · Cited by 15 · see citing works
Improving Factuality of Abstractive Summarization via Contrastive Reward Learning
Chern, I.-C., Wang, Z., Das, S. K., Sharma, B., Liu, P., & Neubig, G. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.6 · Cited by 18 · see citing works
X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization
Chaudhury, S., Swaminathan, S., Gunasekara, C., Crouse, M., Ravishankar, S., Kimura, D., Murugesan, K., Astudillo, R. F., Naseem, T., Kapanipathi, P., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.478 · Cited by 14 · see citing works
FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Feng, S., Balachandran, V., Bai, Y., & Tsvetkov, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.59 · Cited by 21 · see citing works
Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges
Ramprasad, S., Mcinerney, J., Marshall, I. J., & Wallace, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.27 · Cited by 12 · see citing works
On Improving Summarization Factual Consistency from Natural Language Feedback
Liu, Y., Deb, B., Teruel, M., Halfaker, A., Radev, D., & Awadallah, A. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.844 · Cited by 17 · see citing works
R2D2: Robust Data-to-Text with Replacement Detection
Nan, L., Flores, L. J., Zhao, Y., Liu, Y., Benson, L., Zou, W., & Radev, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.464 · Cited by 14 · see citing works
Fact-Driven Abstractive Summarization by Utilizing Multi-Granular Multi-Relational Knowledge
Mao, Q., Li, J., Peng, H. M., He, S., Wang, L., Yu, P. S., & Wang, Z. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3161157 · Cited by 11 · see citing works
A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
Adams, G., Zucker, J. E., & Elhadad, N. (2023) · PubMed · doi:10.48550/arxiv.2303.03948 · Cited by 15 · see citing works
Truth-Conditional Captions for Time Series Data
Jhamtani, H., & Berg-Kirkpatrick, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.55 · Cited by 10 · see citing works
Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality
Dixit, T., Wang, F., & Chen, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.78 · Cited by 12 · see citing works
Factual Error Correction for Abstractive Summaries Using Entity Retrieval
Lee, H., Park, C., Yoon, S., Bui, T., Dernoncourt, F., Kim, J., & Jung, K. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.41 · Cited by 10 · see citing works
Learning with Rejection for Abstractive Text Summarization
Cao, M., Dong, Y., He, J., & Cheung, J. C. K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.663 · Cited by 10 · see citing works
Improving abstractive summarization with energy-based re-ranking
Pernes, D., Mendes, A., & Martins, A. F. T. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.1 · Cited by 9 · see citing works
Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution
Xu, J., & Durrett, G. (2021) · doi:10.18653/v1/2021.acl-long.539 · Cited by 11 · see citing works
Self-citationEvaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization
Dreyer, M., Liu, M., Nan, F., Atluri, S., & Ravi, S. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.156 · Cited by 12 · see citing works
Fact Verification in Knowledge Graphs Using LLMs
Shami, F., Marchesin, S., & Silvello, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730142 · Cited by 8 · see citing works
Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization
Chan, H. P., Zeng, Q., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.402 · Cited by 8 · see citing works
Adams, G., Nguyen, B. H., Smith, J. A., Xia, Y., Xie, S., Ostropolets, A., Deb, B., Chen, Y.-J., Naumann, T., & Elhadad, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.587 · Cited by 9 · see citing works
HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models
Goyal, T., Rajani, N. F., Liu, W., & Kryściński, W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.30 · Cited by 13 · see citing works
Self-citationLeveraging speaker-aware structure and factual knowledge for faithful dialogue summarization
Zhao, L., Xu, W., Zhang, C., & Guo, J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108550 · Cited by 7 · see citing works
Lee, H., Yoo, K. M., Park, J., Lee, H., & Jung, K. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.76 · Cited by 7 · see citing works
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., & Yih, S. W.-T. (2023) · arXiv · doi:10.48550/arxiv.2305.14739 · Cited by 6 · see citing works
“, 2020) and removing noisy training samples (Kang and Hashimoto, 2020; Goyal and Durrett, 2021).”
BackgroundNonFactS: NonFactual Summary Generation for Factuality Evaluation in Document Summarization
Soleimani, A., Monz, C., & Worring, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.400 · Cited by 6 · see citing works
Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
Guo, Y., Clavel, C., Eddine, M. K., & Vazirgiannis, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.386 · Cited by 6 · see citing works
Evaluating and Improving Factuality in Multimodal Abstractive Summarization
Wan, D., & Bansal, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.654 · Cited by 8 · see citing works
Massive-scale Decoding for Text Generation using Lattices
Xu, J., Jonnalagadda, S., & Durrett, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.344 · Cited by 5 · see citing works
Self-citationEchoGen: Generating Conclusions from Echocardiogram Notes
Tang, L., Kooragayalu, S., Wang, Y., Ding, Y., Durrett, G., Rousseau, J. F., & Peng, Y. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.35 · Cited by 5 · see citing works
Self-citationIt Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance
Subramonian, A., Yuan, X., Daumé, H., & Blodgett, S. L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.202 · Cited by 23 · see citing works
Improving Faithfulness by Augmenting Negative Summaries from Fake Documents
Wang, T., Ladhak, F., Durmus, E., & He, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.816 · Cited by 5 · see citing works
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
Kamoi, R., Goyal, T., & Durrett, G. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.11 · Cited by 5 · see citing works
Self-citationCan LMs Generalize to Future Data? An Empirical Analysis on Text Summarization
Cheang, C., Chan, H. P., Wong, D. F., Liu, X., Li, Z., Sun, Y., Liu, S., & Chao, L. S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1007 · Cited by 6 · see citing works
MoFE: Mixture of Factual Experts for Controlling Hallucinations in Abstractive Summarization.Preprint
Choubey, P. K., Vig, J., Liu, W., & Rajani, N. F. (2021) · arXiv (Cornell University) · Cited by 5 · see citing works
Li, Y., Li, L., Hu, D., Li, Y., Zhou, Y., Litvak, M., & Vanetik, N. (2023) · Research Square · doi:10.21203/rs.3.rs-3695932/v1 · Cited by 7 · see citing works
A Survey of Factual Consistency in Summarization from 2021 to 2023
Li, Y., Hao, X., & Li, L. (2023) · International Conference on Computers and Artificial Intelligence Technology (CAIT) · doi:10.1109/cait59945.2023.10468958 · Cited by 4 · see citing works
Perspective-Based Microblog Summarization
Li, C.-Y., Chun, S. A., & Geller, J. (2025) · Information · doi:10.3390/info16040285 · Cited by 4 · see citing works
CoP: Factual Inconsistency Detection by Controlling the Preference
She, S., Geng, X., Huang, S., & Chen, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26589 · Cited by 4 · see citing works
Aligning Factual Consistency for Clinical Studies Summarization through Reinforcement Learning
Tang, X., Cohan, A., & Gerstein, M. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.7 · Cited by 4 · see citing works
WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning
Wu, W., Li, W., Xiao, X., Liu, J., Li, S., & Lyu, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.18 · Cited by 7 · see citing works
HistAlign: Improving Context Dependency in Language Generation by Aligning with History
Wan, D., Zhang, S., & Bansal, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.179 · Cited by 5 · see citing works
Probing Factually Grounded Content Transfer with Factual Ablation
West, P., Quirk, C., Galley, M., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.294 · Cited by 4 · see citing works
Zhu, Y., Li, Y., Li, L., Hu, D., Hao, X., Chen, D., Zhang, X., Zhang, Z., Zhou, Y., Litvak, M., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3567744 · Cited by 4 · see citing works
Unsupervised Token-level Hallucination Detection from Summary Generation By-products
Marfurt, A., & Henderson, J. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.21 · Cited by 5 · see citing works
Scirè, A., Ghonim, K., & Navigli, R. (2024) · arXiv · doi:10.48550/arxiv.2403.02270 · Cited by 2 · see citing works
Employing Internal and External Knowledge to Factuality-Oriented Abstractive Summarization
Gao, Z., Jiang, F., Chu, X., & Li, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_61 · Cited by 2 · see citing works
Evaluating Factuality in Cross-lingual Summarization
Gao, M., Wang, W., Wan, X., & Xu, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.786 · Cited by 2 · see citing works
USB: A Unified Summarization Benchmark Across Tasks and Domains
Krishna, K., Gupta, P., Ramprasad, S., Wallace, B., Bigham, J. P., & Lipton, Z. C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.592 · Cited by 2 · see citing works
Fact-Aware Abstractive Summarization Based on Prompt Information and Re-Ranking
Cai, T., Yang, Y., Wan, Y., Mao, X., & Ju, S. (2024) · International Conference on Machine Learning and Natural Language Processing (MLNLP) · doi:10.1109/mlnlp63328.2024.10800044 · Cited by 2 · see citing works
Cardenas, R., Yao, B., Wang, D., & Hou, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.76 · Cited by 2 · see citing works
Adversarial Fine-Grained Fact Graph for Factuality-Oriented Abstractive Summarization
Gao, Z., Jiang, F., Chu, X., & Li, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_27 · Cited by 2 · see citing works
Mind the Link: Discourse Link-Aware Hallucination Detection in Summarization
Lee, D., Jung, H., & Choi, Y. S. (2025) · Applied Sciences · doi:10.3390/app151910506 · Cited by 1 · see citing works
PMDS: progressive multi-document summarization with iterative summary integration
Yu, S., Yang, C., Qin, Y., Huang, R., Chen, Y., Lin, C., & Gao, W. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2025.104501 · Cited by 1 · see citing works
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization
Zhu, R., Qi, J., & Lau, J. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.377 · Cited by 4 · see citing works
Multi-View Source Ablation for Faithful Summarization
Cao, S., Ma, L., Lu, D., Logan, R. L., Tetreault, J., & Jaimes, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.151 · Cited by 1 · see citing works
Factual Relation Discrimination for Factuality-oriented Abstractive Summarization
Gao, Z., Li, P., Jiang, F., Chu, X., & Zhu, Q. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.69 · Cited by 1 · see citing works
Improving Consistency for Text Summarization with Energy Functions
Zeng, Q., Yin, Q., Li, Z., Gao, Y., Nag, S., Wang, Z., Yin, B., Ji, H., & Zhang, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.798 · Cited by 1 · see citing works
You, Z., & Guo, Y. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105019 · Cited by 2 · see citing works
Dissecting Generation Modes for Abstractive Summarization Models via Ablation and AttributionPreprint
Xu, J., & Durrett, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01518 · Cited by 1 · see citing works
Self-citationTruth-Conditional Captioning of Time Series DataPreprint
Jhamtani, H., & Berg-Kirkpatrick, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01839
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09209 · Cited by 14 · see citing works
Model Intrinsic Features of Fine-tuning based Text Summarization Models for Factual Consistency
Song, J., Park, N., Hwang, B., Yun, J.-W., Joe, S., Gwon, Y., & Yoon, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.872
ECR: An Expertise-Enriched Conclude-Then-Refine Summarization Framework for Professional Articles
Liang, Z., Xie, K., Lu, S., Shi, Y., Yeerpan, T., & Wang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_10
Chen, D., Hu, D., & Li, L. (2024) · International Conference on Communication and Information Processing (ICCIP) · doi:10.1145/3708657.3708772
- Authors
- Goyal, T., & Durrett, G.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2021
- Pages
- 1449-1462
- OpenAlex
- W3152781945
- Cited by
- 100
- References
- 33
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 6 · 2023: 39 · 2022: 35 · 2021: 9
- FWCI
- 10.28
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1814522)
- Nvidia
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- abstractive summarization, factuality, fine-grained annotation, XSum, training data quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
