The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Evaluating the Factual Consistency of Abstractive Text Summarization
Open accessAbstract
Currently used metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents.We propose a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and a generated summary.Training data is generated by applying a series of rule-based transformations to the sentences of source documents.The factual consistency model is then trained jointly for three tasks: 1) identify whether sentences remain factually consistent after transformation, 2) extract a span in the source documents to support the consistency prediction, 3) extract a span in the summary sentence that is inconsistent if one exists.Transferring this model to summaries generated by several state-of-the art models reveals that this highly scalable approach substantially outperforms previous models, including those trained with strong supervision using standard datasets for natural language inference and fact checking.Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In most previous works, factuality has been defined as whether a claim in a text can be supported by evidence from a separate, trustworthy knowledge base, with applications in fact-checking (Thorne et al., 2018b) (where the knowledge base is a large source like Wikipedia) and summarization (Kryscinski et al., 2020) (where the knowledge base is an input document or documents).”
From FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023) · cites this paper as Kryscinski et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, hallucinations are observed over a wide range of tasks beyond pure fact verification (Kryscinski et al., 2020; Maynez et al., 2020).”
From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Kryscinski et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Research suggests that up to 30% of machine-generated abstractive summaries may contain factual inconsistencies [52, 53].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [53]
The cited source says what the claim attributes to it.
In the citing paper
“Results show that the performance of their FactCC model surpasses the classifiers trained on the MNLI or FEVER datasets.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [126]
The cited source says what the claim attributes to it.
In the citing paper
“These methods, such as entity swapping and sentence negation (Kryscinski et al., 2020;Goyal and Durrett, 2021), were designed to target specific error types that occurred in claims from earlier summarization models.”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Kryscinski et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“In response to this challenge, a surge of research explores leveraging data-augmentation methods to construct synthetical data for fine-tuning the classifier, either by rule-based perturbation [79, 152, 266] or generation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [152]
The cited source says what the claim attributes to it.
In the citing paper
“most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization [81, 102, 152, 208, 236, 310], data-to-text[240, 296], and machine translation [389].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [152]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This approach has been widely studied in the summarization literature and recent work in factual precision (Kryscinski et al., 2020; Wang et al., 2020; Gao et al., 2022; Manakul et al., 2023).”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Kryscinski et al., 2020
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2017) · Cited by 11,625
(2004) · Cited by 9,718
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1044 · Cited by 2,932
Research works printed but not linked: 9
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguis- tics.
- Kyunghyun Cho, Bart van Merrienboer, C ¸aglar Gülc ¸ehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724- 1734.
- Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing. Association for Compu- tational Linguistics.
- Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. CoRR, abs/1808.09381. · arXiv
- Ramesh Nallapati, Bowen Zhou, C ¸aglar Gülc ¸ehre, Bing Xiang, et al. 2016. Abstractive text summa- rization using sequence-to-sequence rnns and be- yond. Proceedings of SIGNLL Conference on Com- putational Natural Language Learning.
- Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. CoRR, abs/1804.06451. · arXiv
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proc. of NAACL.
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. CoRR, abs/1803.05355. · arXiv
Cited by
Sorted by citations
Items 101–200 of 693
DiffuSum: Generation Enhanced Extractive Summarization with Diffusion
Zhang, H., Liu, X., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.828 · Cited by 51
FAR-ASS: Fact-aware reinforced abstractive sentence summarization
Zhang, M., Zhou, G., Yu, W., & Liu, W. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2020.102478 · Cited by 36
Abstractive Meeting Summarization: A Survey
Rennard, V., Shang, G., Hunter, J., & Vazirgiannis, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00578 · Cited by 48
Multi-language transfer learning for low-resource legal case summarization
Moro, G., Piscaglia, N., Ragazzi, L., & Italiani, P. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09373-8 · Cited by 40
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization
Laban, P., Kryściński, W., Agarwal, D., Fabbri, A. R., Xiong, C., Joty, S., & Wu, C.-S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.600 · Cited by 44
Self-citationA Critical Evaluation of Evaluations for Long-form Question Answering
Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.181 · Cited by 33
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 112
Balachandran, V., Hajishirzi, H., Cohen, W. W., & Tsvetkov, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.667 · Cited by 33
Faithfulness-Aware Decoding Strategies for Abstractive Summarization
Wan, D., Liu, M., McKeown, K. R., Dreyer, M., & Bansal, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.210 · Cited by 46
Abstractive Summarization of Indian Legal Judgments
Prabhakar, P., Gupta, D., & Pati, P. B. (2022) · OITS International Conference on Information Technology (OCIT) · doi:10.1109/ocit56763.2022.00056 · Cited by 30
A Field Guide to Automatic Evaluation of LLM-Generated Summaries
van Schaik, T. A., & Pugh, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3661346 · Cited by 30
Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2021) · doi:10.18653/v1/2021.acl-long.498 · Cited by 32
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization
Ladhak, F., Durmus, E., Suzgun, M., Zhang, T., Jurafsky, D., McKeown, K., & Hashimoto, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.234 · Cited by 30
Learning to Revise References for Faithful Summarization
Adams, G., Shing, H.-C., Sun, Q., Winestock, C., McKeown, K. R., & Elhadad, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.296 · Cited by 34
Ahmed, K., Khan, M. A., Haq, I., Mazroa, A. A., Syam, M. S., Innab, N., Alajmi, M., & Alkahtani, H. K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124650 · Cited by 29
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.121 · Cited by 29
Leveraging Summary Guidance on Medical Report Summarization
Zhu, Y., Yang, X., Wu, Y., & Zhang, W. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3304376 · Cited by 37
CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems
Chen, Y., Liu, P., Zhong, M., Dou, Z.-Y., Wang, D., Qiu, X., & Huang, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.329 · Cited by 30
Kouris, P., Alexandridis, G., & Stafylopatis, A. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00950-5 · Cited by 33
Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained GenerationPreprint
Mao, Y., Ren, X., Ji, H., & Han, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12723 · Cited by 40
ExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.167 · Cited by 28
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
Sadat, M., Zhou, Z., Lange, L., Araki, J., Gundroo, A., Wang, B., Menon, R. R., Parvez, M. R., & Feng, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.59 · Cited by 55
Logic-Consistency Text Generation from Semantic Parses
Shu, C., Zhang, Y., Dong, X., Shi, P., Tao, Y., & Zhang, R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.388 · Cited by 39
Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators
Liang, C., Deng, Y., Bian, Y., Qin, Z., Wu, B., Chua, T., & Wong, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.390 · Cited by 56
Question-aware transformer models for consumer health question summarization
Yadav, S., Gupta, D., Abacha, A. B., & Demner‐Fushman, D. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104040 · Cited by 27
Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control
Zhang, H., Yavuz, S., Kryściński, W., Hashimoto, K., & Zhou, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.40 · Cited by 31
MLSUM: The Multilingual Summarization Corpus
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.647 · Cited by 64
Utama, P. A., Bambrick, J., Moosavi, N. S., & Gurevych, I. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.199 · Cited by 33
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
Roit, P., Ferret, J., Shani, L., Aharoni, R., Cideron, G., Dadashi, R., Geist, M., Girgin, S., Hussenot, L., Keller, O., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.344 · Cited by 63
Towards Unifying Multi-Lingual and Cross-Lingual Summarization
Wang, J., Meng, F., Zheng, D., Liang, Y., Li, Z., Qu, J., & Zhou, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.843 · Cited by 29
Knowledge Enhanced Pretrained Language Models: A Compreshensive SurveyPreprint
Wei, X., Wang, S., Zhang, D., Bhatia, P., & Arnold, A. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08455 · Cited by 57
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29
“, 2018b) (where the knowledge base is a large source like Wikipedia) and summarization (Kryscinski et al., 2020) (where the knowledge base is an input document or documents).”
BackgroundInfluentialA Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation
Narayan, S., Simões, G., Zhao, Y., Maynez, J., Das, D., Collins, M., & Lapata, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.94 · Cited by 27
Mutual Information Alleviates Hallucinations in Abstractive Summarization
van der Poel, L., Cotterell, R., & Meister, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.399 · Cited by 54
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A. P., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00583 · Cited by 30
Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus
Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., & Fu, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.58 · Cited by 71
Chen, F., Li, Y., Chen, Y., Bian, Z., Duo, L., Zhou, Q., Zhang, L., & ADVANCED Working Group (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70075 · Cited by 30
EA2E: Improving Consistency with Event Awareness for Document-Level Argument Extraction
Zeng, Q., Zhan, Q., & Ji, H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.202 · Cited by 25
An Investigation of Evaluation Methods in Automatic Medical Note Generation
Abacha, A. B., Yim, W., Michalopoulos, G., & Lin, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.161 · Cited by 47
QuestEval: Summarization Asks for Fact-based EvaluationPreprint
Scialom, T., Dray, P.-A., Gallinari, P., Lamprier, S., Piwowarski, B., Staiano, J., & Wang, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12693 · Cited by 22
Saleh, M. E., Wazery, Y. M., & Ali, A. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124153 · Cited by 24
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 33
A Novel Approach for Semantic Extractive Text Summarization
Waseemullah, W., Fatima, Z., Zardari, S., Fahim, M., Siddiqui, M. A., Ibrahim, A. A. A., Nisar, K., & Naz, L. F. (2022) · Applied Sciences · doi:10.3390/app12094479 · Cited by 24
Training Dynamics for Text Summarization Models
Goyal, T., Xu, J., Li, J. J., & Durrett, G. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.163 · Cited by 24
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
Liu, T., Zheng, X., Chang, B., & Sui, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17583 · Cited by 21
Santhanam, S., Hedayatnia, B., Gella, S., Padmakumar, A., Kim, S., Liu, Y., & Hakkani-Tur, D. (2021) · arXiv · doi:10.48550/arxiv.2110.05456 · Cited by 21
Opportunities and Risks of LLMs for Scalable Deliberation with Polis
Small, C. T., Vendrov, I., Durmus, E., Homaei, H., Barry, E., Cornebise, J., Suzman, T., Ganguli, D., & Megill, C. (2023) · arXiv · doi:10.48550/arxiv.2306.11932 · Cited by 20
“Prior work has proposed automated metrics to detect such errors (Maynez et al., 2020; Durmus et al., 2020; Kryscinski et al., 2020; Goyal and Durrett, 2021); however, they do not always correlate well with human judgements (Fabbri et al., 2021; Durmus et al., 2022).”
BackgroundBuilding trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety
Gaur, M., & Sheth, A. (2024) · AI Magazine · doi:10.1002/aaai.12149 · Cited by 34
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 24
Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness
Song, Y.-Z., Chen, Y.-S., & Shuai, H.-H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.120 · Cited by 21
Toward Unifying Text Segmentation and Long Document Summarization
Cho, S., Song, K., Wang, X., Liu, F., & Yu, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.8 · Cited by 33
A dataset for plain language adaptation of biomedical abstracts
Attal, K., Ondov, B., & Demner‐Fushman, D. (2023) · Scientific Data · doi:10.1038/s41597-022-01920-3 · Cited by 24
Text Summarization Techniques and Applications
Dehru, V., Tiwari, P. K., Aggarwal, G., Joshi, B., & Kartik, P. (2021) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/1099/1/012042 · Cited by 19
Exploring Different Methods of Scanners and Parsers
Asmitha, M., & Panda, N. (2024) · International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE) · doi:10.1109/amathe61652.2024.10582089 · Cited by 19
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
Chen, Y., Liu, P., & Qiu, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.179 · Cited by 21
Li, S., & Xu, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103467 · Cited by 18
Zero-shot Faithful Factual Error Correction
Huang, K.-H., Chan, H. P., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.311 · Cited by 26
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations
Wang, L. L., Otmakhova, Y., DeYoung, J., Truong, T. H., Kuehl, B., Bransom, E., & Wallace, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.549 · Cited by 22
Zhang, S., Wan, D., & Bansal, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.120 · Cited by 33
How Far are We from Robust Long Abstractive Summarization?
Koh, H. Y., Ju, J., Zhang, H., Liu, M. Z., & Pan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.172 · Cited by 21
Detecting and Mitigating Hallucinations in Multilingual Summarisation
Qiu, Y., Ziser, Y., Korhonen, A., Ponti, E. M., & Cohen, S. B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.551 · Cited by 35
Sentence salience contrastive learning for abstractive text summarization
Huang, Y., Li, Z., Chen, Z., Zhang, C., & Ma, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127808 · Cited by 18
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters
Schuster, T., Chen, S., Buthpitiya, S., Fabrikant, A., & Metzler, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.28 · Cited by 17
Compressive Summarization with Plausibility and Salience Modeling
Desai, S., Xu, J., & Durrett, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.507 · Cited by 17
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization
Vig, J., Kryściński, W., Goel, K., & Rajani, N. F. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.18 · Cited by 20
DialSummEval: Revisiting Summarization Evaluation for Dialogues
Gao, M., & Wan, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.418 · Cited by 20
Textual entailment as an evaluation metric for abstractive text summarization
Bhuyan, S. S., Mahanta, S. K., Pakray, P., & Favre, B. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100028 · Cited by 18
Automation of Text Summarization Using Hugging Face NLP
Asmitha, M., Danda, A., Bysani, H., Singh, R. P., & Kanchan, S. (2024) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet61516.2024.10593316 · Cited by 18
Play the Shannon Game with Language Models: A Human-Free Approach to Summary Evaluation
Egan, N., Vasilyev, O., & Bohannon, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21304 · Cited by 22
SEASum: Syntax-Enriched Abstractive Summarization
Liu, S., Yang, L., & Cai, X. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116819 · Cited by 17
Discriminative Adversarial Search for Abstractive SummarizationPreprint
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10375 · Cited by 18
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024) · arXiv · doi:10.48550/arxiv.2404.04475 · Cited by 17
Reducing hallucinations of large language models via hierarchical semantic piece
Liu, Y., Yang, Q., Tang, J.-W., Guo, T., Chen, W., Pan, L., Sai, X., Gao, X., Li, Z., Liu, J., et al. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01833-9 · Cited by 21
Chakraborty, S., Chowdhury, R., Shuvo, S. R., Chatterjee, R., & Roy, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-15203-5 · Cited by 21
Discourse Understanding and Factual Consistency in Abstractive Summarization
Gabriel, S., Bosselut, A., Da, J., Holtzman, A., Buys, J., Lo, K., Çelikyılmaz, A., & Choi, Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.34 · Cited by 18
Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation
Gupta, P., Tsvetkov, Y., & Bigham, J. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.338 · Cited by 18
How well do you know your summarization datasets?
Tejaswin, P., Naik, D., & Liu, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.303 · Cited by 25
Using Question Answering Rewards to Improve Abstractive Summarization
Gunasekara, C., Feigenblat, G., Sznajder, B., Aharonov, R., & Joshi, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.47 · Cited by 17
Compute to Tell the Tale: Goal-Driven Narrative Generation
Wong, Y., Fan, S., Guo, Y., Xu, Z., Stephen, K., Sheoran, R., Bhamidipati, A., Barsopia, V., Liu, J., & Kankanhalli, M. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3549202 · Cited by 17
Analyzing and Evaluating Faithfulness in Dialogue Summarization
Wang, B., Zhang, C., Zhang, Y., Chen, Y., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.325 · Cited by 25
Improving Factual Consistency in Summarization with Compression-Based Post-Editing
Fabbri, A. R., Choubey, P. K., Vig, J., Wu, C.-S., & Xiong, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.623 · Cited by 21
Self-citationDong, Y., Wieting, J., & Verga, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.76 · Cited by 16
Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future Directions
Jia, Q., Liu, Y., Ren, S., & Zhu, K. Q. (2023) · ACM Computing Surveys · doi:10.1145/3622933 · Cited by 20
Exploring Decomposition for Table-based Fact Verification
Yang, X., & Zhu, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.90 · Cited by 17
Spurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization
Choubey, P. K., Fabbri, A. R., Vig, J., Wu, C.-S., Liu, W., & Rajani, N. F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.685 · Cited by 26
Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior NotesPreprint
Shing, H.-C., Shivade, C., Pourdamghani, N., Nan, F., Resnik, P., Oard, D. W., & Bhatia, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13498 · Cited by 35
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
Jain, S., Keshava, V., Mysore Sathyendra, S., Fernandes, P., Liu, P., Neubig, G., & Zhou, C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.537 · Cited by 15
Semantic Similarity Based Evaluation for Abstractive News Summarization
Fikri, F. B., Oflazer, K., & Yanikoglu, B. A. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.3 · Cited by 18
Challenges in Domain-Specific Abstractive Summarization and How to Overcome Them
Afzal, A., Vladika, J., Braun, D., & Matthes, F. (2023) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0011744500003393 · Cited by 19
Briman, M. K. H., & Yildiz, B. (2024) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213024500179 · Cited by 16
Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 24
A New Approach to Overgenerating and Scoring Abstractive Summaries
Song, K., Wang, B., Feng, Z., & Liu, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.110 · Cited by 14
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
Tang, X., Fabbri, A. R., Li, H., Mao, Z., Adams, G., Wang, B., Çelikyılmaz, A., Mehdad, Y., & Radev, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.417 · Cited by 21
Enhancing Multi-Document Summarization with Cross-Document Graph-based Information Extraction
Zhang, Z., Elfardy, H., Dreyer, M., Small, K., Ji, H., & Bansal, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.124 · Cited by 15
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report GenerationPreprint
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2010.10042 · Cited by 264
Planning with Entity Chains for Abstractive Summarization.Preprint
Narayan, S., Zhao, Y., Maynez, J., Simões, G., & McDonald, R. (2021) · arXiv (Cornell University) · Cited by 14
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14
“This approach has been widely studied in the summarization literature and recent work in factual precision (Kryscinski et al., 2020; Wang et al., 2020; Gao et al., 2022; Manakul et al., 2023).”
Method
Metadata
- Authors
- Kryscinski, W., McCann, B., Xiong, C., & Socher, R.
- Affiliations
- Salesforce (United States) (US)
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 9332-9346
- OpenAlex
- W2981456979
- Cited by
- 669
- References
- 41
- Citations by year
- 2026: 94 · 2025: 86 · 2024: 56 · 2023: 171 · 2022: 106 · 2021: 118 · 2020: 37
- FWCI
- 41.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, factual consistency, weakly supervised learning, rule-based transformation, span extraction, natural language inference, fact-checking, summarization evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
