CoChat
Sign Up
Cited by 21Topic ModelingAdvanced Text Analysis TechniquesNatural Language Processing Techniques

Towards Improving Faithfulness in Abstractive Summarization

Open access
Chen, Xiuying, Li, Mingzhe, Gao, Xin, Zhang, Xiangliang·2022·Advances in Neural Information Processing Systems 35
DOI 10.52202/068431-1780
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite the success achieved in neural abstractive summarization based on pre-trained language models, one unresolved issue is that the generated summaries are not always faithful to the input document. There are two possible causes of the unfaithfulness problem: (1) the summarization model fails to understand or capture the gist of the input text, and (2) the model over-relies on the language model to generate fluent but inadequate words. In this work, we propose a Faithfulness Enhanced Summarization model (FES), which is designed for addressing these two problems and improving faithfulness in abstractive summarization. For the first problem, we propose to use question-answering (QA) to examine whether the encoder fully grasps the input document and can answer the questions on the key information in the input. The QA attention on the proper input words can also be used to stipulate how the decoder should attend to the source. For the second problem, we introduce a max-margin loss defined on the difference between the language and the summarization model, aiming to prevent the overconfidence of the language model. Extensive experiments on two benchmark summarization datasets, CNN/DM and XSum, demonstrate that our model significantly outperforms strong baselines. The evaluation of factual consistency also shows that our model generates more faithful summaries than baselines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/068431-1780

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior studies in conditional text generation have highlighted the issue of over-confidence which stems from an excessive focus on the partially generated content, often prioritizing fluency at the expense of faithfully adhering to the source context.

    “Prior studies in conditional text generation [45, 212] have highlighted the issue of over-confidence which stems from an excessive focus on the partially generated content, often prioritizing fluency at the expense of faithfully adhering to the source context.”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [45]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 43

  1. [1][1] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL , 2017. -
  2. [2][2] Min Sun, Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, and Jing Tang. A unified model for extractive and abstractive summarization using inconsistency loss. In Proc. of ACL , 2018. -
  3. [3][3] Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. Global encoding for abstractive summarization. In Proc. of ACL , 2018. -
  4. [4][4] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proc. of EMNLP , 2019. -
  5. [5][5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of AACL , 2019. -
  6. [6][6] Wojciech Kry´ sci´ nski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proc. of EMNLP , 2020. -
  7. [7][7] Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. Questeval: Summarization asks for fact-based evaluation. In Proc. of EMNLP , 2021. -
  8. [8][8] Esin Durmus, He He, and Mona Diab. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proc. of ACL , 2020. -
  9. [9][9] Wenhao Wu, Wei Li, Xinyan Xiao, Jiachen Liu, Ziqiang Cao, Sujian Li, Hua Wu, and Haifeng Wang. Bass: Boosting abstractive summarization with unified semantic graph. In Proc. of ACL , 2021. -
  10. [10][10] Shuyang Cao and Lu Wang. Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proc. of EMNLP , 2021. -
  11. [11][11] Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 , 2020. - · arXiv
  12. [12][12] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NeurIPS , 2015. -
  13. [13][13] Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proc. of EMNLP , 2018. -
  14. [14][14] Xiuying Chen, Shen Gao, Chongyang Tao, Yan Song, Dongyan Zhao, and Rui Yan. Iterative document representation learning towards summarization with polishing. EMNLP , 2018. -
  15. [15][15] Xiuying Chen, Zhangming Chan, Shen Gao, Meng-Hsuan Yu, Dongyan Zhao, and Rui Yan. Learning towards abstractive timeline summarization. In Proc. of IJCAI , 2019. -
  16. [16][16] Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. Capturing relations between scientific papers: An abstractive model for related work section generation. In ACL , 2021. -
  17. [17][17] Linqing Liu, Yao Lu, Min Yang, Qiang Qu, Jia Zhu, and Hongyan Li. Generative adversarial network for abstractive text summarization. In Proc. of AAAI , 2018. -
  18. [18][18] Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. In Proc. of MNLP , 2018. -
  19. [19][19] Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. Deep communicating agents for abstractive summarization. In Proc. of AACL , 2018. -
  20. [20][20] Hao Zhou, Weidong Ren, Gongshen Liu, Bo Su, and Wei Lu. Entity-aware abstractive multidocument summarization. In Findings of ACL , 2021. -
  21. [21][21] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. Proc. of ICML , 2020. -
  22. [22][22] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proc. of ACL , 2020. -
  23. [23][23] Hanqi Jin, Tianming Wang, and Xiaojun Wan. Semsum: Semantic dependency guided neural abstractive summarization. In Proc. of AAAI , 2020. -
  24. [24][24] Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. Faithful to the original: Fact aware neural abstractive summarization. In Proc. of AAAI , 2018. -
  25. [25][25] Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proc. of COLING , 2018. -
  26. [26][26] Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. Enhancing factual consistency of abstractive summarization. In Proc. of AACL , 2021. -
  27. [27][27] Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. Focus attention: Promoting faithfulness and diversity in summarization. In Proc. of ACL , pages 6078-6095, 2021. -
  28. [29][29] Bernd Bohnet and Joakim Nivre. A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing. In Proc. of EMNLP , 2012. -
  29. [30][30] Jun Hatori, Takuya Matsuzaki, Yusuke Miyao, and Jun'ichi Tsujii. Incremental joint approach to word segmentation, pos tagging, and dependency parsing in chinese. In Proc. of ACL , 2012. -
  30. [31][31] Zhenghua Li, Min Zhang, Wanxiang Che, Ting Liu, and Wenliang Chen. Joint optimization for chinese pos tagging and dependency parsing. IEEE/ACM transactions on audio, speech, and language processing , 2013. -
  31. [32][32] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 , 2016. - · arXiv
  32. [33][33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NIPS , 2017. -
  33. [34][34] Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. Text generation from knowledge graphs with graph transformers. arXiv preprint arXiv:1904.02342 , 2019. - · arXiv
  34. [35][35] Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. Proc. of ICLR , 2017. -
  35. [36][36] Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. Handling divergent reference texts when evaluating table-to-text generation. In Proc. of ACL , 2019. -
  36. [37][37] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proc. of ACL , 2020. -
  37. [38][38] Rongxiang Weng, Heng Yu, Xiangpeng Wei, and Weihua Luo. Towards enhancing faithfulness for neural machine translation. In Proc. of EMNLP , 2020. -
  38. [39][39] Mengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, and Jie Zhou. Prevent the language model from being overconfident in neural machine translation. In Proc. of ACL , 2021. -
  39. [40][40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Proc. of JMLR , 2019. -
  40. [41][41] Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. Gsum: A general framework for guided neural abstractive summarization. In Proc. of AACL , 2021. -
  41. [42][42] Yixin Liu and Pengfei Liu. Simcls: A simple framework for contrastive learning of abstractive summarization. In Proc. of ACL , 2021. -
  42. [43][43] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-ofthe-art natural language processing. In Proc. of EMNLP , 2020. -
  43. [45][45] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In Proc. of ICLR , 2020.

Cited by

Sorted by citations

Authors
Chen, X., Li, M., Gao, X., & Zhang, X.
ORCID
Zhang, Xiangliang
Venue
Advances in Neural Information Processing Systems 35
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2022
Pages
24516-24528
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/068431-1780
OpenAlex
W7133226401
Cited by
21
References
0
Citations by year
2026: 3 · 2025: 3 · 2024: 4 · 2023: 10 · 2022: 1
FWCI
2.34
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
King Abdullah University of Science and Technology (FCC/1/1976- 45-01, BAS/1/1635-01-01, URF/1/4663-01-01, FCC/1/1976‐44‐01)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
abstractive summarization, faithfulness, summarization evaluation, factual accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review