Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 10

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization

Zhang, Shiyue, Wan, David, Bansal, Mohit·2023·Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2023.acl-long.120
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.acl-long.120

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 3 The unfaithfulness error distributions of 16 extractive summarization systems.
Figure 3 The unfaithfulness error distributions of 16 extractive summarization systems.CC-BY · From the PDF
Image 2
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 54

  1. Regina Barzilay, Kathleen R. McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of the 37th Annual Meeting of the Association for Com- putational Linguistics, pages 550-557, College Park, Maryland, USA. Association for Computational Lin- guistics. · DOI
  2. Ann L Brown and Jeanne D Day. 1983. Macrorules for summarizing texts: The development of exper- tise. Journal of verbal learning and verbal behavior, 22(1):1-14.
  3. Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  4. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). · DOI
  5. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics. · DOI
  6. Jackie CK Cheung. 2008. Comparing abstractive and extractive summarization of evaluative text: contro- versiality and content selection. B. Sc.(Hons.) Thesis in the Department of Computer Science of the Faculty of Science, University of British Columbia, 47.
  7. Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Routledge.
  8. Trevor Cohn and Mirella Lapata. 2008. Sentence com- pression beyond word deletion. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 137-144, Manch- ester, UK. Coling 2008 Organizing Committee.
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop, pages 177-190. Springer.
  10. Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021. Towards question-answering as an automatic metric for evaluating the content quality of a sum- mary. Transactions of the Association for Computa- tional Linguistics, 9:774-789. · DOI · Paper page
  11. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit- Sum: Extractive summarization as a contextual ban- dit. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 3739-3748, Brussels, Belgium. Association for Com- putational Linguistics. · DOI
  12. Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2021. Analyzing the abstractiveness-factuality tradeoff with nonlin- ear abstractiveness constraints. arXiv preprint arXiv:2108.02859. · arXiv
  13. Tim Fischer. 2021. Finding factual inconsistencies in abstractive summaries. Master's thesis, Universität Hamburg.
  14. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew E Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language process- ing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6.
  15. Matthew Gentzkow, Jesse M Shapiro, and Daniel F Stone. 2015. Media bias in the marketplace: Theory. In Handbook of media economics, volume 1, pages 623-645. Elsevier.
  16. Tanya Goyal and Greg Durrett. 2020. Evaluating factu- ality in generation with dependency-level entailment. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 3592-3603, Online. Association for Computational Linguistics. · DOI · Paper page
  17. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. Snac: Coherence error detection for narrative sum- marization. arXiv preprint arXiv:2205.09641. · arXiv
  18. Vishal Gupta and Gurpreet Singh Lehal. 2010. A survey of text summarization extractive techniques. Jour- nal of emerging technologies in web intelligence, 2(3):258-268.
  19. Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28.
  20. Walter Kintsch and Teun van Dijk. 1978. Cognitive psychology and discourse: Recalling and summariz- ing stories. Current trends in text linguistics, pages 61-80.
  21. Haoran Li, Arash Einolghozati, Srinivasan Iyer, Bhar- gavi Paranjape, Yashar Mehdad, Sonal Gupta, and Marjan Ghazvininejad. 2021. EASE: Extractive- abstractive summarization end-to-end using the infor- mation bottleneck principle. In Proceedings of the Third Workshop on New Frontiers in Summarization, pages 85-95, Online and in Dominican Republic. Association for Computational Linguistics. · DOI
  22. Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 1430-1441, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  23. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  25. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional the- ory of text organization. Text-interdisciplinary Jour- nal for the Study of Discourse, 8(3):243-281.
  26. Lluís Màrquez, Marta Recasens, and Emili Sapena. 2013. Coreference resolution: An empirical study based on semeval-2010 shared task 1. Lang. Resour. Eval., 47(3):661-694. · DOI
  27. Mani Maybury. 1999. Advances in automatic text sum- marization. MIT press.
  28. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
  29. Hidetsugu Nanba and Manabu Okumura. 2000. Pro- ducing more readable extracts by revising them. In COLING 2000 Volume 2: The 18th International Conference on Computational Linguistics.
  30. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summariza- tion with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  31. Ani Nenkova and Kathleen McKeown. 2012. A survey of text summarization techniques. In Mining text data, pages 43-76. Springer.
  32. Cailin O'Connor and James Owen Weatherall. 2019. The misinformation age. In The Misinformation Age. Yale University Press.
  33. Vishakh Padmakumar and He He. 2021. Unsupervised extractive summarization using pointwise mutual in- formation. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 2505-2512, Online. Association for Computational Linguistics. · DOI
  34. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for bet- ter content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84, Copenhagen, Denmark. Association for Computational Linguistics. · DOI
  35. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt- sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA).
  36. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. As- sociation for Computational Linguistics. · DOI
  37. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
  38. Qian Ruan, Malte Ostendorff, and Georg Rehm. 2022. HiStruct+: Improving extractive text summarization with hierarchical structure information. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1292-1308, Dublin, Ireland. As- sociation for Computational Linguistics. · DOI
  39. Horacio Saggion and Thierry Poibeau. 2013. Auto- matic text summarization: Past, present and future. In Multi-source, multilingual information extraction and summarization, pages 3-21. Springer.
  40. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  41. Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 21-29, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
  42. David Wan and Mohit Bansal. 2022. FactPEGASUS: Factuality-aware pre-training and fine-tuning for ab- stractive summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1010-1028, Seattle, United States. Association for Computational Lin- guistics. · DOI
  43. Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020b. Heterogeneous graph neural networks for extractive document summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6209-6219, Online. Association for Computational Linguistics. · DOI
  44. Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 100-110, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  45. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text sum- marization. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computa- tional Linguistics. · DOI
  46. Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2022. Sequence level contrastive learning for text summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11556-11565. · DOI
  47. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020a. PEGASUS: Pre-training with ex- tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339.
  48. PMLR.
  49. Shiyue Zhang and Mohit Bansal. 2021. Finding a bal- anced degree of automation for summary evaluation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6617-6632, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI
  50. Hao Zheng and Mirella Lapata. 2019. Sentence cen- trality revisited for unsupervised summarization. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6236- 6247, Florence, Italy. Association for Computational Linguistics. · DOI
  51. Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational Linguistics. · DOI
  52. Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2019. Searching for effective neural extractive summarization: What works and what's next. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1049-1058, Florence, Italy. Association for Computational Linguistics. · DOI
  53. Deyu Zhou, Linsen Guo, and Yulan He. 2018a. Neural storyline extraction model for storyline generation from news articles. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pages 1727-1736, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  54. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018b. Neural docu- ment summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 654-663, Melbourne, Australia. Association for Computational Linguistics. · DOI

Cited by

See all 10

Metadata

Authors
Zhang, S., Wan, D., & Bansal, M.
Venue
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2023
Pages
2153-2174
DOI
10.18653/v1/2023.acl-long.120
Cited by
10
References
18

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization. https://app.cochat.ai/papers/10.18653/v1/2023.acl-long.120
The paper
Zhang, S., Wan, D., & Bansal, M. (2023). Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2023.acl-long.120

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.