CoChat
Sign Up
PreprintCited by 1,185Topic ModelingAdvanced Text Analysis TechniquesNatural Language Processing Techniques

On Faithfulness and Factuality in Abstractive Summarization

Open access
Maynez, Joshua, Narayan, Shashi, Bohnet, Bernd, McDonald, Ryan·2020·arXiv
DOI 10.48550/arxiv.2005.00661arXiv 2005.00661
Open this paper in CoChat — free accountOpen PDF

Abstract

It is well known that the standard likelihood training and approximate decoding objectives in neural text generation models lead to less human-like responses for open-ended tasks such as language modeling and story generation. In this paper we have analyzed limitations of these models for abstractive document summarization and found that these models are highly prone to hallucinate content that is unfaithful to the input document. We conducted a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce. Our human annotators found substantial amounts of hallucinated content in all model generated summaries. However, our analysis does show that pretrained models are better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in generating faithful and factual summaries as evaluated by humans. Furthermore, we show that textual entailment measures better correlate with faithfulness than standard metrics, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2005.00661

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The 'implicit representation' of knowledge in LLM has been shown to generate non-factual information despite linguistically coherent answers (i.e. 'hallucination') as a response to the input prompt.

    “Nevertheless, the “implicit representation” of knowledge in LLM has also been shown to generate non-factual information despite linguistically coherent answers (i.e. “hallucination”) as a response to the input prompt (Maynez et al. 2020, Raunak et al. 2021, Ji et al. 2023).”

    From Biomedical knowledge graph-optimized prompt generation for large language models (2024) · cites this paper as Maynez et al. 2020

    Section 5.3, “Assessment of Factual Hallucinations” (page unavailable), reports that over 90% of BERTS2S hallucinations were erroneous, supporting the point that generated text can be non-factual. But the study concerns abstractive summaries of documents, not LLM answers to prompts, and it does not establish that an implicit representation of knowledge causes the errors; it says pretraining helps generate more factual summaries. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • When a language model is incorrect, it generates an alternative response that is typically plausible looking (e.g., a similar but incorrect entity).

    “In those cases where the model is incorrect, they instead generate an alternative response which is typically plausible looking (e.g., a similar entity, but an incorrect one).”

    From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Maynez et al., 2020

    Section 2.1 describes generated summaries that misrepresent source information, and Appendix E gives an incorrect name, “Zac Goldwin,” where the article names Zac Goldsmith. These passages support the similar-but-incorrect-entity example, but the paper studies abstractive summarization and does not establish that incorrect language-model responses are typically plausible-looking alternatives in general. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • LLMs have a propensity to generate incorrect or fabricated information ('hallucinations').

    “However, a known challenge with LLMs is their propensity to generate incorrect or fabricated information (“hallucinations”) [21].”

    From SKiM-GPT: combining biomedical literature-based discovery with large language model hypothesis evaluation (2025) · cites this paper as [21]

    The BERTS2S conclusions report frequent hallucinations in generated summaries and find that over 90% of extrinsic hallucinations assessed were erroneous. The paper studies abstractive summarization systems, however, rather than establishing a general propensity of LLMs across tasks. Section 5.3 also cautions that hallucinations are not necessarily erroneous. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Hallucination is a general problem in language model generations that appears across many tasks, from summarization to open-domain dialogue, and has not been resolved by simply scaling up training data or model size.

    “Hallucination is a general problem in language model generations that appears across many tasks, from summarization (Maynez et al., 2020) to opendomain dialogue (Roller et al., 2020), and has not been resolved by simply scaling up training data or model size (Zhang et al., 2023).”

    From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Maynez et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Convolutional Sequence to Sequence Learning

    Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 2× · see citing works

    “The Topic-aware Convolutional Sequence to Sequence model (TCONVS2S) introduced by Narayan et al. (2018a) is an abstractive system which is conditioned on the article’s topics and based entirely on Convolutional Neural Networks (Gehring et al., 2017).”
    Method

Flagged first, then by citations

Research works printed but not linked: 53

  1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob De- vlin, and Michael Collins. 2019. Synthetic QA cor- pora generation with roundtrip consistency. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6168- 6173, Florence, Italy.
  2. Kristjan Arumae and Fei Liu. 2019. Guiding extractive summarization with question-answering rewards. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2566-2577, Minneapolis, Minnesota.
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd Interna- tional Conference on Learning Representations, San Diego, CA, USA.
  4. Michele Banko, Michael J. Cafarella, Stephen Soder- land, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Pro- ceedings of the 20th International Joint Conference on Artifical Intelligence, pages 2670-2676, Hyder- abad, India.
  5. Ann L. Brown and Jeanne D. Day. 1983. Macrorules for summarizing texts: The development of exper- tise. Journal of Verbal Learning and Verbal Be- haviour, 22(1):1-14.
  6. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics, pages 675-686, Melbourne, Australia.
  7. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2978-2988, Florence, Italy.
  8. Hoa Trang Dang. 2005. Overview of DUC 2005. In Proceedings of the Document Understanding Con- ference, pages 1-12.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186, Minneapolis, Min- nesota.
  10. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Co- hen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884-4895, Flo- rence, Italy.
  11. Teun A. van Dijk and Walter Kintsch. 1978. Cognitive psychology and discourse: Recalling and summariz- ing stories. In Wolfgang U. Dressler, editor, Current Trends in Textlinguistics, pages 61-80.
  12. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understand- ing and generation. In Advances in Neural Infor- mation Processing Systems 32, pages 13063-13075.
  13. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An in- teresting but challenging application for natural lan- guage inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2214-2220, Florence, Italy.
  14. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium.
  15. Ben Goodrich, Vinay Rao, Peter J. Liu, and Moham- mad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 166-175, New York, NY, USA.
  16. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics, pages 1631-1640, Berlin, Germany.
  17. Hardy, Shashi Narayan, and Andreas Vlachos. 2019. HighRES: Highlight-based reference-less evaluation of summarization. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 3381-3392, Florence, Italy.
  18. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Informa- tion Processing Systems 28, pages 1693-1701.
  19. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degener- ation. In Proceedings of the 8th International Con- ference on Learning Representations, Virtual Con- ference, Formerly Addis Ababa Ethiopia.
  20. Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019. Sample efficient text sum- marization using a single pre-trained transformer. CoRR, abs/1905.08836. · arXiv
  21. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 540-551, Hong Kong, China.
  22. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the fac- tual consistency of abstractive text summarization. CoRR, abs/1910.12840. · arXiv
  23. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. CoRR, abs/1808.06226. · arXiv
  24. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Jacob De- vlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question an- swering research. Transactions of the Association for Computational Linguistics, 7:453-466.
  25. J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):159-174.
  26. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas.
  27. Chin Yew Lin and Eduard Hovy. 2003. Automatic eval- uation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, pages 150-157.
  28. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In Proceedings of the 6th Inter- national Conference on Learning Representations, Vancouver Canada.
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. · arXiv
  30. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don't give me the details, just the summary! Topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium.
  31. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summariza- tion with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1747-1759, New Orleans, Louisiana.
  32. Shashi Narayan, Gonc ¸alo Simoes, Ji Ma, Hannah Craighead, and Ryan T. McDonald. 2020. QURI- OUS: Question generation pretraining for text gen- eration. CoRR, abs/2004.11026. · arXiv
  33. Ani Nenkova. 2005. Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference. In Proceedings of the 20th National Conference on Artificial Intelligence - Volume 3, pages 1436-1441.
  34. Ani Nenkova and Kathleen McKeown. 2011. Auto- matic summarization. Foundations and Trends in Information Retrieval, 5(2-3):103-233.
  35. Ani Nenkova and Rebecca Passonneau. 2004. Evaluat- ing content selection in summarization: The Pyra- mid method. In Proceedings of the Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, pages 145-152, Boston, Massachusetts, USA.
  36. Timothy Niven and Hung-Yu Kao. 2019. Probing neu- ral network comprehension of natural language ar- guments. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy.
  37. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 646-653, New Orleans, Louisiana.
  38. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive sum- marization. In Proceedings of the 6th International Conference on Learning Representations, Vancou- ver, BC, Canada.
  39. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. To appear in Transac- tions of the Association for Computational Linguis- tics, abs/1907.12461. · arXiv
  40. Natalie Schluter. 2017. The limits of automatic sum- marisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 41- 45, Valencia, Spain.
  41. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics, pages 1073-1083, Vancouver, Canada.
  42. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning, pages 843-861, Hong Kong, China.
  43. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2019. MASS: Masked sequence to se- quence pre-training for language generation. In Pro- ceedings of the 36th International Conference on Machine Learning, Long Beach, California.
  44. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Sys- tems 27, pages 3104-3112.
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
  46. Alex Wang, Kyunghyun Cho, and Michael Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual Conference, Formerly Seattle, USA.
  47. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2020. Neu- ral text generation with unlikelihood training. In Proceedings of the 8th International Conference on Learning Representations, Virtual Conference, For- merly Addis Ababa Ethiopia.
  48. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy.
  49. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112-1122, New Orleans, Louisiana.
  50. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark.
  51. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rud- nick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. · arXiv
  52. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237. · arXiv
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proceed- ings of the 8th International Conference on Learn- ing Representations, Virtual Conference, Formerly Addis Ababa Ethiopia.

Cited by

Sorted by citations

Authors
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R.
Affiliations
Google (United States) (US)
ORCID
Maynez, Joshua
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
1906-1919
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2005.00661
arXiv
2005.00661
OpenAlex
W3021338988
Cited by
1,185
References
1
Citations by year
2026: 288 · 2025: 232 · 2024: 144 · 2023: 256 · 2022: 130 · 2021: 112 · 2020: 21 · 2019: 2
FWCI
49.57
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
abstractive summarization, hallucinations, faithfulness, factual accuracy, neural text generation, pre-trained language models, human evaluation, ROUGE, natural language inference, automatic evaluation metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review