CoChat
Sign Up
Cited by 335Topic ModelingAdvanced Text Analysis TechniquesNatural Language Processing Techniques

Neural Text Summarization: A Critical Evaluation

Open access
Kryscinski, Wojciech, Keskar, Nitish Shirish, McCann, Bryan, Xiong, Caiming, Socher, Richard·2019·Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
DOI 10.18653/v1/d19-1051
Open this paper in CoChat — free accountOpen PDF

Abstract

Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d19-1051

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 75

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
  2. Chris Callison-Burch, Cameron S. Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In WMT@ACL, pages 136-158. Association for Com- putational Linguistics.
  3. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in ma- chine translation research. In EACL. The Associa- tion for Computer Linguistics.
  4. Danqi Chen, Jason Bolton, and Christopher D. Man- ning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In ACL (1). The Association for Computer Linguistics.
  5. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In ACL (1), pages 675-686. Association for Computational Linguistics.
  6. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with at- tentive recurrent neural networks. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016.
  7. Eric Chu and Peter J. Liu. 2018. Unsupervised neural multi-document abstractive summarization. CoRR, abs/1810.05739. · arXiv
  8. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers).
  9. Carlos A Colmenares, Marina Litvak, Amin Mantrach, and Fabrizio Silvestri. 2015. Heads: Headline gen- eration as sequence prediction using an abstract feature-rich space. In HLT-NAACL, pages 133-142.
  10. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -November 4, 2018.
  11. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to head- line generation. In HLT-NAACL.
  12. Katja Filippova and Yasemin Altun. 2013. Overcom- ing the lack of parallel data in sentence compression. In Proceedings of EMNLP, pages 1481-1491. Cite- seer.
  13. Kavita Ganesan. 2018. ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks. CoRR, abs/1803.01937. · arXiv
  14. Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-up abstractive summarization. In EMNLP, pages 4098-4109. Association for Com- putational Linguistics.
  15. Dimitra Gkatzia and Saad Mahamood. 2015. A snap- shot of NLG evaluation practices 2005 -2014. In ENLG 2015 -Proceedings of the 15th European Workshop on Natural Language Generation, 10-11 September 2015, University of Brighton, Brighton, UK, pages 57-60.
  16. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that re- quire simple lexical inferences. In ACL (2), pages 650-655. Association for Computational Linguis- tics.
  17. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image un- derstanding in visual question answering. In CVPR, pages 6325-6334. IEEE Computer Society.
  18. Yvette Graham. 2015. Re-evaluating automatic sum- marization with BLEU and 192 shades of ROUGE. In EMNLP, pages 128-137. The Association for Computational Linguistics.
  19. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Vol- ume 1 (Long Papers).
  20. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2018, Mel- bourne, Australia, July 15-20, 2018, Volume 1: Long Papers.
  21. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in nat- ural language inference data. In NAACL-HLT (2), pages 107-112. Association for Computational Lin- guistics.
  22. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015. Teaching ma- chines to read and comprehend. In NIPS.
  23. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi- document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden.
  24. Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers.
  25. Yichen Jiang and Mohit Bansal. 2018. Closed-book training to improve summarization encoder memory. In EMNLP, pages 4067-4077. Association for Com- putational Linguistics.
  26. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In EMNLP, pages 5010-5015. Association for Compu- tational Linguistics.
  27. Chris Kedzie, Kathleen R. McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In EMNLP, pages 1818-1828. Association for Computational Linguistics.
  28. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2018. Abstractive summarization of reddit posts with multi-level memory networks. CoRR, abs/1811.00783. · arXiv
  29. Mahnaz Koupaee and William Yang Wang. 2018. Wik- ihow: A large scale text summarization dataset. CoRR, abs/1810.09305. · arXiv
  30. Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In EMNLP, pages 1808-1817. Association for Computational Linguistics.
  31. Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. 2016. Reason- ing in vector space: An exploratory study of ques- tion answering. In ICLR.
  32. Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the SIGDIAL 2016 Conference, The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 13-15 September 2016, Los Angeles, CA, USA.
  33. Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018. Improving neural abstractive docu- ment summarization with structural regularization. In EMNLP, pages 4078-4087. Association for Com- putational Linguistics.
  34. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out, page 10.
  35. Zachary C. Lipton and Jacob Steinhardt. 2019. Trou- bling trends in machine learning scholarship. ACM Queue, 17(1):80.
  36. Feifan Liu and Yang Liu. 2010. Exploring correlation between ROUGE and human evaluation on meeting summaries. IEEE Trans. Audio, Speech & Language Processing, 18(1):187-196.
  37. Jingyun Liu, Jackie Chi Kit Cheung, and Annie Louis. 2019. What comes next? extractive sum- marization by next-sentence prediction. CoRR, abs/1901.03859. · arXiv
  38. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancou- ver, BC, Canada, April 30 -May 3, 2018, Confer- ence Track Proceedings.
  39. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based se- quence model for extractive summarization of docu- ments. In AAAI.
  40. Ramesh Nallapati, Bowen Zhou, C ¸aglar Gülc ¸ehre, Bing Xiang, et al. 2016a. Abstractive text sum- marization using sequence-to-sequence rnns and be- yond. Proceedings of SIGNLL Conference on Com- putational Natural Language Learning.
  41. Ramesh Nallapati, Bowen Zhou, and Mingbo Ma. 2016b. Classify or select: Neural architectures for extractive document summarization. CoRR, abs/1611.04244. · arXiv
  42. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summa- rization with reinforcement learning. In NAACL- HLT, pages 1747-1759. Association for Computa- tional Linguistics.
  43. Shashi Narayan, Nikos Papasarantopoulos, Mirella La- pata, and Shay B. Cohen. 2017. Neural extrac- tive summarization with side information. CoRR, abs/1704.04530. · arXiv
  44. Ani Nenkova and Rebecca J. Passonneau. 2004. Evalu- ating content selection in summarization: The pyra- mid method. In Human Language Technology Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2-7, 2004.
  45. Joel Larocca Neto, Alex A Freitas, and Celso AA Kaestner. 2002. Automatic text summarization us- ing a machine learning approach. In Brazilian Sym- posium on Artificial Intelligence, pages 205-215. Springer.
  46. Jun-Ping Ng and Viktoria Abrecht. 2015. Better sum- marization evaluation with word embeddings for ROUGE. CoRR, abs/1508.06034. · arXiv
  47. Benjamin Nye and Ani Nenkova. 2015. Identification and characterization of newsworthy verbs in world news. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Denver, Colorado, USA, May 31 -June 5, 2015.
  48. Karolina Owczarzak, Peter A. Rankel, Hoa Trang Dang, and John M. Conroy. 2012. Assessing the ef- fect of inconsistent assessors on summarization eval- uation. In ACL (2), pages 359-362. The Association for Computer Linguistics.
  49. Rebecca J. Passonneau, Emily Chen, Weiwei Guo, and Dolores Perin. 2013. Automated pyramid scoring of summaries using distributional semantics. In ACL (2), pages 143-147. The Association for Computer Linguistics.
  50. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. CoRR, abs/1804.06451. · arXiv
  51. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive sum- marization. In ICLR.
  52. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In *SEM@NAACL-HLT, pages 180-191. As- sociation for Computational Linguistics.
  53. Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT, pages 186-191. Association for Computational Linguistics.
  54. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3).
  55. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evalu- ating natural language generation systems. Compu- tational Linguistics, 35(4):529-558.
  56. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. Proceedings of EMNLP.
  57. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
  58. John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. 2015. Gradient estimation using stochastic computation graphs. In NIPS.
  59. Raphael Schumann. 2018. Unsupervised abstrac- tive sentence summarization using length controlled variational autoencoder. CoRR, abs/1809.05233. · arXiv
  60. D. Sculley, Jasper Snoek, Alexander B. Wiltschko, and Ali Rahimi. 2018. Winner's curse? on pace, progress, and empirical rigor. In ICLR (Workshop). OpenReview.net.
  61. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In ACL.
  62. Elaheh ShafieiBavani, Mohammad Ebrahimi, Ray- mond K. Wong, and Fang Chen. 2018. A graph- theoretic summary evaluation for rouge. In EMNLP, pages 762-767. Association for Computational Lin- guistics.
  63. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural net- works. In NIPS.
  64. Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hi- rao, and Masaaki Nagata. 2016. Neural headline generation on abstract meaning representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016.
  65. Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph- based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers.
  66. Liling Tan, Jon Dehdari, and Josef van Genabith. 2015. An awkward disparity between BLEU / RIBES scores and human judgements in machine transla- tion. In WAT, pages 74-81. Workshop on Asian Translation.
  67. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 Decem- ber 2017, Long Beach, CA, USA, pages 6000-6010.
  68. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In NIPS.
  69. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learn- ing. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intel- ligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018.
  70. Yongqin Xian, Bernt Schiele, and Zeynep Akata. 2017. Zero-shot learning -the good, the bad and the ugly. In CVPR, pages 3077-3086. IEEE Computer Soci- ety.
  71. Jiacheng Xu and Greg Durrett. 2019. Neural extrac- tive text summarization with syntactic compression. CoRR, abs/1902.00863. · arXiv
  72. Yinfei Yang and Ani Nenkova. 2014. Detecting information-dense texts in multiple news domains. In Proceedings of the Twenty-Eighth AAAI Confer- ence on Artificial Intelligence, July 27 -31, 2014, Québec City, Québec, Canada.
  73. Fangfang Zhang, Jin-ge Yao, and Rui Yan. 2018. On the abstractiveness of neural document summariza- tion. In EMNLP, pages 785-790. Association for Computational Linguistics.
  74. Liang Zhou, Chin-Yew Lin, Dragos Stefan Munteanu, and Eduard H. Hovy. 2006. Paraeval: Using para- phrases to evaluate summaries automatically. In Hu- man Language Technology Conference of the North American Chapter of the Association of Compu- tational Linguistics, Proceedings, June 4-9, 2006, New York, New York, USA.
  75. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural docu- ment summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics, ACL 2018, Melbourne, Australia, July 15- 20, 2018, Volume 1: Long Papers.

Cited by

Sorted by citations

Authors
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R.
Affiliations
Salesforce (United States) (US)
ORCID
Venue
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Publisher
Association for Computational Linguistics
Published
2019
Pages
540-551
DOI
10.18653/v1/d19-1051
OpenAlex
W2969286400
Cited by
335
References
3
Citations by year
2026: 9 · 2025: 23 · 2024: 21 · 2023: 73 · 2022: 59 · 2021: 88 · 2020: 56 · 2019: 6
FWCI
19.99
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
abstractive summarization, extractive summarization, summarization evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review