Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
In this work, we model abstractive text summarization using Attentional Encoder-Decoder Recurrent Neural Networks, and show that they achieve state-of-the-art performance on two different corpora.We propose several novel models that address critical problems in summarization that are not adequately modeled by the basic architecture, such as modeling key-words, capturing the hierarchy of sentence-toword structure, and emitting words that are rare or unseen at training time.Our work shows that many of our proposed models contribute to further improvement in performance.We also propose a new dataset consisting of multi-sentence summaries, and establish performance benchmarks for further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“AGGREFACT (Tang et al., 2023a) is a factual consistency evaluation benchmark for new summarization, targeting CNN(/DM) (Nallapati et al., 2016) and XSum (Narayan et al., 2018).”
From MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024) · cites this paper as Nallapati et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“The dataset we use from (Stiennon et al., 2022) (filtered from (Nallapati et al., 2016)) comes with train, validation and test splits, which we use throughout our work.”
From Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023) · cites this paper as Nallapati et al., 2016
In the citing paper
“For example, on CNN/DailyMail [83] and XSUM [84], fine-tuned models like Brio [75] and Pegasus [136] have much better performance than any LLMs w.r.t.ROUGE, but LLMs like OPT [137] perform far better in human evaluation considering all aspects including faithfulness, coherence, and relevance [138].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [83]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Training: For all the models we discuss below, we used word2vec vectors (Mikolov et al., 2013) trained on the same corpus to initialize the model embeddings, but we allowed the embeddings to be updated during training.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,986
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
LexRank: Graph-based Lexical Centrality as Salience in Text Summarization
Erkan, G., & Radev, D. R. (2004) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1523 · Cited by 1,700
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1053 · Cited by 1,172
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096
Abstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1012 · Cited by 881
On Using Very Large Target Vocabulary for Neural Machine Translation
Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1001 · Cited by 862
Neural Summarization by Extracting Sentences and Words
Cheng, J., & Lapata, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1046 · Cited by 790
Addressing the Rare Word Problem in Neural Machine Translation
Luong, T., Sutskever, I., Le, Q., Vinyals, O., & Zaremba, W. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1002 · Cited by 619
A Hierarchical Neural Autoencoder for Paragraphs and Documents
Li, J., Luong, T., & Jurafsky, D. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1107 · Cited by 490
Gulcehre, C., Ahn, S., Nallapati, R., Zhou, B., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1014 · Cited by 478
Self-citationLCSTS: A Large Scale Chinese Short Text Summarization Dataset
Hu, B., Chen, Q., & Zhu, F. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1229 · Cited by 316
Graph-based keyword extraction for single-document summarization
Litvak, M., & Last, M. (2008) · Proceedings of the Workshop on Multi-source Multilingual Information Extraction and Summarization - MMIES '08 · doi:10.3115/1613172.1613178 · Cited by 272
Headline generation based on statistical translation
Banko, M., Mittal, V. O., & Witbrock, M. J. (2000) · Proceedings of the 38th Annual Meeting on Association for Computational Linguistics - ACL '00 · doi:10.3115/1075218.1075259 · Cited by 240
Automatic Text Summarization Using a Machine Learning Approach
Neto, J. L., Freitas, A. A., & Kaestner, C. A. A. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-36127-8_20 · Cited by 232
Extractive summarization using supervised and semi-supervised learning
Wong, K.-F., Wu, M., & Li, W. (2008) · Proceedings of the 22nd International Conference on Computational Linguistics - COLING '08 · doi:10.3115/1599081.1599205 · Cited by 208
Sentence compression beyond word deletion
Cohn, T., & Lapata, M. (2008) · Proceedings of the 22nd International Conference on Computational Linguistics - COLING '08 · doi:10.3115/1599081.1599099 · Cited by 169
A Hierarchical Neural Autoencoder for Paragraphs and Documents
Li, J., Luong, M.-T., & Jurafsky, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.01057 · Cited by 169
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R. J., Darrell, T. J., & Saenko, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00487 · Cited by 136
Overcoming the Lack of Parallel Data in Sentence Compression
Filippova, K., & Altun, Y. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1155 · Cited by 132 · cited 1×
“A vast majority of past work in summarization has been extractive, which consists of identifying key sentences or passages in the source document and reproducing them as summary (Neto et al., 2002; Erkan and Radev, 2004; Wong et al., 2008a; Filippova and Altun, 2013; Colmenares et al., 2015; Litvak and Last, 2008; K.…”
BackgroundLong story short – Global unsupervised models for keyphrase based meeting summarization
Riedhammer, K., Favre, B., & Hakkani-Tür, D. Z. (2010) · Speech Communication · doi:10.1016/j.specom.2010.06.002 · Cited by 104
Zajic, D. M., Dorr, B. J., & Schwartz, R. M. (2004) · Cited by 93
Title Generation with Quasi-Synchronous Grammar
Woodsend, K., Feng, Y., & Lapata, M. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 63
HEADS: Headline Generation as Sequence Prediction Using an Abstract Feature-Rich Space
Colmenares, C. A., Litvak, M., Mantrach, A., & Silvestri, F. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1014 · Cited by 54
On Using Very Large Target Vocabulary for Neural Machine Translation
Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.2007 · Cited by 49
LCSTS: A Large Scale Chinese Short Text Summarization DatasetPreprint
Hu, B., Chen, Q., & Zhu, F. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.05865 · Cited by 23
Self reinforcement for important passage retrieval
Ribeiro, R., Marujo, L., de Matos, D. M., Neto, J. P. D. S., Gershman, A. V., & Carbonell, J. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484134 · Cited by 17
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · arXiv · doi:10.48550/arxiv.1509.00685
End-to-End Attention-based Large Vocabulary Speech Recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1508.04395
Research works printed but not linked: 1
- Kam-Fai Wong, Mingli Wu, and Wenjie Li. 2008b. Extractive summarization using supervised and semi-supervised learning. In Proceedings of the 22nd Annual Meeting of the Association for Computational Linguistics , pages 985-992.
Cited by
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850
“Sequence to sequence learning has been successful in many tasks such as machine translation, speech recognition (Sutskever et al., 2014; Chorowski et al., 2015) and text summarization (Rush et al., 2015; Nallapati et al., 2016; Shen et al., 2016) amongst others.”
Backgroundfairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4009 · Cited by 2,344
Metadata
- Authors
- Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B.
- Affiliations
- IBM (United States) (US)
- Venue
- Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning
- Published
- 2016
- Pages
- 280-290
- OpenAlex
- W2963929190
- Cited by
- 2,361
- References
- 32
- Citations by year
- 2026: 72 · 2025: 166 · 2024: 172 · 2023: 321 · 2022: 266 · 2021: 395 · 2020: 330 · 2019: 361 · 2018: 210 · 2017: 61 · 2016: 6
- Type
- conference abstract
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Computational Natural Language Learning (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, sequence-to-sequence models, recurrent neural networks, attention mechanism, encoder-decoder architecture, summarization datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.