Abstractive summarization of long texts by representing multiple compositionalities with temporal hierarchical pointer generator network
Abstract
In order to tackle the problem of abstractive summarization of long multi-sentence texts, it is critical to construct an efficient model, which can learn and represent multiple compositionalities better. In this paper, we introduce a temporal hierarchical pointer generator network that can represent multiple compositionalities in order to handle longer sequences of texts with a deep structure. We demonstrate how a multilayer gated recurrent neural network organizes itself with the help of an adaptive timescale in order to represent the compositions. The temporal hierarchical network is implemented with a multiple timescale architecture where the timescale of each layer is also learned during the training process through error backpropagation through time. We evaluate our proposed model using an Introduction-Abstract summarization dataset from scientific articles and the CNN/Daily Mail summarization benchmark dataset. The results illustrate that, we successfully implement a summary generation system for long texts by using the multiple timescale with adaptation concept. We also show that we have improved the summary generation system with our proposed model on the benchmark dataset.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
(2004) · Cited by 9,718 · see citing works
(2010) · Cited by 8,261 · see citing works
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
The Automatic Creation of Literature Abstracts
Luhn, H. P. (1958) · IBM Journal of Research and Development · doi:10.1147/rd.22.0159 · Cited by 3,278 · see citing works
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. H. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073465 · Cited by 1,591 · see citing works
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312 · see citing works
On the Properties of Neural Machine Translation: Encoder-Decoder ApproachesPreprint
Cho, K., van Merriënboer, B., Bahdanau, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.1259 · Cited by 1,112 · see citing works
Kriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093 · see citing works
Cortical tracking of hierarchical linguistic structures in connected speech
Ding, N., Melloni, L., Zhang, H., Tian, X., & Poeppel, D. (2016) · Nature Neuroscience · doi:10.1038/nn.4186 · Cited by 1,059 · see citing works
Hierarchical modularity in human brain functional networks
David Meunier, A. (2009) · Frontiers in Neuroinformatics · doi:10.3389/neuro.11.037.2009 · Cited by 660 · see citing works
International 1989 Joint Conference on Neural Networks
(1989) · doi:10.1109/ijcnn.1989 · Cited by 609 · see citing works
A Hierarchical Neural Autoencoder for Paragraphs and Documents
Li, J., Luong, M.-T., & Jurafsky, D. (2015) · doi:10.3115/v1/p15-1107 · Cited by 507 · see citing works
Yamashita, Y., & Tani, J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000220 · Cited by 468 · see citing works
Nenkova, A. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601984715 · Cited by 457 · see citing works
Dias, M. R., Ribeiro, R., & Pinto, H. S. (2019) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2024.2 · Cited by 355 · see citing works
Automatic summarising: The state of the art
Jones, K. S. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.03.009 · Cited by 354 · see citing works
The challenges of automatic summarization
Hahn, U., & Mani, I. (2000) · Computer · doi:10.1109/2.881692 · Cited by 352 · see citing works
Document Embedding with Paragraph VectorsPreprint
Dai, A. M., Olah, C., & Le, Q. V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.07998 · Cited by 352 · see citing works
A Field Guide to Dynamical Recurrent Networks
Kolen, J. F., & Kremer, S. C. (2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037 · Cited by 338 · see citing works
LCSTS: A Large Scale Chinese Short Text Summarization Dataset
Hu, B., Chen, Q., & Zhu, F. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1229 · Cited by 325 · see citing works
Simple Recurrent Units for Highly Parallelizable Recurrence
Lei, T., Zhang, Y., Wang, S. I., Dai, H., & Artzi, Y. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1477 · Cited by 315 · see citing works
Text summarization using unsupervised deep learning
Yousefi-Azar, M., & Hamey, L. (2017) · Expert systems with applications · doi:10.1016/j.eswa.2016.10.017 · Cited by 232 · see citing works
A Hierarchical Neural Autoencoder for Paragraphs and DocumentsPreprint
Li, J., Luong, M.-T., & Jurafsky, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.01057 · Cited by 169 · see citing works
Evaluation challenges in large-scale document summarization
Radev, D., Teufel, S., Saggion, H., et al. (2003) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1075096.1075144 · Cited by 154 · see citing works
Overcoming the Lack of Parallel Data in Sentence Compression
Filippova, K., & Altun, Y. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1155 · Cited by 132 · see citing works
Sequence-to-Sequence RNNs for Text SummarizationPreprint
Nallapati, R. M., Xiang, B., & Zhou, B. (2016) · arXiv (Cornell University) · Cited by 120 · see citing works
Multilevel structure in behaviour and in the brain: a model of Fuster's hierarchy
Botvinick, M. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2056 · Cited by 110 · see citing works
A Neural Attention Model for Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Cited by 89 · see citing works
A topic modeling based approach to novel document automatic summarization
Wu, Z., Li, L., Li, G., et al. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.04.054 · Cited by 79 · see citing works
Extractive vs. NLG-based abstractive summarization of evaluative text
Carenini, G., & Cheung, J. C. K. (2008) · doi:10.3115/1708322.1708330 · Cited by 69 · see citing works
Hearst, M. A., & Ostendorf, M. (2003) · Cited by 68 · see citing works
Motor primitive and sequence self-organization in a hierarchical recurrent neural network
Paine, R. W., & Tani, J. (2004) · Neural Networks · doi:10.1016/j.neunet.2004.08.005 · Cited by 67 · see citing works
Artificial Neural Networks and Machine Learning – ICANN 2012
Bhattacharya, B. I. 0.-0.-5.-0. S., Coyle, D., Maguire, L., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33269-2 · Cited by 66 · see citing works
Sample Efficient Text Summarization Using a Single Pre-Trained TransformerPreprint
Khandelwal, U., Clark, K., Jurafsky, D., & Kaiser, Ł. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08836 · Cited by 61 · see citing works
Towards Abstraction from Extraction: Multiple Timescale Gated Recurrent Unit for Summarization
Kim, M., Moirangthem, D. S., & Lee, M. (2016) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w16-1608 · Cited by 31 · see citing works
Self-citationJeong, H., Ko, Y., & Seo, J. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.05.001 · Cited by 27 · see citing works
Diversifying customer review rankings
Krestel, R., & Dokoohaki, N. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.02.008 · Cited by 21 · see citing works
Toward abstraction from multi-modal data: Empirical studies on multiple time-scale recurrent models
Zhong, J., Cangelosi, A., & Ogata, T. (2017) · doi:10.1109/ijcnn.2017.7966312 · Cited by 12 · see citing works
Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network
Heinrich, S., Weber, C., & Wermter, S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33269-2_70 · Cited by 9 · see citing works
Temporal hierarchies in multilayer gated recurrent neural networks for language models
Moirangthem, D. S., & Lee, M. (2017) · doi:10.1109/ijcnn.2017.7966115 · Cited by 9 · see citing works
Self-citationMoirangthem, D. S., Son, J., & Lee, M. (2017) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w17-2616 · Cited by 4 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
- Authors
- Moirangthem, D. S., & Lee, M.
- Affiliations
- Kyungpook National University (KR)
- Corresponding author
- Lee, Minho
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2020-04
- Volume
- 124
- Pages
- 1-11
- ISSN
- 0893-6080 (ISSN-L)
- OpenAlex
- W2998517938
- Cited by
- 30
- References
- 60
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 10 · 2023: 8 · 2022: 2 · 2021: 5 · 2020: 1
- Type
- conference abstract
- Language
- en
- Open access
- closed
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Institute for Information and Communications Technology Promotion (IITP)
- Ministry of Trade, Industry and Energy
- Ministry of Science and ICT, South Korea (10073162, 2016-0-00564)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- abstractive summarization, long text summarization, pointer-generator network, gated recurrent neural networks, backpropagation through time, CNN/DailyMail
- MeSH terms
- Machine Learning, Periodicals as Topic, Word Processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review