Text Segmentation by Cross Segment Attention
Open accessAbstract
Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents, which are commonly used to help downstream tasks such as information retrieval or text summarization.In this work, we propose three transformer-based architectures and provide comprehensive comparisons with previously proposed approaches on three standard datasets.We establish a new state-of-the-art, reducing in particular the error rates by a large margin in all cases.We further analyze model sizes and find that we can build models with many fewer parameters while keeping good performance, thus facilitating real-world applications.Early life and marriage: Franklin Delano Roosevelt was born on January 30, 1882, in the Hudson Valley town of Hyde Park, New York, to businessman James Roosevelt I and his second wife, Sara Ann Delano.(...) Aides began to refer to her at the time as "the president's girlfriend", and gossip linking the two romantically appeared in the newspapers.(...) Legacy: Roosevelt is widely considered to be one of the most important figures in the history of the United States, as well as one of the most influential figures of the 20th century.(...) Roosevelt has also appeared on several U.S. Postage stamps.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Neural Legal Judgment Prediction in English
Chalkidis, I., Androutsopoulos, I., & Aletras, N. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1424 · Cited by 352 · see citing works
Text Segmentation as a Supervised Learning Task
Koshorek, O., Cohen, A., Mor, N., Rotman, M., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2075 · Cited by 167
Flagged first, then by citations
Research works printed but not linked: 33
- Pinkesh Badjatiya, Litton J. Kurisinkel, Manish Gupta, and Vasudeva Varma. 2018. Attention-based neural text segmentation. CoRR, abs/1808.09935. · arXiv
- Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Ma- chine Learning, 34(1):177-210.
- Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Pro- ceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing, Philadelphia, PA, USA, August 20-23, 2006, pages 535-541. · DOI · Paper page
- Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged cor- pus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.
- Freddy Y. Y. Choi. 2000. Advances in domain inde- pendent linear text segmentation. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, NAACL 2000, pages 26-33, Stroudsburg, PA, USA. Associa- tion for Computational Linguistics.
- Jacob Eisenstein. 2009. Hierarchical text segmentation from multi-scale lexical cohesion. In Human Lan- guage Technologies: Conference of the North Amer- ican Chapter of the Association of Computational Linguistics, Proceedings,, pages 353-361.
- Marti Hearst. 1997. TextTiling: segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33-64.
- Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learn- ing Workshop.
- Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Gabriel Murray. 2019. Discourse analysis and its ap- plications. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics: Tutorial Abstracts, pages 12-17, Florence, Italy. As- sociation for Computational Linguistics. · DOI
- Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385-435. · DOI
- Jing Li, Aixin Sun, and Shafiq Joty. 2018. Segbot: A generic neural text segmentation model with pointer network. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intel- ligence, IJCAI-18, pages 4166-4172. International Joint Conferences on Artificial Intelligence Organi- zation. · DOI
- Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summa- rization. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 137-147, Los Angeles. Association for Computational Linguistics. · DOI
- Fernando Llopis, Antonio Ferrández Rodríguez, and José Luis Vicedo González. 2002. Text segmenta- tion for efficient information retrieval. In Proceed- ings of the Third International Conference on Com- putational Linguistics and Intelligent Text Process- ing, CICLing '02, pages 373-380, Berlin, Heidel- berg. Springer-Verlag.
- Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711.05101. · arXiv
- Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. 2020a. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.02819. · arXiv
- Michal Lukasik, Himanshu Jain, Aditya Menon, Se- ungyeon Kim, Srinadh Bhojanapalli, Felix Yu, and Sanjiv Kumar. 2020b. Semantic label smoothing for sequence to sequence problems. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing.
- Michal Lukasik and Richard Zens. 2018. Content ex- plorer: Recommending novel entities for a docu- ment writer. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 3371-3380, Brussels, Belgium. Associa- tion for Computational Linguistics. · DOI
- William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional the- ory of text organization. Text -Interdisciplinary Journal for the Study of Discourse, 8(3):243-281.
- Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT '94, pages 114-119, Stroudsburg, PA, USA. Association for Computational Linguistics. · DOI
- Pedro Mota, Maxine Eskenazi, and Luísa Coheur. 2019. BeamSeg: A joint model for multi-document seg- mentation and topic identification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 582-592. Asso- ciation for Computational Linguistics.
- Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1532-1543, Doha, Qatar. Asso- ciation for Computational Linguistics. · DOI · Paper page
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. CoRR, abs/1802.05365. · arXiv
- Lev Pevzner and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text seg- mentation. Comput. Linguist., 28(1):19-36.
- Gennady Shtekh, Polina Kazakova, Nikita Nikitinsky, and Nikolay Skachkov. 2018. Applying topic seg- mentation to document-level information retrieval. In Proceedings of the 14th Central and Eastern European Software Engineering Conference Russia, CEE-SECR '18, pages 6:1-6:6, New York, NY, USA. ACM. · DOI
- Radu Soricut and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical infor- mation. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, pages 228-235.
- Masao Utiyama and Hitoshi Isahara. 2001. A statis- tical model for domain-independent text segmenta- tion. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, pages 499-506.
- Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
- Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Proceedings of the 28th International Conference on Neural Informa- tion Processing Systems -Volume 2, NIPS'15, pages 2692-2700, Cambridge, MA, USA. MIT Press.
- Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward fast and accurate neural discourse segmen- tation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing EMNLP-18, pages 962-967. Association for Com- putational Linguistics.
- Jason Wei and Kai Zou. 2019. EDA: Easy data aug- mentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382-6388, Hong Kong, China. Association for Computational Linguistics. · DOI
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rud- nick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. · arXiv
- Ngo Xuan Bach, Nguyen Le Minh, and Akira Shi- mazu. 2012. A reranking model for discourse seg- mentation using subtree features. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 160-168, Seoul, South Korea. Association for Computational Linguistics.
- Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HI- BERT: Document level pre-training of hierarchical bidirectional transformers for document summariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059-5069. · DOI
Cited by
Sorted by citations
Membox: Weaving Topic Continuity into Long-Range Memory for LLM Agents
Tao, D., Ma, G., Huang, Y., & Jiang, M. (2026) · arXiv · doi:10.48550/arxiv.2601.03785
“Sub-sequent frameworks like MemoryBank (Zhong et al., 2024) and Ret-LLM (Modarressi et al., 2023) employed embedding-based indexing, while MemGPT (Packer et al., 2023) and SCM (Wang et al., 2023) introduced hierarchical or controller-based architectures for better long-term information management.”
- Authors
- Lukasik, M., Dadachev, B., Papineni, K., & Simões, G.
- Affiliations
- Google (United States) (US)
- ORCID
- Simões, Gonçalo
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 4707-4716
- OpenAlex
- W3022669779
- Cited by
- 68
- References
- 4
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 12 · 2023: 18 · 2022: 10 · 2021: 10 · 2020: 4
- FWCI
- 4.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Keywords
- text segmentation, document segmentation, discourse segmentation, transformer models, constituency parsing, model compression, parameter efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
