CoChat
Sign Up
Cited by 85Topic ModelingText and Document Classification TechnologiesComputational and Text Analysis Methods

SECTOR: A Neural Model for Coherent Topic Segmentation and Classification

Open access
Arnold, Sebastian, Schneider, Rudolf, Cudré-Mauroux, Philippe, Gers, Felix A., Löser, Alexander·2019·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00261
Open this paper in CoChat — free account

Abstract

When searching for information, a human reader first glances over a document, spots relevant sections, and then focuses on a few sentences for resolving her intention. However, the high variance of document structure complicates the identification of the salient topic of a given section at a glance. To tackle this challenge, we present SECTOR, a model to support machine reading systems by segmenting documents into coherent sections and assigning topic labels to each section. Our deep neural network architecture learns a latent topic embedding over the course of a document. This can be leveraged to classify local topics from plain text and segment a document at topic shifts. In addition, we contribute WikiSection, a publicly available data set with 242k labeled sections in English and German from two distinct domains: diseases and cities. From our extensive evaluation of 20 architectures, we report a highest score of 71.6% F1 for the segmentation and classification of 30 topics from the English city domain, scored by our SECTOR long short-term memory model with Bloom filter embeddings and bidirectional segmentation. This is a significant improvement of 29.5 points F1 over state-of-the-art CNN classifiers with baseline segmentation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00261

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 85 · Cited by 85

Research works printed but not linked: 58

  1. Shashank Agarwal and Hong Yu. 2009. Automatically classifying sentences in full-text biomedical articles into introduction, methods, results and discussion. Bioinformatics , 25(23):31743180.
  2. Yamen Ajjour, Wei-Fan Chen, Johannes Kiesel, Henning Wachsmuth, and Benno Stein. 2017. Unit segmentation of argumentative texts. In Proceedings of the 4th Workshop on Argument Mining , pages 118-128.
  3. Alexander A. Alemi and Paul Ginsparg. 2015. Text segmentation based on semantic word embeddings. CoRR , cs.CL/1503.05543v1.
  4. James Allan. 2002. Introduction to topic detection and tracking. In Topic Detection and Tracking , pages 1-16. Springer.
  5. Loulwah AlSumait, Daniel Barbará, and Carlotta Domeniconi. 2008. On-line LDA: Adaptive topic models for mining text streams with applications to topic detection and tracking. In Eighth IEEE International Conference on Data Mining , pages 3-12. IEEE.
  6. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In ICLR 2017: 5th International Conference on Learning Representations .
  7. M. Bayomi, K. Levacher, M. R. Ghorab, and S. Lawless. 2015. OntoSeg: A novel approach to text segmentation using ontological similarity. In 2015 International Conference on Data Mining Workshop , pages 1274-1283. IEEE.
  8. Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Machine Learning , 34(1):177-210.
  9. Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. 2016. Automatic labelling of topics with neural embeddings. In Proceedings of the 26th International Conference on Computational Linguistics , pages 953-963.
  10. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research , 3(Jan):9931022.
  11. Harr Chen, S. R. K. Branavan, Regina Barzilay, and David R. Karger. 2009. Global models of document structure using latent permutations. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 371-379. ACL.
  12. Freddy Y. Y. Choi. 2000. Advances in domain independent linear text segmentation. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference , pages 26-33. ACL.
  13. Philipp Cimiano, Antje Schultz, Sergej Sizov, Philipp Sorg, and Steffen Staab. 2009. Explicit versus latent concept models for cross-language information retrieval. In Proceedings of the 21st International Joint Conference on Artifical Intelligence , volume 9, pages 1513-1518.
  14. Daniel Cohen, Liu Yang, and W. Bruce Croft. 2018. WikiPassageQA: A benchmark collection for research on non-factoid answer passage retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 1165-1168. ACM.
  15. Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017. Very deep convolutional networks for text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics , volume 1, pages 1107-1116. ACL.
  16. Gaël Dias, Elsa Alves, and José Gabriel Pereira Lopes. 2007. Topic segmentation algorithms for text summarization and passage retrieval: An exhaustive evaluation. In Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence , volume 7, pages 1334-1340.
  17. Adji B. Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2017. TopicRNN: A recurrent neural network with long-range semantic dependency. In ICLR 2017: 5th International Conference on Learning Representations .
  18. Lan Du, Wray Buntine, and Mark Johnson. 2013. Topic segmentation with a structured topic model. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 190-200.
  19. Jacob Eisenstein and Regina Barzilay. 2008. Bayesian unsupervised topic segmentation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing , pages 334-343. ACL.
  20. George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais. 1987. The vocabulary problem in human-system communication. Communications of the ACM , 30(11):964-971.
  21. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence , pages 1606-1611.
  22. Felix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural Compututation , 12(10):2451-2471.
  23. Goran Glavaš, Federico Nanni, and Simone Paolo Ponzetto. 2016. Unsupervised text segmentation using semantic relatedness graphs. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics , pages 125130. ACL.
  24. Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks , volume 385. Springer, Berlin Heidelberg.
  25. Marti A. Hearst. 1997. TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics , 23(1):33-64.
  26. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. WikiReading: A novel large-scale language understanding task over Wikipedia. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics , volume 1, pages 1535-1545. ACL.
  27. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1367-1377.
  28. Julia Hirschberg and Christopher D. Manning. 2015. Advances in natural language processing. Science , 349(6245):261-266.
  29. Hoa T. Le, Christophe Cerisara, and Alexandre Denis. 2018. Do convolutional networks need to be deep for text classification? In Association for the Advancement of Artificial Intelligence 2018 Workshop on Affective Content Analysis , pages 29-36.
  30. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation , 9(8):1735-1780.
  31. Xiangji Huang, Fuchun Peng, Dale Schuurmans, Nick Cercone, and Stephen E. Robertson. 2003. Applying machine learning to text segmentation for information retrieval. Information Retrieval , 6(3-4):333-362.
  32. Minwoo Jeong and Ivan Titov. 2010. Multidocument topic segmentation. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management , pages 1119-1128. ACM.
  33. Bin Jiang. 2012. Head/tail breaks: A new classification scheme for data with a heavy-tailed distribution. The Professional Geographer , 65(3):482-494.
  34. Hubert Jin, Rich Schwartz, Sreenivasa Sista, and Frederick Walls. 1999. Topic tracking for radio, TV broadcast and newswire. In Proceedings of the DARPA Broadcast News Workshop , pages 199-204. Morgan Kaufmann.
  35. Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. In European Conference on Machine Learning , pages 137-142. Springer.
  36. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , pages 17461751.
  37. Giridhar Kumaran and James Allan. 2004. Text classification and named entities for new event detection. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 297-304. ACM.
  38. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning , volume 32, pages 1188-1196.
  39. Kalev Leetaru and Philip A. Schrodt. 2013. GDELT: Global data on events, location, and tone, 1979-2012. In ISA Annual Convention , volume 2, pages 1-49. Citeseer.
  40. Jialu Liu, Xiang Ren, Jingbo Shang, Taylor Cassidy, Clare R. Voss, and Jiawei Han. 2016. Representing documents via latent keyphrase inference. In Proceedings of the 25th International Conference on World Wide Web , pages 10571067.
  41. Xiaoyong Liu and W. Bruce Croft. 2002. Passage retrieval based on language models. In Proceedings of the Eleventh International Conference on Information and Knowledge Management , pages 375-382. ACM.
  42. Sean MacAvaney, Andrew Yates, Arman Cohan, Luca Soldaini, Kai Hui, Nazli Goharian, and Ophir Frieder. 2018. Characterizing question facets for complex answer retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 1205-1208. ACM Press.
  43. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR , cs.CL/1301.3781v3.
  44. Marwa Naili, Anja Habacha Chaïbi, and Henda Hajjami Ben Ghézala. 2017. Comparative study of word embedding methods in topic segmentation. In Proceedings of the 21st International Conference Knowledge-Based and Intelligent Information & Engineering Systems , volume 112 of Procedia Computer Science , pages 340-349. Elsevier.
  45. Federico Nanni, Bhaskar Mitra, Matt Magnusson, and Laura Dietz. 2017. Benchmark for complex answer retrieval. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval , pages 293-296. ACM.
  46. Mark E. J. Newman. 2006. Finding community structure in networks using the eigenvectors of matrices. Physical Review E , 74(3):036104.
  47. Tiziano Piccardi, Michele Catasta, Leila Zia, and Robert West. 2018. Structuring Wikipedia articles with section recommendations. In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 665-674. ACM.
  48. Yashoteja Prabhu and Manik Varma. 2014. FastXML: A fast, accurate and stable treeclassifier for extreme multi-label learning. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 263-272. ACM.
  49. Martin Riedl and Chris Biemann. 2012. TopicTiling: A text segmentation algorithm based on LDA. In Proceedings of ACL 2012 Student Research Workshop , pages 37-42. ACL.
  50. Cicero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing , pages 626-634. ACL.
  51. Imran Sehikh, Dominique Fohr, and Irina Illina. 2017. Topic segmentation in ASR transcripts using bidirectional RNNs for change detection. In Automatic Speech Recognition and Understanding Workshop , pages 512-518. IEEE.
  52. Joan Serrà and Alexandros Karatzoglou. 2017. Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks. In Proceedings of the Eleventh ACM Conference on Recommender Systems , pages 279-287. ACM.
  53. Thomas Pellissier Tanon, Denny Vrandecic, Sebastian Schaffert, Thomas Steiner, and Lydia Pintscher. 2016. From Freebase to Wikidata: The great migration. In Proceedings of the 25th International Conference on World Wide Web , pages 1419-1428.
  54. Michael Tepper, Daniel Capurro, Fei Xia, Lucy Vanderwende, and Meliha Yetisgen-Yildiz. 2012. Statistical section segmentation in freetext clinical records. In Proceedings of the Eighth International Conference on Language Resources and Evaluation , pages 2001-2008.
  55. George Tsatsaronis, Michael Schroeder, Georgios Paliouras, Yannis Almirantis, Ion Androutsopoulos, Eric Gaussier, Patrick Gallinari, Thierry Artieres, Michael R. Alvers, and Matthias Zschunke. 2012. BioASQ: A challenge on large-scale biomedical semantic indexing and question answering. In AAAI Technical Report FS-12-05 Information Retrieval and Knowledge Discovery in Biomedical Text , pages 92-98.
  56. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2009. Mining multi-label data. In Data Mining and Knowledge Discovery Handbook , pages 667-685. Springer.
  57. Masao Utiyama and Hitoshi Isahara. 2001. A statistical model for domain-independent text segmentation. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics , pages 499-506. ACL.
  58. Liang Wang, Sujian Li, Yajuan Lyu, and Houfeng Wang. 2017. Learning to rank semantic coherence for topic segmentation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 13401344. ACL.

Authors
Arnold, S., Schneider, R., Cudré-Mauroux, P., Gers, F. A., & Löser, A.
Affiliations
  • Berliner Hochschule für Technik (DE)
  • University of Fribourg (CH)
Corresponding author
  • Arnold, Sebastian
  • Schneider, Rudolf
  • Cudré-Mauroux, Philippe
  • Gers, Felix A.
  • Löser, Alexander
ORCID
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2019-11
Volume
7
Pages
169-184
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl_a_00261
OpenAlex
W2914694065
Cited by
85
References
85
Citations by year
2026: 2 · 2025: 13 · 2024: 9 · 2023: 22 · 2022: 12 · 2021: 16 · 2020: 10 · 2019: 1
FWCI
4.50
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • Bundesministerium für Wirtschaft und Energie
  • Horizon 2020 Framework Programme (732328)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
topic classification, sectors, long short-term memory, document structure, topic shift

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review