The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
An analysis of the coherence of descriptors in topic modeling
Open accessAbstract
We evaluate the coherence and generality of topic descriptors found by LDA and NMF.Six new and existing corpora were specifically compiled for this evaluation.A new coherence measure using word2vec-modeled term vector similarity is proposed.NMF regularly produces more coherent topics, where term weighting is influential.NMF may be more suitable for topic modeling of niche or non-mainstream corpora. In recent years, topic modeling has become an established method in the analysis of text corpora, with probabilistic techniques such as latent Dirichlet allocation (LDA) commonly employed for this purpose. However, it might be argued that adequate attention is often not paid to the issue of topic coherence, the semantic interpretability of the top terms usually used to describe discovered topics. Nevertheless, a number of studies have proposed measures for analyzing such coherence, where these have been largely focused on topics found by LDA, with matrix decomposition techniques such as Non-negative Matrix Factorization (NMF) being somewhat overlooked in comparison. This motivates the current work, where we compare and analyze topics found by popular variants of both NMF and LDA in multiple corpora in terms of both their coherence and associated generality, using a combination of existing and new measures, including one based on distributional semantics. Two out of three coherence measures find NMF to regularly produce more coherent topics, with higher levels of generality and redundancy observed with the LDA topic descriptors. In all cases, we observe that the associated term weighting strategy plays a major role. The results observed with NMF suggest that this may be a more suitable topic modeling method when analyzing certain corpora, such as those associated with niche or non-mainstream domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Learning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,868
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9 · Cited by 12,751
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
ALGORITHMS FOR NON-NEGATIVE MATRIX FACTORIZATION
Lee, D. D., & Seung, H. S. (2000) · neural information processing systems · Cited by 9,876
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049
ALGORITHMS FOR NON-NEGATIVE MATRIX FACTORIZATION
Seung, D., & Lee, L.-W. (2001) · Cited by 4,753
Software Framework for Topic Modelling with Large Corpora
Rek, R. R. U., & Sojka, P. (2010) · doi:10.13140/2.1.2393.1847 · Cited by 3,647
NLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
Blei, D. M., & Lafferty, J. D. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143859 · Cited by 3,173
Proceedings of the 23rd international conference on Machine learning
Cohen, W. W., & Moore, A. (2006) · Cited by 2,647
Optimizing Semantic Coherence in Topic Models
Mimno, D., Wallach, H., Talley, E. M., Leenders, M., & McCallum, A. (2011) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 2,493
Unsupervised Learning by Probabilistic Latent Semantic Analysis
Hofmann, T. (2001) · Machine Learning · doi:10.1023/a:1007617005950 · Cited by 2,456
Exploring the Space of Topic Coherence Measures
Röder, M., Both, A., & Hinneburg, A. (2015) · Proceedings of the Eighth ACM International Conference on Web Search and Data Mining · doi:10.1145/2684822.2685324 · Cited by 2,224
(2009) · Cited by 1,900
NLTK: The Natural Language Toolkit
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,844
Projected Gradient Methods for Nonnegative Matrix Factorization
Lin, C.-J. (2007) · Neural Computation · doi:10.1162/neco.2007.19.10.2756 · Cited by 1,737
Online Learning for Latent Dirichlet Allocation
Hoffman, M. D., Blei, D. M., & Bach, F. (2010) · Cited by 1,363
The author-topic model for authors and documents
Rosen‐Zvi, M., Griffiths, T. L., Steyvers, M., & Smyth, P. (2004) · Uncertainty in Artificial Intelligence · doi:10.5555/1036843.1036902 · Cited by 1,280
Steyvers, M., & Griffiths, T. (2015) · Handbook of Latent Semantic Analysis · doi:10.4324/9780203936399.ch21 · Cited by 1,011
Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI 2010)
Adams, R. P., Dahl, G. E., & Murray, I. (2010) · Cited by 966
Lafferty, J., & Blei, D. M. (2005) · neural information processing systems · Cited by 917
Hierarchical Topic Models and the Nested Chinese Restaurant Process
Griffiths, T. L., Jordan, M. I., Tenenbaum, J. B., & Blei, D. M. (2003) · Cited by 912
Evaluation methods for topic models
Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553515 · Cited by 863
Automatic evaluation of topic coherence
Newman, D. J., Lau, J. H., Grieser, K., & Baldwin, T. (2010) · Cited by 846
Normalized (pointwise) mutual information in collocation extraction
Bouma, G. (2009) · Cited by 845
Rethinking LDA: Why Priors Matter
Wallach, H., Mimno, D., & McCallum, A. (2009) · neural information processing systems · Cited by 816
Exploring Topic Coherence over Many Models and Many Topics
Stevens, K., Kegelmeyer, P., Andrzejewski, D., & Buttler, D. J. (2012) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 712
SVD based initialization: A head start for nonnegative matrix factorization
Boutsidis, C., & Gallopoulos, E. (2008) · Pattern Recognition · doi:10.1016/j.patcog.2007.09.010 · Cited by 695
Koehn, P., & Hoang, H. T. (2007) · The Association for Computational Linguistics · Cited by 673
New York Times Annotated Corpus
Sandhaus, E., & Linguistic Data Consortium (2012) · TIB Data Manager · doi:10.57702/qbrk4fuh · Cited by 586
Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality
Lau, J. H., Newman, D., & Baldwin, T. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1056 · Cited by 571
Boilerplate detection using shallow text features
Kohlschütter, C., Fankhauser, P., & Nejdl, W. (2010) · Proceedings of the third ACM international conference on Web search and data mining · doi:10.1145/1718487.1718542 · Cited by 476
Learning Topic Models -- Going beyond SVD
Arora, S., Ge, R., & Moitra, A. (2012) · 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2012.49 · Cited by 302
Evaluating Topic Coherence Using Distributional Semantics
Αλέτρας, Ν., & Stevenson, M. (2013) · Cited by 276
Relation between PLSA and NMF and implications
Gaussier, E., & Goutte, C. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076148 · Cited by 264
Proceedings of the 24th Annual International Conference on Machine Learning (ICML 2007)
Jaeger, M. (2007) · Cited by 218
Advances in Neural Information Processing Systems 17 (NIPS 2004)
Allan, M., & Williams, C. K. I. (2005) · Cited by 194
Modeling General and Specific Aspects of Documents with a Probabilistic Topic Model
Chemudugunta, C., Smyth, P., & Steyvers, M. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0035 · Cited by 186
Saha, A., & Sindhwani, V. (2012) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2124295.2124376 · Cited by 173
A Practical Algorithm for Topic Modeling with Provable Guarantees
Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag, D. A., Wu, Y., & Zhu, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.4777 · Cited by 161
Evaluating topic coherence measuresPreprint
Rosner, F., Hinneburg, A., Röder, M., Nettling, M., & Both, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.6397 · Cited by 112
Petrović, S., Osborne, M., & Lavrenko, V. P. (2010) · The Association for Computational Linguistics · Cited by 80
On collocations and topic models
Lau, J. H., Baldwin, T., & Newman, D. J. (2013) · ACM Transactions on Speech and Language Processing · doi:10.1145/2483969.2483972 · Cited by 78
An Analysis of Interactions within and between Extreme Right Communities in Social Media
O’Callaghan, D., Greene, D., Conway, M., Carthy, J., & Cunningham, P. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-45392-2_5 · Cited by 56
Self-citationAtzmueller, M., Chin, A., Helić, D., & Andreas Hotho, A. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-45392-2 · Cited by 14
Cited by
Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation
Syed, S., & Spruit, M. (2017) · 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2017.61 · Cited by 400
Topic modeling algorithms and applications: A survey
Abdelrazek, A., Eid, Y., Gawish, E., Medhat, W., & Hassan, A. (2023) · Information Systems · doi:10.1016/j.is.2022.102131 · Cited by 304
Gamage, D., Ghasiya, P., Bonagiri, V., Whiting, M. E., & Sasahara, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517446 · Cited by 79
“If Topic Coherence-Word2Vec (TC-W2V) metric is used with NMF, it can find the optimal number of topics automatically [44].”
MethodMüller-Hansen, F., Callaghan, M. W., & Minx, J. C. (2020) · Energy Research & Social Science · doi:10.1016/j.erss.2020.101691 · Cited by 32
Xue, Y., Chinapah, V., & Zhu, C. (2025) · Education Sciences · doi:10.3390/educsci15060650 · Cited by 30
Metadata
- Authors
- O’Callaghan, D., Greene, D., Carthy, J., & Cunningham, P.
- Venue
- Expert Systems with Applications
- Publisher
- Elsevier BV
- Published
- 2015-08
- Volume
- 42
- Issue
- 13
- Pages
- 5645-5657
- ISSN
- 0957-4174 (ISSN-L)
- OpenAlex
- W1988680150
- Cited by
- 370
- References
- 53
- FWCI
- 14.39
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.