The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Topic Modeling for Short Texts with Auxiliary Word Embeddings
Abstract
For many applications that require semantic understanding of short texts, inferring discriminative and coherent latent topics from short texts is a critical and fundamental task. Conventional topic models largely rely on word co-occurrences to derive topics from a collection of documents. However, due to the length of each document, short texts are much more sparse in terms of word co-occurrences. Data sparsity therefore becomes a bottleneck for conventional topic models to achieve good results on short texts. On the other hand, when a human being interprets a piece of short text, the understanding is not solely based on its content words, but also her background knowledge (e.g., semantically related words). The recent advances in word embedding offer effective learning of word semantic relations from a large corpus. Exploiting such auxiliary word embeddings to enrich topic modeling for short texts is the main focus of this paper. To this end, we propose a simple, fast, and effective topic model for short texts, named GPU-DMM. Based on the Dirichlet Multinomial Mixture (DMM) model, GPU-DMM promotes the semantically related words under the same topic during the sampling process by using the generalized Polya urn (GPU) model. In this sense, the background knowledge about word semantic relatedness learned from millions of external documents can be easily exploited to improve topic modeling for short texts. Through extensive experiments on two real-world short text collections in two languages, we show that GPU-DMM achieves comparable or better topic representations than state-of-the-art models, measured by topic coherence. The learned topic representation leads to the best accuracy in text classification task, which is used as an indirect evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Probabilistic Latent Semantic Indexing
Hofmann, T. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130370 · Cited by 3,988
Probabilistic latent semantic indexing
Hofmann, T. (1999) · Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/312624.312649 · Cited by 3,976
Text classification from labeled and unlabeled documents using EM
Nigam, K., Mccallum, A. K., Thrun, S., & Mitchell, T. (2000) · Machine Learning · doi:10.1023/a:1007692713085 · Cited by 2,736 · cited 2×
“Another strategy is to restrict the document-topic distribution, such that each short text is sampled from a single topic, known as mixture of unigrams or Dirichlet Multinomial Mixture (DMM) model [26,40,42].”
BackgroundOptimizing Semantic Coherence in Topic Models
Mimno, D., Wallach, H., Talley, E. M., Leenders, M., & McCallum, A. (2011) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 2,493
(2009) · Cited by 1,900
Predicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
TwitterRank: finding topic-sensitive influential twitterers
Weng, J., Lim, E.-P., Jiang, J., & He, Q. (2010) · Proceedings of the third ACM international conference on Web search and data mining · doi:10.1145/1718487.1718520 · Cited by 1,742
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695
Collaborative topic modeling for recommending scientific articles
Wang, C., & Blei, D. M. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020480 · Cited by 1,527
Comparing Twitter and Traditional Media Using Topic Models
Zhao, W. X., Jiang, J., Weng, J., He, J., Lim, E.-P., Yan, H., & Li, X. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-20161-5_34 · Cited by 1,214
Empirical study of topic modeling in Twitter
Hong, L., & Davison, B. D. (2010) · Proceedings of the First Workshop on Social Media Analytics · doi:10.1145/1964858.1964870 · Cited by 1,120
A biterm topic model for short texts
Yan, X., Guo, J., Lan, Y., & Cheng, X. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488514 · Cited by 1,088
Automatic evaluation of topic coherence
Newman, D. J., Lau, J. H., Grieser, K., & Baldwin, T. (2010) · Cited by 846
A Scalable Hierarchical Distributed Language Model
Mnih, A., & Hinton, G. E. (2008) · Cited by 838
Phan, X.-H., Nguyen, L.-M., & Horiguchi, S. (2008) · Proceedings of the 17th international conference on World Wide Web · doi:10.1145/1367497.1367510 · Cited by 740
Short text classification in twitter to improve information filtering
Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., & Demirbas, M. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835643 · Cited by 727
Characterizing Microblogs with Topic Models
Ramage, D., Dumais, S., & Liebling, D. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14026 · Cited by 702
A dirichlet multinomial mixture model-based approach for short text clustering
Yin, J., & Wang, J. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623715 · Cited by 561
Fast collapsed gibbs sampling for latent dirichlet allocation
Porteous, I., Newman, D., Ihler, A., Asuncion, A., Smyth, P., & Welling, M. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401960 · Cited by 560
BTM: Topic Modeling over Short Texts
Cheng, X., Yan, X., Lan, Y., & Guo, J. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2313872 · Cited by 556
Neural Probabilistic Language Models
Bengio, Y., Schwenk, H., Senécal, J.-S., Morin, F., & Gauvain, J.-L. (2006) · Studies in Fuzziness and Soft Computing · doi:10.1007/10985687_6 · Cited by 540
Improving LDA topic models for microblogs via tweet pooling and automatic labeling
Mehrotra, R., Sanner, S., Buntine, W., & Xie, L. (2013) · Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2484028.2484166 · Cited by 461
Short Text Similarity with Word Embeddings
Kenter, T., & de Rijke, M. (2015) · Proceedings of the 24th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2806416.2806475 · Cited by 440
Improving Topic Models with Latent Feature Word Representations
Nguyen, D. Q., Billingsley, R., Du, L., & Johnson, M. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00140 · Cited by 315
Gaussian LDA for Topic Models with Word Embeddings
Das, R., Zaheer, M., & Dyer, C. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1077 · Cited by 313
Natural Language Processing (Almost) from Scratch
CollobertRonan, WestonJason, BottouLéon, KarlenMichael, KavukcuogluKoray, & KuksaPavel (2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078186 · Cited by 306
Transferring topical knowledge from auxiliary long texts for short text clustering
Jin, O., Liu, N. N., Zhao, K., Yu, Y., & Yang, Q. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063689 · Cited by 218
Mining topics in documents: standing on the shoulders of big data
Chen, Z., & Liu, B. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623622 · Cited by 162
Short and sparse text topic modeling via self-aggregation
Quan, X., Kit, C., Ge, Y., & Pan, S. J. (2015) · CityU Scholars · Cited by 159
Discovering coherent topics using general knowledge
Chen, Z., Mukherjee, A., Liu, B., Hsu, M., Castellanos, M., & Ghosh, R. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505519 · Cited by 130
Learning to Reweight Terms with Distributed Representations
Zheng, G., & Callan, J. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767700 · Cited by 120
Topic-driven reader comments summarization
Ma, Z., Sun, A., Yuan, Q., & Cong, G. (2012) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2396761.2396798 · Cited by 117
Self-citationShort text classification using very few words
Sun, A. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2348283.2348511 · Cited by 91
Self-citationTracking trends: incorporating term volume into temporal topic models
Hong, L., Yin, D., Guo, J. F., & Davison, B. D. (2011) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020485 · Cited by 58
Cited by
Short Text Topic Modeling Techniques, Applications, and Performance: A Survey
Qiang, J., Qian, Z., Li, Y., Yuan, Y., & Wu, X. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2992485 · Cited by 314
Topic Modeling Using Latent Dirichlet allocation: A Survey
Chauhan, U., & Shah, A. (2022) · ACM Computing Surveys · doi:10.1145/3462478 · Cited by 270
More Accounts, Fewer Links: How Algorithmic Curation Impacts Media Exposure in Twitter Timelines
Bandy, J., & Diakopoulos, N. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449152 · Cited by 54
“Another solution involves restrictions on the document-topic distribution, namely, using a Gibbs Sampling Dirichlet Multinomial Mixture (GSDMM) model [58, 101, 102] that assumes each text belongs to a single topic.”
BackgroundMethod
Metadata
- Authors
- Li, C., Wang, H., Zhang, Z., Sun, A., & Ma, Z.
- Venue
- Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2016-07-07
- Pages
- 165-174
- OpenAlex
- W2340381866
- Cited by
- 292
- References
- 40
- FWCI
- 18.70
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.