The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Topic Modeling in Embedding Spaces
Open accessAbstract
Topic modeling analyzes documents to learn meaningful patterns of words. However, existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. To this end, we develop the embedded topic model (etm), a generative model of documents that marries traditional topic models with word embeddings. More specifically, the etm models each word with a categorical distribution whose natural parameter is the inner product between the word’s embedding and an embedding of its assigned topic. To fit the etm, we develop an efficient amortized variational inference algorithm. The etm discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms existing document models, such as latent Dirichlet allocation, in terms of both topic quality and predictive performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 5×
“We focus on the continuous bag-of-words (CBOW) variant of word embeddings (Mikolov et al., 2013b).”
BackgroundMethodInfluentialAuto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Self-citationDistributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
Variational Inference: A Review for Statisticians
Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1285773 · Cited by 3,811
Self-citationAn Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,791
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.4082 · Cited by 2,405
Variational Inference: A Review for Statisticians
Blei, D. M., Kucukelbir, A., & McAuliffe, J. (2017) · Figshare · doi:10.6084/m9.figshare.5203696 · Cited by 1,919
Self-citationNeural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,582
Optimizing Semantic Coherence in Topic Models
Mimno, D., Wallach, H., Talley, E. M., Leenders, M., & McCallum, A. (2011) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 1,570
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533223 · Cited by 1,566
Online Learning for Latent Dirichlet Allocation
Hoffman, M. D., Blei, D. M., & Bach, F. (2010) · Cited by 1,363
Self-citationStochastic variational inference
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2502622 · Cited by 1,333
Self-citationThe author-topic model for authors and documents
Rosen‐Zvi, M., Griffiths, T. L., Steyvers, M., & Smyth, P. (2004) · Uncertainty in Artificial Intelligence · doi:10.5555/1036843.1036902 · Cited by 1,280
A correlated topic model of Science
Blei, D. M., & Lafferty, J. (2018) · doi:10.1184/r1/6587330.v1 · Cited by 1,088
Self-citationA correlated topic model of Science
Blei, D. M., & Lafferty, J. D. (2007) · The Annals of Applied Statistics · doi:10.1214/07-aoas114 · Cited by 953
Self-citationEvaluation methods for topic models
Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553515 · Cited by 863
Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality
Lau, J. H., Newman, D., & Baldwin, T. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1056 · Cited by 571
Neural Probabilistic Language Models
Bengio, Y., Schwenk, H., Senécal, J.-S., Morin, F., & Gauvain, J.-L. (2006) · Studies in Fuzziness and Soft Computing · doi:10.1007/10985687_6 · Cited by 540
Learning word embeddings efficiently with noise-contrastive estimation
Mnih, A., & Kavukcuoglu, K. (2013) · neural information processing systems · Cited by 446
Amortized Inference in Probabilistic Reasoning
Gershman, S. J., & Goodman, N. D. (2014) · eScholarship (California Digital Library) · Cited by 384
Improving Topic Models with Latent Feature Word Representations
Nguyen, D. Q., Billingsley, R., Du, L., & Johnson, M. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00140 · Cited by 315
Gaussian LDA for Topic Models with Word Embeddings
Das, R., Zaheer, M., & Dyer, C. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1077 · Cited by 313
A model for analogical reasoning
Rumelhart, D. E., & Abrahamson, A. A. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90023-6 · Cited by 253
Boyd-Graber, J., Hu, Y., & Mimno, D. (2017) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000030 · Cited by 246
Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vecPreprint
Moody, C. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.02019 · Cited by 234
Word Embedding for Understanding Natural Language: A Survey
Li, Y., & Yang, T. (2018) · Studies in Big Data · doi:10.1007/978-3-319-53817-4_4 · Cited by 226
Logistic-normal distributions:Some properties and uses
ATCHISON, J., & SHEN, S. M. (1980) · Biometrika · doi:10.1093/biomet/67.2.261 · Cited by 188
Word Features for Latent Dirichlet Allocation
Petterson, J., Buntine, W., Narayanamurthy, S., Caetano, T. S., & Smola, A. (2010) · Cited by 112
A Correlated Topic Model Using Word Embeddings
Xun, G., Li, Y., Zhao, W. X., Gao, J., & Zhang, A. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/588 · Cited by 103
Distilled Wasserstein Learning for Word Embedding and Topic ModelingPreprint
Xu, H., Wang, W., Liu, W., & Carin, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04705 · Cited by 91
Nonparametric Spherical Topic Modeling with Word Embeddings
Batmanghelich, K. N., Saeedi, A., Narasimhan, K., & Gershman, S. J. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2087 · Cited by 91
Incorporating Word Correlation Knowledge into Topic Modeling
Xie, P., Yang, D., & Xing, E. P. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1074 · Cited by 85
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic ModelingPreprint
Zhang, H., Chen, B., Guo, D., & Zhou, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01328 · Cited by 85
Jointly Learning Word Embeddings and Latent Topics
Shi, B., Lam, W. P., Jameel, S., Schockaert, S., & Lai, K. P. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080806 · Cited by 81
Topic Discovery for Short Texts Using Word Embeddings
Xun, G., Gopalakrishnan, V., Ma, F., Li, Y., Gao, J., & Zhang, A. (2016) · Proceedings · doi:10.1109/icdm.2016.0176 · Cited by 48
Rudolph, M. (2018) · Cambridge University Engineering Department Publications Database · doi:10.7916/d8nz9rht · Cited by 33
A Word Embeddings Informed Focused Topic Model
Zhao, H., Du, L., & Buntine, W. (2017) · Asian Conference on Machine Learning · Cited by 31
Bunk, S., & Krestel, R. (2018) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3197026.3197043 · Cited by 24
A Neural Framework for Generalized Topic Models.Preprint
Card, D., Tan, C., & Smith, N. A. (2017) · arXiv (Cornell University) · Cited by 16
Cited by
The neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
Topic modeling algorithms and applications: A survey
Abdelrazek, A., Eid, Y., Gawish, E., Medhat, W., & Hassan, A. (2023) · Information Systems · doi:10.1016/j.is.2022.102131 · Cited by 304
DestVI identifies continuums of cell types in spatial transcriptomics data
Lopez, R., Li, B., Keren-Shaul, H., Boyeau, P., Kedmi, M., Pilzer, D., Jelinski, A., Yofe, I., David, E., Wagner, A., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01272-8 · Cited by 276
Topic Modeling Using Latent Dirichlet allocation: A Survey
Chauhan, U., & Shah, A. (2022) · ACM Computing Surveys · doi:10.1145/3462478 · Cited by 270
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence
Bianchi, F., Terragni, S., & Hovy, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.96 · Cited by 269
Metadata
- Authors
- Dieng, A. B., Ruiz, F. J. R., & Blei, D. M.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2020-12
- Volume
- 8
- Pages
- 439-453
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2961152326
- Cited by
- 780
- References
- 45
- FWCI
- 33.81
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.