Stochastic collapsed variational Bayesian inference for latent Dirichlet allocation
Open accessThe DOI matches its Crossref record; 27 of the 30 references it cites were found in a registry or an index.
Abstract
There has been an explosion in the amount of digital text information available in recent years, leading to challenges of scale for traditional inference algorithms for topic models. Recent advances in stochastic variational inference algorithms for latent Dirichlet allocation (LDA) have made it feasible to learn topic models on very large-scale corpora, but these methods do not currently take full advantage of the collapsed representation of the model. We propose a stochastic algorithm for collapsed variational Bayesian inference for LDA, which is simpler and more efficient than the state of the art method. In experiments on large-scale text corpora, the algorithm was found to converge faster and often to a better solution than previous methods. Human-subject experiments also demonstrated that the method can learn coherent topics in seconds on small corpora, facilitating the use of topic models in interactive document analysis software.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As outlined in [19], the SCVB0 inference algorithm is essentially equivalent to variants of the online-EM [23] algorithm with a modified prior.”
From Salmon provides fast and bias-aware quantification of transcript expression (2017) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049 · cited 2×
“collapsed space are more efficient at improving held-out log probability than their uncollapsed counterparts, both per iteration and in wall-clock time per iteration [12, 24, 3].”
BackgroundMethodA View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187
Hierarchical Dirichlet Processes
Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000302 · Cited by 2,120
(2009) · Cited by 1,900
Online Learning for Latent Dirichlet Allocation
Hoffman, M. D., Blei, D. M., & Bach, F. (2010) · Cited by 1,469
Stochastic variational inference
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2502622 · Cited by 1,333
Julia: A Fast Dynamic Language for Technical Computing
Anderson, T. A., Liu, H., Kuper, L., Totoni, E., Vitek, J., & Shpeisman, T. (2012) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2017.4 · Cited by 627
Fast collapsed gibbs sampling for latent dirichlet allocation
Porteous, I., Newman, D., Ihler, A., Asuncion, A., Smyth, P., & Welling, M. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401960 · Cited by 560
Self-citationA Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation
Teh, Y. W., Newman, D., & Welling, M. (2007) · doi:10.21236/ada629956 · Cited by 490
Self-citationDistributed Algorithms for Topic Models
Newman, D. J., Asuncion, A., Smyth, P., & Welling, M. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1755845 · Cited by 483
On-Line Expectation–Maximization Algorithm for latent Data Models
Cappé, O., & Moulines, E. (2009) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2009.00698.x · Cited by 477
On Smoothing and Inference for Topic Models
Asuncion, A., Welling, M., Smyth, P., & Teh, Y. W. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2662 · Cited by 434
Self-citationEfficient methods for topic model inference on streaming document collections
Yao, L., Mimno, D., & McCallum, A. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557121 · Cited by 410
Uncertainty in artificial intelligence
Parsons, S. (1994) · The Knowledge Engineering Review · doi:10.1017/s0269888900006597 · Cited by 406
An architecture for parallel topic models
Smola, A., & Narayanamurthy, S. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920931 · Cited by 399
Stability of Stochastic Approximation under Verifiable Conditions
Andrieu, C., Moulines, É., & Priouret, P. (2005) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012902417267 · Cited by 194
Sparse Stochastic Inference for Latent Dirichlet allocationPreprint
Mimno, D., Hoffman, M., & Blei, D. M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6425 · Cited by 123
Topic Models over Text Streams: A Study of Batch and Online Unsupervised Learning
Banerjee, A., & Basu, S. (2007) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972771.40 · Cited by 87
Computational historiography: Data mining in a century of classics journals
Mimno, D. (2012) · Journal on Computing and Cultural Heritage · doi:10.1145/2160165.2160168 · Cited by 83
Topic models: A novel method for modeling couple and family text data.
Atkins, D. C., Rubin, T. N., Steyvers, M., Doeden, M. A., Baucom, B. R., & Christensen, A. (2012) · Journal of Family Psychology · doi:10.1037/a0029607 · Cited by 70
Online Inference for the Infinite Topic-Cluster Model: Storylines from Streaming Text
Ahmed, A. A., Ho, Q., Teo, C. H., Eisenstein, J., Smola, A. J., & Xing, E. P. (2018) · Figshare · doi:10.1184/r1/6476222 · Cited by 64
Minka, T. (2004) · Cited by 49
Rethinking Collapsed Variational Bayes Inference for LDAPreprint
Sato, I., & Nakagawa, H. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6435 · Cited by 35
Practical collapsed variational bayes inference for hierarchical dirichlet process
Sato, I., Kurihara, K., & Nakagawa, H. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339550 · Cited by 28
Reconstructing Pompeian HouseholdsPreprint
Mimno, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.3747 · Cited by 12
Research works printed but not linked: 2
- [10] B. Carpenter. Integrating out multinomial parameters in latent Dirichlet allocation and naive bayes for collapsed Gibbs sampling. Technical report, LingPipe, 2010. -
- [17] T. Minka. Power EP. Technical report, Microsoft Research, Cambridge, UK, 2004. -
Cited by
Salmon provides fast and bias-aware quantification of transcript expression
Patro, R., Duggal, G., Love, M. I., Irizarry, R. A., & Kingsford, C. (2017) · Nature Methods · doi:10.1038/nmeth.4197 · Cited by 12,700
Joint Modeling of User Check-in Behaviors for Real-time Point-of-Interest Recommendation
Yin, H., Cui, B., Zhou, X., Wang, W., Huang, Z., & Sadiq, S. (2017) · ACM Transactions on Information Systems · doi:10.1145/2873055 · Cited by 171
RNA Sequencing Data: Hitchhiker's Guide to Expression Analysis
den Berge, K. V., Hembach, K., Soneson, C., Tiberi, S., Clement, L., Love, M. I., Patro, R., & Robinson, M. D. (2019) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-072018-021255 · Cited by 180
Patro, R., Duggal, G., & Kingsford, C. (2012) · KiltHub Repository · doi:10.1184/r1/6702233 · Cited by 100
Topic modelling for theme park online reviews: analysis of Disneyland
Luo, J. M., Vu, H. Q., Li, G., & Law, R. (2020) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2020.1740138 · Cited by 95
Metadata
- Authors
- Foulds, J., Boyles, L., DuBois, C., Smyth, P., & Welling, M.
- Affiliations
- University of California, Irvine (US)
- University of Amsterdam (NL)
- Venue
- Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2013-08-11
- Pages
- 446-454
- OpenAlex
- W2001975024
- Cited by
- 126
- References
- 30
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 2 · 2023: 8 · 2022: 8 · 2021: 11 · 2020: 8 · 2019: 14 · 2018: 10 · 2017: 13 · 2016: 15 · 2015: 16 · 2014: 14 · 2013: 2
- FWCI
- 10.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Methods and Mixture Models
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- latent Dirichlet allocation, topic modeling, stochastic variational inference, topic coherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.