The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Evaluation methods for topic models
Open accessAbstract
A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049
Hierarchical Dirichlet Processes
Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000302 · Cited by 2,120
Marginal Likelihood from the Gibbs Output
Chib, S. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476635 · Cited by 1,905
Approximate Bayesian Inference with the Weighted Likelihood Bootstrap
Newton, M. A., & Raftery, A. E. (1994) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1994.tb01956.x · Cited by 1,487
The author-topic model for authors and documents
Rosen‐Zvi, M., Griffiths, T. L., Steyvers, M., & Smyth, P. (2004) · Uncertainty in Artificial Intelligence · doi:10.5555/1036843.1036902 · Cited by 1,280
Sequential Monte Carlo Samplers
Del Moral, P., Doucet, A., & Jasra, A. (2006) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2006.00553.x · Cited by 1,239
Neal, R. M. (2001) · Statistics and Computing · doi:10.1023/a:1008923215028 · Cited by 1,172
Topic modeling: beyond bag-of-words
Wallach, H. M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143967 · Cited by 1,058
Self-citationPachinko allocation: DAG-structured mixture models of topic correlations
Li, W., & McCallum, A. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143917 · Cited by 609
Cited by
(2009) · Cited by 1,900
stm : An R Package for Structural Topic Models
Roberts, M. E., Stewart, B. M., & Tingley, D. (2019) · Journal of Statistical Software · doi:10.18637/jss.v091.i02 · Cited by 1,714
LDAvis: A method for visualizing and interpreting topics
Sievert, C., & Shirley, K. (2014) · Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces · doi:10.3115/v1/w14-3110 · Cited by 1,366
topicmodels : An R Package for Fitting Topic Models
Grün, B., & Hornik, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v040.i13 · Cited by 1,072
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
Topic Modeling in Embedding Spaces
Dieng, A. B., Ruiz, F. J. R., & Blei, D. M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00325 · Cited by 780
Tirunillai, S., & Tellis, G. J. (2014) · Journal of Marketing Research · doi:10.1509/jmr.12.0106 · Cited by 774
A Model of Text for Experimentation in the Social Sciences
Roberts, M. E., Stewart, B. M., & Airoldi, E. M. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1141684 · Cited by 728
What are developers talking about? An analysis of topics and trends in Stack Overflow
Barua, A., Thomas, S. W., & Hassan, A. E. (2014) · Empirical Software Engineering · doi:10.1007/s10664-012-9231-y · Cited by 612
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
Denny, M. J., & Spirling, A. (2018) · Political Analysis · doi:10.1017/pan.2017.44 · Cited by 463
Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation
Syed, S., & Spruit, M. (2017) · 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2017.61 · Cited by 400
An analysis of the coherence of descriptors in topic modeling
O’Callaghan, D., Greene, D., Carthy, J., & Cunningham, P. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.02.055 · Cited by 370
Topic modelling for qualitative studies
Nikolenko, S. I., Koltcov, S., & Koltsova, O. (2017) · Journal of Information Science · doi:10.1177/0165551515617393 · Cited by 332
Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., & McCallum, A. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 2 - EMNLP '09 · doi:10.3115/1699571.1699627 · Cited by 328
Self-citationHacker, J., vom Brocke, J., Handali, J., Otto, M., & Schneider, J. (2020) · European Journal of Information Systems · doi:10.1080/0960085x.2020.1814680 · Cited by 287
Latent dirichlet allocation: stability and applications to studies of user-generated content
Koltcov, S., Koltsova, O., & Nikolenko, S. (2014) · Proceedings of the 2014 ACM conference on Web science · doi:10.1145/2615569.2615680 · Cited by 280
Sparse Additive Generative Models of Text
Eisenstein, J., Ahmed, A. A., & Xing, E. P. (2018) · Figshare · doi:10.1184/r1/6476342 · Cited by 277
Text Mining in Organizational Research
Kobayashi, V. B., Mol, S. T., Berkers, H. A., Kismihók, G., & Den Hartog, D. N. (2018) · Organizational Research Methods · doi:10.1177/1094428117722619 · Cited by 275
Topic Modeling Using Latent Dirichlet allocation: A Survey
Chauhan, U., & Shah, A. (2022) · ACM Computing Surveys · doi:10.1145/3462478 · Cited by 270
Topic Modeling: A Comprehensive Review
Kherwa, P., & Bansal, P. (2018) · ICST Transactions on Scalable Information Systems · doi:10.4108/eai.13-7-2018.159623 · Cited by 264
Text-mining the signals of climate change doubt
Boussalis, C., & Coan, T. G. (2016) · Global Environmental Change · doi:10.1016/j.gloenvcha.2015.12.001 · Cited by 263
Investigating task performance of probabilistic topic models: an empirical study of PLSA and LDA
Lu, Y., Mei, Q., & Zhai, C. (2011) · Information Retrieval · doi:10.1007/s10791-010-9141-9 · Cited by 259
Text mining for information systems researchers: An annotated topic modeling tutorial
University of Liechtenstein, Debortoli, S., Müller, O., IT University of Copenhagen, Junglas, I., Florida State University, vom Brocke, J., & University of Liechtenstein (2016) · Communications of the Association for Information Systems · doi:10.17705/1cais.03907 · Cited by 199
Fischer-Preßler, D., Schwemmer, C., & Fischbach, K. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.05.012 · Cited by 133
Cohen, R., Elhadad, M., & Elhadad, N. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-10 · Cited by 121
Mental health responses to COVID-19 around the world
Olff, M., Primasari, I., Qing, Y., Coimbra, B. M., Hovnanyan, A., Grace, E., Williamson, R. E., Hoeboer, C. M., & Consortium, T. G.-C. (2021) · European Journal of Psychotraumatology · doi:10.1080/20008198.2021.1929754 · Cited by 108
Enhancing scientific discoveries in molecular biology with deep generative models
Lopez, R., Gayoso, A., & Yosef, N. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199198 · Cited by 82
Redundancy-Aware Topic Modeling for Patient Record Notes
Cohen, R., Aviram, I., Elhadad, M., & Elhadad, N. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087555 · Cited by 71
Müller-Hansen, F., Callaghan, M. W., & Minx, J. C. (2020) · Energy Research & Social Science · doi:10.1016/j.erss.2020.101691 · Cited by 32
The impact of peer review on the contribution potential of scientific papers
Matsui, A., Chen, E., Wang, Y., & Ferrara, E. (2021) · PeerJ · doi:10.7717/peerj.11999 · Cited by 21
Metadata
- Authors
- Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D.
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 1105-1112
- OpenAlex
- W2144100511
- Cited by
- 863
- References
- 14
- FWCI
- 20.68
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.