CoChat
Sign Up
Cited by 108Topic ModelingComputational and Text Analysis MethodsNatural Language Processing Techniques

TopicGPT: A Prompt-based Topic Modeling Framework

Open access
Pham, Chau, Hoyle, Alexander, Sun, Simeng, Resnik, Philip, Iyyer, Mohit·2024·Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
DOI 10.18653/v1/2024.naacl-long.164
Open this paper in CoChat — free accountOpen PDF

Abstract

Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, Mohit Iyyer. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.naacl-long.164

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The template-based intermediate representation preserves narrative elements while abstracting away surface wording, forcing downstream stages to reason over narrative content rather than stylistic surface.

    “Templates convert prose into structured narrative fields, forcing downstream stages to reason over narrative content rather than stylistic surface.”

    From StoryScope: Investigating idiosyncrasies in AI fiction (2026) · cites this paper as Pham et al., 2024

    The Introduction and §3.2 describe generating topic labels and assigning them to documents, while Appendix J contains prompt templates for those tasks. Searches for narrative fields, an intermediate representation, and downstream reasoning over content rather than style found no passage addressing the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 43

  1. David Andrzejewski and Xiaojin Zhu. 2009. Latent Dirichlet Allocation with Topic-in-Set Knowledge. In Proceedings of the NAACL HLT 2009 Workshop on Semi-supervised Learning for Natural Language Processing, pages 43-48, Boulder, Colorado. Associ- ation for Computational Linguistics.
  2. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
  3. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. Ad- vances in neural information processing systems, 22.
  4. Jason Chuang, Sonal Gupta, Christopher Manning, and Jeffrey Heer. 2013. Topic model diagnostics: As- sessing domain relevance via topical alignment. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Ma- chine Learning Research, pages 612-620, Atlanta, Georgia, USA. PMLR.
  5. Caitlin Doogan and Wray Buntine. 2021. Topic model or topic twaddle? re-evaluating semantic inter- pretability measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3824-3848, Online. Association for Computational Linguistics. · DOI
  6. Ryan J. Gallagher, Kyle Reing, David Kale, and Greg Ver Steeg. 2017. Anchored Correlation Ex- planation: Topic Modeling with Minimal Domain Knowledge. Transactions of the Association for Com- putational Linguistics, 5:529-542.
  7. Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. 2003. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16.
  8. Tom Griffiths. 2002. Gibbs sampling in the genera- tive model of latent dirichlet allocation. Standford University, 518(11):1-3.
  9. Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. 2021. Is Automated Topic Model Evalu- ation Broken? The Incoherence of Coherence. In Advances in Neural Information Processing Systems, volume 34, pages 2018-2033.
  10. Alexander Miserlis Hoyle, Pranav Goel, Rupak Sarkar, and Philip Resnik. 2022. Are neural topic models broken? In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 5321-5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  11. Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three ap- proaches to qualitative content analysis. Qualitative health research, 15(9):1277-1288.
  12. Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of classification, 2:193-218.
  13. Jagadeesh Jagarlamudi, Hal Daumé III, and Raghaven- dra Udupa. 2012. Incorporating Lexical Priors into Topic Models. In Proceedings of the 13th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 204-213, Avignon, France. Association for Computational Linguistics.
  14. Damir Korenčić, Strahil Ristov, Jelena Repar, and Jan Šnajder. 2021. A Topic Coverage Approach to Eval- uation of Topic Models. IEEE Access, 9:123280- 123312. ArXiv:2012.06274 [cs]. · DOI
  15. Helvi Kyngäs. 2020. Inductive content analysis. The application of content analysis in nursing science research, pages 13-21.
  16. Jey Han Lau, Karl Grieser, David Newman, and Tim- othy Baldwin. 2011. Automatic labelling of topic models. In Proceedings of the 49th annual meet- ing of the association for computational linguistics: human language technologies, pages 1536-1545.
  17. Marsha M Linehan, Judith L Goodstein, Stevan L Nielsen, and John A Chiles. 1983. Reasons for stay- ing alive when you are thinking of killing yourself: the reasons for living inventory. Journal of consult- ing and clinical psychology, 51(2):276.
  18. Sengjie Liu and Christopher G Healey. 2023. Abstrac- tive summarization of large document collections using gpt. arXiv preprint arXiv:2310.05690. · arXiv
  19. Marina Meilȃ. 2007. Comparing clusterings-an in- formation based distance. Journal of multivariate analysis, 98(5):873-895.
  20. Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020. Dis- criminative Topic Mining via Category-Name Guided Text Embedding. In Proceedings of The Web Con- ference 2020, pages 2121-2132. ArXiv:1908.07162 · DOI
  21. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net.
  22. David Mimno, Wei Li, and Andrew McCallum. 2007. Mixtures of hierarchical topics with pachinko alloca- tion. In Proceedings of the 24th international confer- ence on Machine learning, pages 633-640.
  23. David Newman, Jey Han Lau, Karl Grieser, and Tim- othy Baldwin. 2010. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chap- ter of the association for computational linguistics, pages 100-108.
  24. Sergey I Nikolenko, Sergei Koltcov, and Olessia Koltsova. 2017. Topic modelling for qualitative stud- ies. Journal of Information Science, 43(1):88-102.
  25. John Paisley, Chong Wang, David M Blei, and Michael I Jordan. 2014. Nested hierarchical dirichlet processes. IEEE transactions on pattern analysis and machine intelligence, 37(2):256-270.
  26. Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Leah Findlater, and Kevin Seppi. 2016. Alto: Active learn- ing with topic overviews for speeding label induc- tion and document labeling. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1158-1169.
  27. Daniel Ramage, Christopher D. Manning, and Susan Dumais. 2011. Partially labeled topic models for in- terpretable text mining. In Proceedings of the 17th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 457-465, San Diego California USA. ACM. · DOI
  28. William M Rand. 1971. Objective criteria for the evalu- ation of clustering methods. Journal of the American Statistical association, 66(336):846-850.
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
  30. Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423.
  31. Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728-1736, Online. Association for Computa- tional Linguistics. · DOI
  32. Akash Srivastava and Charles Sutton. 2017. Autoen- coding variational inference for topic models. arXiv preprint arXiv:1703.01488. · arXiv
  33. Alexander Strehl and Joydeep Ghosh. 2002. Cluster ensembles-a knowledge reuse framework for com- bining multiple partitions. Journal of machine learn- ing research, 3(Dec):583-617. 11 2966
  34. Robert H Tai, Lillian R Bentley, Xin Xia, Jason M Sitt, Sarah C Fankhauser, Ana M Chicas-Mosier, and Barnas G Monteith. 2023. Use of large language models to aid analysis of textual data. bioRxiv, pages 2023-07.
  35. Yee Whye Teh, Michael I Jordan, Matthew J Beal, and David M Blei. 2006. Hierarchical Dirichlet Pro- cesses. Journal of the American Statistical Asso- ciation, 101(476):1566-1581. · DOI
  36. Laure Thompson and David Mimno. 2020. Topic Mod- eling with Contextualized Word Representation Clus- ters. ArXiv:2010.12626 [cs. · arXiv
  37. Danya F Vears and Lynn Gillam. 2022. Inductive con- tent analysis: A guide for beginning qualitative re- searchers. Focus on Health Professional Education: A Multi-disciplinary Journal, 23(1):111-127.
  38. Nguyen Xuan Vinh, Julien Epps, and James Bailey. 2009. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th annual international confer- ence on machine learning, pages 1073-1080.
  39. Hanna Wallach, David Mimno, and Andrew McCallum. 2009. Rethinking lda: Why priors matter. Advances in neural information processing systems, 22.
  40. Xiaojun Wan and Tianming Wang. 2016. Automatic labeling of topic models using text summaries. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2297-2305, Berlin, Germany. Association for Computational Linguistics. · DOI
  41. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. · arXiv
  42. Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023. ClusterLLM: Large Language Models as a Guide for Text Clustering. ArXiv:2305.14871 [cs]. · DOI
  43. Ying Zhao. 2005. Criterion Functions for Document Clustering. Ph.D. thesis, University of Minnesota, USA. AAI3180039. 12 2967 Prompt template for refining (merging) topics You will receive a list of topics that belong to the same level of a topic hierarchy. Your task is to merge topics that are paraphrases or near duplicates of one another. "None" if is needed. Merging topics ("[Taxes" and "

Cited by

Sorted by citations

  • StoryScope: Investigating idiosyncrasies in AI fiction

    Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., & Wieting, J. (2026) · arXiv · doi:10.48550/arxiv.2604.03136

    “10 The templates serve as a controlled intermediate representation (Pham et al., 2024; Wang et al., 2023): by converting prose into structured fields, the subsequent comparison stage reasons over narrative content rather than stylistic surface (see §B for details).”
    Self-citation

Authors
Pham, C., Hoyle, A., Sun, S., Resnik, P., & Iyyer, M.
Affiliations
  • University of Massachusetts Amherst (US)
  • University of Maryland, College Park (US)
ORCID
Venue
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2024
Pages
2956-2984
DOI
10.18653/v1/2024.naacl-long.164
OpenAlex
W4401043789
Cited by
108
References
0
Citations by year
2026: 57 · 2025: 40 · 2024: 9 · 2023: 1 · 2018: 1
FWCI
31.28
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
Funders
National Science Foundation (2202506, 2046248)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
topic modeling, prompt learning, large language models, topic generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review