Evaluation of unsupervised static topic models’ emergence detection ability
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.
Abstract
Detecting emerging topics is crucial for understanding research trends, technological advancements, and shifts in public discourse. While unsupervised topic modeling techniques such as Latent Dirichlet allocation (LDA), BERTopic, and CoWords clustering are widely used for topic extraction, their ability to retrospectively detect emerging topics without relying on ground truth labels has not been systematically compared. This gap largely stems from the lack of a dedicated evaluation metric for measuring emergence detection. In this study, we introduce a quantitative evaluation metric to assess the effectiveness of topic models in detecting emerging topics. We evaluate three topic modeling approaches using both qualitative analysis and our proposed emergence detection metric. Our results indicate that, qualitatively, CoWords identifies emerging topics earlier than LDA and BERTopics. Quantitatively, our evaluation metric demonstrates that LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0%. These findings highlight the strengths and limitations of different topic models for emergence detection, while our proposed metric provides a robust framework for future benchmarking in this area.
Overview
CoChat Summary
TL;DR
The paper introduces an unsupervised metric for retrospectively evaluating static topic models’ ability to detect emerging topics by comparing local and global model agreement at the document level. Across WoS, ACL, and Enron, CoWords qualitatively detects emergence earliest, while LDA quantitatively outperforms BERTopic.
Key findings
- Qualitatively, CoWords identifies emerging topics earlier than LDA and BERTopic; LDA captures steady trends, while BERTopic detects weak early signals but shows delayed and more volatile recognition.
- Quantitatively, LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0% (abstract); the results section reports LDA at 80.1% versus BERTopic at 56.6% on ACL, with F1 increasing as segment size grows.
- TWOR matching consistently retrieves more semantically coherent topics than KL divergence, which prioritizes similar temporal trends; for example, WoS TWOR matches diabetes-related terms while KL divergence retrieves bioelectromagnetic terms.
- Topic extraction quality varies by dataset: WoS and ACL produce clearer, more interpretable topics, while Enron yields broader and more ambiguous topics.
Methods
- Pipeline: preprocess time-stamped documents, train static topic models (CoWords, LDA, BERTopic) with 100 topics and 20 top words, extract prevalence trends, match topics hierarchically via Top Words Overlap Rate (TWOR) and KL divergence, then perform qualitative and quantitative emergence analysis.
- Data: Web of Science biomedical publications (171,499 documents), ACL anthology publications (63,199 after preprocessing), and Enron emails (439,819 after preprocessing), covering 1990–2020, 1980–2022, and 1998–2001 respectively.
- Quantitative metric: train a global model on the full dataset as a silver standard and local models on time segments with 80/20 train/test splits; classify documents as emerging if associated with topics showing positive growth rate, then compute precision, recall, and F1 agreement between local and global models. CoWords is not applicable because it does not produce document-topic distributions.
Limitations
- Stated: the study is retrospective and does not predict future topics; only three topic models are examined; datasets are domain-specific (scientific articles and emails); CoWords cannot be evaluated by the quantitative metric due to lacking document-topic distributions; the metric depends on topic granularity, the global model may overfit dominant trends, and external validation is absent.
- Stated: the qualitative conclusion that CoWords detects emergence earliest is based on limited qualitative results and may not generalize at larger scale.
- Evident: the abstract reports LDA’s average F1 as 80.6% while the results section reports 80.1%, an inconsistency in the paper’s own numbers.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
0 Valid · 5 Not valid · 10 Not checked · 65 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The WoS dataset captures the emergence of epidemiology-related topics, as shown in Fig. 3.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Qualitative analysis
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present a case study where our models detect an emerging topic related to Bias and Fairness in NLP research.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Qualitative analysis
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When evaluating model performance in detecting emerging topics, our proposed metric showed that LDA consistently outperformed BERTopic in terms of agreement between local and global models.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Conclusions
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Enron dataset produces broader, more ambiguous topics, reflecting the informal and unstructured nature of email communications, where multiple themes may coexist in a single document.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Discussion > Dataset difference in topic extraction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results indicate that LDA achieves a higher average F1 score (80.1%) compared to BERTopic (56.6%), suggesting that LDA more consistently detects emerging topics over time.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Quantitative analysis
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 80 claims and to verify the 11 references still open.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 22
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluation of unsupervised static topic models’ emergence detection ability > Abstract
“Quantitatively, our evaluation metric demonstrates that LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0%.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Quantitative analysis
“Results indicate that LDA achieves a higher average F1 score (80.1%) compared to BERTopic (56.6%), suggesting that LDA more consistently detects emerging topics over time.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluation of unsupervised static topic models’ emergence detection ability > Topic extraction pipeline > Matched topics emergence analysis
“(6) gk(ti,tj)=(Cik−Cjk)/Cik,in which Cik is the number of documents associated with topic k at time ti, reflecting how frequently the topic appears in that period. Topic k is emerging is gk(ti,tj)>0, k is not emerging otherwise.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Topic extraction pipeline > Matched topics emergence analysis
“Given a topic trend, determining if a topic is emerging in a certain time period fundamentally relies on if the trend shows an upward trajectory.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
THE STRUCTURE OF SCIENTIFIC REVOLUTIONS
MCGUIRE, J. E. (1963) · Philosophical Books · doi:10.1111/j.1468-0149.1963.tb00813.x · Cited by 8,424
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Blei, D. M., & Lafferty, J. D. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143859 · Cited by 3,173
Density-Based Clustering Based on Hierarchical Density Estimates
Campello, R. J. G. B., Moulavi, D., & Sander, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-37456-2_14 · Cited by 2,680
Exploring the Space of Topic Coherence Measures
Röder, M., Both, A., & Hinneburg, A. (2015) · Proceedings of the Eighth ACM International Conference on Web Search and Data Mining · doi:10.1145/2684822.2685324 · Cited by 2,224
Callon, M., Courtial, J. P., & Laville, F. (1991) · Scientometrics · doi:10.1007/bf02019280 · Cited by 2,090
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 16
Storkey, A. (2004) · neural information processing systems · Cited by 2,085
From translations to problematic networks: An introduction to co-word analysis
Callon, M., Courtial, J.-P., Turner, W. A., & Bauin, S. (1983) · Social Science Information · doi:10.1177/053901883022002003 · Cited by 1,951
(2009) · Cited by 1,900
Three frames for innovation policy: R&D, systems of innovation and transformative change
Schot, J., & Steinmueller, W. E. (2018) · Research Policy · doi:10.1016/j.respol.2018.08.011 · Cited by 1,801
Meme-tracking and the dynamics of the news cycle
Leskovec, J., Backstrom, L., & Kleinberg, J. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557077 · Cited by 1,530
BERTopic: Neural topic modeling with a class-based TF-IDF procedure
Grootendorst, M. (2022) · arXiv · doi:10.48550/arxiv.2203.05794 · Cited by 1,423
The Enron Corpus: A New Dataset for Email Classification Research
Klimt, B., & Yang, Y. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30115-8_22 · Cited by 1,157
Topic Modeling in Embedding Spaces
Dieng, A. B., Ruiz, F. J. R., & Blei, D. M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00325 · Cited by 780
Li, K., Rollins, J., & Yan, E. (2018) · Scientometrics · doi:10.1007/s11192-017-2622-5 · Cited by 732
Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes
Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2004) · UCL Discovery (University College London) · Cited by 589
Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality
Lau, J. H., Newman, D., & Baldwin, T. (2014) · Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/v1/e14-1056 · Cited by 571
Topic modeling algorithms and applications: A survey
Abdelrazek, A., Eid, Y., Gawish, E., Medhat, W., & Hassan, A. (2023) · Information Systems · doi:10.1016/j.is.2022.102131 · Cited by 304
Topics over Time: A NonMarkov ContinuousTime Model of Topical Trends
Wang, X., & McCallum, A. (2006) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 291
Exploring the Political Agenda of the European Parliament Using a Dynamic Topic Modeling Approach
Greene, D., & Cross, J. P. (2017) · Political Analysis · doi:10.1017/pan.2016.7 · Cited by 223
Nonnegative Matrix Factorization for Interactive Topic Modeling and Document Clustering
Kuang, D., Choo, J., & Park, H. (2014) · Springer eBooks · doi:10.1007/978-3-319-09259-1_7 · Cited by 180
Research on the semantic-based co-word analysis
Wang, Z., Li, G., Li, C., & Li, A. (2011) · Scientometrics · doi:10.1007/s11192-011-0563-y · Cited by 143
Chen, B., Tsutsui, S., Ding, Y., & Ma, F. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.10.003 · Cited by 133
Emerging topic detection for organizations from microblogs
Chen, Y., Amiri, H., Li, Z., & Chua, T. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484057 · Cited by 129
Visualizing research trends and research theme evolution in E-learning field: 1999–2018
Bai, Y., Li, H., & Yong, L. (2020) · Scientometrics · doi:10.1007/s11192-020-03760-7 · Cited by 77
Time-aware link prediction to explore network effects on temporal knowledge evolution
Choudhury, N., & Uddin, S. (2016) · Scientometrics · doi:10.1007/s11192-016-2003-5 · Cited by 69
Research addressing emerging technological ideas has greater scientific impact
Kwon, S., Liu, X., Porter, A. L., & Youtie, J. (2019) · Research Policy · doi:10.1016/j.respol.2019.103834 · Cited by 60
Evolution and diffusion of information literacy topics
Li, Y., Chen, Y., & Wang, Q. (2021) · Scientometrics · doi:10.1007/s11192-021-03925-y · Cited by 54
A Topic Coverage Approach to Evaluation of Topic Models
Korenčić, D., Ristov, S., Repar, J., & Šnajder, J. (2021) · IEEE Access · doi:10.1109/access.2021.3109425 · Cited by 45
Novelty Detection: A Perspective from Natural Language Processing
Ghosal, T., Saikh, T., Biswas, T., Ekbal, A., & Bhattacharyya, P. (2021) · Computational Linguistics · doi:10.1162/coli_a_00429 · Cited by 43
Yashuang, Q., Zhu, N., Zhai, Y., & Ding, Y. (2018) · Scientometrics · doi:10.1007/s11192-018-2693-y · Cited by 38
Mining the evolutionary process of knowledge through multiple relationships between keywords
Zhang, X., Xie, Q. S., Song, C., & Song, M. (2022) · Scientometrics · doi:10.1007/s11192-022-04272-2 · Cited by 33
Real-time Change Point Detection using On-line Topic Models.
Wang, Y., & Goutte, C. (2018) · NPARC · Cited by 22
Ghosal, T., Edithal, V., Ekbal, A., Bhattacharyya, P., Chivukula, S. S. S. K., & Tsatsaronis, G. (2020) · Natural Language Engineering · doi:10.1017/s1351324920000194 · Cited by 20
BERTrend: neural topic modeling for emerging trends detection
Boutaleb, A., Picault, J., & Grosjean, G. (2024) · Proceedings of the Workshop on the Future of Event Detection (FuturED) · doi:10.18653/v1/2024.futured-1.1 · Cited by 16
A Comparative Empirical Evaluation of Topic Modeling Techniques
Kherwa, P., & Bansal, P. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-981-15-5148-2_26 · Cited by 12
Mckinnon, T., & Rubino, C. R. G. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39b9r9za6x48 · Cited by 12
ATEM: A Topic Evolution Model for the Detection of Emerging Topics in Scientific Archives
Rahimi, H., Naacke, H., Constantin, C., & Amann, B. (2024) · Studies in Computational Intelligence · doi:10.1007/978-3-031-53472-0_28 · Cited by 9
Document Level Novelty Detection: Textual Entailment Lends a Helping Hand
Saikh, T., Ghosal, T., Ekbal, A., & Bhattacharyya, P. (2017) · International conference natural language processing · Cited by 9
A Quest to Detect Novelty Using Deep Neural Nets
Nandi, D., & Basak, R. (2020) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt49239.2020.9225588 · Cited by 8
Changepoint Analysis of Topic Proportions in Temporal Text DataPreprint
Bose, A., & Mukherjee, S. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.00827 · Cited by 7
Monitoring geometrical properties of word embeddings for detecting the emergence of new topics.
Christophe, C., Velcin, J., Cugliari, J., Boumghar, M., & Suignard, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.76 · Cited by 6
Incremental Learning of Topic Models for Finding Trend Topics in Scientific Publications
Gerasimenko, N. A., Chernyavsky, A. S., Никифорова, М. А., Nikitin, M. D., & Vorontsov, K. V. (2022) · Doklady Mathematics · doi:10.1134/s1064562422060084 · Cited by 6
Beyond Topic Modeling: Comparative Evaluation of Topic Interpretation by Large Language Models
de Melo, T., & Merialdo, P. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66336-9_16 · Cited by 6
Studying the Evolution of Scientific Topics and their Relationships
Uban, A. S., Caragea, C., & Dinu, L. P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.167 · Cited by 3
Finding early signals of emerging trends in text through topic modeling and anomaly detection
Redyuk, S. (2018) · DiVA (Digitala Vetenskapliga Arkivet) · Cited by 2
(2010)
A topic coverage approach to evaluation of topic models
Korenčić, D., Ristov, S., Repar, J., & Šnajder, J. (2020) · arXiv · doi:10.48550/arxiv.2012.06274
Topic Modeling in Embedding Spaces
Dieng, A. B., Ruiz, F. J. R., & Blei, D. M. (2019) · arXiv · doi:10.48550/arxiv.1907.04907
Research works printed but not linked: 1
- Rohatgi, 2022, ACL anthology corpus with full text
Cited by
Demir, O., & Saran, M. (2026) · IEEE Access · doi:10.1109/access.2026.3687277 · Cited by 1
Identifying and evaluating emerging scientific trends through temporal n-gram analysis
Štimec, G., Novalija, I., Brank, J., Šturm, J., Topal, O., Gheorghe, M., Grobelnik, M., & Roman, D. (2026) · Scientometrics · doi:10.1007/s11192-026-05744-5
Holistic Evaluations of Topic ModelsPreprint
Compton, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.23364
Methods and models for the analysis of scientific activity
Gubanov, D., Zhilyakova, L. Y., Kuznetsov, O. P., & Chkhartishvili, A. G. (2025) · Ontology of Designing · doi:10.18287/2223-9537-2025-15-4-578-597
From fine-tuning to prompting : a paradigm shift in knowledge graph construction
Li, X. (2026) · Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands
Self-citation
Metadata
- Authors
- Li, X., Esposito, C. D., Groth, P., Sitruk, J., Szatmari, B., & Wijnberg, N.
- Affiliations
- University of Amsterdam (NL)
- University of Johannesburg (ZA)
- Corresponding author
- Li, Xue
- Venue
- PeerJ Computer Science
- Publisher
- PeerJ
- Published
- 2025-05-30
- Volume
- 11
- Pages
- e2875
- ISSN
- 2376-5992 (ISSN-L)
- PMC
- PMC12192802
- OpenAlex
- W4410902617
- Semantic Scholar
- cb976e050e82231b604297e2a7a38807d0328433
- Cited by
- 9
- References
- 52
- Citations by year
- 2026: 6 · 2025: 3
- FWCI
- 1.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PeerJ Computer Science (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Dutch Research Council (MVI.19.032)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- unsupervised topic modeling, latent Dirichlet allocation, BERTopic, evaluation metrics, benchmarking, research trends, topic emergence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.