Evaluation of unsupervised static topic models’ emergence detection ability
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.
Abstract
Detecting emerging topics is crucial for understanding research trends, technological advancements, and shifts in public discourse. While unsupervised topic modeling techniques such as Latent Dirichlet allocation (LDA), BERTopic, and CoWords clustering are widely used for topic extraction, their ability to retrospectively detect emerging topics without relying on ground truth labels has not been systematically compared. This gap largely stems from the lack of a dedicated evaluation metric for measuring emergence detection. In this study, we introduce a quantitative evaluation metric to assess the effectiveness of topic models in detecting emerging topics. We evaluate three topic modeling approaches using both qualitative analysis and our proposed emergence detection metric. Our results indicate that, qualitatively, CoWords identifies emerging topics earlier than LDA and BERTopics. Quantitatively, our evaluation metric demonstrates that LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0%. These findings highlight the strengths and limitations of different topic models for emergence detection, while our proposed metric provides a robust framework for future benchmarking in this area.
Overview
CoChat Summary
TL;DR
The paper introduces an unsupervised metric for retrospectively evaluating static topic models’ ability to detect emerging topics by comparing local and global model agreement at the document level. Across WoS, ACL, and Enron, CoWords qualitatively detects emergence earliest, while LDA quantitatively outperforms BERTopic.
Key findings
- Qualitatively, CoWords identifies emerging topics earlier than LDA and BERTopic; LDA captures steady trends, while BERTopic detects weak early signals but shows delayed and more volatile recognition.
- Quantitatively, LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0% (abstract); the results section reports LDA at 80.1% versus BERTopic at 56.6% on ACL, with F1 increasing as segment size grows.
- TWOR matching consistently retrieves more semantically coherent topics than KL divergence, which prioritizes similar temporal trends; for example, WoS TWOR matches diabetes-related terms while KL divergence retrieves bioelectromagnetic terms.
- Topic extraction quality varies by dataset: WoS and ACL produce clearer, more interpretable topics, while Enron yields broader and more ambiguous topics.
Methods
- Pipeline: preprocess time-stamped documents, train static topic models (CoWords, LDA, BERTopic) with 100 topics and 20 top words, extract prevalence trends, match topics hierarchically via Top Words Overlap Rate (TWOR) and KL divergence, then perform qualitative and quantitative emergence analysis.
- Data: Web of Science biomedical publications (171,499 documents), ACL anthology publications (63,199 after preprocessing), and Enron emails (439,819 after preprocessing), covering 1990–2020, 1980–2022, and 1998–2001 respectively.
- Quantitative metric: train a global model on the full dataset as a silver standard and local models on time segments with 80/20 train/test splits; classify documents as emerging if associated with topics showing positive growth rate, then compute precision, recall, and F1 agreement between local and global models. CoWords is not applicable because it does not produce document-topic distributions.
Limitations
- Stated: the study is retrospective and does not predict future topics; only three topic models are examined; datasets are domain-specific (scientific articles and emails); CoWords cannot be evaluated by the quantitative metric due to lacking document-topic distributions; the metric depends on topic granularity, the global model may overfit dominant trends, and external validation is absent.
- Stated: the qualitative conclusion that CoWords detects emergence earliest is based on limited qualitative results and may not generalize at larger scale.
- Evident: the abstract reports LDA’s average F1 as 80.6% while the results section reports 80.1%, an inconsistency in the paper’s own numbers.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures









Verification
Claims the paper makes
0 Valid · 5 Not valid · 10 Not checked · 65 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The WoS dataset captures the emergence of epidemiology-related topics, as shown in Fig. 3.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Qualitative analysis
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We present a case study where our models detect an emerging topic related to Bias and Fairness in NLP research.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Qualitative analysis
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When evaluating model performance in detecting emerging topics, our proposed metric showed that LDA consistently outperformed BERTopic in terms of agreement between local and global models.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Conclusions
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Enron dataset produces broader, more ambiguous topics, reflecting the informal and unstructured nature of email communications, where multiple themes may coexist in a single document.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Discussion > Dataset difference in topic extraction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results indicate that LDA achieves a higher average F1 score (80.1%) compared to BERTopic (56.6%), suggesting that LDA more consistently detects emerging topics over time.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Quantitative analysis
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
25 more claims
Create a free account to see all 80 claims and to verify the 11 references still open.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 22
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Evaluation of unsupervised static topic models’ emergence detection ability > Abstract
“Quantitatively, our evaluation metric demonstrates that LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0%.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Results > Quantitative analysis
“Results indicate that LDA achieves a higher average F1 score (80.1%) compared to BERTopic (56.6%), suggesting that LDA more consistently detects emerging topics over time.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluation of unsupervised static topic models’ emergence detection ability > Topic extraction pipeline > Matched topics emergence analysis
“(6) gk(ti,tj)=(Cik−Cjk)/Cik,in which Cik is the number of documents associated with topic k at time ti, reflecting how frequently the topic appears in that period. Topic k is emerging is gk(ti,tj)>0, k is not emerging otherwise.”
§ Evaluation of unsupervised static topic models’ emergence detection ability > Topic extraction pipeline > Matched topics emergence analysis
“Given a topic trend, determining if a topic is emerging in a certain time period fundamentally relies on if the trend shows an upward trajectory.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
THE STRUCTURE OF SCIENTIFIC REVOLUTIONS
MCGUIRE, J. E. (1963) · Philosophical Books · doi:10.1111/j.1468-0149.1963.tb00813.x · Cited by 8,424
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Blei, D. M., & Lafferty, J. D. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143859 · Cited by 3,173
Density-Based Clustering Based on Hierarchical Density Estimates
Campello, R. J. G. B., Moulavi, D., & Sander, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-37456-2_14 · Cited by 2,680
Exploring the Space of Topic Coherence Measures
Röder, M., Both, A., & Hinneburg, A. (2015) · Proceedings of the Eighth ACM International Conference on Web Search and Data Mining · doi:10.1145/2684822.2685324 · Cited by 2,224
Callon, M., Courtial, J. P., & Laville, F. (1991) · Scientometrics · doi:10.1007/bf02019280 · Cited by 2,090
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 16
Storkey, A. (2004) · neural information processing systems · Cited by 2,085
Research works printed but not linked: 1
- Rohatgi, 2022, ACL anthology corpus with full text
Cited by
Demir, O., & Saran, M. (2026) · IEEE Access · doi:10.1109/access.2026.3687277 · Cited by 1
Identifying and evaluating emerging scientific trends through temporal n-gram analysis
Štimec, G., Novalija, I., Brank, J., Šturm, J., Topal, O., Gheorghe, M., Grobelnik, M., & Roman, D. (2026) · Scientometrics · doi:10.1007/s11192-026-05744-5
Holistic Evaluations of Topic ModelsPreprint
Compton, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.23364
Methods and models for the analysis of scientific activity
Gubanov, D., Zhilyakova, L. Y., Kuznetsov, O. P., & Chkhartishvili, A. G. (2025) · Ontology of Designing · doi:10.18287/2223-9537-2025-15-4-578-597
From fine-tuning to prompting : a paradigm shift in knowledge graph construction
Li, X. (2026) · Centrum Wiskunde & Informatica (CWI), the national research institute for mathematics and computer science in the Netherlands
Self-citation
Metadata
- Authors
- Li, X., Esposito, C. D., Groth, P., Sitruk, J., Szatmari, B., & Wijnberg, N.
- Affiliations
- University of Amsterdam (NL)
- University of Johannesburg (ZA)
- Corresponding author
- Li, Xue
- Venue
- PeerJ Computer Science
- Publisher
- PeerJ
- Published
- 2025-05-30
- Volume
- 11
- Pages
- e2875
- ISSN
- 2376-5992 (ISSN-L)
- PMC
- PMC12192802
- OpenAlex
- W4410902617
- Semantic Scholar
- cb976e050e82231b604297e2a7a38807d0328433
- Cited by
- 9
- References
- 52
- Citations by year
- 2026: 6 · 2025: 3
- FWCI
- 1.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PeerJ Computer Science (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Dutch Research Council (MVI.19.032)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- unsupervised topic modeling, latent Dirichlet allocation, BERTopic, evaluation metrics, benchmarking, research trends, topic emergence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation of unsupervised static topic models’ emergence detection ability. https://app.cochat.ai/papers/10.7717/peerj-cs.2875
- The paper
- Li, X., Esposito, C. D., Groth, P., Sitruk, J., Szatmari, B., & Wijnberg, N. (2025). Evaluation of unsupervised static topic models’ emergence detection ability. PeerJ Computer Science. https://doi.org/10.7717/peerj-cs.2875
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.