CoChat

Evaluation of unsupervised static topic models’ emergence detection ability

Open access

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; none of the 13 sources its claims cite is checked for support yet.

Abstract

Detecting emerging topics is crucial for understanding research trends, technological advancements, and shifts in public discourse. While unsupervised topic modeling techniques such as Latent Dirichlet allocation (LDA), BERTopic, and CoWords clustering are widely used for topic extraction, their ability to retrospectively detect emerging topics without relying on ground truth labels has not been systematically compared. This gap largely stems from the lack of a dedicated evaluation metric for measuring emergence detection. In this study, we introduce a quantitative evaluation metric to assess the effectiveness of topic models in detecting emerging topics. We evaluate three topic modeling approaches using both qualitative analysis and our proposed emergence detection metric. Our results indicate that, qualitatively, CoWords identifies emerging topics earlier than LDA and BERTopics. Quantitatively, our evaluation metric demonstrates that LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0%. These findings highlight the strengths and limitations of different topic models for emergence detection, while our proposed metric provides a robust framework for future benchmarking in this area.

Overview

CoChat Summary

TL;DR

The paper introduces an unsupervised metric for retrospectively evaluating static topic models’ ability to detect emerging topics by comparing local and global model agreement at the document level. Across WoS, ACL, and Enron, CoWords qualitatively detects emergence earliest, while LDA quantitatively outperforms BERTopic.

Key findings

  • Qualitatively, CoWords identifies emerging topics earlier than LDA and BERTopic; LDA captures steady trends, while BERTopic detects weak early signals but shows delayed and more volatile recognition.
  • Quantitatively, LDA achieves an average F1 score of 80.6% in emergence detection, outperforming BERTopic by 24.0% (abstract); the results section reports LDA at 80.1% versus BERTopic at 56.6% on ACL, with F1 increasing as segment size grows.
  • TWOR matching consistently retrieves more semantically coherent topics than KL divergence, which prioritizes similar temporal trends; for example, WoS TWOR matches diabetes-related terms while KL divergence retrieves bioelectromagnetic terms.
  • Topic extraction quality varies by dataset: WoS and ACL produce clearer, more interpretable topics, while Enron yields broader and more ambiguous topics.

Methods

  • Pipeline: preprocess time-stamped documents, train static topic models (CoWords, LDA, BERTopic) with 100 topics and 20 top words, extract prevalence trends, match topics hierarchically via Top Words Overlap Rate (TWOR) and KL divergence, then perform qualitative and quantitative emergence analysis.
  • Data: Web of Science biomedical publications (171,499 documents), ACL anthology publications (63,199 after preprocessing), and Enron emails (439,819 after preprocessing), covering 1990–2020, 1980–2022, and 1998–2001 respectively.
  • Quantitative metric: train a global model on the full dataset as a silver standard and local models on time segments with 80/20 train/test splits; classify documents as emerging if associated with topics showing positive growth rate, then compute precision, recall, and F1 agreement between local and global models. CoWords is not applicable because it does not produce document-topic distributions.

Limitations

  • Stated: the study is retrospective and does not predict future topics; only three topic models are examined; datasets are domain-specific (scientific articles and emails); CoWords cannot be evaluated by the quantitative metric due to lacking document-topic distributions; the metric depends on topic granularity, the global model may overfit dominant trends, and external validation is absent.
  • Stated: the qualitative conclusion that CoWords detects emergence earliest is based on limited qualitative results and may not generalize at larger scale.
  • Evident: the abstract reports LDA’s average F1 as 80.6% while the results section reports 80.1%, an inconsistency in the paper’s own numbers.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.7717/peerj-cs.2875

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Pipeline for extracting and matching topics with three models.
Figure 1 Pipeline for extracting and matching topics with three models.CC BY · PubMed Central
Figure 2 Independent quantitative measure between global and local model.
Figure 2 Independent quantitative measure between global and local model.CC BY · PubMed Central
Figure 3 Selected match for WoS. The top 10 words for each method are as follows. CoWords: {infection, mortality, transmission, virus, spread, infect, vector, incidence, viral, epidemic}, LDA: {mouse, infection, antioxidant, virus, observed, stimulation, respond, protection, viral, infect}, BERTopic: {epidemic, infection, virus, viral, vaccination, model, transmission, vaccine, infectious, infect}.
Figure 3 Selected match for WoS. The top 10 words for each method are as follows. CoWords: {infection, mortality, transmission, virus, spread, infect, vector, incidence, viral, epidemic}, LDA: {mouse, infection, antioxidant, virus, observed, stimulation, respond, protection, viral, infect}, BERTopic: {epidemic, infection, virus, viral, vaccination, model, transmission, vaccine, infectious, infect}.CC BY · PubMed Central
Figure 4 Selected match for ACL. The top 10 words for each method are as follows. CoWords: {segmentation, tag, Chinese, character, segment, boundary, tagging, tagger, wordlevel, partofspeech_tagged}, LDA: {accuracy, segmentation, character, segment, rich, morphology, morpheme, unsupervise, convention, lefttoright}, BERTopic: {segmentation, chinese, word, character, model, tagging, partofspeech_tagged, ngram, language, accuracy}.
Figure 4 Selected match for ACL. The top 10 words for each method are as follows. CoWords: {segmentation, tag, Chinese, character, segment, boundary, tagging, tagger, wordlevel, partofspeech_tagged}, LDA: {accuracy, segmentation, character, segment, rich, morphology, morpheme, unsupervise, convention, lefttoright}, BERTopic: {segmentation, chinese, word, character, model, tagging, partofspeech_tagged, ngram, language, accuracy}.CC BY · PubMed Central
Figure 5 Selected match for Enron. The top 10 words for each method are as follows. CoWords: {email, agreement, question, receive, meeting, schedule, request, contact, file, list}, LDA: {agreement, contract, review, document, draft, bind, title, attorney, signature, wrong}, BERTopic: {abb_transformer, abb, existence, agreement, override, transformer, signature, initial, word, option}.
Figure 5 Selected match for Enron. The top 10 words for each method are as follows. CoWords: {email, agreement, question, receive, meeting, schedule, request, contact, file, list}, LDA: {agreement, contract, review, document, draft, bind, title, attorney, signature, wrong}, BERTopic: {abb_transformer, abb, existence, agreement, override, transformer, signature, initial, word, option}.CC BY · PubMed Central
Figure 6 Smoothed trends for the topic: Bias and Fairness in NLP research. CoWords: {bias, gender, mitigate, age, demographic, biased, debiase, fairness, female, male}, LDA: {gender, mitigate, transe, people, production, rapidly, expectation, game, progress, produce}, BERTopic: {gender, pronoun, bias, pronoun_resolution, stereotype, female, debiase, language, gendere, stereotypical}.
Figure 6 Smoothed trends for the topic: Bias and Fairness in NLP research. CoWords: {bias, gender, mitigate, age, demographic, biased, debiase, fairness, female, male}, LDA: {gender, mitigate, transe, people, production, rapidly, expectation, game, progress, produce}, BERTopic: {gender, pronoun, bias, pronoun_resolution, stereotype, female, debiase, language, gendere, stereotypical}.CC BY · PubMed Central
Figure 7 Average F1 score for LDA and BERTopic given different segment sizes of the ACL dataset. The overall average F1 score for LDA is 80.1%, and for BERTopic is 56.6%.
Figure 7 Average F1 score for LDA and BERTopic given different segment sizes of the ACL dataset. The overall average F1 score for LDA is 80.1%, and for BERTopic is 56.6%.CC BY · PubMed Central
Figure 8 Heatmap of F1 scores for LDA on segment size of 2. For each cell, the model is trained on the training data from the row time periods (2-year span) and tested on the test data from the column time periods (2-year span).
Figure 8 Heatmap of F1 scores for LDA on segment size of 2. For each cell, the model is trained on the training data from the row time periods (2-year span) and tested on the test data from the column time periods (2-year span).CC BY · PubMed Central
Figure 9 Heatmaps of F1 scores for LDA (top) and BERTopic (bottom), with the segment size of 3, meaning local models are trained on training data from a 3-year span and tested on test data from a 3-year span.
Figure 9 Heatmaps of F1 scores for LDA (top) and BERTopic (bottom), with the segment size of 3, meaning local models are trained on training data from a 3-year span and tested on test data from a 3-year span.CC BY · PubMed Central

Verification

Claims the paper makes

0 Valid · 5 Not valid · 10 Not checked · 65 Own results and interpretations

25 more claims

Create a free account to see all 80 claims and to verify the 11 references still open.

See all claims — free account
  • It may state two separate claims as one. 35
  • Probably not a substantial claim. 22

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 1

  1. Rohatgi, 2022, ACL anthology corpus with full text

Cited by

See all 9

Metadata

Authors
Li, X., Esposito, C. D., Groth, P., Sitruk, J., Szatmari, B., & Wijnberg, N.
Affiliations
  • University of Amsterdam (NL)
  • University of Johannesburg (ZA)
Corresponding author
Li, Xue
Venue
PeerJ Computer Science
Publisher
PeerJ
Published
2025-05-30
Volume
11
Pages
e2875
ISSN
2376-5992 (ISSN-L)
OpenAlex
W4410902617
Cited by
9
References
52
Citations by year
2026: 6 · 2025: 3
FWCI
1.71
Type
journal article
Language
en
Open access
Locations
  • PeerJ Computer Science (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Dutch Research Council (MVI.19.032)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
unsupervised topic modeling, latent Dirichlet allocation, BERTopic, evaluation metrics, benchmarking, research trends, topic emergence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.