Identifying and evaluating emerging scientific trends through temporal n-gram analysis
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index; none of the 30 sources its claims cite is checked for support yet.
Abstract
The rapid growth of scientific publishing creates both opportunities and challenges for understanding emerging research trends. Traditional approaches to trend detection, such as citation analyses or latent topic models, often lag behind real developments or lack interpretability. To address these limitations, this study proposes a lightweight and transparent methodology for detecting and evaluating emerging scientific concepts through temporal n -gram analysis. From the OpenAlex corpus of scholarly metadata, n -grams are extracted from titles and abstracts, and aggregated into quarterly frequency series. Bursty dynamics are identified using complementary detection algorithms, including Kleinberg’s state-machine model and a moving average convergence divergence (MACD) approach. Evaluation uses a human-centered framework in which participants judge temporal plots of anonymized n -grams, providing both pairwise comparisons and graded burstiness scores. These judgments are compared with algorithmic rankings, and alignment between large language models (LLMs) and human raters is examined. We further contrast burst dynamics across fast- and slow-evolving subfields and verify the robustness of the approach to corpus-growth effects through a normalization experiment. By integrating scalable temporal analysis and multi-perspective evaluation, this work contributes a framework for monitoring and characterizing the trajectory of emerging research topics. The approach is intended as an interpretable tool for digital libraries, funding agencies, and research analysts seeking to navigate fast-evolving scientific landscapes.
Overview
CoChat Summary
TL;DR
The paper proposes temporal n-gram analysis of OpenAlex titles and abstracts, using Kleinberg and MACD burst detection plus human/LLM evaluation of anonymized frequency plots. It finds algorithmic bursts align with recognizable AI terminology and Kleinberg agrees more with human majority in binary comparisons, but human slider ratings show low consensus and diverge from LLM/algorithmic scores.
Key findings
- The dataset contains 358,912 unique n-grams across 252 subfields, with 1,564,182 n-grams after preprocessing, covering 60 quarterly periods from 2010-Q1 to 2024-Q4; only n-grams appearing in at least 1,000 documents were retained.
- In Artificial Intelligence, LLM-related terms dominate 2023–2024 rankings (e.g., “llms,” “large language models,” “chatgpt,” “prompt”), while Archaeology shows sparse, low-intensity bursts in 2023 (Kleinberg detected only one qualifying burst) and a 2024 cluster around Dresden porcelain. MACD is more sensitive to steady rises; Kleinberg emphasizes sharp, statistically significant deviations but is noisier in fast-moving fields.
- Binary A/B voting: explicit LLM prompting reached 73% unanimity and Fleiss’ κ = 0.743, versus 42% and κ = 0.449 for non-explicit LLMs; human raters reached 8% unanimity across all 12 annotators, averaging 31.3% across 200 random subsets of five. Human majority agreed with explicit LLM majority on 81.1% of items (χ² = 37.29, p < 0.001) and with non-explicit LLM majority on 77.9% (χ² = 26.58, p < 0.001). Agreement with burst methods was higher for Kleinberg than MACD: human majority 82.0% vs 67.4%; explicit LLM 80.0% vs 62.0%; non-explicit LLM 72.0% vs 63.0%.
- Slider scoring: explicit LLM group had highest reliability (ρ = 0.635, weighted κ = 0.656, RMSE = 1.60), followed by non-explicit LLM (ρ = 0.612, κ = 0.493, RMSE = 1.08), while human raters were weaker (ρ = 0.475, κ = 0.435, RMSE = 32.04). Group mean slider scores showed weak human–LLM correspondence (humans vs non-explicit: ρ = 0.138, κ = 0.060; humans vs explicit: ρ = −0.035, κ = −0.075), while the two LLM groups correlated strongly (ρ = 0.823, κ = 0.208). Document-normalized frequencies left top-ranked AI bursty terms largely unchanged, with minor reordering among lower-ranked terms.
Methods
- Data: OpenAlex June 2025 snapshot of approximately 267 million scholarly works; analysis subset of about 147 million works from 2010–2024. Titles and abstracts were tokenized, lowercased, stopword-filtered, and used to extract 1- to 5-grams. Each entry was organized as (n-gram, field, subfield) using the primary OpenAlex topic, producing quarterly count series from 2010 to 2024; only n-grams with document frequency ≥ 1,000 were kept.
- Burst detection: Kleinberg’s state-machine model was run with scaling ratio s = 5.7 and transition cost 1.0, retaining bursts of duration ≥ 2 quarters with W(I) > 0. MACD used short/long/signal spans of 24, 48, and 12 quarters, with histogram values scaled by the square root of the 99.5th percentile of positive counts (fallback: historical maximum); positive contributions were aggregated. Kleinberg and MACD scores were stored independently rather than combined, in a PostgreSQL schema with per-point and global scores.
- Evaluation: human and LLM raters judged anonymized time-series charts without labels or scale information. Binary A/B voting used 12 human raters and 5 LLMs per prompting condition on 100 preselected n-gram pairs. Slider scoring used 10 human raters and 5 LLMs per prompting condition on 50 n-grams rated 0–100. LLMs were GPT-5, GPT-5-nano, Gemini 2.5 Pro, Gemini 2.5 Flash, and Claude Sonnet 4, tested under explicit and non-explicit prompts with deterministic configurations (temperature ≈ 0) and JSON-schema outputs. Analyses used unanimity, Fleiss’ κ, pairwise χ² tests, Spearman correlations, weighted κ, RMSE, leave-one-out consensus accuracy, and majority voting.
Limitations
- Stated in the paper: surface n-grams treat synonymous terms as independent signals, introducing redundancy; the human evaluation relied on a relatively small and homogeneous rater pool; preprocessing could be improved with domain- or language-specific stopwords, removal of markup, acronym expansion, and synonym handling; lexical bursts do not by themselves signal conceptual novelty or lasting scientific impact and may reflect hype cycles, relabeling, or publication-volume/discourse changes; the ≥1,000-document threshold scopes the framework to terms already visible in the literature and misses early-stage, niche, or weak cross-disciplinary terms; direct empirical comparison with neural/embedding-based methods is complicated by different units of analysis and the absence of a shared benchmark with ground-truth emergence labels.
- Evident from the text (not framed as formal limitations): human slider ratings showed low consensus and scale-calibration variability; the explicit LLM prompt imposed a narrower definition of burstiness centered on a single dominant peak, while human raters appeared to use a broader perceptual interpretation including sustained increases and multiple peaks; binary and slider tasks diverged in human–LLM alignment; some top-ranked n-grams remained artifacts such as repeated tokens (“strong strong”) or formatting-related terms.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





















Verification
Claims the paper makes
0 Valid · 2 Not valid · 19 Not checked · 148 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We verify that the detected burst signals are driven by relative changes in term usage rather than by corpus growth, through a normalization experiment comparing raw and document-frequency normalized rankings.”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Introduction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Correlations ranged from zero (voter_2 ↔ voter_6, ≈ -0.01 ) to strong (voter_4 ↔ voter_9, ≈ 0.84 ), high- lighting wide dispersion in how individuals mapped perceived burstiness.”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Slider scoring
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pairs with lower agreement also exhibited longer median decision times (Spearman = -0.518, p < 0.001 ), consistent with greater cognitive difficulty on visually ambiguous comparisons.”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Consensus reference: majority voting
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Archaeology results exhibit comparatively low-intensity and diffuse burst patterns in 2023, particularly under the Kleinberg method, which identifies only one qualifying burst.”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Algorithmic burst detection results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, five of the human-LLM disagreement cases occurred on items where the human panel split exactly 6-6, whereas the explicit LLM ensemble still returned a majority decision, suggesting that the prompt imposes a more decisive rule set on borderline cases.”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Consensus reference: majority voting
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
74 more claims
Create a free account to see all 169 claims and to verify the 25 references still open.
- It may state two separate claims as one. 58
- Probably not a substantial claim. 47
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Consensus reference: majority voting
“Thirteen pairs reached the minimum observed agreement of 53.3%,only marginally above chance”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Results and evaluation
“The first was a binary A/B voting task, in which annotators selected which of two charts appeared more bursty (12 human raters, 5 LLMs per prompting condition).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Additional information
“This supports the conclusion that the main burst signals are robust to corpus growth effects and are driven by relative changes in term usage rather than absolute frequency scaling (Fig. 14).”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Conclusion and future work
“Figure: 14 Fig. 14 Fig. 14 Pairwise agreement matrix across LLM (Non-Explicit prompts) group for the binary A/B task”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Results and evaluation
“The second was a slider-based scoring task, where annotators rated the perceived degree of burstiness on a 0-100 scale (10 human raters, 5 LLMs per prompting condition).”
§ Identifying and evaluating emerging scientific trends through temporal n-gram analysis > Conclusion and future work
“Score this time-series chart's "burstiness" from 1-10.”
Citations
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
A NEW MEASURE OF RANK CORRELATION
KENDALL, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,269
Chen, C. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20317 · Cited by 6,433
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Blei, D. M., & Lafferty, J. D. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143859 · Cited by 3,173
DBpedia: A Nucleus for a Web of Open Data
Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., & Ives, Z. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-76298-0_52 · Cited by 2,454
Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey
Jelodar, H., Wang, Y., Yuan, C., Feng, X., Jiang, X., Li, Y., & Zhao, L. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6894-4 · Cited by 1,961
BERTopic: Neural topic modeling with a class-based TF-IDF procedure
Grootendorst, M. (2022) · arXiv · doi:10.48550/arxiv.2203.05794 · Cited by 1,423
Bursty and hierarchical structure in streams
Kleinberg, J. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775061 · Cited by 1,337
Cited by
Still being checked.
Metadata
- Authors
- Štimec, G., Novalija, I., Brank, J., Šturm, J., Topal, O., Gheorghe, M., Grobelnik, M., & Roman, D.
- Affiliations
- Jožef Stefan Institute (SI)
- Bucharest University of Economic Studies (RO)
- SINTEF (NO)
- Corresponding author
- Novalija, Inna
- Venue
- Scientometrics
- Published
- 2026-09-30
- Volume
- 131
- Issue
- 11
- Pages
- 7793-7831
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W7214956221
- Semantic Scholar
- 04662dcf05875f042057ad148bc62e05b174d155
- Cited by
- 0
- References
- 38
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientometrics (publishedVersion, cc-by)
- DiRROS repository (University of Maribor) (publishedVersion, cc-by)
- dCOBISS.SI Digital Repository (publishedVersion, cc-by)
- Funders
- HORIZON EUROPE Framework Programme (101120237)
- PNPR (760049, 101189771, 101092639)
- ARIS (GC-0001)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- emerging research trends, burstiness, MACD, large language models, OpenAlex, scholarly metadata, digital libraries
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Identifying and evaluating emerging scientific trends through temporal n-gram analysis. https://app.cochat.ai/papers/10.1007/s11192-026-05744-5
- The paper
- Štimec, G., Novalija, I., Brank, J., Šturm, J., Topal, O., Gheorghe, M., Grobelnik, M., & Roman, D. (2026). Identifying and evaluating emerging scientific trends through temporal n-gram analysis. Scientometrics. https://doi.org/10.1007/s11192-026-05744-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.