CoChat

Identifying and evaluating emerging scientific trends through temporal n-gram analysis

Open access

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index; none of the 30 sources its claims cite is checked for support yet.

Abstract

The rapid growth of scientific publishing creates both opportunities and challenges for understanding emerging research trends. Traditional approaches to trend detection, such as citation analyses or latent topic models, often lag behind real developments or lack interpretability. To address these limitations, this study proposes a lightweight and transparent methodology for detecting and evaluating emerging scientific concepts through temporal n -gram analysis. From the OpenAlex corpus of scholarly metadata, n -grams are extracted from titles and abstracts, and aggregated into quarterly frequency series. Bursty dynamics are identified using complementary detection algorithms, including Kleinberg’s state-machine model and a moving average convergence divergence (MACD) approach. Evaluation uses a human-centered framework in which participants judge temporal plots of anonymized n -grams, providing both pairwise comparisons and graded burstiness scores. These judgments are compared with algorithmic rankings, and alignment between large language models (LLMs) and human raters is examined. We further contrast burst dynamics across fast- and slow-evolving subfields and verify the robustness of the approach to corpus-growth effects through a normalization experiment. By integrating scalable temporal analysis and multi-perspective evaluation, this work contributes a framework for monitoring and characterizing the trajectory of emerging research topics. The approach is intended as an interpretable tool for digital libraries, funding agencies, and research analysts seeking to navigate fast-evolving scientific landscapes.

Overview

CoChat Summary

TL;DR

The paper proposes temporal n-gram analysis of OpenAlex titles and abstracts, using Kleinberg and MACD burst detection plus human/LLM evaluation of anonymized frequency plots. It finds algorithmic bursts align with recognizable AI terminology and Kleinberg agrees more with human majority in binary comparisons, but human slider ratings show low consensus and diverge from LLM/algorithmic scores.

Key findings

  • The dataset contains 358,912 unique n-grams across 252 subfields, with 1,564,182 n-grams after preprocessing, covering 60 quarterly periods from 2010-Q1 to 2024-Q4; only n-grams appearing in at least 1,000 documents were retained.
  • In Artificial Intelligence, LLM-related terms dominate 2023–2024 rankings (e.g., “llms,” “large language models,” “chatgpt,” “prompt”), while Archaeology shows sparse, low-intensity bursts in 2023 (Kleinberg detected only one qualifying burst) and a 2024 cluster around Dresden porcelain. MACD is more sensitive to steady rises; Kleinberg emphasizes sharp, statistically significant deviations but is noisier in fast-moving fields.
  • Binary A/B voting: explicit LLM prompting reached 73% unanimity and Fleiss’ κ = 0.743, versus 42% and κ = 0.449 for non-explicit LLMs; human raters reached 8% unanimity across all 12 annotators, averaging 31.3% across 200 random subsets of five. Human majority agreed with explicit LLM majority on 81.1% of items (χ² = 37.29, p < 0.001) and with non-explicit LLM majority on 77.9% (χ² = 26.58, p < 0.001). Agreement with burst methods was higher for Kleinberg than MACD: human majority 82.0% vs 67.4%; explicit LLM 80.0% vs 62.0%; non-explicit LLM 72.0% vs 63.0%.
  • Slider scoring: explicit LLM group had highest reliability (ρ = 0.635, weighted κ = 0.656, RMSE = 1.60), followed by non-explicit LLM (ρ = 0.612, κ = 0.493, RMSE = 1.08), while human raters were weaker (ρ = 0.475, κ = 0.435, RMSE = 32.04). Group mean slider scores showed weak human–LLM correspondence (humans vs non-explicit: ρ = 0.138, κ = 0.060; humans vs explicit: ρ = −0.035, κ = −0.075), while the two LLM groups correlated strongly (ρ = 0.823, κ = 0.208). Document-normalized frequencies left top-ranked AI bursty terms largely unchanged, with minor reordering among lower-ranked terms.

Methods

  • Data: OpenAlex June 2025 snapshot of approximately 267 million scholarly works; analysis subset of about 147 million works from 2010–2024. Titles and abstracts were tokenized, lowercased, stopword-filtered, and used to extract 1- to 5-grams. Each entry was organized as (n-gram, field, subfield) using the primary OpenAlex topic, producing quarterly count series from 2010 to 2024; only n-grams with document frequency ≥ 1,000 were kept.
  • Burst detection: Kleinberg’s state-machine model was run with scaling ratio s = 5.7 and transition cost 1.0, retaining bursts of duration ≥ 2 quarters with W(I) > 0. MACD used short/long/signal spans of 24, 48, and 12 quarters, with histogram values scaled by the square root of the 99.5th percentile of positive counts (fallback: historical maximum); positive contributions were aggregated. Kleinberg and MACD scores were stored independently rather than combined, in a PostgreSQL schema with per-point and global scores.
  • Evaluation: human and LLM raters judged anonymized time-series charts without labels or scale information. Binary A/B voting used 12 human raters and 5 LLMs per prompting condition on 100 preselected n-gram pairs. Slider scoring used 10 human raters and 5 LLMs per prompting condition on 50 n-grams rated 0–100. LLMs were GPT-5, GPT-5-nano, Gemini 2.5 Pro, Gemini 2.5 Flash, and Claude Sonnet 4, tested under explicit and non-explicit prompts with deterministic configurations (temperature ≈ 0) and JSON-schema outputs. Analyses used unanimity, Fleiss’ κ, pairwise χ² tests, Spearman correlations, weighted κ, RMSE, leave-one-out consensus accuracy, and majority voting.

Limitations

  • Stated in the paper: surface n-grams treat synonymous terms as independent signals, introducing redundancy; the human evaluation relied on a relatively small and homogeneous rater pool; preprocessing could be improved with domain- or language-specific stopwords, removal of markup, acronym expansion, and synonym handling; lexical bursts do not by themselves signal conceptual novelty or lasting scientific impact and may reflect hype cycles, relabeling, or publication-volume/discourse changes; the ≥1,000-document threshold scopes the framework to terms already visible in the literature and misses early-stage, niche, or weak cross-disciplinary terms; direct empirical comparison with neural/embedding-based methods is complicated by different units of analysis and the absence of a shared benchmark with ground-truth emergence labels.
  • Evident from the text (not framed as formal limitations): human slider ratings showed low consensus and scale-calibration variability; the explicit LLM prompt imposed a narrower definition of burstiness centered on a single dominant peak, while human raters appeared to use a broader perceptual interpretation including sustained increases and multiple peaks; binary and slider tasks diverged in human–LLM alignment; some top-ranked n-grams remained artifacts such as repeated tokens (“strong strong”) or formatting-related terms.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11192-026-05744-5

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Three-level methodology for identifying and evaluating emerging scientific trends
Fig. 1 Three-level methodology for identifying and evaluating emerging scientific trendsCC-BY · From the PDF
Image 2
CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Fig. 4 System implementation architecture diagram
Fig. 4 System implementation architecture diagramCC-BY · From the PDF
Fig. 5 Dynamic filters for burst assessment
Fig. 5 Dynamic filters for burst assessmentCC-BY · From the PDF
Fig. 6 Inspection of the n -gram 'large language' for the year 2024
Fig. 6 Inspection of the n -gram 'large language' for the year 2024CC-BY · From the PDF
Fig. 7 Binary voting page, first pair
Fig. 7 Binary voting page, first pairCC-BY · From the PDF
Image 8
CC-BY · From the PDF
Fig. 9 Temporal frequency of the n -gram 'LLMs' from 2010 to 2024 based on counts within the Artificial Intelligence subfield. The shaded area (2023-2024) corresponds to the period used for the burst detection analysis, indicating a sharp growth in publication frequency associated with large language models
Fig. 9 Temporal frequency of the n -gram 'LLMs' from 2010 to 2024 based on counts within the Artificial Intelligence subfield. The shaded area (2023-2024) corresponds to the period used for the burst detection analysis, indicating a sharp growth in publication frequency associated with large language modelsCC-BY · From the PDF
Image 10
CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Fig. 13 Pairwise agreement matrix across human raters group for the binary A/B task
Fig. 13 Pairwise agreement matrix across human raters group for the binary A/B taskCC-BY · From the PDF
Image 15
CC-BY · From the PDF
Fig. 14 Pairwise agreement matrix across LLM (Non-Explicit prompts) group for the binary A/B task
Fig. 14 Pairwise agreement matrix across LLM (Non-Explicit prompts) group for the binary A/B taskCC-BY · From the PDF
Image 17
CC-BY · From the PDF
Fig. 15 Pairwise agreement matrix across LLM (Explicit prompts) group for the binary A/B task
Fig. 15 Pairwise agreement matrix across LLM (Explicit prompts) group for the binary A/B taskCC-BY · From the PDF
Image 19
CC-BY · From the PDF
Fig. 16 Pairwise Spearman correlation matrix across LLM (Non-Explicit) group for the slider score assignment task
Fig. 16 Pairwise Spearman correlation matrix across LLM (Non-Explicit) group for the slider score assignment taskCC-BY · From the PDF
Image 21
CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 2 Not valid · 19 Not checked · 148 Own results and interpretations

74 more claims

Create a free account to see all 169 claims and to verify the 25 references still open.

See all claims — free account
  • It may state two separate claims as one. 58
  • Probably not a substantial claim. 47
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 39 references

Cited by

Still being checked.

Metadata

Authors
Štimec, G., Novalija, I., Brank, J., Šturm, J., Topal, O., Gheorghe, M., Grobelnik, M., & Roman, D.
Affiliations
  • Jožef Stefan Institute (SI)
  • Bucharest University of Economic Studies (RO)
  • SINTEF (NO)
Corresponding author
Novalija, Inna
Venue
Scientometrics
Published
2026-09-30
Volume
131
Issue
11
Pages
7793-7831
ISSN
  • 0138-9130 (ISSN-L)
  • 1588-2861
OpenAlex
W7214956221
Cited by
0
References
38
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientometrics (publishedVersion, cc-by)
  • DiRROS repository (University of Maribor) (publishedVersion, cc-by)
  • dCOBISS.SI Digital Repository (publishedVersion, cc-by)
Funders
  • HORIZON EUROPE Framework Programme (101120237)
  • PNPR (760049, 101189771, 101092639)
  • ARIS (GC-0001)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
emerging research trends, burstiness, MACD, large language models, OpenAlex, scholarly metadata, digital libraries

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Identifying and evaluating emerging scientific trends through temporal n-gram analysis. https://app.cochat.ai/papers/10.1007/s11192-026-05744-5
The paper
Štimec, G., Novalija, I., Brank, J., Šturm, J., Topal, O., Gheorghe, M., Grobelnik, M., & Roman, D. (2026). Identifying and evaluating emerging scientific trends through temporal n-gram analysis. Scientometrics. https://doi.org/10.1007/s11192-026-05744-5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.