The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 4 of the 36 sources its claims cite were checked for support.
Measuring the Evolution of a Scientific Field through Citation Frames
Open accessAbstract
Citations have long been used to characterize the state of a scientific field and to identify influential works. However, writers use citations for different purposes, and this varied purpose influences uptake by future scholars. Unfortunately, our understanding of how scholars use and frame citations has been limited to small-scale manual citation analysis of individual papers. We perform the largest behavioral study of citations to date, analyzing how scientific works frame their contributions through different types of citations and how this framing affects the field as a whole. We introduce a new dataset of nearly 2,000 citations annotated for their function, and use it to develop a state-of-the-art classifier and label the papers of an entire field: Natural Language Processing. We then show how differences in framing affect scientific uptake and reveal the evolution of the publication venues and the field as a whole. We demonstrate that authors are sensitive to discourse structure and publication venue when citing, and that how a paper frames its work through citations is predictive of the citation count it will receive. Finally, we use changes in citation framing to show that the field of NLP is undergoing a significant increase in consensus.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces a six-class citation-function scheme and a classifier to label 134,127 citations across 21,474 NLP papers, showing citation framing varies by section and venue and predicts future impact. It argues NLP has shifted toward rapid discovery science, with declining comparison/contrast citations and increasing concentration on shared methods.
Key findings
- Citation framing follows paper structure: BACKGROUND citations dominate introductions and related work, USES citations rise in methodology/evaluation, COMPARISON OR CONTRAST increases in results/discussion, and FUTURE citations are most frequent in conclusions.
- Venue matters: journals have the highest percentage of BACKGROUND citations, conferences devote proportionally more to comparison/contrast, and workshops use more BACKGROUND with relatively little comparison; SemEval workshops are primarily USES citations.
- A paper’s citation framing significantly improves prediction of its 5-year citation count: adding framing lowers AIC from 35,693.400 to 35,567.630 (p ≤ 0.01) on 10,434 papers; USES (0.055*), COMPARISON OR CONTRAST (0.025*), EXTENDS (0.021), MOTIVATION (0.014*), and BACKGROUND (0.013*) are positive predictors, while FUTURE is not significant.
- NLP shows increased consensus: comparison/contrast citations decline (r = −0.899, p ≤ 0.01), non-methodological citations decline (r = −0.663, p ≤ 0.01), and incoming citations concentrate on the same compared works (r = 0.734, p ≤ 0.01) and used works (r = 0.889, p ≤ 0.01); workshops also become more conference-like over time, especially long-running workshops.
Methods
- Built a six-class citation-function scheme (BACKGROUND, MOTIVATION, USES, EXTENSION, COMPARISON OR CONTRAST, FUTURE) and annotated 1,969 citations: 1,436 from 52 fully labeled papers (mean 27.6 citations/paper) plus 533 supplemental contexts from 133 papers; combined with Teufel’s data to create 3,083 instances.
- Trained a Random Forest classifier using structural, lexical/grammatical, field, usage, topic-model, bootstrapped-pattern, and selectional-preference features; evaluated with paper-level cross-validation and macro-averaged F1; applied it to 21,474 ACL Anthology papers, yielding 134,127 citations.
- Used negative binomial regression to predict cumulative 5-year citation counts for 10,434 papers, and Jensen-Shannon divergence to compare venue citation framing over time.
Limitations
- The paper states errors remain; a main challenge is incorporating information external to the citing sentence, and citation context identification is important for improving performance.
- The paper states some instances were excluded due to malformed context, ParsCit wrong spans, or numeric citation style; these errors were <4% and no further correction was performed in the larger unannotated data. Citation sentiment was omitted because negative sentiment is rare and subjective to classify.
- Evident from the text: the classifier and analyses focus on ACL/NLP data, so generalization to other fields is not established; rare classes were oversampled during annotation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
36 Own result · 17 No basis given · 13 Own interpretation · 11 No full text of the sources · 8 Sources not found · 7 Own method · 5 Not checked yet · 1 Supported · 1 Partly supported · 1 Mixed support · accuracy not yet measured
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“we constructed a negative binomial regression to predict the cumulative number of citations a paper received within the first five years after publication, which is known to be highly representative of the eventual citation count (Wang et al., 2013;Stern, 2014).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Disputedcites Wang et al., 2013 sourceExperimental — accuracy not yet measured
The paper’s Abstract (page unavailable) says papers with the same number of citations within five years can have widely different long-term impact. It acknowledges that early and long-term citations correlate for most papers, but says the correlation breaks down for the papers with the greatest long-term impact. The discussion (page unavailable) likewise describes short-term citations as lacking predictive power, contrary to the claim that five-year counts are highly representative of eventual counts. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Supportedcites Stern, 2014 source
The Review of Literature on Predicting Citations reports a 0.871 rank correlation between citations accumulated in the first five years and total citations at 31 years. Stern’s Results also find that early citation totals strongly predict later citation rankings for economics and political science articles. The evidence concerns how well early counts represent later relative citation impact, not whether the raw counts are similar. (LLM, conf=0.88)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We propose that the increased use of shared evaluations, and the statistical methodology borrowed originally from electrical engineering (Hall et al., 2008;Anderson et al., 2012) has led NLP to undergo a shift towards rapid discovery science.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 8 The Growth of Rapid Discovery Science
What the sources say
- Partly supportedcites Hall et al., 2008 sourceExperimental — accuracy not yet measured
Section 4.1 says speech recognition, historically an electrical-engineering field, made early use of statistical methods and suggests spoken-language researchers helped bring those methods into computational linguistics. The paper does not establish increased use of shared evaluations; the shift toward rapid discovery science is presented as the citing authors’ own proposal, not a finding of this source. Page numbers are unavailable in our copy. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites Anderson et al., 2012
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, this trend in increasing similarity was seen much more for both long-running workshops, suggesting that multiple-iteration workshops create their own conference-like norms that attract more conference-like papers each year.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Experimental Setup
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The number of ACL workshops increased from around ten in the 1990s to over 100 by 2010, with many workshops having multiple iterations across the years.
- Rests on an assumption: The growth of the ACL community has been accompanied by the creation of new publication venues.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, examining the feature weighting in the random forest shows that features for structure (e.g., section number), topic, and selectional preference comprised most of the 100 highest-weighted features (76%).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Workshops, by contrast, have relatively little comparison and instead use more BACKGROUND; the experimental nature of workshop papers presumably results in fewer potential prospects for compar-ison.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that similar venue types have similar distributions of citation framing.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, the SemEval workshops focus on rapidly developing new systems for a shared task, which is reflected in the papers framing as primarily USES citations and relatively little COMPARE OR CONTRAST, as the venue's shared task provides the broader framing connecting papers to related work.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, we observe that selectional preference and bootstrapped lexicon features had the largest impact on performance; both features capture local information indicating this type of information is important for recognizing function.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, we found that many citation contexts with alignments-such as this onecontain signals of both comparison and contrast; for our intended analyses, we considered this alignment signalling more important than whether the author was comparing or contrasting.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 2.1 Classification Scheme
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Journals have the highest percentage of BACKGROUND citations, suggesting that their extra space and wider temporal scope lends itself more to positioning.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 27
Every source the paper cites for this claim was checked and supports it.
In the paper
“Collins claims this shift characterizes natural sciences, but not many social sciences, which are instead more likely to engage in continual contesting and turnover of core methods and assumptions (Evans et al., 2016).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 8 The Growth of Rapid Discovery Science
What the sources say
- Supportedcites Evans et al., 2016 source
The Literature Review (page unavailable) attributes to Collins a contrast between high-consensus, rapid-discovery hard sciences and fields where debate persists. The section on stability at the core (page unavailable) says preparadigmatic disciplines have an ever-shifting core, and the Discussion and Conclusions reports lower consensus and stability in the social sciences. Together, these support the paraphrase. (LLM, conf=0.88)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ultimately, 805 new bootstrapped patterns were added for the citing sentence, 669 for the preceding context, and 1159 for the following context, a total of over four times the number of manually curated patterns.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Consistent with prior work, the majority of citations are BACKGROUND (Moravcsik and Murugesan, 1975;Spiegel-Rüsing, 1977;Teufel et al., 2006b).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other non-conjunctive classifiers we tried resulted in substantially lower Macro F1: Naive Bayes, 0.286; knearest neighbor, 0.255 (k=3); and Linear-kernel SVM, 0.393 (C=1).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We began with the largest publicly-available list of citation patterns (Teufel, 2000) and extended it with 132 new patterns and 13 new lexical categories based on a manual analysis of the corpus.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
What the sources say
- Not matchedcites Teufel, 2000
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 6: The average cited paper receives an increasing number of USES and COMPARISON OR CONTRAST citations per year showing that field increasingly builds upon the same set of papers, providing a methodological lineage.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in 1991, authors compared with a diffuse group of parsing papers, e.g., (Shieber, 1988;Pereira and Warren, 1983;Haas, 1989), with such papers receiving at most three citations that year; whereas in 2000, most comparisons were to a core set of parsing papers, e.g., (Collins, 1999;Buchholz et al., 1999;Collins, 1997), with a much sharper (lower entropy) distribution of citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Knowledge of how a paper frames its contributions helps improve predicting its future impact, with a statistically significant improvement in AIC when the distribution of citation functions is added (likelihood ratio test, p ≤ 0.01).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8 shows which types of citations are significantly predictive of higher impact (p ≤ 0.01).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not observe this effect in how authors frame”
- Also stated in: § 7 Predicting Future Impact · plain
“their work as COMPARISON OR CONTRAST or EX-TENDS, with both having significant positive effects.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 7 Predicting Future Impact · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8: Regression models for predicting the total number of citations five years after publication show that a paper's citation framing provides a statistically significant improvement in model fit and reveals which type of framing yields more cited papers.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All papers with at least five years of publication history in the anthology were considered, yielding a set of 10,434 papers.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) the way in which an author frames their work aids in predicting its future impact as the number of citations its receives, with the community favoring works that integrate many new technologies and also relate to prior work through comparison and contrast”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 2: Expected percentage of citation functions per section shows a clear narrative trajectory across sections.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite an increase in BACKGROUND citations, the total percentage of non-methodological citations still declines (r= -0.663, p ≤ 0.01), with authors instead increasingly including more USES citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5: Changes in the average citation frame in ACL papers reveals a continued decline in the percentage of COM-PARISON OR CONTRAST and increase in USES citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(2) ACL workshops have evolved to become more like the mainstream conferences, with multi-iteration workshops being quicker to establish conference-like norms”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Patterns provide a powerful mechanism for capturing regularity in citation usage (Dong and Schäfer, 2011).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
What the sources say
- Not matchedcites Dong and Schäfer, 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“However, the authors note that us- ing the citing sentence alone is the correct context in 80% of the instances, so we view our implementation as a close approximation.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, a citation in a context describing system performances and results is likely to be a COMPARE OR CONTRAST, whereas one describing methodology is more likely to be USES.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that authors are sensitive to discourse structure and publication venue when citing”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Following best practices for annotating citations (Athar, 2014), annotators saw an extended context before and after the citing sentence, provided from the output of ParsCit.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
What the sources say
- Not matchedcites Athar, 2014
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“To avoid semantic drift (Riloff and Jones, 1999), a bootstrapped pattern was only included as a feature if the majority of its occurrences were with a single citation function.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
What the sources say
- Not checked — no full text of the sourcecites Riloff and Jones, 1999
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also note that when performing crossvalidation, we compute the bootstrapped patterns and prototypical argument features using only contexts from the training data.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“All models were trained using a Random Forest classifier, which is robust to overfitting even with large numbers of features (Fernández-Delgado et al., 2014).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
What the sources say
- Not checked — no full text of the sourcecites Fernández-Delgado et al., 2014 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Annotated data is crucial for developing high accuracy for rare citation classes.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“When combining the two datasets, we omit the data labeled with their BACKGROUNDequivalent class to reduce the effects of a large majority class and because instances of the FUTURE class are merged into BACKGROUND according to their scheme.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Scientific papers commonly follow a structured section narrative to frame their contributions: Introduction, Methodology, Results, and Discussion (Skelton, 1994;Nwogu, 1997).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 4 Narrative Structure of Citation Function
What the sources say
- Not checked — no full text of the sourcecites Skelton, 1994
- Not checked — no full text of the sourcecites Nwogu, 1997 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation is performed using crossvalidation where each fold leaves out all citations of a single paper.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“All citations were then doubly-annotated by two trained annotators with expertise in NLP using the Brat tool (Stenetorp et al., 2012) and were then fully adjudicated to ensure quality.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
What the sources say
- Not checked — no full text of the sourcecites Stenetorp et al., 2012
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Stratifying by paper instead of instance is critical: since multiple citations may appear in the same sentence, instance-based stratification would leak information between training and test.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all three compared systems, we use identical parameter values as reported in the papers.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The resulting model has a variance inflation factor of < 10 for all variables.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlimited extra space for citations acted to modify authors' citation framing behavior; given unlimited space, authors chose to include proportionally more non-methodological citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: ACL venues began allowing unlimited references in 2010 for ACL, 2011 for NAACL, etc.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(4) the NLP field as a whole has seen increased consensus in what constitutes valid workwith a reduced need for positioning and excessive comparison-demonstrating its shift towards rapid discovery science.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, a differentiation of citation functions affords a deeper understanding of how scholars develop arguments for different publication venues as well as how these venues may demand different forms of knowledge representation and arguments over time.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As such, each venue has a distinct genre of writing, from the tentative results of workshop papers to journal papers with substantial synthesis.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“AIC measures each model's goodness of fit in proportion to the number of independent variables; when comparing models, the model with the minimal AIC is preferred (Akaike, 1974).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not checked — no full text of the sourcecites Akaike, 1974 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Multi-author papers are known to receive higher citation counts (Gazni and Didegah, 2011), partially due to the effects of self-citation (Fowler and Aksnes, 2007), and therefore we include the number of authors on the paper.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Each publication venue has its own expectation for the types of work it accepts, e.g., the degree of polish or depth of experiments.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Each section in the narrative adopts argumentative moves designed to convince the reader of the work's claims (Swales, 1986;Swales, 1990).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 4 Narrative Structure of Citation Function
What the sources say
- Not checked — no full text of the sourcecites Swales, 1986 source
- Not matchedcites Swales, 1990
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, these modes of citation help us understand the state of research efforts and their evolution more broadly for entire scientific fields like NLP.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“The scholarly narrative told through citations provides the reader with support for its claims and technical competence (Latour, 1987, p. 34).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not matchedcites Latour, 1987
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These trends show the increased incorporation of prior work to form a lineage of method technologies as well as show increased consensus on which works are sufficient for comparing against in order to establish a claim.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This framing could affect how the work is perceived and, ultimately, how it is received and cited within the com-munity (Shi et al., 2010).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not checked — no full text of the sourcecites Shi et al., 2010 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simply put, citation functions help frame an article's reception.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Collins (1994) proposed that a field can undergo a particular shift, referring to it as rapid discovery science, when the field (a) reaches high consensus on research topics as well as methods and technologies, and (b) then develops genealogies of methods and technologies that continually improve on one another.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 8 The Growth of Rapid Discovery Science
What the sources say
- Not checked — no full text of the sourcecites Collins (1994) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we include the publication venue, using the individual conference or workshop in which the paper was published to control for variations in prestige between venues.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Authors cite works for different reasons (or function), so regarding them as equivalent signals is potentially problematic.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“we regress out the number of citations from the citation function counts (Kutner et al., 2004;O'brien, 2007).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not matchedcites Kutner et al., 2004
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites O'brien, 2007 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper relies on this without support.
The number of ACL workshops increased from around ten in the 1990s to over 100 by 2010, with many workshops having multiple iterations across the years.
“The number of workshops has increased substantially with the growth of the field, from around ten workshops in the 1990s to over 100 workshops by 2010, with many workshops having multiple iterations across the years.”
The paper relies on this without support.
The growth of the ACL community has been accompanied by the creation of new publication venues.
“The growth of the ACL community has been accompanied by the creation of new publication venues.”
The paper relies on this without support.
ACL venues began allowing unlimited references in 2010 for ACL, 2011 for NAACL, etc.
“This transition corresponds to the date at which ACL venues began allowing unlimited references (2010 for ACL, 2011 for NAACL, etc.).”
The paper relies on this without support.
Many fluff citations exist, while some less common ones are substantively relevant to the paper's argument.
“Many fluff citations exist, while some less common ones are substantively relevant to the paper's argument.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“ACL-ARC is a dataset of citation intents released by Jurgens et al. (2018). The dataset is based on a sample of papers from the ACL Anthology Reference Corpus (Bird et al., 2008) and includes 1,941 citation instances from 186 papers and is annotated by domain experts in the NLP field.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“previous state-of-the-art results of Jurgens et al. (2018) 54.6”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our model achieves a new state-ofthe-art on an existing ACL anthology dataset (ACL-ARC) with a 13.3% absolute increase in F1 score, without relying on external linguistic resources or hand-engineered features as done in existing methods.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A citation context includes text spans in a citing paper describing a referenced work and has been shown to be the primary signal in intent classification (Teufel et al., 2006; Abu-Jbara et al., 2013; Jurgens et al., 2018).”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jurgens et al. (2018) expanded all pre-existing featurebased efforts on citation intent classification by proposing a comprehensive set of engineered features, including boostrapped patterns, topic modeling, dependency-based, and metadata features for the task.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, for the Background and Future classes, no grammatical features were provided by Jurgens and colleagues (2018), rendering a direct comparison with our results unfeasible.”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Existing models for this problem are feature-based, modeling the citation context with respect to a set of predefined handengineered features (such as linguistic patterns or cue phrases) and ignoring other signals that could improve prediction.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“ACL-ARC is a dataset of citation intents released by Jurgens et al. (2018). The dataset is based on a sample of papers from the ACL Anthology Reference Corpus (Bird et al., 2008) and includes 1,941 citation instances from 186 papers and is annotated by domain experts in the NLP field.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Our implementation of this model achieves a macro-averaged F1 score of 0.526 using 10-fold cross-validation”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“We note that these results were obtained without using hand-curated features or additional linguistic resources as used in Jurgens et al. (2018).”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“We also experimented with adding features used in Jurgens et al. (2018) to our best model and not only we did not see any improvements, but we observed at least 1.7% decline in performance.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“followed by Jurgens et al. (2018)”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Jurgens et al. (2018) incorporate a variety of features, ranging from pattern-based features to topic-modeling features, to citation graph features. They also incorporate section titles and relative section position in the paper as features.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Jurgens et al. (2018) proposed a six category system for citation intents.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“An early contribution to CIC is the ACL-ARC citations dataset (Jurgens et al., 2018), which provides nearly 2000 manually annotated citation contexts from the NLP domain.”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“potentially refined by citation type or class, such as background citations and method citations (Teufel et al., 2006; AbuRa'ed et al., 2017; Jurgens et al., 2018; Pride and Knoth, 2020; Lauscher et al., 2022).”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Jurgens et al., 2018
In the citing paper
“ACL-ARC (Jurgens et al., 2018) and SciCite (Cohan et al., 2019) assign intent labels (e.g.Comparison, Extension, etc.) to sentences from scientific papers that cite other papers.”
From SciBERT: A Pretrained Language Model for Scientific Text (2019) · cites this paper as Jurgens et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
“The resulting citation function dataset contains 3,083 instances.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
“we apply our classifier trained on our combined dataset (2600 citation instances) to the ACL Anthology”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
“MOTIVATION 98 CONTINUATION 73 FUTURE 68”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
“EXTENSION Extends P 's data, methods, etc.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
“We map their 12 function classes into our six classes (see Appendix A).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Conflict
“CoCoGM Comparison or Contrast CoCo Comparison or Contrast CoCoR0 Comparison or Contrast CoCoXY Background PBas Extends PUse Uses PModi Extends PMot Motivation PSim Comparison or Contrast PSup Comparison or Contrast Neut Background CoMetN Comparison or Contrast CoGoaN Comparison or Contrast”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Frame Analysis: An Essay on the Organization of Experience.
Gamson, W. A., & Goffman, E. (1975) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2064022 · Cited by 11,891
A Caution Regarding Rules of Thumb for Variance Inflation Factors
O’brien, R. M. (2007) · Quality & Quantity · doi:10.1007/s11135-006-9018-6 · Cited by 11,217
(1987) · Cited by 8,955
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Research works printed but not linked: 5
- Fernández-Delgado Manuel, 2014, Journal of Machine Learning Research (JMLR), 15, 3133
- Haas Andrew, 1989, Computational Linguistics, 15, 219
- Leech Geoffrey, 1992, Language Research, 28, 1
- Pedregosa Fabian, 2011, The Journal of Machine Learning Research (JMLR), 12, 2825
- Skelton John, 1994, British Journal of General Practice, 44, 455
Cited by
Sorted by citations
Items 201–234 of 234
Analysis of Use of Figures and Tables in Computer Vision Papers Using Image Recognition Technique
Yamamoto, S., Suzuki, R., Shinagawa, S., Kataoka, H., & Morishima, S. (2021) · Journal of the Japan Society for Precision Engineering · doi:10.2493/jjspe.87.995
A descriptive analysis of cite units from the perspectives of content and linguistic expressions
Watanabe, K., Kageura, K., & Sekine, S. (2021) · Library and Information Science Research E-Journal · doi:10.32655/libres.2021.2.2
Snapshot-Guided Domain Adaptation for ELECTRA
Cheng, D., Huang, S., Liu, J., Zhan, Y., Sun, H., Wei, F., Deng, D., & Zhang, Q. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.163 · Cited by 1
Automating Citation Intent Classification on SciCite Dataset
Lenin, A., Chauhan, P., Mehta, T., Baliyan, N., & Sharma, D. K. (2023) · International Conference on Computer, Electronics & Electrical Engineering & their Applications (IC2E3) · doi:10.1109/ic2e357697.2023.10262812
Improving Peer-Review Score Prediction via Pretrained Model with Intermediate-Task Training
Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2024) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom60618.2024.10418356
Multi-sentence and multi-intent classification using RoBERTa and graph convolutional neural networkPreprint
Ravi, K., Singh, A. V., & Gautam, C. (2024) · Research Square · doi:10.21203/rs.3.rs-3960194/v1
Citation Intent Classification Using Transformers
C, R. G. H., Raj, K. S., & Kumar, M. A. (2024) · Students Conference on Engineering and Systems · doi:10.1109/sces61914.2024.10652428 · Cited by 1
-, D. B., -, K. S., -, A. B., & -, S. M. A. H. (2024) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2024.v06i06.32271
The development of a Web-based in-text citation analysis tool
Yoganingrum, A., Maryati, I., Yaniasih, Y., Karsanti, H. T., Puspita, G. C., & Marlina, E. (2025) · Library Hi Tech News · doi:10.1108/lhtn-12-2024-0208
Chronological Evaluation of Novel Methodology Extraction from AI Literature
Ghosh, M., Ganguly, D., Basuchowdhuri, P., & Naskar, S. K. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702589
Marua, J. (2025) · Research Square · doi:10.21203/rs.3.rs-6747329/v1
Chatrinan, K., Noraset, T., & Tuarob, S. (2025) · IEEE Access · doi:10.1109/access.2025.3586729
Cui, H., Wang, Y., & Li, K. (2025) · International Conference on Scientometrics and Informetrics (ISSI) · doi:10.51408/issi2025_147
Contrasting citation patterns across academic disciplines in French doctoral dissertations
Cisel, M., Antognelli, P., & Virgaux, P. (2025) · Studies in Graduate and Postdoctoral Education · doi:10.1108/sgpe-05-2025-0045
Multi-intent prediction of scientific literature based on heterogeneous graph fusion network
Quan, Z., Mao, J., & Li, G. (2025) · Scientometrics · doi:10.1007/s11192-025-05493-x
Datasets for NLP tasks on scholarly articles
Ahmad, R. A., D’Souza, J., Zloch, M., & Otto, W. (2022) · Open Research Knowledge Graph (ORKG) · doi:10.48366/r280270
Fine-Tuning SciBERT for Citation Intent Classification: An Empirical Study
Firdaus, J., & Ruskanda, F. Z. (2025) · IEEE International Conference on Data and Software Engineering (ICoDSE) · doi:10.1109/icodse68111.2025.11351694
Box-enhanced context fusion for citation intent classification
Yang, J., & Du, Z. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108962
Causal Backdoor Adjustment for Citation Intent Classification
Song, Y., Wan, L., Zhang, Z., Wang, D., & Zhuang, F. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2868-3_9
Citation Intent Classification via Exponential Borda Fusion and SciBERT
Barchane, M., Belefqih, S., Benlahmar, E. H., Zahour, O., & Zellou, A. (2026) · Algorithms · doi:10.3390/a19080612
LitBench: A Graph-Centric Large Language Model Benchmarking Tool For Literature Tasks
Varvarigos, A., Maatouk, A., Zhang, J., Bui, N., Chen, J., Tassiulas, L., & Ying, R. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3785679 · Cited by 1
CiteFuncRanker: an LLM-based pairwise ranking framework for multi-functional citation analysis
Wang, Y., Ruan, X., Lyu, D., & Cheng, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05764-1
An, T. (2026) · Research Square · doi:10.21203/rs.3.rs-10449790/v1
Yaniasih, Y., Yoganingrum, A., Bisri, A., Marlina, E., Puspita, G. C., Maryati, I., Noorbani, M. A., & Rachmawati, R. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101866
Goyal, R., & Garg, S. (2026) · Springer eBooks · doi:10.1007/978-981-95-7441-4_9
Latent Chain-of-Thought Reasoning with Adaptive Scratchpad for Citation Intent Classification
Song, Y., Li, Y., Zhang, Z., Wang, D., & Zhuang, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_32
Zhang, L., Wan, Z., Cheng, M., Shi, S., & Gou, Z. (2026) · Scientometrics · doi:10.1007/s11192-026-05813-9
A data-driven comparative analysis on citation intents across scientific disciplines
Phan, T.-A., Nam, S., & Jung, J. J. (2026) · PLoS ONE · doi:10.1371/journal.pone.0358327
Does Discussion Matter? Interaction Dynamics in Open Peer Review
Wei, S., & Bi, R. (2026) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3785651.3831562
MOTEGI, K., Ito, K., & Matsubara, S. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846499
FROM CITATION COUNTS TO COMPUTATIONAL SCIENCE: METHODS, DATA, AND APPLICATIONS IN SCIENTOMETRICS
Kuchanskyi, O., & Amangeldi, M. (2026) · doi:10.59787/2413-5488-2026-55-3-93-107
Song, Y., Lu, X., Wan, L., & Zhang, Z. (2026) · Applied Sciences · doi:10.3390/app162110848
From citation intent to knowledge contribution: Classifying what cited papers actually contribute
Quan, Z., Liang, Z., Ma, M., Wei, J., Li, G., & Mao, J. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70127
Metadata
- Authors
- Jurgens, D., Kumar, S., Hoover, R., McFarland, D., & Jurafsky, D.
- Affiliations
- University of Michigan (US)
- Stanford University (US)
- Stanford University,
- Corresponding author
- Jurgens, David
- Kumar, Srijan
- Hoover, Raine
- Jurafsky, Dan
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 391-406
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2808556605
- Cited by
- 212
- References
- 79
- Citations by year
- 2026: 16 · 2025: 25 · 2024: 21 · 2023: 42 · 2022: 34 · 2021: 43 · 2020: 19 · 2019: 12
- FWCI
- 14.11
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Science Foundation (1633036)
- Brown Institute for Media Innovation
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Keywords
- citation functions, citation analysis, citation counts, field evolution, publication venues, discourse structure, citation classification, natural language processing, disciplinary consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
