The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 4 of the 36 sources its claims cite were checked for support.
Measuring the Evolution of a Scientific Field through Citation Frames
Open accessAbstract
Citations have long been used to characterize the state of a scientific field and to identify influential works. However, writers use citations for different purposes, and this varied purpose influences uptake by future scholars. Unfortunately, our understanding of how scholars use and frame citations has been limited to small-scale manual citation analysis of individual papers. We perform the largest behavioral study of citations to date, analyzing how scientific works frame their contributions through different types of citations and how this framing affects the field as a whole. We introduce a new dataset of nearly 2,000 citations annotated for their function, and use it to develop a state-of-the-art classifier and label the papers of an entire field: Natural Language Processing. We then show how differences in framing affect scientific uptake and reveal the evolution of the publication venues and the field as a whole. We demonstrate that authors are sensitive to discourse structure and publication venue when citing, and that how a paper frames its work through citations is predictive of the citation count it will receive. Finally, we use changes in citation framing to show that the field of NLP is undergoing a significant increase in consensus.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces a six-class citation-function scheme and a classifier to label 134,127 citations across 21,474 NLP papers, showing citation framing varies by section and venue and predicts future impact. It argues NLP has shifted toward rapid discovery science, with declining comparison/contrast citations and increasing concentration on shared methods.
Key findings
- Citation framing follows paper structure: BACKGROUND citations dominate introductions and related work, USES citations rise in methodology/evaluation, COMPARISON OR CONTRAST increases in results/discussion, and FUTURE citations are most frequent in conclusions.
- Venue matters: journals have the highest percentage of BACKGROUND citations, conferences devote proportionally more to comparison/contrast, and workshops use more BACKGROUND with relatively little comparison; SemEval workshops are primarily USES citations.
- A paper’s citation framing significantly improves prediction of its 5-year citation count: adding framing lowers AIC from 35,693.400 to 35,567.630 (p ≤ 0.01) on 10,434 papers; USES (0.055*), COMPARISON OR CONTRAST (0.025*), EXTENDS (0.021), MOTIVATION (0.014*), and BACKGROUND (0.013*) are positive predictors, while FUTURE is not significant.
- NLP shows increased consensus: comparison/contrast citations decline (r = −0.899, p ≤ 0.01), non-methodological citations decline (r = −0.663, p ≤ 0.01), and incoming citations concentrate on the same compared works (r = 0.734, p ≤ 0.01) and used works (r = 0.889, p ≤ 0.01); workshops also become more conference-like over time, especially long-running workshops.
Methods
- Built a six-class citation-function scheme (BACKGROUND, MOTIVATION, USES, EXTENSION, COMPARISON OR CONTRAST, FUTURE) and annotated 1,969 citations: 1,436 from 52 fully labeled papers (mean 27.6 citations/paper) plus 533 supplemental contexts from 133 papers; combined with Teufel’s data to create 3,083 instances.
- Trained a Random Forest classifier using structural, lexical/grammatical, field, usage, topic-model, bootstrapped-pattern, and selectional-preference features; evaluated with paper-level cross-validation and macro-averaged F1; applied it to 21,474 ACL Anthology papers, yielding 134,127 citations.
- Used negative binomial regression to predict cumulative 5-year citation counts for 10,434 papers, and Jensen-Shannon divergence to compare venue citation framing over time.
Limitations
- The paper states errors remain; a main challenge is incorporating information external to the citing sentence, and citation context identification is important for improving performance.
- The paper states some instances were excluded due to malformed context, ParsCit wrong spans, or numeric citation style; these errors were <4% and no further correction was performed in the larger unannotated data. Citation sentiment was omitted because negative sentiment is rare and subjective to classify.
- Evident from the text: the classifier and analyses focus on ACL/NLP data, so generalization to other fields is not established; rare classes were oversampled during annotation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
36 Own result · 17 No basis given · 13 Own interpretation · 11 No full text of the sources · 8 Sources not found · 7 Own method · 5 Not checked yet · 1 Supported · 1 Partly supported · 1 Mixed support · accuracy not yet measured
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“we constructed a negative binomial regression to predict the cumulative number of citations a paper received within the first five years after publication, which is known to be highly representative of the eventual citation count (Wang et al., 2013;Stern, 2014).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Disputedcites Wang et al., 2013 sourceExperimental — accuracy not yet measured
The paper’s Abstract (page unavailable) says papers with the same number of citations within five years can have widely different long-term impact. It acknowledges that early and long-term citations correlate for most papers, but says the correlation breaks down for the papers with the greatest long-term impact. The discussion (page unavailable) likewise describes short-term citations as lacking predictive power, contrary to the claim that five-year counts are highly representative of eventual counts. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Supportedcites Stern, 2014 source
The Review of Literature on Predicting Citations reports a 0.871 rank correlation between citations accumulated in the first five years and total citations at 31 years. Stern’s Results also find that early citation totals strongly predict later citation rankings for economics and political science articles. The evidence concerns how well early counts represent later relative citation impact, not whether the raw counts are similar. (LLM, conf=0.88)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We propose that the increased use of shared evaluations, and the statistical methodology borrowed originally from electrical engineering (Hall et al., 2008;Anderson et al., 2012) has led NLP to undergo a shift towards rapid discovery science.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 8 The Growth of Rapid Discovery Science
What the sources say
- Partly supportedcites Hall et al., 2008 sourceExperimental — accuracy not yet measured
Section 4.1 says speech recognition, historically an electrical-engineering field, made early use of statistical methods and suggests spoken-language researchers helped bring those methods into computational linguistics. The paper does not establish increased use of shared evaluations; the shift toward rapid discovery science is presented as the citing authors’ own proposal, not a finding of this source. Page numbers are unavailable in our copy. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites Anderson et al., 2012
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, this trend in increasing similarity was seen much more for both long-running workshops, suggesting that multiple-iteration workshops create their own conference-like norms that attract more conference-like papers each year.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Experimental Setup
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The number of ACL workshops increased from around ten in the 1990s to over 100 by 2010, with many workshops having multiple iterations across the years.
- Rests on an assumption: The growth of the ACL community has been accompanied by the creation of new publication venues.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, examining the feature weighting in the random forest shows that features for structure (e.g., section number), topic, and selectional preference comprised most of the 100 highest-weighted features (76%).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Workshops, by contrast, have relatively little comparison and instead use more BACKGROUND; the experimental nature of workshop papers presumably results in fewer potential prospects for compar-ison.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that similar venue types have similar distributions of citation framing.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, the SemEval workshops focus on rapidly developing new systems for a shared task, which is reflected in the papers framing as primarily USES citations and relatively little COMPARE OR CONTRAST, as the venue's shared task provides the broader framing connecting papers to related work.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, we observe that selectional preference and bootstrapped lexicon features had the largest impact on performance; both features capture local information indicating this type of information is important for recognizing function.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, we found that many citation contexts with alignments-such as this onecontain signals of both comparison and contrast; for our intended analyses, we considered this alignment signalling more important than whether the author was comparing or contrasting.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 2.1 Classification Scheme
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Journals have the highest percentage of BACKGROUND citations, suggesting that their extra space and wider temporal scope lends itself more to positioning.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 27
Every source the paper cites for this claim was checked and supports it.
In the paper
“Collins claims this shift characterizes natural sciences, but not many social sciences, which are instead more likely to engage in continual contesting and turnover of core methods and assumptions (Evans et al., 2016).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 8 The Growth of Rapid Discovery Science
What the sources say
- Supportedcites Evans et al., 2016 source
The Literature Review (page unavailable) attributes to Collins a contrast between high-consensus, rapid-discovery hard sciences and fields where debate persists. The section on stability at the core (page unavailable) says preparadigmatic disciplines have an ever-shifting core, and the Discussion and Conclusions reports lower consensus and stability in the social sciences. Together, these support the paraphrase. (LLM, conf=0.88)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ultimately, 805 new bootstrapped patterns were added for the citing sentence, 669 for the preceding context, and 1159 for the following context, a total of over four times the number of manually curated patterns.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Consistent with prior work, the majority of citations are BACKGROUND (Moravcsik and Murugesan, 1975;Spiegel-Rüsing, 1977;Teufel et al., 2006b).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other non-conjunctive classifiers we tried resulted in substantially lower Macro F1: Naive Bayes, 0.286; knearest neighbor, 0.255 (k=3); and Linear-kernel SVM, 0.393 (C=1).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We began with the largest publicly-available list of citation patterns (Teufel, 2000) and extended it with 132 new patterns and 13 new lexical categories based on a manual analysis of the corpus.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
What the sources say
- Not matchedcites Teufel, 2000
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 6: The average cited paper receives an increasing number of USES and COMPARISON OR CONTRAST citations per year showing that field increasingly builds upon the same set of papers, providing a methodological lineage.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in 1991, authors compared with a diffuse group of parsing papers, e.g., (Shieber, 1988;Pereira and Warren, 1983;Haas, 1989), with such papers receiving at most three citations that year; whereas in 2000, most comparisons were to a core set of parsing papers, e.g., (Collins, 1999;Buchholz et al., 1999;Collins, 1997), with a much sharper (lower entropy) distribution of citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Knowledge of how a paper frames its contributions helps improve predicting its future impact, with a statistically significant improvement in AIC when the distribution of citation functions is added (likelihood ratio test, p ≤ 0.01).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8 shows which types of citations are significantly predictive of higher impact (p ≤ 0.01).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we did not observe this effect in how authors frame”
- Also stated in: § 7 Predicting Future Impact · plain
“their work as COMPARISON OR CONTRAST or EX-TENDS, with both having significant positive effects.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 7 Predicting Future Impact · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 8: Regression models for predicting the total number of citations five years after publication show that a paper's citation framing provides a statistically significant improvement in model fit and reveals which type of framing yields more cited papers.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All papers with at least five years of publication history in the anthology were considered, yielding a set of 10,434 papers.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) the way in which an author frames their work aids in predicting its future impact as the number of citations its receives, with the community favoring works that integrate many new technologies and also relate to prior work through comparison and contrast”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 2: Expected percentage of citation functions per section shows a clear narrative trajectory across sections.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite an increase in BACKGROUND citations, the total percentage of non-methodological citations still declines (r= -0.663, p ≤ 0.01), with authors instead increasingly including more USES citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 5: Changes in the average citation frame in ACL papers reveals a continued decline in the percentage of COM-PARISON OR CONTRAST and increase in USES citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(2) ACL workshops have evolved to become more like the mainstream conferences, with multi-iteration workshops being quicker to establish conference-like norms”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Patterns provide a powerful mechanism for capturing regularity in citation usage (Dong and Schäfer, 2011).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
What the sources say
- Not matchedcites Dong and Schäfer, 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“However, the authors note that us- ing the citing sentence alone is the correct context in 80% of the instances, so we view our implementation as a close approximation.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, a citation in a context describing system performances and results is likely to be a COMPARE OR CONTRAST, whereas one describing methodology is more likely to be USES.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that authors are sensitive to discourse structure and publication venue when citing”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Following best practices for annotating citations (Athar, 2014), annotators saw an extended context before and after the citing sentence, provided from the output of ParsCit.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
What the sources say
- Not matchedcites Athar, 2014
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“To avoid semantic drift (Riloff and Jones, 1999), a bootstrapped pattern was only included as a feature if the majority of its occurrences were with a single citation function.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.1 Features
What the sources say
- Not checked — no full text of the sourcecites Riloff and Jones, 1999
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also note that when performing crossvalidation, we compute the bootstrapped patterns and prototypical argument features using only contexts from the training data.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“All models were trained using a Random Forest classifier, which is robust to overfitting even with large numbers of features (Fernández-Delgado et al., 2014).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
What the sources say
- Not checked — no full text of the sourcecites Fernández-Delgado et al., 2014 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Annotated data is crucial for developing high accuracy for rare citation classes.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“When combining the two datasets, we omit the data labeled with their BACKGROUNDequivalent class to reduce the effects of a large majority class and because instances of the FUTURE class are merged into BACKGROUND according to their scheme.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Scientific papers commonly follow a structured section narrative to frame their contributions: Introduction, Methodology, Results, and Discussion (Skelton, 1994;Nwogu, 1997).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 4 Narrative Structure of Citation Function
What the sources say
- Not checked — no full text of the sourcecites Skelton, 1994
- Not checked — no full text of the sourcecites Nwogu, 1997 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation is performed using crossvalidation where each fold leaves out all citations of a single paper.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“All citations were then doubly-annotated by two trained annotators with expertise in NLP using the Brat tool (Stenetorp et al., 2012) and were then fully adjudicated to ensure quality.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
What the sources say
- Not checked — no full text of the sourcecites Stenetorp et al., 2012
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Stratifying by paper instead of instance is critical: since multiple citations may appear in the same sentence, instance-based stratification would leak information between training and test.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all three compared systems, we use identical parameter values as reported in the papers.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The resulting model has a variance inflation factor of < 10 for all variables.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlimited extra space for citations acted to modify authors' citation framing behavior; given unlimited space, authors chose to include proportionally more non-methodological citations.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: ACL venues began allowing unlimited references in 2010 for ACL, 2011 for NAACL, etc.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(4) the NLP field as a whole has seen increased consensus in what constitutes valid workwith a reduced need for positioning and excessive comparison-demonstrating its shift towards rapid discovery science.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, a differentiation of citation functions affords a deeper understanding of how scholars develop arguments for different publication venues as well as how these venues may demand different forms of knowledge representation and arguments over time.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As such, each venue has a distinct genre of writing, from the tentative results of workshop papers to journal papers with substantial synthesis.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“AIC measures each model's goodness of fit in proportion to the number of independent variables; when comparing models, the model with the minimal AIC is preferred (Akaike, 1974).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not checked — no full text of the sourcecites Akaike, 1974 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Multi-author papers are known to receive higher citation counts (Gazni and Didegah, 2011), partially due to the effects of self-citation (Fowler and Aksnes, 2007), and therefore we include the number of authors on the paper.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Each publication venue has its own expectation for the types of work it accepts, e.g., the degree of polish or depth of experiments.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 5 Venues and Citation Patterns
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Each section in the narrative adopts argumentative moves designed to convince the reader of the work's claims (Swales, 1986;Swales, 1990).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 4 Narrative Structure of Citation Function
What the sources say
- Not checked — no full text of the sourcecites Swales, 1986 source
- Not matchedcites Swales, 1990
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, these modes of citation help us understand the state of research efforts and their evolution more broadly for entire scientific fields like NLP.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“The scholarly narrative told through citations provides the reader with support for its claims and technical competence (Latour, 1987, p. 34).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not matchedcites Latour, 1987
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These trends show the increased incorporation of prior work to form a lineage of method technologies as well as show increased consensus on which works are sufficient for comparing against in order to establish a claim.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This framing could affect how the work is perceived and, ultimately, how it is received and cited within the com-munity (Shi et al., 2010).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not checked — no full text of the sourcecites Shi et al., 2010 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simply put, citation functions help frame an article's reception.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Collins (1994) proposed that a field can undergo a particular shift, referring to it as rapid discovery science, when the field (a) reaches high consensus on research topics as well as methods and technologies, and (b) then develops genealogies of methods and technologies that continually improve on one another.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 8 The Growth of Rapid Discovery Science
What the sources say
- Not checked — no full text of the sourcecites Collins (1994) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we include the publication venue, using the individual conference or workshop in which the paper was published to control for variations in prestige between venues.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Authors cite works for different reasons (or function), so regarding them as equivalent signals is potentially problematic.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“we regress out the number of citations from the citation function counts (Kutner et al., 2004;O'brien, 2007).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 7 Predicting Future Impact
What the sources say
- Not matchedcites Kutner et al., 2004
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites O'brien, 2007 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper relies on this without support.
The number of ACL workshops increased from around ten in the 1990s to over 100 by 2010, with many workshops having multiple iterations across the years.
“The number of workshops has increased substantially with the growth of the field, from around ten workshops in the 1990s to over 100 workshops by 2010, with many workshops having multiple iterations across the years.”
The paper relies on this without support.
The growth of the ACL community has been accompanied by the creation of new publication venues.
“The growth of the ACL community has been accompanied by the creation of new publication venues.”
The paper relies on this without support.
ACL venues began allowing unlimited references in 2010 for ACL, 2011 for NAACL, etc.
“This transition corresponds to the date at which ACL venues began allowing unlimited references (2010 for ACL, 2011 for NAACL, etc.).”
The paper relies on this without support.
Many fluff citations exist, while some less common ones are substantively relevant to the paper's argument.
“Many fluff citations exist, while some less common ones are substantively relevant to the paper's argument.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“ACL-ARC is a dataset of citation intents released by Jurgens et al. (2018). The dataset is based on a sample of papers from the ACL Anthology Reference Corpus (Bird et al., 2008) and includes 1,941 citation instances from 186 papers and is annotated by domain experts in the NLP field.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“previous state-of-the-art results of Jurgens et al. (2018) 54.6”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our model achieves a new state-ofthe-art on an existing ACL anthology dataset (ACL-ARC) with a 13.3% absolute increase in F1 score, without relying on external linguistic resources or hand-engineered features as done in existing methods.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A citation context includes text spans in a citing paper describing a referenced work and has been shown to be the primary signal in intent classification (Teufel et al., 2006; Abu-Jbara et al., 2013; Jurgens et al., 2018).”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jurgens et al. (2018) expanded all pre-existing featurebased efforts on citation intent classification by proposing a comprehensive set of engineered features, including boostrapped patterns, topic modeling, dependency-based, and metadata features for the task.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, for the Background and Future classes, no grammatical features were provided by Jurgens and colleagues (2018), rendering a direct comparison with our results unfeasible.”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Jurgens et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Existing models for this problem are feature-based, modeling the citation context with respect to a set of predefined handengineered features (such as linguistic patterns or cue phrases) and ignoring other signals that could improve prediction.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“ACL-ARC is a dataset of citation intents released by Jurgens et al. (2018). The dataset is based on a sample of papers from the ACL Anthology Reference Corpus (Bird et al., 2008) and includes 1,941 citation instances from 186 papers and is annotated by domain experts in the NLP field.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Our implementation of this model achieves a macro-averaged F1 score of 0.526 using 10-fold cross-validation”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“We note that these results were obtained without using hand-curated features or additional linguistic resources as used in Jurgens et al. (2018).”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“We also experimented with adding features used in Jurgens et al. (2018) to our best model and not only we did not see any improvements, but we observed at least 1.7% decline in performance.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“followed by Jurgens et al. (2018)”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Jurgens et al. (2018) incorporate a variety of features, ranging from pattern-based features to topic-modeling features, to citation graph features. They also incorporate section titles and relative section position in the paper as features.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Jurgens et al. (2018) proposed a six category system for citation intents.”
From Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“An early contribution to CIC is the ACL-ARC citations dataset (Jurgens et al., 2018), which provides nearly 2000 manually annotated citation contexts from the NLP domain.”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Jurgens et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“potentially refined by citation type or class, such as background citations and method citations (Teufel et al., 2006; AbuRa'ed et al., 2017; Jurgens et al., 2018; Pride and Knoth, 2020; Lauscher et al., 2022).”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Jurgens et al., 2018
In the citing paper
“ACL-ARC (Jurgens et al., 2018) and SciCite (Cohan et al., 2019) assign intent labels (e.g.Comparison, Extension, etc.) to sentences from scientific papers that cite other papers.”
From SciBERT: A Pretrained Language Model for Scientific Text (2019) · cites this paper as Jurgens et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
“The resulting citation function dataset contains 3,083 instances.”
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.3 Results and Discussion
“we apply our classifier trained on our combined dataset (2600 citation instances) to the ACL Anthology”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Measuring the Evolution of a Scientific Field through Citation Frames > 9 Conclusion
“MOTIVATION 98 CONTINUATION 73 FUTURE 68”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Class Description
“EXTENSION Extends P 's data, methods, etc.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Measuring the Evolution of a Scientific Field through Citation Frames > 3.2 Experimental Setup
“We map their 12 function classes into our six classes (see Appendix A).”
§ Measuring the Evolution of a Scientific Field through Citation Frames > Conflict
“CoCoGM Comparison or Contrast CoCo Comparison or Contrast CoCoR0 Comparison or Contrast CoCoXY Background PBas Extends PUse Uses PModi Extends PMot Motivation PSim Comparison or Contrast PSup Comparison or Contrast Neut Background CoMetN Comparison or Contrast CoGoaN Comparison or Contrast”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Frame Analysis: An Essay on the Organization of Experience.
Gamson, W. A., & Goffman, E. (1975) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2064022 · Cited by 11,891
A Caution Regarding Rules of Thumb for Variance Inflation Factors
O’brien, R. M. (2007) · Quality & Quantity · doi:10.1007/s11135-006-9018-6 · Cited by 11,217
(1987) · Cited by 8,955
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252
Research works printed but not linked: 5
- Fernández-Delgado Manuel, 2014, Journal of Machine Learning Research (JMLR), 15, 3133
- Haas Andrew, 1989, Computational Linguistics, 15, 219
- Leech Geoffrey, 1992, Language Research, 28, 1
- Pedregosa Fabian, 2011, The Journal of Machine Learning Research (JMLR), 12, 2825
- Skelton John, 1994, British Journal of General Practice, 44, 455
Cited by
Sorted by citations
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
The Diversity–Innovation Paradox in Science
Hofstra, B., Kulkarni, V., Galvez, S. M.-N., He, B. D., Jurafsky, D., & McFarland, D. A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915378117 · Cited by 1,000
Self-citationS2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457
Structural Scaffolds for Citation Intent Classification in Scientific Publications
Cohan, A., Ammar, W., van Zuylen, M., & Cady, F. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1361 · Cited by 214
(2021) · doi:10.1145/3462757.3466088 · Cited by 151
How status of research papers affects the way they are read and cited
Teplitskiy, M., Duede, E., Menietti, M., & Lakhani, K. R. (2022) · Research Policy · doi:10.1016/j.respol.2022.104484 · Cited by 153
Scientific document summarization via citation contextualization and scientific discourse
Cohan, A., & Goharian, N. (2017) · International Journal on Digital Libraries · doi:10.1007/s00799-017-0216-8 · Cited by 100
SciREX: A Challenge Dataset for Document-Level Information Extraction
Jain, S., van Zuylen, M., Hajishirzi, H., & Beltagy, I. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.670 · Cited by 159
New directions in science emerge from disconnection and discord
Lin, Y., Evans, J. A., & Wu, L. (2021) · Journal of Informetrics · doi:10.1016/j.joi.2021.101234 · Cited by 86
Counterfactual Inference for Text Classification Debiasing
Chen, Q., Feng, F., Wen, L., Ma, C., & Xie, P. (2021) · doi:10.18653/v1/2021.acl-long.422 · Cited by 80
Hou, Y., Jochim, C., Gleize, M., Bonin, F., & Ganguly, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1513 · Cited by 79
Creative Destruction: The Structural Consequences of Scientific Curation
McMahan, P., & McFarland, D. A. (2021) · American Sociological Review · doi:10.1177/0003122421996323 · Cited by 72
Self-citationScientific Text Sentiment Analysis using Machine Learning Techniques
Raza, H., Faizan, M., Hamza, A., Mushtaq, A., & Akhtar, N. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0101222 · Cited by 67
Citation Intent Classification Using Word Embedding
Roman, M., Shahid, A., Khan, S., Koubâa, A., & Yu, L. (2021) · IEEE Access · doi:10.1109/access.2021.3050547 · Cited by 59
What Types of Novelty Are Most Disruptive?
Leahey, E., Lee, J., & Funk, R. J. (2023) · American Sociological Review · doi:10.1177/00031224231168074 · Cited by 66
Domain-Independent Extraction of Scientific Concepts from Research Articles
Brack, A., D’Souza, J., Hoppe, A., Auer, S., & Ewerth, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_17 · Cited by 60
Wang, Y., & Zhang, C. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101091 · Cited by 44
A meta-analysis of semantic classification of citations
Kunnath, S. N., Herrmannova, D., Pride, D., & Knoth, P. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00159 · Cited by 43
ELLE: Efficient Lifelong Pre-training for Emerging Data
Qin, Y., Zhang, J., Lin, Y., Liu, Z., Li, P., Sun, M., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.220 · Cited by 67
Shi, S., Hu, K., Xie, J., Guo, Y., & Wu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103531 · Cited by 44
Citation Recommendation Using Distributed Representation of Discourse Facets in Scientific Articles
Kobayashi, Y., Shimbo, M., & Matsumoto, Y. (2018) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3197026.3197059 · Cited by 45
End-to-End Construction of NLP Knowledge Graph
Mondal, I., Hou, Y., & Jochim, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.165 · Cited by 53
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · arXiv · doi:10.48550/arxiv.1903.10676 · Cited by 41
Understanding the meanings of citations using sentiment, role, and citation function classifications
Budi, I. D., & Yaniasih, Y. (2022) · Scientometrics · doi:10.1007/s11192-022-04567-4 · Cited by 41
Aljohani, N. R., Fayoumi, A. G., & Hassan, S. (2021) · Journal of Information Science · doi:10.1177/0165551521991022 · Cited by 41
Metrics and mechanisms: Measuring the unmeasurable in the science of science
Wu, L., Kittur, A., Youn, H., Milojević, S., Leahey, E., Fiore, S. M., & Ahn, Y. (2022) · Journal of Informetrics · doi:10.1016/j.joi.2022.101290 · Cited by 44
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 54
Automatic Classification of Algorithm Citation Functions in Scientific Literature
Tuarob, S., Kang, S. W., Wettayakorn, P., Pornprasit, C., Sachati, T., Hassan, S., & Haddawy, P. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2913376 · Cited by 39
Efficient Domain Adaptation of Language Models via Adaptive Tokenization
Sachidananda, V., Kessler, J. S., & Lai, Y.-A. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.16 · Cited by 47
Lauscher, A., Ko, B., Kuehl, B., Johnson, S., Cohan, A., Jurgens, D., & Lo, K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.137 · Cited by 38
Self-citationS2ORC: The Semantic Scholar Open Research CorpusPreprint
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02782 · Cited by 115
Continual Training of Language Models for Few-Shot Learning
Ke, Z., Lin, H., Shao, Y., Xu, H., Shu, L., & Liu, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.695 · Cited by 35
Utilizing BERT for biomedical and clinical text mining
Zhu, R., Tu, X., & Huang, J. X. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-819314-3.00005-7 · Cited by 34
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yao, Y., Huang, S., Wang, W., Dong, L., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.40 · Cited by 43
Knowledge Inheritance for Pre-trained Language Models
Qin, Y., Lin, Y., Jing, Y., Zhang, J., Han, X., Zhang, Z., Su, Y., Liu, Z., Li, P., Sun, M., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.288 · Cited by 37
Wang, M., Zhang, J., Jiao, S., Zhang, X., Zhu, N., & Chen, G. (2020) · Scientometrics · doi:10.1007/s11192-020-03677-1 · Cited by 32
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation
Diao, S., Xu, R., Su, H., Jiang, Y., Song, Y., & Zhang, T. (2021) · doi:10.18653/v1/2021.acl-long.259 · Cited by 35
How New Ideas Diffuse in Science
Cheng, M., Smith, D. S., Ren, X., Cao, H., Smith, S., & McFarland, D. A. (2023) · American Sociological Review · doi:10.1177/00031224231166955 · Cited by 32
Self-citationMapping the carbon finance research: A bibliometric exploration of innovation in sustainable finance
Roshid, M. M., Karim, R., Waaje, A., Islam, S., & Bhowmik, R. C. (2025) · Development and sustainability in economics and finance. · doi:10.1016/j.dsef.2025.100048 · Cited by 31
An Authoritative Approach to Citation Classification
Pride, D., & Knoth, P. (2020) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3383583.3398617 · Cited by 31
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.-W. D., Wei, X., Arnold, A., & Ren, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.351 · Cited by 38
Zhao, H., Luo, Z., Feng, C., Zheng, A., & Liu, X. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1524 · Cited by 31
Towards employing native information in citation function classification
Zhang, Y., Zhao, R., Wang, Y., Chen, H., Mahmood, A., Zaib, M., Zhang, W. E., & Sheng, Q. Z. (2022) · Scientometrics · doi:10.1007/s11192-021-04242-0 · Cited by 28
Exploring the applicability of large language models to citation context analysis
Nishikawa, K., & Koshiba, H. (2024) · Scientometrics · doi:10.1007/s11192-024-05142-9 · Cited by 29
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.-W. D., Wei, X., Arnold, A., & Ren, X. (2022) · doi:10.18653/v1/2022.bigscience-1.1 · Cited by 29
Quantifying the rise and fall of scientific fields
Singh, C. K., Barme, E., Ward, R., Tupikina, L., & Santolini, M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270131 · Cited by 25
CitePrompt: Using Prompts to Identify Citation Intent in Scientific Papers
Lahiri, A., Sanyal, D. K., & Mukherjee, I. (2023) · 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl57899.2023.00017 · Cited by 24
Identifying Referential Intention with Heterogeneous Contexts
Yu, W., Yu, M., Zhao, T., & Jiang, M. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380175 · Cited by 24
Scientia Potentia Est—On the Role of Knowledge in Computational Argumentation
Lauscher, A., Wachsmuth, H., Gurevych, I., & Glavaš, G. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00525 · Cited by 29
Explaining Relationships Between Scientific Documents
Luu, K., Wu, X., Koncel-Kedziorski, R., Lo, K., Cachola, I. A., & Smith, N. A. (2021) · doi:10.18653/v1/2021.acl-long.166 · Cited by 27
AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain
Hong, J., Kim, T., Lim, H., & Choo, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.385 · Cited by 26
CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding
Wright, D., & Augenstein, I. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.157 · Cited by 28
Deep learning meets bibliometrics: A survey of citation function classification
Zhang, Y., Wang, Y., Sheng, Q. Z., Yao, L., Chen, H., Wang, K., Mahmood, A., Zhang, W. E., Zaib, M., Sagar, S., et al. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101608 · Cited by 26
Structure-inducing pre-training
McDermott, M. B. A., Yap, B., Szolovits, P., & Žitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00647-z · Cited by 29
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient FrameworkPreprint
Yao, X., Zheng, Y., Yang, X., & Yang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04130 · Cited by 26
The use of citation context to detect the evolution of research topics: a large-scale analysis
Jebari, C., Herrera‐Viedma, E., & Cobo, M. J. (2021) · Scientometrics · doi:10.1007/s11192-020-03858-y · Cited by 21
Jiang, X., & Liu, J. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24748 · Cited by 20
Multi-task learning model for citation intent classification in scientific publications
Qi, R., Jia, W., Shao, Z., Li, Z.-G., Chen, H., Sun, Y., & Li, S. (2023) · Scientometrics · doi:10.1007/s11192-023-04858-4 · Cited by 21
Prompting Strategies for Citation Classification
Kunnath, S. N., Pride, D., & Knoth, P. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615018 · Cited by 19
TDM-CFC: Towards Document-Level Multi-label Citation Function Classification
Zhang, Y., Wang, Y., Sheng, Q. Z., Mahmood, A., Zhang, W. E., & Zhao, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91560-5_26 · Cited by 19
GraphCite: Citation Intent Classification in Scientific Publications via Graph Embeddings
Berrebbi, D., Huynh, N., & Balalau, O. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524657 · Cited by 18
Runhui, L., Li, Y., Ze, J., Qiqi, X., & Chen, X. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103933 · Cited by 18
Prester, J., Wagner, G., Schryen, G., & Hassan, N. R. (2020) · Decision Support Systems · doi:10.1016/j.dss.2020.113432 · Cited by 18
Healthcare: A Growing Role for Large Language Models and Generative AI
Pahune, S. A., & Rewatkar, N. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55573 · Cited by 19
Woo, S., & Walsh, J. P. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00303 · Cited by 16
A simulation-based analysis of the impact of rhetorical citations in science
Bao, H., & Teplitskiy, M. (2024) · Nature Communications · doi:10.1038/s41467-023-44249-0 · Cited by 18
How Data Scientists Review the Scholarly Literature
Mysore, S., Jasim, M., Song, H., Akbar, S., Randall, A. K. C., & Mahyar, N. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578309 · Cited by 19
Self-supervised Regularization for Text Classification
Zhou, M., Li, Z., & Xie, P. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00389 · Cited by 14
Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications
Lucy, L., Dodge, J., Bamman, D., & Keith, K. A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.433 · Cited by 14
Beyond networks: Aligning qualitative and computational science studies
Cambrosio, A., Cointet, J., & Abdo, A. H. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00055 · Cited by 17
Wittek, M., Bartenhagen, C., & Berthold, F. (2022) · Social Networks · doi:10.1016/j.socnet.2022.05.001 · Cited by 15
Aljohani, N. R., Fayoumi, A. G., & Hassan, S. (2021) · Applied Sciences · doi:10.3390/app112210970 · Cited by 13
Zheng, A., Zhao, H., Luo, Z., Feng, C., Liu, X., & Ye, Y. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102638 · Cited by 13
Dattolo, A., & Corbatto, M. (2021) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24578 · Cited by 12
Citation recommendation based on argumentative zoning of user queries
Ma, S.-T., Zhang, C.-Z., Zhang, H., & Gao, Z. (2025) · J. Informetrics · doi:10.1016/j.joi.2024.101607 · Cited by 15
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding
Fang, H., & Xie, P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00521 · Cited by 14
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Zhang, Z., Wang, S., Yi‐chong, X., Fang, Y., Yu, W., Liu, Y., Zhao, H. Y., Zhu, C., & Zeng, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.416 · Cited by 17
Citation Function Classification Based on Ontologies and Convolutional Neural Networks
Bakhti, K., Niu, Z., Yousif, A., & Nyamawe, A. S. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-95522-3_10 · Cited by 12
A Context-based Framework for Resource Citation Classification in Scientific Literatures
Zhao, H., Luo, Z., Feng, C., & Ye, Y. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331348 · Cited by 12
Koloveas, P., Chatzopoulos, S., Vergoulis, T., & Tryfonopoulos, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05409-8_13 · Cited by 15
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
Wang, X., Zhou, W., Zhang, Q., Zhou, J., Gao, S., Wang, J., Zhang, M., Gao, X., Chen, Y. W., & Gui, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.35 · Cited by 14
CiteBench: A Benchmark for Scientific Citation Text Generation
Funkquist, M., Kuznetsov, I., Hou, Y., & Gurevych, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.455 · Cited by 11
Yan, E., Chen, Z., & Li, K. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00040 · Cited by 11
ACT: An Annotation Platform for Citation Typing at Scale
Pride, D., Knoth, P., & Harag, J. (2019) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2019.00055 · Cited by 11
Ghosal, T., Varanasi, K. K., & Kordoni, V. (2023) · Scientometrics · doi:10.1007/s11192-023-04811-5 · Cited by 12
A New Scheme for Citation Classification based on Convolutional Neural Networks
Bakhti, K., Niu, Z., & Nyamawe, A. S. (2018) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2018-141 · Cited by 10
Sentiment and intent classification of in-text citations using BERT
Visser, R., & Dunaiski, M. (2022) · EPiC series in computing · doi:10.29007/wk21 · Cited by 10
Hybrid Data Augmentation for Citation Function Classification
Zhang, Y., Wang, Y., Sheng, Q. Z., Mahmood, A., Zhang, W. E., & Zhao, R. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191695 · Cited by 11
Semi-Automatic Annotation for Citation Function Classification
Bakhti, K., Niu, Z., & Nyamawe, A. S. (2018) · International Conference on Control, Artificial Intelligence, Robotics & Optimization (ICCAIRO) · doi:10.1109/iccairo.2018.00016 · Cited by 10
Roman, M., Shahid, A., Uddin, M. I., Hua, Q., & Maqsood, S. (2021) · Complexity · doi:10.1155/2021/5554874 · Cited by 10
Contextualised segment-wise citation function classification
Jiang, X., & Chen, J. (2023) · Scientometrics · doi:10.1007/s11192-023-04778-3 · Cited by 10
Scientific document processing: challenges for modern learning methods
Kashyap, A. R., Yang, Y., & Kan, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00352-7 · Cited by 13
Opinion Mining by Convolutional Neural Networks for Maximizing Discoverability of Nanomaterials
Xie, T., Wan, Y., Wang, H., Østrøm, I., Wang, S., He, M., Deng, R. D., Wu, X., Grazian, C., Kit, C., et al. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00746 · Cited by 11
Wang, Y., Yi, X., & Zhang, C. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101550 · Cited by 9
Ma, Y., Qing, L., Kang, Y., Liu, J., Zhang, Y., Cheng, Q., Lu, W., & Liu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127226 · Cited by 9
Keyphrase Extraction in Scholarly Digital Library Search Engines
Patel, K., Caragea, C., Wu, J. F., & Giles, C. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59618-7_12 · Cited by 9
Studying the Evolution of Scientific Topics and their Relationships
Uban, A. S., Caragea, C., & Dinu, L. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.167 · Cited by 9
An empirical study of the design choices for local citation recommendation systems
Medić, Z., & Šnajder, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116852 · Cited by 10
Pre-trained transformer-based citation context-aware citation network embeddings
Ohagi, M., & Aizawa, A. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3533290 · Cited by 9
Metadata
- Authors
- Jurgens, D., Kumar, S., Hoover, R., McFarland, D., & Jurafsky, D.
- Affiliations
- University of Michigan (US)
- Stanford University (US)
- Stanford University,
- Corresponding author
- Jurgens, David
- Kumar, Srijan
- Hoover, Raine
- Jurafsky, Dan
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2018-12
- Volume
- 6
- Pages
- 391-406
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2808556605
- Cited by
- 212
- References
- 79
- Citations by year
- 2026: 16 · 2025: 25 · 2024: 21 · 2023: 42 · 2022: 34 · 2021: 43 · 2020: 19 · 2019: 12
- FWCI
- 14.11
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Science Foundation (1633036)
- Brown Institute for Media Innovation
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Keywords
- citation functions, citation analysis, citation counts, field evolution, publication venues, discourse structure, citation classification, natural language processing, disciplinary consensus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
