The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 19 of the 28 sources its claims cite were checked for support.
Structural Scaffolds for Citation Intent Classification in Scientific Publications
Open accessAbstract
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, Field Cady. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
Overview
CoChat Summary
TL;DR
The paper proposes a multitask neural model with two “structural scaffolds”—predicting section title and citation worthiness—to improve citation intent classification. It reports a new state of the art on ACL-ARC (67.9 macro F1, +13.3 absolute over prior SOTA) and introduces SciCite, a larger multi-domain citation intent dataset.
Key findings
- On ACL-ARC, the best model (`BiLSTM-Attn w/ ELMo + both scaffolds`) reaches 67.9 macro F1, a 13.3 absolute improvement over the previous state of the art (Jurgens et al., 2018) at 54.6.
- Each scaffold helps: section title scaffold reaches 56.9 F1 and citation worthiness scaffold 56.3 F1, versus 51.8 for the BiLSTM-Attn baseline; using both scaffolds reaches 63.1 F1 without ELMo.
- SciCite contains 11,020 instances from 6,627 papers across computer science and medicine, with categories Background (0.58), Method (0.29), and Result comparison; ACL-ARC has 1,941 instances from 186 papers. Best SciCite result is 84.0 macro F1 with ELMo plus both scaffolds.
- Adding Jurgens et al. (2018) hand-engineered features to the best model did not improve performance and caused at least a 1.7% decline.
Methods
- The model is a multitask neural framework: a shared BiLSTM with attention encodes the citation context, using 100-dimensional GloVe and 1,024-dimensional ELMo representations; task-specific MLPs produce outputs for the main citation intent task and two auxiliary scaffold tasks.
- The two scaffolds are citation worthiness (whether a sentence needs a citation) and section title prediction (normalized to introduction, related work, method, experiments, conclusion); training uses a weighted joint loss, AdaDelta, gradient clipping at 5.0, and early stopping.
- Data include ACL-ARC (1,941 instances, 186 papers, six intent categories) and SciCite (11,020 instances, 6,627 papers, three categories), with SciCite crowdsourced via Figure Eight and a separate expert-annotated test set of 1,861 instances; scaffold data sizes are about 47K/50K for ACL-ARC and 91K/73K for SciCite.
Limitations
- The paper states that errors occur on rare categories, e.g., FUTUREWORK has the fewest data points and is hardest to classify, and some errors may require additional context; an attempt to encode extended surrounding context with a BiLSTM concatenated to the citation vector caused a large decline in overall performance.
- The paper states future work includes replacing ELMo with other contextualized representations such as BERT.
- Evident from the text: evaluation is limited to ACL-ARC (NLP) and SciCite (computer science and medicine); SciCite annotations are crowdsourced with quality controls, and only a 100-sample subset was re-annotated by an expert, with 86% agreement.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
16 Valid · 7 Not valid · 3 Not checked · 64 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“previous state-of-the-art results of Jurgens et al. (2018) 54.6”
What the sources say
- Disputedcites Jurgens et al., 2018 sourceExperimental — accuracy not yet measured
Table 7 reports a macro F1 of 0.530 for the authors’ classifier, not 0.546. The Experimental Setup describes evaluation on a combined citation-function dataset, so the paper also does not establish 54.6 as an ACL-ARC result. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our model achieves a new state-ofthe-art on an existing ACL anthology dataset (ACL-ARC) with a 13.3% absolute increase in F1 score, without relying on external linguistic resources or hand-engineered features as done in existing methods.”
- Also stated in: § 1 Introduction · plain
“we achieve a new state-of-the-art of 67.9% F1 on the ACLARC citations benchmark, an absolute 13.3% increase over the previous state-of-the-art (Jurgens et al., 2018)”
- Also stated in: § 4.3 Results · plain
“The best result is achieved when we also add ELMovectors (Peters et al., 2018) to the input representations in 'BiLSTM-Attn w/ ELMo + both scaffolds', achieving an F1 of 67.9, a major improvement from the previous state-of-the-art results of Jurgens et al. (2018) 54.6 ( ∆=13 . 3 ).”
What the sources say
- Partly supportedcites Jurgens et al., 2018 sourceExperimental — accuracy not yet measured
Section 3.3 (page unavailable) says the paper’s methods outperformed the closest state of the art, supporting the general performance claim. It does not report a 13.3-percentage-point absolute F1 increase in the passages found; Section 3.2 describes evaluation on a combined citation-function dataset rather than establishing that figure for ACL-ARC. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our implementation of this model achieves a macro-averaged F1 score of 0.526 using 10-fold cross-validation”
What the sources say
- Supportedcites Jurgens et al., 2018 source
The citation identifies the model being implemented: Section 3.2 describes Jurgens et al.’s classifier and its evaluation using macro-averaged F1. The 0.526 score and 10-fold cross-validation are stated as results of the citing authors’ own implementation, not attributed to Jurgens et al. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“followed by Jurgens et al. (2018)”
What the sources say
- Supportedcites Jurgens et al., 2018 source
The citation identifies the Jurgens et al. system used as a comparison baseline. Section 3.2 describes their citation-function classifier; the 'second best' ranking on SciCite is the citing paper's own comparison, not a result attributed to Jurgens et al. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We note that these results were obtained without using hand-curated features or additional linguistic resources as used in Jurgens et al. (2018).”
What the sources say
- Supportedcites Jurgens et al., 2018 source
Section 3.1 (page not supplied) says Jurgens et al. added 132 patterns and 13 lexical categories based on manual analysis, supporting the reference to hand-curated features. Section 3.2 (page not supplied) says they used CoreNLP for syntactic processing and pretrained GloVe vectors, supporting the reference to additional linguistic resources. The citing authors’ statement about how they obtained their own results is not attributed to Jurgens et al. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
39 more claims
Create a free account to see all 90 claims and every source verdict.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 14
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“section titles have frequently been used as auxiliary or additional features within multitask learning frameworks for CIC (Cohan et al., 2019; Qi et al., 2023; Zhang et al., 2022)”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Cohan et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
From SciBERT: A Pretrained Language Model for Scientific Text (2019) · cites this paper as Cohan et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Incorporating auxiliary scaffold tasks via multitask learning has been studied before for improving span-labeling and text classification (Swayamdipta et al., 2018b; Cohan et al., 2019).”
From TLDR: Extreme Summarization of Scientific Documents (2020) · cites this paper as Cohan et al., 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“SciBERT (Beltagy et al., 2019), pretrained on scientific corpora and equipped with a specialized scientific vocabulary – SciVOCAB –, consistently outperforms general-pur- pose models in processing academic texts (Cohan et al., 2019).”
From CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses (2025) · cites this paper as Cohan et al., 2019
Experimental — accuracy not yet measuredIn the citing paper
“In addition, SCIBERT achieves new SOTA results on ACL-ARC (Cohan et al., 2019), and the NER part of SciERC (Luan et al., 2018).”
From SciBERT: A Pretrained Language Model for Scientific Text (2019) · cites this paper as Cohan et al., 2019
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3 Results
“| Previous SOTA (Jurgens et al., 2018) | 79.6 |”
§ 4.4 Analysis
“| Previous SOTA (Jurgens et al., 2018) | 77.9 | 92.9 | 84.7 | 91.5 | 63.1 | 74.7 | 79.1 | 77.3 | 78.2 | 82.8 | 77.8 | 79.2 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Citation contexts were annotated by 850 crowdsource workers who made a total of 29,926 annotations and individually made between 4 and 240 annotations. Each sentence was annotated, on average, 3.74 times. This resulted in a total 9,159 crowdsourced instances”
Reported 3.74 · recomputed 3.267387
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“…(GloVe, Pennington et al., 2014) and contextualized embeddings (ELMo, Peters et al., 2018), i.e.: We then use a bidirectional long short-term memory (Hochreiter and Schmidhuber, 1997) (BiL-STM) network with hidden size of d 2 to obtain a contextual representation of each token vector with respect…”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“For example, at the time of finalizing the camera ready version of this paper, Beltagy et al. (2019) showed that a BERT contextualized representation model (Devlin et al., 2018) trained on scientific text can achieve promising results on the SciCite dataset.”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
(2010) · International Conference on Machine Learning · Cited by 13,995
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 4×
“We experiment with two variants of this model: with and without using the contex-tualized word vector representations (ELMo) of Peters et al. (2018).”
BackgroundMethodInfluentialCaruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · cited 2×
“Multitask learning as defined by Caruana (1997) is an approach to inductive transfer learning that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias.”
MethodADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322 · cited 1×
“We compute the gradient of the loss for each mini-batch and tune model parameters using the AdaDelta optimizer (Zeiler, 2012) with gradient clipping threshold of 5.0.”
MethodInfluentialAllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M., Schmitz, M., & Zettlemoyer, L. (2018) · Proceedings of Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,252 · cited 1×
“We implement our proposed scaffold framework using the AllenNLP library (Gardner et al., 2018).”
Method
Cited by
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
“In addition, SCIBERT achieves new SOTA results on ACLARC (Cohan et al., 2019), and the NER part of SciERC (Luan et al., 2018).”
BackgroundMethodResultInfluentialSelf-citationS2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457
“Other tasks that leverage citation contexts include classifying citation intent (Teufel et al., 2006; Jurgens et al., 2018; Cohan et al., 2019), identifying citation sentiment (Athar and Teufel, 2012), identifying meaningful citations (Valen-zuela et al., 2015), extracting key phrases (Caragea et…”
BackgroundTLDR: Extreme Summarization of Scientific Documents
Cachola, I., Lo, K., Cohan, A., & Weld, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.428 · Cited by 169
“Incorporating auxiliary scaffold tasks via multitask learning has been studied before for improving span-labeling and text classification (Swayamdipta et al., 2018b; Cohan et al., 2019).”
MethodSelf-citationA Survey of Joint Intent Detection and Slot Filling Models in Natural Language Understanding
Weld, H., Huang, X., Long, S., Poon, J., & Han, S. C. (2023) · ACM Computing Surveys · doi:10.1145/3547138 · Cited by 167
Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021) · bioRxiv (published in Quantitative Science Studies) · doi:10.1101/2021.03.15.435418 · Cited by 202
“…with RNN (Recursive Neural Network) architectures such as BidGRU (Bidirectional Gated Recurrent Unit, an architecture similar to the approach of Cohan et al. (2019) for citation intent classification), we obtained significant improvements with the more recently introduced ELMo (Embeddings from…”
Method
Metadata
- Authors
- Cohan, A., Ammar, W., van Zuylen, M., & Cady, F.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
- Published
- 2019
- Pages
- 3586-3596
- OpenAlex
- W2933655602
- Semantic Scholar
- 95ebe37c856e914b760bc5db63561f461ec444cc
- Cited by
- 214
- References
- 29
- Citations by year
- 2026: 19 · 2025: 26 · 2024: 26 · 2023: 47 · 2022: 38 · 2021: 35 · 2020: 16 · 2019: 7
- FWCI
- 76.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- citation intent classification, scientific publications, structural scaffolds, citation analysis, bibliographic metadata
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Structural Scaffolds for Citation Intent Classification in Scientific Publications. https://app.cochat.ai/papers/10.18653/v1/n19-1361
- The paper
- Cohan, A., Ammar, W., van Zuylen, M., & Cady, F. (2019). Structural Scaffolds for Citation Intent Classification in Scientific Publications. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). https://doi.org/10.18653/v1/n19-1361
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.