The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index; 15 of the 29 sources its claims cite were checked for support.
CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses
Open accessAbstract
Abstract Understanding the motivations underlying scholarly citations is essential to evaluate research impact and promote transparent scholarly communication. This study introduces CiteFusion , an ensemble framework designed to address the multi-class Citation Intent Classification task on two benchmark datasets: SciCite and ACL-ARC. The framework employs a one-vs-all decomposition of the multi-class task into class-specific binary subtasks, leveraging complementary pairs of SciBERT and XLNet models, independently tuned, for each citation intent. The outputs of these base models are aggregated through a feedforward neural network meta-classifier to reconstruct the original classification task. To enhance interpretability, SHAP ( SHapley Additive exPlanations ) is employed to analyze token-level contributions, and interactions among base models, providing transparency into the classification dynamics of CiteFusion, and insights about the kind of misclassifications of the ensemble. In addition, this work investigates the semantic role of structural context by incorporating section titles, as framing devices, into input sentences, assessing their positive impact on classification accuracy. CiteFusion ultimately demonstrates robust performance in imbalanced and data-scarce scenarios: experimental results show that CiteFusion achieves state-of-the-art performance, with Macro-F1 scores of 89.60% on SciCite, and 76.24% on ACL-ARC. Furthermore, to ensure interoperability and reusability, citation intents from both datasets schemas are mapped to Citation Typing Ontology (CiTO) object properties, highlighting some overlaps. Finally, we describe and release a web-based application that classifies citation intents leveraging the CiteFusion models developed on SciCite.
Overview
CoChat Summary
TL;DR
CiteFusion is an ensemble framework for citation intent classification that decomposes multi-class tasks into one-vs-all binary subtasks, pairing SciBERT and XLNet per class and aggregating with a feedforward neural network meta-classifier. It reports state-of-the-art Macro-F1 of 89.60% on SciCite and 76.24% on ACL-ARC, with SHAP analyses and section titles used as framing devices.
Key findings
- CiteFusion (WS) achieves Macro-F1 89.60%, Micro-F1/accuracy 90.80%, and weighted F1 90.87% on SciCite; on ACL-ARC it achieves Macro-F1 76.24%, Micro-F1/accuracy 81.29%, and weighted F1 80.77%.
- Including section titles consistently improved performance: mean improvements were ΔA = 0.73 and ΔMF1 = 0.93 on SciCite, and ΔA = 1.86 and ΔMF1 = 1.55 on ACL-ARC.
- The FFNN meta-classifier consistently outperformed other aggregation strategies except in the ACL-ARC WoS setting, where Random Forest performed marginally better; across 10 additional WS runs, SciCite mean accuracy was 90.53 (σ = 0.14) and Macro-F1 89.34 (σ = 0.18), while ACL-ARC mean accuracy was 80.44 and Macro-F1 73.71 (σ = 1.61).
- SHAP analyses showed SciBERT emphasizes domain-specific scientific tokens while XLNet relies on more general terms; section titles reshaped base-model attention, and dataset labels were mapped to CiTO object properties.
Methods
- Design: one-vs-all decomposition into class-specific binary subtasks; for each class, SciBERT and XLNet were independently fine-tuned, their positive probabilities concatenated into a 2KD-dimensional vector, and a feedforward neural network meta-classifier produced the final multi-class prediction; voting, StackingC, a geometric framework, RF, SVM, LR, and KNN were also compared.
- Data: SciCite (11,020 instances; Background 58%, Method 29%, Result Comparison 13%) and ACL-ARC (1,941 instances; Background 51%, Extends 4%, Uses 18%, Motivation 5%, Compare/Contrast 18%, Future Work 4%), evaluated in WS (with section titles) and WoS (without section titles) settings.
- Analysis: Macro-F1 and accuracy as main metrics; 10 additional runs for computational instability; SHAP at level-0 token level and level-1 meta-classifier level; error analysis using attribution mass and confusion matrix.
Limitations
- The paper states that SHAP cannot fully account for feature dependencies in transformer-based models, that reproducibility/replicability was not evaluated on alternative platforms, hardware configurations, or different library versions, and that computational instability analyses were performed only under the WS setting due to limited computational resources.
- The paper states that class imbalance and data scarcity limit underrepresented classes (Result in SciCite; Extends, Motivation, and Future in ACL-ARC), and that annotation subjectivity, ambiguous citation contexts, and the absence of a definitive gold standard remain open challenges.
- Evident from the text: ACL-ARC’s small size produces higher variance, especially in Macro-F1, and the deployed web application uses only the SciCite-trained models because they were considered more reliable than those tuned on ACL-ARC.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 3 Not valid · 10 Not checked · 158 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior works further suggest that contextual cues are particularly effective when placed at the beginning of a sentence, therefore as prefixes (Pimentel et al., 2021).”
What the sources say
- Partly supportedcites Pimentel et al., 2021 sourceExperimental — accuracy not yet measured
The Introduction reports stronger disambiguatory signals at the beginnings of words than at their endings. It studies segments within words, not contextual cues placed at the beginnings of sentences as prefixes, so it does not establish the claim at that level. (LLM, conf=0.85)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“citations are inherently complex and heterogeneous: they reflect a wide range of motivations and rhetorical functions within scholarly writing (Pride, 2022; Tahamtan & Bornmann, 2019)”
What the sources say
- Not checked — no full text of the sourcecites Pride, 2022 source
- Supportedcites Tahamtan & Bornmann, 2019 source
In Section 4 (p. 43), the review describes citing motivation as multidimensional and says scholars cite for a variety of scientific and nonscientific reasons. It also describes many citation functions that cannot all be captured by broad categories; Section 3.1 (p. 18) notes that even a single citation can serve more than one function. These passages support the paraphrase. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“experimental results show that CiteFu- sion achieves state-of-the-art performance, with Macro-F1 scores of 89.60% on SciCite,”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 76.24% on ACL-ARC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For SciCite, the mean improvements across all the aggregation functions and models utilized in the experiments were ΔA = 0.73 and ΔMF1 = 0.93 .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
97 more claims
Create a free account to see all 182 claims and to verify the 3 references still open.
- Probably not a substantial claim. 33
- It may state two separate claims as one. 51
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5932 Scientometrics (2025) 130:5911–5981
“for the Method class, accuracy and F1 scores are 89.60% and 91.65% , respectively; for the Background class, 91.06% and 91.85%; and for the Result class, 91.89% and 85.30%.”
§ 5944 Scientometrics (2025) 130:5911–5981
“C ite Fu si on (W S) 94 .6 7 91 .3 4 95 .5 9 91 .6 5 91 .8 5 85 .3 0”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Discussion
“FFNN 89.56 88.22 76.98 71.46”
§ 5944 Scientometrics (2025) 130:5911–5981
“C ite Fu si on (W oS ) 93 .6 5 89 .8 9 95 .5 9 89 .5 8 91 .0 0 84 .8 6 89 .7 3 89 .7 7 87 .4 0 89 .7 3 89 .6 8 88 .4 8”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5932 Scientometrics (2025) 130:5911–5981
“for the Background class, accuracy and F1 scores are 92.96% and 86.84% , respectively; for the Uses class, 69.23% and 73.47%; for the CompareOrContrast class, 72.00% and 78.26% ; for the Extends class, 80.00% and 88.89% ; for the Motivation class, 42.86% and 50.00%; and for the Future class, 80.00% and 80.00%.”
§ 5944 Scientometrics (2025) 130:5911–5981
“C ite Fu si on (W S) 85 .6 1 86 .8 4 90 .6 5 73 .4 7 92 .8 1 78 .2 6 99 .2 8 88 .8 9 95 .6 8 50 .0 0 98 .5 6 80 .0 0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5932 Scientometrics (2025) 130:5911–5981
“FFNNs consistently outperformed all other aggregators, except for the WoS setting in ACL-ARC, in which Random Forest (RF) performed better.”
§ Discussion
“StackingC 90.10 88.81 82.73 73.91”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5934 Scientometrics (2025) 130:5911–5981
“Mean (Std) 90.53 (0.14) 89.34 (0.18) 80.44 (0.79) 73.71 (1.61)”
§ 5936 Scientometrics (2025) 130:5911–5981
“the mean accuracy and Macro-F1 scores are 80.44 ( � = 0.44 ) and 73.71 ( � = 1.61 ), respectively.”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Frame Analysis: An Essay on the Organization of Experience.
Davis, M. S., & Goffman, E. (1975) · Contemporary Sociology · doi:10.2307/2064021 · Cited by 4,537
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · cited 1×
“Enhanced explainability of model predictions, as emphasized by Pan et al. (2024), is crucial for developing robust and reliable systems suitable for deployment in production environments.”
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Morgan, M. C. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.17575540 · Cited by 1,283
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257
A comprehensive review on ensemble deep learning: Opportunities and challenges
Mohammed, A., & Kora, R. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.01.014 · Cited by 1,127 · cited 3×
“An interesting research direction in classification tasks is related to ensemble strategies (ES), in which a set of weak - or baseline - learners are employed by means of an aggregation function to produce a single output through the different predictions of the base models (Mohammed & Kora, 2023).”
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
Jiang, D., Ren, X., & Lin, B. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.792 · Cited by 957
Research works printed but not linked: 2
- Bertin, M., & Atanassova, I. (2014). A study of lexical distribution in citation contexts through the IMRaD standard. Proceedings of the first workshop on bibliometric-enhanced information retrieval (BIR) Co-Located with the 36th European Conference on Information Retrieval (ECIR 2014), 5–12.
- Seewald, A. K. (2002). How to Make Stacking Better and Faster While Also Taking Care of an Unknown Weakness. Proceedings of the 19th International Conference on Machine Learning, 554–561.
Cited by
Causal Backdoor Adjustment for Citation Intent Classification
Song, Y., Wan, L., Zhang, Z., Wang, D., & Zhuang, F. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2868-3_9
Citation Intent Classification via Exponential Borda Fusion and SciBERT
Barchane, M., Belefqih, S., Benlahmar, E. H., Zahour, O., & Zellou, A. (2026) · Algorithms · doi:10.3390/a19080612
CiteFuncRanker: an LLM-based pairwise ranking framework for multi-functional citation analysis
Wang, Y., Ruan, X., Lyu, D., & Cheng, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05764-1
“More recent transformer-based and ensemble approaches have further advanced classification performance on these benchmarks(Bezerra et al., 2026; Paolini et al., 2025).”
Islam, M. S. (2026) · Research Square · doi:10.21203/rs.3.rs-10795203/v1
Quantifying scholars’ marginal contributions to average-based journal metrics
Aviv-Reuven, S., Bronstein, J., & Rosenfeld, A. (2026) · Scientometrics · doi:10.1007/s11192-026-05811-x
Metadata
- Authors
- Paolini, L., Vahdati, S., Di Iorio, A., Wardenga, R., Heibi, I., & Peroni, S.
- Affiliations
- University of Bologna (IT)
- Center for Scalable Data Analytics and Artificial Intelligence (DE)
- Technische Universität Dresden (DE)
- Corresponding author
- Paolini, Lorenzo
- Venue
- Scientometrics
- Published
- 2025-10-27
- Volume
- 130
- Issue
- 11
- Pages
- 5911-5981
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W4415589270
- Cited by
- 8
- References
- 62
- Citations by year
- 2026: 8
- FWCI
- 5.07
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientometrics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by)
- arXiv (Cornell University)
- RENATE (cc-by)
- Funders
- Alma Mater Studiorum - Università di Bologna
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- citation intent classification, scientific citations, ensemble learning, SciBERT, XLNet, SHAP, meta-classifier, class imbalance, research impact assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses. https://app.cochat.ai/papers/10.1007/s11192-025-05418-8
- The paper
- Paolini, L., Vahdati, S., Di Iorio, A., Wardenga, R., Heibi, I., & Peroni, S. (2025). CiteFusion: an ensemble framework for citation intent classification harnessing dual-model binary couples and SHAP analyses. Scientometrics. https://doi.org/10.1007/s11192-025-05418-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.