Evaluating the Accuracy of scite, a Smart Citation Index
Open accessThe DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index; 7 of the 10 sources its claims cite were checked for support.
Abstract
Objectives: Citations do not always equate endorsement, therefore it is important to understand the context of a citation. Researchers may heavily rely on a paper they cite, they may refute it entirely, or they may mention it only in passing, so an accurate classification of a citation is valuable for researchers and users. While AI solutions have emerged to provide a more nuanced meaning, the accuracy of these tools has yet to be determined. This project seeks to assess the accuracy of scite in assessing the meaning of citations in a sample of publications. Methods: Using a previously established sample of systematic reviews that cited retracted publications, we conducted known item searching in scite, a tool that uses machine learning to categorize the meaning of citations. scite's interpretation of the citation's meaning was recorded, as was our assessment of the citation’s meaning. Citations were classified as mentioning, supporting or contrasting. Recall, precision, and f-measure were calculated to describe the accuracy of scite's assessment in comparison to human assessment. Results: From the original sample of 324 citations, 98 citations were classified in scite. Of these, scite found that 2 were supporting and 96 were mentioning, while we determined that 42 were supporting, 39 were mentioning, and 17 were contrasting. Supporting citations had high precision and low recall, while mentioning citations had high recall and low precision. F-measures ranged between 0.0 and 0.58, representing low classification accuracy. Conclusions: In our sample, the overall accuracy of scite's assessments was low. scite was less able to classify supporting and contrasting citations, and instead labeled them as mentioning. Although there is potential and enthusiasm for AI to make engagement with literature easier and more immediate, the results generated from AI differed significantly from the human interpretation.
Overview
CoChat Summary
TL;DR
This study assessed scite’s accuracy in classifying citation meaning using a sample of systematic reviews that cited retracted pharmacy publications. Overall accuracy was low: scite classified only 98 of 324 citations and labeled most as mentioning, while human reviewers identified many more supporting and contrasting citations.
Key findings
- From 324 references to retracted publications in 286 systematic reviews, 11 were not found in scite, leaving 313 references; 97 citations (31%) were not represented in scite, and 118 (37.7%) were unclassified due to lack of full-text access.
- Of the 98 citations scite classified, scite identified 2 (2%) as supporting, 96 (98%) as mentioning, and 0 (0%) as contrasting; human reviewers classified 42 as supporting, 39 as mentioning, and 17 as contrasting.
- Supporting citations had precision 1.0, recall 0.05, and F-measure 0.096; contrasting citations had precision 0.0, recall 0.0, and F-measure 0.0; mentioning citations had precision 0.41, recall 1.0, and F-measure 0.58.
- The authors conclude that scite’s overall accuracy was low in this sample, with scite less able to classify supporting and contrasting citations and instead labeling them as mentioning.
Methods
- The study used a previously established sample of systematic reviews that cited retracted publications in pharmacy, identified via Retraction Watch and known item searching in Scopus and Web of Science Core Collection.
- Reviewers used a Qualtrics data extraction form to record scite’s citation classification and their own assessment; citations were classified as mentioning, supporting, or contrasting, with disagreements or uncertainty reviewed collectively.
- Data were summarized in R 4.1.3, and scite’s accuracy was assessed by calculating precision, recall, and F-measure against human assessment.
Limitations
- The paper states that the sample was limited to one discipline and a specific study design, focused on systematic reviews citing at least one retracted publication, and was derived from a larger research project rather than selected solely to assess scite; these factors limit generalizability.
- The paper states that retraction is a relatively rare and extreme publication state, so the citations studied are not indicative of most publications.
- Evident from the text: only 98 of 324 citations were classified by scite, and 31% of citations were not represented in scite while 37.7% were unclassified, which the authors describe as a significant barrier to use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 0 Not valid · 1 Not checked · 43 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous research has found that the inclusion of retracted publications in systematic reviews can significantly alter the overall findings of the review. 13”
What the sources say
- Partly supportedcites [13] 13. Garmendia CA, Nassar Gorra L, Rodriguez AL, Trepka MJ, Veledar E, Madhivanan P. Evaluation of the Inclusion of Studies Identified by the FDA as Having Falsified Data in the Results of Meta-analyses: The Example of the Apixaban Trials [published correction appears in JAMA Intern Med. 2021 Mar 1;181(3):409]. JAMA Intern Med. 2019;179(4):582-584. doi:10.1001/jamainternmed.2018.7661 sourceExperimental — accuracy not yet measured
The Results and Discussion report that publications containing falsified data changed the conclusions of 12 of 22 meta-analyses. The paper does not establish that those publications were retracted, so it supports the effect on meta-analysis findings but not the claim's specific description of them as retracted publications. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This is broadly in alignment with scite's overall assessments, as the creators indicate that 'the average distribution of citation statements [is] 92.6% mentioning, 6.5% supporting, and 0.8% contrasting statements.' 5”
What the sources say
- Supportedcites [5] 5. Nicholson JM, Mordaunt M, Lopez P, Uppala A, Rosati D, Rodrigues NP, et al. scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quant Sci Stud. 2021 Nov 5;2(3):882-98. doi:10.1162/qss a 0 0146 source
The “Task modeling and training data” section estimates that 92.6% of citation statements are mentioning, 6.5% supporting, and 0.8% disputing, based on manual annotations. This supports the claim that the vast majority are mentioning; the citing sentence calls the paper’s “disputing” category “contrasting.” (LLM, conf=0.94)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were an additional 40 citations that scite misclassified which we assessed as supporting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 31% of cases, the citation was not represented at all, while in 38% of cases the citation was represented but marked as unclassified due to lack of full-text access.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, while we agreed with scite's assessment of 39 citations as mentioning, we found 57 citations were marked as mentioning when they were in fact contrasting or supporting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
28 more claims
Create a free account to see all 48 claims and every source verdict.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 8
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Cited works that were retracted
Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.
A note, not an issue: for example, a cited work that was retracted only after this paper was published, which its authors could not have known.
In the paper
“7. Brown SJ, Bakker CJ, Theis-Mahon NR. Retracted publications in pharmacy systematic reviews. J Med Libr Assoc. 2022 Feb 11;110(1):47-55. doi:10.5195/jmla.2022.1280”
Cited work 10.5195/jmla.2022.1280 · retraction date not known · this paper published 2023-09-13
Citations
Cites
1 retracted
Retracted publications in pharmacy systematic reviews
Brown, S. J., Bakker, C., & Theis-Mahon, N. R. (2022) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2022.1280 · Cited by 41
Self-citationRetractedR: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Citation Indexing for Studying Science
GARFIELD, E. (1970) · Nature · doi:10.1038/227669a0 · Cited by 391
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
Why and how do journals retract articles? An analysis of Medline retractions 1988–2008
Wager, E., & Williams, P. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2010.040964 · Cited by 263
Citation Analysis and Discourse Analysis Revisited
White, H. D. (2004) · Applied Linguistics · doi:10.1093/applin/25.1.89 · Cited by 260
An Introduction to Machine Learning
Rebala, G., Ravi, A., & Churiwala, S. (2019) · Springer eBooks · doi:10.1007/978-3-030-15729-6 · Cited by 237
Using artificial intelligence methods for systematic review in health sciences: A systematic review
Blaizot, A., Veettil, S. K., Saidoung, P., Moreno‐Garcia, C. F., Wiratunga, N., Aceves‐Martins, M., Lai, N. M., & Chaiyakunapruk, N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1553 · Cited by 223
Moylan, E. C., & Kowalczuk, M. K. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012047 · Cited by 212
Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021) · bioRxiv (published in Quantitative Science Studies) · doi:10.1101/2021.03.15.435418 · Cited by 202
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188
Wallace, B. C., Noel-Storr, A., Marshall, I. J., Cohen, A. M., Smalheiser, N. R., & Thomas, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx053 · Cited by 186
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Reasons for article retraction in anesthesiology: a comprehensive analysis
Nair, S., Yean, C., Yoo, J., Leff, J., Delphin, E., & Adams, D. C. (2020) · Canadian Journal of Anesthesia/Journal canadien d'anesthésie · doi:10.1007/s12630-019-01508-3 · Cited by 90
Retractions in cancer research: a systematic survey
Bozzo, A., Bali, K., Evaniew, N., & Ghert, M. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0031-1 · Cited by 83
A systematic review of retracted publications in emergency medicine
Chauvin, A., De Villelongue, C., Pateron, D., & Yordanov, Y. (2019) · European Journal of Emergency Medicine · doi:10.1097/mej.0000000000000491 · Cited by 47
Zhang, Y., Liang, S., Feng, Y., Wang, Q., Sun, F., Chen, S., Yang, Y., He, X., Zhu, H., & Pan, H. (2022) · Systematic Reviews · doi:10.1186/s13643-021-01881-5 · Cited by 39
Garmendia, C. A., Nassar Gorra, L., Rodriguez, A. L., Trepka, M. J., Veledar, E., & Madhivanan, P. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.7661 · Cited by 35
Brody, S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.1331 · Cited by 24
Research works printed but not linked: 2
- [14]14. Cochrane Library. Managing potentially problematic studies [Internet]. Cochrane Database of Systematic Reviews: editorial policies. [cited 2023 Feb 10]. Available from: https://www.cochranelibrary.com/cdsr/editorial-policiesproblematic-studies
- [16]16. Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. 4.S1 Technical Supplement to Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions Version 63 [Internet]. Chichester, UK: John Wiley Sons, Ltd; 2022 [cited 2023 Feb 10]. Available from: https://training.cochrane.org/handbook/current/chapter-04-technical-supplementsearching-and-selecting-studies
Cited by
Natural language processing with transformers: a review
Țucudean, G., Bucos, M., Drăgulescu, B., & Căleanu, C. D. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2222 · Cited by 71
Bakker, C., Theis-Mahon, N. R., Brown, S. J., & Zeegers, M. P. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02316-z · Cited by 7
Self-citationStevens, E. R., & Laynor, G. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1434427 · Cited by 5
All You Can “E”: E-resources Food For Thought
Flynn, C. M. (2025) · Journal of Electronic Resources Librarianship · doi:10.1080/1941126x.2025.2455799 · Cited by 4
Mazel, N. (2025) · Library Trends · doi:10.1353/lib.2025.a961197 · Cited by 2
Metadata
- Authors
- Bakker, C., Theis-Mahon, N., & Brown, S. J.
- Affiliations
- University of Regina (CA)
- University of Minnesota (US)
- Venue
- Hypothesis: Research Journal for Health Information Professionals
- Publisher
- IUPUI University Library
- Published
- 2023-09-13
- Volume
- 35
- Issue
- 2
- ISSN
- 1093-5665 (ISSN-L)
- 2688-1268
- OpenAlex
- W4386699143
- Cited by
- 10
- References
- 13
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 4 · 2023: 1
- FWCI
- 1.87
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Hypothesis Research Journal for Health Information Professionals (publishedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Keywords
- Scite, citation context, retracted article, systematic reviews, machine learning, AI accuracy, human-AI comparison, literature search tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.