CoChat

Evaluating the Accuracy of scite, a Smart Citation Index

Open access
Bakker, Caitlin, Theis-Mahon, Nicole, Brown, Sarah Jane · 2023 · Hypothesis: Research Journal for Health Information Professionals

The DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index; 7 of the 10 sources its claims cite were checked for support.

Abstract

Objectives: Citations do not always equate endorsement, therefore it is important to understand the context of a citation. Researchers may heavily rely on a paper they cite, they may refute it entirely, or they may mention it only in passing, so an accurate classification of a citation is valuable for researchers and users. While AI solutions have emerged to provide a more nuanced meaning, the accuracy of these tools has yet to be determined. This project seeks to assess the accuracy of scite in assessing the meaning of citations in a sample of publications. Methods: Using a previously established sample of systematic reviews that cited retracted publications, we conducted known item searching in scite, a tool that uses machine learning to categorize the meaning of citations. scite's interpretation of the citation's meaning was recorded, as was our assessment of the citation’s meaning. Citations were classified as mentioning, supporting or contrasting. Recall, precision, and f-measure were calculated to describe the accuracy of scite's assessment in comparison to human assessment. Results: From the original sample of 324 citations, 98 citations were classified in scite. Of these, scite found that 2 were supporting and 96 were mentioning, while we determined that 42 were supporting, 39 were mentioning, and 17 were contrasting. Supporting citations had high precision and low recall, while mentioning citations had high recall and low precision. F-measures ranged between 0.0 and 0.58, representing low classification accuracy. Conclusions: In our sample, the overall accuracy of scite's assessments was low. scite was less able to classify supporting and contrasting citations, and instead labeled them as mentioning. Although there is potential and enthusiasm for AI to make engagement with literature easier and more immediate, the results generated from AI differed significantly from the human interpretation.

Overview

CoChat Summary

TL;DR

This study assessed scite’s accuracy in classifying citation meaning using a sample of systematic reviews that cited retracted pharmacy publications. Overall accuracy was low: scite classified only 98 of 324 citations and labeled most as mentioning, while human reviewers identified many more supporting and contrasting citations.

Key findings

  • From 324 references to retracted publications in 286 systematic reviews, 11 were not found in scite, leaving 313 references; 97 citations (31%) were not represented in scite, and 118 (37.7%) were unclassified due to lack of full-text access.
  • Of the 98 citations scite classified, scite identified 2 (2%) as supporting, 96 (98%) as mentioning, and 0 (0%) as contrasting; human reviewers classified 42 as supporting, 39 as mentioning, and 17 as contrasting.
  • Supporting citations had precision 1.0, recall 0.05, and F-measure 0.096; contrasting citations had precision 0.0, recall 0.0, and F-measure 0.0; mentioning citations had precision 0.41, recall 1.0, and F-measure 0.58.
  • The authors conclude that scite’s overall accuracy was low in this sample, with scite less able to classify supporting and contrasting citations and instead labeling them as mentioning.

Methods

  • The study used a previously established sample of systematic reviews that cited retracted publications in pharmacy, identified via Retraction Watch and known item searching in Scopus and Web of Science Core Collection.
  • Reviewers used a Qualtrics data extraction form to record scite’s citation classification and their own assessment; citations were classified as mentioning, supporting, or contrasting, with disagreements or uncertainty reviewed collectively.
  • Data were summarized in R 4.1.3, and scite’s accuracy was assessed by calculating precision, recall, and F-measure against human assessment.

Limitations

  • The paper states that the sample was limited to one discipline and a specific study design, focused on systematic reviews citing at least one retracted publication, and was derived from a larger research project rather than selected solely to assess scite; these factors limit generalizability.
  • The paper states that retraction is a relatively rare and extreme publication state, so the citations studied are not indicative of most publications.
  • Evident from the text: only 98 of 324 citations were classified by scite, and 31% of citations were not represented in scite while 37.7% were unclassified, which the authors describe as a significant barrier to use.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18060/26528

Cites

21 of 22 references verified1 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 0 Not valid · 1 Not checked · 43 Own results and interpretations

28 more claims

Create a free account to see all 48 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 8
  • It may state two separate claims as one. 8

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

1 of 1

Citations

Cites

1 retracted

Research works printed but not linked: 2

  1. [14]14. Cochrane Library. Managing potentially problematic studies [Internet]. Cochrane Database of Systematic Reviews: editorial policies. [cited 2023 Feb 10]. Available from: https://www.cochranelibrary.com/cdsr/editorial-policiesproblematic-studies
  2. [16]16. Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. 4.S1 Technical Supplement to Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions Version 63 [Internet]. Chichester, UK: John Wiley Sons, Ltd; 2022 [cited 2023 Feb 10]. Available from: https://training.cochrane.org/handbook/current/chapter-04-technical-supplementsearching-and-selecting-studies

Cited by

See all 10

Metadata

Authors
Bakker, C., Theis-Mahon, N., & Brown, S. J.
Affiliations
  • University of Regina (CA)
  • University of Minnesota (US)
Venue
Hypothesis: Research Journal for Health Information Professionals
Published
2023-09-13
Volume
35
Issue
2
ISSN
  • 1093-5665 (ISSN-L)
  • 2688-1268
OpenAlex
W4386699143
Cited by
10
References
13
Citations by year
2026: 1 · 2025: 4 · 2024: 4 · 2023: 1
FWCI
1.87
Type
journal article
Language
en
Open access
Locations
Hypothesis Research Journal for Health Information Professionals (publishedVersion, cc-by-nc-nd)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
Keywords
Scite, citation context, retracted article, systematic reviews, machine learning, AI accuracy, human-AI comparison, literature search tools

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.