CoChat

Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level

Open access

Published version of a preprint

The DOI matches its Crossref record; 87 of the 91 references it cites were found in a registry or an index; 1 of the 30 sources its claims cite were checked for support.

Abstract

Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.

Verification finished with gaps

Some checks didn’t finish. The results below cover the steps that did.

  1. Registry recordVerified
  2. Full textRead the full text
  3. Reference list87 references found
  4. Reference check87 of 87 references checked
  5. StatisticsRecomputed
  6. Claims86 claims anchored to a quote
  7. Cited sourcesChecked
  8. Self-checkDidn’t finish — the rest of the report still applies.
  9. VerificationFindings verified

Overview

CoChat Summary

## TL;DR

The paper introduces Relative Citation Ratio (RCR), an article-level, field-normalized citation metric built from co-citation networks and benchmarked to a peer group. The authors report that RCR correlates strongly with expert judgments, behaves better than several existing bibliometrics in their tests, and is available through the iCite web tool.

## Key findings

  • Co-citation networks were used to define an article’s field; in a validation study, diagnostic words were more likely to be shared between an article and papers in its co-citation network than between that article and papers in the same journal, based on 1,397 articles and 249,981 pairwise comparisons within co-citation networks versus 28,516,576 comparisons with same-journal articles.
  • In a benchmark set of 35,837 papers published in 2009 by NIH R01 recipients, FCRs based on co-citation networks had lower variance than FCRs based on cited or citing networks, and lower variance than ACRs or JIFs (p < 10−4 for the variance comparisons reported).
  • RCR correlated well with subject matter expert ratings across three validation datasets: 2,193 Faculty of 1000-rated articles, 430 STPI-rated articles, and 290 NIH-rated articles.
  • In an NIH case study of 88,835 articles from 3,089 continuously funded R01 principal investigators, 53% of investigators in the top RCR quintile stayed there across two 4-year periods, and 53% of those in the bottom quintile stayed at the bottom; among more than 200,000 articles analyzed for FCR stability, only 0.2% had an RCR drop of 0.1 or more between 2012 and 2014, and less than 2% dropped at all.

## Methods

  • The metric was built from an article’s co-citation network: papers citing the reference article were used to assemble the set of co-cited papers, whose journal citation rates were averaged to form the field citation rate (FCR); the article’s article citation rate (ACR) was then normalized by FCR and benchmarked to a peer group to yield RCR.
  • Citation analyses used the Thomson Reuters Web of Science dataset from 2002–2012 (extended to 2014 for FCR stability analyses), focusing on journals in which NIH R01-funded researchers published; grant and PI data came from NIH RePORTER, SPIRES, and IMPAC-II.
  • Validation included cosine-similarity content analysis with IN-SPIRE on 1,397 articles from six journals, expert-comparison analyses, and regression/variance tests; RCR benchmarking used OLS for large-scale analyses and quantile regression as an alternative for smaller, skewed settings.

## Limitations

  • The paper states that RCR cannot be calculated immediately after publication; enough citations must accrue first, and the authors note that 93% of co-citation-network-based FCRs stabilize after a paper has been cited five times.
  • The authors explicitly say they have not determined whether small differences in RCR, such as 1.0 versus 1.2, reliably reflect meaningful differences in influence.
  • The paper also states that citation metrics cannot fully capture article value, are not a substitute for reading the paper, and RCR is not designed as a measure of long-term impact or as a metric for applied research; these are stated limitations of scope.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pbio.1002541

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig 1 Properties of co-citation networks. (A) Schematic of a co-citation network. The reference article (RA) (red, middle row) cites previous papers from the literature (orange, bottom row); subsequent papers cite the RA (blue, top row). The co-citation network is the set of papers that appear alongside the article in the subsequent citing papers (green, middle row). The field citation rate is calculated as the mean of the latter articles’ journal citation rates. (B) Growth of co-citation networks over time. Three RAs published in 2006 (red dots) were cited 5 (top row), 9 (middle row), or 31 times (bottom row) by 2011. Three intervals were chosen to illustrate the growth of the corresponding co-citation networks: 2006–2007, 2006–2009, and 2006–2011 (the first, second, and third columns, respectively). Each article in one of the three co-citation networks is shown as a separate green dot; the edges (connections between dots) indicates their presence together in the same reference list. (C) Cluster algorithm-based content analysis of the 215 papers in the co-citation network of a sample RA (panel B, bottom network series) identified a changing pattern of relevance to different subdisciplines over time. This RA described the identification of new peptides of possible clinical utility due to their similarity to known conotoxins. Papers in the co-citation network of this RA focused on (1) α-conotoxin mechanisms of action, (2) structure and evolution of conotoxins, (3) cyclotide biochemistry, (4) conotoxin phylogenetics, and (5) identification and synthesis of lantibiotics. (D) Growth of an article’s co-citation network is proportional to the number of times it has been cited. Each point is the average network size of 1,000 randomly chosen papers with between 1 and 100 citations (error bars represent the standard error of the mean). Each paper is only counted once, even if it is co-cited with the article of interest multiple times. An average of 17.8 new papers is added to the co-citation network for each additional citation. This suggests substantial duplication of articles within a co-citation network, since on average 32.4 papers (median of 30) are referenced in each citing article.
Fig 1 Properties of co-citation networks. (A) Schematic of a co-citation network. The reference article (RA) (red, middle row) cites previous papers from the literature (orange, bottom row); subsequent papers cite the RA (blue, top row). The co-citation network is the set of papers that appear alongside the article in the subsequent citing papers (green, middle row). The field citation rate is calculated as the mean of the latter articles’ journal citation rates. (B) Growth of co-citation networks over time. Three RAs published in 2006 (red dots) were cited 5 (top row), 9 (middle row), or 31 times (bottom row) by 2011. Three intervals were chosen to illustrate the growth of the corresponding co-citation networks: 2006–2007, 2006–2009, and 2006–2011 (the first, second, and third columns, respectively). Each article in one of the three co-citation networks is shown as a separate green dot; the edges (connections between dots) indicates their presence together in the same reference list. (C) Cluster algorithm-based content analysis of the 215 papers in the co-citation network of a sample RA (panel B, bottom network series) identified a changing pattern of relevance to different subdisciplines over time. This RA described the identification of new peptides of possible clinical utility due to their similarity to known conotoxins. Papers in the co-citation network of this RA focused on (1) α-conotoxin mechanisms of action, (2) structure and evolution of conotoxins, (3) cyclotide biochemistry, (4) conotoxin phylogenetics, and (5) identification and synthesis of lantibiotics. (D) Growth of an article’s co-citation network is proportional to the number of times it has been cited. Each point is the average network size of 1,000 randomly chosen papers with between 1 and 100 citations (error bars represent the standard error of the mean). Each paper is only counted once, even if it is co-cited with the article of interest multiple times. An average of 17.8 new papers is added to the co-citation network for each additional citation. This suggests substantial duplication of articles within a co-citation network, since on average 32.4 papers (median of 30) are referenced in each citing article.CC0 · PubMed Central
Fig 2 Text similarity of articles is defined more accurately by their co-citation networks than by the journals in which they appear. (A, B) The text in each of 1,397 RAs was compared, either with the text in each corresponding co-citation network or separately with the collection of articles appearing in the same journal. Both primary and review articles are included. Cosine similarity scores were then calculated using either the top 100 terms (A) or all terms appearing in at least ten documents (B). Filled circles in green, co-citation network comparison; filled circles in shades of red, multidisciplinary journal comparison; filled circles in shades of blue, disciplinary journal comparison. Curves shifted to the right show more text similarity: RAs are least similar to papers in the same multidisciplinary journals, more similar to papers in the same disciplinary journal, and most similar to papers in their co-citation network.
Fig 2 Text similarity of articles is defined more accurately by their co-citation networks than by the journals in which they appear. (A, B) The text in each of 1,397 RAs was compared, either with the text in each corresponding co-citation network or separately with the collection of articles appearing in the same journal. Both primary and review articles are included. Cosine similarity scores were then calculated using either the top 100 terms (A) or all terms appearing in at least ten documents (B). Filled circles in green, co-citation network comparison; filled circles in shades of red, multidisciplinary journal comparison; filled circles in shades of blue, disciplinary journal comparison. Curves shifted to the right show more text similarity: RAs are least similar to papers in the same multidisciplinary journals, more similar to papers in the same disciplinary journal, and most similar to papers in their co-citation network.CC0 · PubMed Central
Fig 3 Algorithm for calculating the Relative Citation Ratio (RCR). (A) Article citation rate (ACR) is calculated as the total citations divided by the number of years excluding the calendar year of publication (Supporting Equation S1 in S1 Text), when few, if any, citations accrue (S2 Fig). Numbers in the bars correspond to the number of citations in that year. (B) Box-and-whisker plots of 88,835 NIH-funded papers (published between 2003 and 2010), summarizing their ACR, journal impact factor (matched to the article’s year of publication), and field citation rate (FCR). Boxes show the 25th–75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles. (C) Correlation of FCR as generated in 2012 versus 2 y later in 2014 for the same set of articles, as a function of the number of starting citations in 2012. Data were sliced by the number of initial citations in 2012, to assess stability as a function of the number of citing articles (and thereby the starting size of the network). Each point, correlation coefficient for >1,000 articles. Between 2012 and 2014, articles accrued a median of 5 additional citations. The inclusion of the full span of years ensures a representative spread of ACRs at each value of the independent axis. Furthermore, since papers in this analysis receive a nearly identical number of citations in their ninth year as in their first full year after publication (S2 Fig), the FCRs of articles published later in the chosen time frame (2003 to 2010) do not undergo substantially more change than those published earlier. (D) Generate an expectation for ACRs based on a preselected benchmark group, by regressing the ACR of the benchmark papers onto their FCRs (Supporting Equations S3, S4 in S1 Text), one regression each publication year. The graphed examples were sampled from a random distribution for illustrative purposes. (E) The coefficients from each year’s regression equation transforms the FCRs of papers published in the same year into expected citation rates (ECRs) (Supporting Equation S5 in S1 Text). Each paper’s RCR is its ACR/ECR ratio. A portfolio’s RCR is simply the average of the individual articles’ RCRs (Supporting Equation S6 in S1 Text).
Fig 3 Algorithm for calculating the Relative Citation Ratio (RCR). (A) Article citation rate (ACR) is calculated as the total citations divided by the number of years excluding the calendar year of publication (Supporting Equation S1 in S1 Text), when few, if any, citations accrue (S2 Fig). Numbers in the bars correspond to the number of citations in that year. (B) Box-and-whisker plots of 88,835 NIH-funded papers (published between 2003 and 2010), summarizing their ACR, journal impact factor (matched to the article’s year of publication), and field citation rate (FCR). Boxes show the 25th–75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles. (C) Correlation of FCR as generated in 2012 versus 2 y later in 2014 for the same set of articles, as a function of the number of starting citations in 2012. Data were sliced by the number of initial citations in 2012, to assess stability as a function of the number of citing articles (and thereby the starting size of the network). Each point, correlation coefficient for >1,000 articles. Between 2012 and 2014, articles accrued a median of 5 additional citations. The inclusion of the full span of years ensures a representative spread of ACRs at each value of the independent axis. Furthermore, since papers in this analysis receive a nearly identical number of citations in their ninth year as in their first full year after publication (S2 Fig), the FCRs of articles published later in the chosen time frame (2003 to 2010) do not undergo substantially more change than those published earlier. (D) Generate an expectation for ACRs based on a preselected benchmark group, by regressing the ACR of the benchmark papers onto their FCRs (Supporting Equations S3, S4 in S1 Text), one regression each publication year. The graphed examples were sampled from a random distribution for illustrative purposes. (E) The coefficients from each year’s regression equation transforms the FCRs of papers published in the same year into expected citation rates (ECRs) (Supporting Equation S5 in S1 Text). Each paper’s RCR is its ACR/ECR ratio. A portfolio’s RCR is simply the average of the individual articles’ RCRs (Supporting Equation S6 in S1 Text).CC0 · PubMed Central
Fig 4 RCRs correspond with expert reviewer scores. (A–C) Bubble plots of reviewer scores versus RCR for three different datasets. Articles are binned by reviewer score; bubble area is proportionate to the number of articles in that bin. (A) F1000 scores for 2,193 R01-funded papers published in 2009. Faculty reviewers rated the articles on a scale of one to three (“good,” “very good,” and “exceptional”, respectively); those scores were summed into a composite F1000 score for each article (S3 Fig). (B) Reviewer scores of 430 HHMI and NIH-funded papers collected by STPI. (C) Scores of 290 R01-funded articles reviewed by experts from the NIH Intramural Research Program. Black line, linear regression. (D) Box-and-whisker plots illustrating the distribution of journal impact factors (JIFs) citations per year (CPY) and RCRs for two areas of NIH-funded research from 2007–2011. Cell biology, n = 5,936; neurological function, n = 5,417. *** p < 0.001, Kruskal-Wallis with Dunn’s multiple comparison test. n.s., not significant. Mean represented by a “+.” (E) Comparison of RCRs (orange) and Thompson Reuters ratios (blue) [17,28] for the same 544 articles with a low denominator. Data points are partially transparent to allow coordinates with multiple papers (darker) to be more clearly identified.
Fig 4 RCRs correspond with expert reviewer scores. (A–C) Bubble plots of reviewer scores versus RCR for three different datasets. Articles are binned by reviewer score; bubble area is proportionate to the number of articles in that bin. (A) F1000 scores for 2,193 R01-funded papers published in 2009. Faculty reviewers rated the articles on a scale of one to three (“good,” “very good,” and “exceptional”, respectively); those scores were summed into a composite F1000 score for each article (S3 Fig). (B) Reviewer scores of 430 HHMI and NIH-funded papers collected by STPI. (C) Scores of 290 R01-funded articles reviewed by experts from the NIH Intramural Research Program. Black line, linear regression. (D) Box-and-whisker plots illustrating the distribution of journal impact factors (JIFs) citations per year (CPY) and RCRs for two areas of NIH-funded research from 2007–2011. Cell biology, n = 5,936; neurological function, n = 5,417. *** p < 0.001, Kruskal-Wallis with Dunn’s multiple comparison test. n.s., not significant. Mean represented by a “+.” (E) Comparison of RCRs (orange) and Thompson Reuters ratios (blue) [17,28] for the same 544 articles with a low denominator. Data points are partially transparent to allow coordinates with multiple papers (darker) to be more clearly identified.CC0 · PubMed Central
Fig 5 iCite, a publicly available tool for calculating RCR and accessing related citation information. (A) Screenshot of a sample iCite result. Four hundred sample PMIDs from papers published over a 4-y window were entered into the iCite tool, which returned the maximum, mean +/− standard error of the mean (SEM), and median values for both CPY and RCR; weighted RCR is equal to the sum of the RCRs for this group. The box-and-whisker plot shows the distribution of article RCRs; bar graphs show the number of publications per year and weighted RCR per year, respectively. (B) Sample data download for an iCite result. iCite returns the total number of citations, number of CPY, expected CPY based on an NIH benchmark, FCR, Relative Citation Ratio, and percentile ranking in a downloadable Excel format for each PMID entered, as well as the corresponding title, author information, and year/journal of publication.
Fig 5 iCite, a publicly available tool for calculating RCR and accessing related citation information. (A) Screenshot of a sample iCite result. Four hundred sample PMIDs from papers published over a 4-y window were entered into the iCite tool, which returned the maximum, mean +/− standard error of the mean (SEM), and median values for both CPY and RCR; weighted RCR is equal to the sum of the RCRs for this group. The box-and-whisker plot shows the distribution of article RCRs; bar graphs show the number of publications per year and weighted RCR per year, respectively. (B) Sample data download for an iCite result. iCite returns the total number of citations, number of CPY, expected CPY based on an NIH benchmark, FCR, Relative Citation Ratio, and percentile ranking in a downloadable Excel format for each PMID entered, as well as the corresponding title, author information, and year/journal of publication.CC0 · PubMed Central
Fig 6 RCR-based evaluation of two NIH-funded research programs. (A) Bar graph showing the percentage of papers that experience a drop in RCR from 2012 to 2014. Black bars, decrease in RCR; grey bars, decrease in RCR of 0.1 or more. (B) Box-and-whisker plots showing the distribution of RCR values for articles describing the human microbiome, published with support from the Human Microbiome Project (HMP) of the NIH Common Fund or another source (other). Boxes show the 25th–75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles.
Fig 6 RCR-based evaluation of two NIH-funded research programs. (A) Bar graph showing the percentage of papers that experience a drop in RCR from 2012 to 2014. Black bars, decrease in RCR; grey bars, decrease in RCR of 0.1 or more. (B) Box-and-whisker plots showing the distribution of RCR values for articles describing the human microbiome, published with support from the Human Microbiome Project (HMP) of the NIH Common Fund or another source (other). Boxes show the 25th–75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles.CC0 · PubMed Central
Fig 7 Properties of RCRs at the article and investigator level. (A, B) Frequency distribution of article-level RCRs (A) and JIFs (B), from 88,835 papers (authored by 3,089 R01-funded principal investigators [PIs]) for which co-citation networks were generated. Article RCRs are well fit by a log-normal distribution (R2 = 0.99), and JIFs less so (R2 = 0.79). (C) Box-and-whisker plots summarizing JIFs for the same papers, binned by impact factor quintile (line, median; box, 25th–75th percentiles; whiskers, 10th–90th percentiles). (D) RCR for the same papers using the same bins by JIF quintile (same scale as C). Although the median RCR for each bin generally corresponds to the impact factor quintile, there is a wide range of article RCRs in each category. (E) Box-and-whisker plots summarizing RCRs of these same papers published in selected journals. In each journal, there are papers with article RCRs surpassing the median RCR of the highest impact factor journals (left three). The impact factor of each journal is shown above. (F, G) Frequency distribution of investigator-level RCRs (F) and JIFs (G), representing the mean values for papers authored by each of 3,089 R01-funded PIs. Dashed line in (F), mode of RCR for PIs.
Fig 7 Properties of RCRs at the article and investigator level. (A, B) Frequency distribution of article-level RCRs (A) and JIFs (B), from 88,835 papers (authored by 3,089 R01-funded principal investigators [PIs]) for which co-citation networks were generated. Article RCRs are well fit by a log-normal distribution (R2 = 0.99), and JIFs less so (R2 = 0.79). (C) Box-and-whisker plots summarizing JIFs for the same papers, binned by impact factor quintile (line, median; box, 25th–75th percentiles; whiskers, 10th–90th percentiles). (D) RCR for the same papers using the same bins by JIF quintile (same scale as C). Although the median RCR for each bin generally corresponds to the impact factor quintile, there is a wide range of article RCRs in each category. (E) Box-and-whisker plots summarizing RCRs of these same papers published in selected journals. In each journal, there are papers with article RCRs surpassing the median RCR of the highest impact factor journals (left three). The impact factor of each journal is shown above. (F, G) Frequency distribution of investigator-level RCRs (F) and JIFs (G), representing the mean values for papers authored by each of 3,089 R01-funded PIs. Dashed line in (F), mode of RCR for PIs.CC0 · PubMed Central
Fig 8 Scientific mobility of investigators’ influence relative to their field. Color intensity is proportional to the percentage of PIs in each quintile. (A) The 3,089 investigators who were continuously funded by at least one R01 were ranked by their articles’ average RCR in each time window and split into quintiles. From left to right, investigators starting in different quintiles were tracked to see their rank in the next 4-y period. (B) This panel shows the same analysis, but the number of published articles was multiplied by their average RCR to calculate an influence-weighted article count. PIs were ranked by this aggregate score and split into quintiles. (C) Scatter plot illustrating the relationship between PI RCR at earlier and later time frames. Black points, actual RCR values; black line, linear regression of actual RCR values. Red points, random assignment model (PI RCRs for the second 4-y period are reshuffled and randomly assigned); red line, linear regression of modeled data.
Fig 8 Scientific mobility of investigators’ influence relative to their field. Color intensity is proportional to the percentage of PIs in each quintile. (A) The 3,089 investigators who were continuously funded by at least one R01 were ranked by their articles’ average RCR in each time window and split into quintiles. From left to right, investigators starting in different quintiles were tracked to see their rank in the next 4-y period. (B) This panel shows the same analysis, but the number of published articles was multiplied by their average RCR to calculate an influence-weighted article count. PIs were ranked by this aggregate score and split into quintiles. (C) Scatter plot illustrating the relationship between PI RCR at earlier and later time frames. Black points, actual RCR values; black line, linear regression of actual RCR values. Red points, random assignment model (PI RCRs for the second 4-y period are reshuffled and randomly assigned); red line, linear regression of modeled data.CC0 · PubMed Central

Verification

Claims the paper makes

1 Valid · 1 Not valid · 18 Not checked · 66 Own results and interpretations

50 more claims

Create a free account to see all 86 claims and to verify the 29 references still open.

See all claims — free account
  • Probably not a substantial claim. 18
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

This check didn’t finish.

Citations

Cites

Show all 87 references

Cited by

Metadata

Authors
Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M.
Affiliations
National Institutes of Health (US)
Corresponding author
  • Hutchins, B. Ian
  • Yuan, Xin
  • Anderson, James M.
  • Santangelo, George M.
Venue
PLoS Biology
Published
2016-09-06
Volume
14
Issue
9
Pages
e1002541
ISSN
  • 1544-9173 (ISSN-L)
  • 1545-7885
OpenAlex
W2953304564
Cited by
500
References
91
Citations by year
2026: 43 · 2025: 51 · 2024: 52 · 2023: 62 · 2022: 68 · 2021: 48 · 2020: 44 · 2019: 45 · 2018: 36 · 2017: 32 · 2016: 17
FWCI
40.48
Type
journal article
Language
en
Open access
Locations
  • PLoS Biology (publishedVersion, public-domain)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, public-domain)
  • PubMed Central (publishedVersion, public-domain)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (publishedVersion)
  • Figshare (submittedVersion, cc-by)
Funders
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico (573871/2008-6)
  • Fundação de Amparo à Pesquisa do Estado de Minas Gerais (573871/2008-6)
  • National Institutes of Health (Z99 OD999999)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
relative citation ratio, citation analysis, co-citation networks, field normalization, iCite
MeSH terms
Authorship, Humans, National Institutes of Health (U.S.), United States, Bibliometrics, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.