Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level
Open accessPublished version of a preprint
The DOI matches its Crossref record; 87 of the 91 references it cites were found in a registry or an index; 1 of the 30 sources its claims cite were checked for support.
Abstract
Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.
Verification finished with gaps
Some checks didn’t finish. The results below cover the steps that did.
- Registry recordVerified
- Full textRead the full text
- Reference list87 references found
- Reference check87 of 87 references checked
- StatisticsRecomputed
- Claims86 claims anchored to a quote
- Cited sourcesChecked
- Self-checkDidn’t finish — the rest of the report still applies.
- VerificationFindings verified
Overview
CoChat Summary
## TL;DR
The paper introduces Relative Citation Ratio (RCR), an article-level, field-normalized citation metric built from co-citation networks and benchmarked to a peer group. The authors report that RCR correlates strongly with expert judgments, behaves better than several existing bibliometrics in their tests, and is available through the iCite web tool.
## Key findings
- Co-citation networks were used to define an article’s field; in a validation study, diagnostic words were more likely to be shared between an article and papers in its co-citation network than between that article and papers in the same journal, based on 1,397 articles and 249,981 pairwise comparisons within co-citation networks versus 28,516,576 comparisons with same-journal articles.
- In a benchmark set of 35,837 papers published in 2009 by NIH R01 recipients, FCRs based on co-citation networks had lower variance than FCRs based on cited or citing networks, and lower variance than ACRs or JIFs (p < 10−4 for the variance comparisons reported).
- RCR correlated well with subject matter expert ratings across three validation datasets: 2,193 Faculty of 1000-rated articles, 430 STPI-rated articles, and 290 NIH-rated articles.
- In an NIH case study of 88,835 articles from 3,089 continuously funded R01 principal investigators, 53% of investigators in the top RCR quintile stayed there across two 4-year periods, and 53% of those in the bottom quintile stayed at the bottom; among more than 200,000 articles analyzed for FCR stability, only 0.2% had an RCR drop of 0.1 or more between 2012 and 2014, and less than 2% dropped at all.
## Methods
- The metric was built from an article’s co-citation network: papers citing the reference article were used to assemble the set of co-cited papers, whose journal citation rates were averaged to form the field citation rate (FCR); the article’s article citation rate (ACR) was then normalized by FCR and benchmarked to a peer group to yield RCR.
- Citation analyses used the Thomson Reuters Web of Science dataset from 2002–2012 (extended to 2014 for FCR stability analyses), focusing on journals in which NIH R01-funded researchers published; grant and PI data came from NIH RePORTER, SPIRES, and IMPAC-II.
- Validation included cosine-similarity content analysis with IN-SPIRE on 1,397 articles from six journals, expert-comparison analyses, and regression/variance tests; RCR benchmarking used OLS for large-scale analyses and quantile regression as an alternative for smaller, skewed settings.
## Limitations
- The paper states that RCR cannot be calculated immediately after publication; enough citations must accrue first, and the authors note that 93% of co-citation-network-based FCRs stabilize after a paper has been cited five times.
- The authors explicitly say they have not determined whether small differences in RCR, such as 1.0 versus 1.2, reliably reflect meaningful differences in influence.
- The paper also states that citation metrics cannot fully capture article value, are not a substitute for reading the paper, and RCR is not designed as a measure of long-term impact or as a metric for applied research; these are stated limitations of scope.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig 4 RCRs correspond with expert reviewer scores. (A–C) Bubble plots of reviewer scores versus RCR for three different datasets. Articles are binned by reviewer score; bubble area is proportionate to the number of articles in that bin. (A) F1000 scores for 2,193 R01-funded papers published in 2009. Faculty reviewers rated the articles on a scale of one to three (“good,” “very good,” and “exceptional”, respectively); those scores were summed into a composite F1000 score for each article (S3 Fig). (B) Reviewer scores of 430 HHMI and NIH-funded papers collected by STPI. (C) Scores of 290 R01-funded articles reviewed by experts from the NIH Intramural Research Program. Black line, linear regression. (D) Box-and-whisker plots illustrating the distribution of journal impact factors (JIFs) citations per year (CPY) and RCRs for two areas of NIH-funded research from 2007–2011. Cell biology, n = 5,936; neurological function, n = 5,417. *** p < 0.001, Kruskal-Wallis with Dunn’s multiple comparison test. n.s., not significant. Mean represented by a “+.” (E) Comparison of RCRs (orange) and Thompson Reuters ratios (blue) [17,28] for the same 544 articles with a low denominator. Data points are partially transparent to allow coordinates with multiple papers (darker) to be more clearly identified.](https://figures.cochat.ai/paper-figures/e792b94297fa4418b200b0579a2ea0b9/relative-citation-ratio-rcr-a-new-metric-that-uses-citation-rates-to-measure-figure-4.jpg)


![Fig 7 Properties of RCRs at the article and investigator level. (A, B) Frequency distribution of article-level RCRs (A) and JIFs (B), from 88,835 papers (authored by 3,089 R01-funded principal investigators [PIs]) for which co-citation networks were generated. Article RCRs are well fit by a log-normal distribution (R2 = 0.99), and JIFs less so (R2 = 0.79). (C) Box-and-whisker plots summarizing JIFs for the same papers, binned by impact factor quintile (line, median; box, 25th–75th percentiles; whiskers, 10th–90th percentiles). (D) RCR for the same papers using the same bins by JIF quintile (same scale as C). Although the median RCR for each bin generally corresponds to the impact factor quintile, there is a wide range of article RCRs in each category. (E) Box-and-whisker plots summarizing RCRs of these same papers published in selected journals. In each journal, there are papers with article RCRs surpassing the median RCR of the highest impact factor journals (left three). The impact factor of each journal is shown above. (F, G) Frequency distribution of investigator-level RCRs (F) and JIFs (G), representing the mean values for papers authored by each of 3,089 R01-funded PIs. Dashed line in (F), mode of RCR for PIs.](https://figures.cochat.ai/paper-figures/7111239ea73541aeace7efe3f230790e/relative-citation-ratio-rcr-a-new-metric-that-uses-citation-rates-to-measure-figure-7.jpg)

Verification
Claims the paper makes
1 Valid · 1 Not valid · 18 Not checked · 66 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Strikingly, this analysis showed that diagnostic words are much more likely to be shared between an article and the papers in its co-citation network than between that same article and the papers that appear alongside it in its journal of publication (Fig 2).”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Co-citation Networks Represent an Article’s Area of Influence
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Abstract
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though journals with the highest impact factors have the highest median RCR”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Quantifying How Past Influence Predicts Future Performance We next undertook a large case study of all 88,835 articles published by NIH investigators who maintained continuous R01 funding from fiscal year (FY) 2003 through FY2010 to ask how the RCR of publications from individual investigators changed over this 8-y interval. Each of these investigators had succeeded at least once in renewing one or more of their projects through the NIH competitive peer review process. In aggregate, the RCR values for these articles are well matched to a log-normal distribution (Fig 7); in contrast, as noted previously by others, the distribution of impact factors of the journals in which they were published is non-normal (Fig 7A and 7B) [53,54]. Sorting into quintiles based on JIF demonstrates that, though journals with the highest impact factors have the highest median RCR, influential publications can be found in virtually all journals (Fig 7C and 7D); these data call into question the assertion that journal of publication is the strongest quality signal for researchers who are young or otherwise unknown to a field [55]. Focusing on a dozen representative journals with a wide range of JIFs further substantiates the finding that influential science appears in many venues and reveals noteworthy departures from the correlation between JIF and median RCR (see S1 Text). For example, NIH-funded articles in both Organic Letters (JIF = 4.7) and the Journal of the Acoustical Society of America (JIF = 1.6) have a higher median RCR than those in Nucleic Acids Research (JIF = 7.1; Fig 7E). As part of this case study, we also calculated the average RCR and average JIF for papers published by each of the 3,089 NIH R01 principal investigators (PIs) represented in the dataset of 88,835 articles. In aggregate, the average RCR and JIF values for NIH R01 PIs exhibited log-normal distributions (Fig 7F and 7G) with substantially different hierarchical ordering (S8 Fig). This raised a further question concerning PIs with RCR values near the mode of the log-normal distribution (dashed line in Fig 7F): as measured by the ability to publish work that influences their respective fields, to what extent does their performance fluctuate? We addressed this question by dividing the 8-y window (FY2003 through FY2010) in half. Average RCRs in the first time period (FY2003 through FY2006) were sorted into quintiles, and the percentage of PIs in the second time period (FY2007 through FY2010) that remained in the same quintile, or moved to a higher or lower quintile, was calculated (Fig 8). The position of PIs in these quintiles appeared to be relatively immobile; 53% of PIs in the top quintile remained at the top, and 53% of those in the bottom quintile remained at the bottom (Fig 8A). For each PI, we also calculated a weighted RCR (the number of articles multiplied by their average RCR); comparing on this basis yielded almost identical results (Fig 8B). It is worth noting that average FCRs for investigators were extremely stable from one 4-y period to the next (Pearson r = 0.92, Table 2), Since FCRs are the quantitative representation of co-citation networks, this further suggests that each co-citation network is successfully capturing the corresponding investigator’s field of research. Another possible interpretation of the above data is that PI RCRs perform an unbiased random walk from their initial state with a large diffusion rate. Considered from this frame, it could be said that 47% of PIs who started in the top quintile moved out of it during the second 4-y period we analyzed. To test this hypothesis directly, we performed a mean reversion test, which determines whether or not the set of values under consideration will return to an average, or mean, value over time. If drift in PI RCR were simply a random walk, then the change in RCR should by definition be independent of starting RCR, and plotting these two values against each other should result in a straight line with a slope of zero. However, the results show that change in RCR is dependent on starting RCR value (p < 0.001, linear regression analysis, n = 3,089, Fig 8C). Furthermore, randomly shuffling PI RCRs from the second 4-y period gives a slope that is significantly different than that observed for the real data (p < 0.001, extra sum-of-squares F-test, n = 3,089, Fig 8C), ruling out the possibility that these values are randomly sampled from the same distribution in each time interval.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The RCR metric is ranking invariant when the same number of uncited articles is added to two groups of equal size (Supporting Equations S7–S9 in S1 Text).”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Expert Validation of RCR as a Measure of Influence For the work presented here, we chose as a benchmark the full set of 311,497 RAs published from 2002 through 2012 by NIH R01 awardees. To measure the degree of correspondence between our method and expert opinion, we compared RCRs generated by OLS benchmarking of ACR/FCR values with three independent sets of postpublication evaluations by subject matter experts (details in S1 Text). We compared RCR with expert rankings (Fig 4) for 2,193 articles published in 2009 and evaluated by Faculty of 1000 members (Fig 4A and S3 Fig), as well as rankings of 430 Howard Hughes Medical Institute- or NIH-funded articles published between 2005 and 2011 and evaluated in a study conducted by the Science and Technology Policy Institute (STPI, Fig 4B and S4 Fig), and finally, 290 articles published in 2009 by extramurally funded NIH investigators and evaluated by NIH intramural investigators in a study of our own design (Fig 4C; S5, S6 and S7 Figs). All three approaches demonstrate that RCR values are well correlated with reviewers’ judgments. We asked experts in the latter study to provide, in addition to an overall score, scores for several independent subcriteria: likely impact of the research, importance of the question being addressed, robustness of the study, appropriateness of the methods, and human health relevance. Random Forest analysis [38] indicated that their scores for likely impact were weighted most heavily in determining their overall evaluation (S6 Fig). In addition to correlating with expert opinion, RCR is ranking invariant, which is considered to be a desirable property of bibliometric indicators [39,40]. In short, an indicator is ranking invariant when it is used to place two groups of articles in hierarchical order, and the relative positions in that order do not change when uncited articles are added to each group. The RCR metric is ranking invariant when the same number of uncited articles is added to two groups of equal size (Supporting Equations S7–S9 in S1 Text). RCR is also ranking invariant when the same proportion of uncited articles is added to two groups of unequal size (Supporting Equations S10–S11 in S1 Text). This demonstrates that the RCR method can be used effectively and safely in evaluating the relative influence of large groups of publications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average TR ratio for these papers is 1.67, and the average RCR is 0.67.”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Comparison of RCR to Existing Bibliometrics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 86 claims and to verify the 29 references still open.
- Probably not a substantial claim. 18
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The RCR methodology was validated using citation data from about 90,000 published papers emanating from NIH-funded research and comparing calculated RCRs to the opinions on manuscript reach of recognized experts in selected fields.”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As one of several methods to quantify publication quality or its impact on the field, the RCR has performed better than journal impact factor, citations per year, or the Thompson–Reuters ratio [11,26,27].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The wRCR was considered a measure of influence, with higher values representing the most cited publications 22 .”
From Global mapping of randomized controlled trials in dentistry (2024) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“the Relative Citation Ratio (RCR) was developed to quantify the influence of a research article that is article level and independent of the scientific field [11].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“That the median RCR of Network publications exceeded 1.0 and the weighted RCR substantially exceeded the total number of publications upon which the analysis was based, demonstrate that the network’s publications have had a greater-than-average influence in their fields [11].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
This check didn’t finish.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
(1999) · Cited by 12,854
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · cited 4×
“In recent years, decisionmakers have increasingly turned to numerical approaches such as counting first or corresponding author publications, using the impact factor of the journals in which those publications appear, and computing Hirsch or H-index values (2).”
BackgroundMethodInfluentialCo‐citation in the scientific literature: A new measure of the relationship between two documents
Small, H. (1973) · Journal of the American Society for Information Science · doi:10.1002/asi.4630240406 · Cited by 5,345 · cited 1×
“There are a priori three types of article-linked citation networks [35].”
BackgroundKoenker, R. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511754098 · Cited by 4,551
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828 · cited 2×
“These authors [41] also found that RCR is better correlated with expert opinion than citation percentiles, which is the measure bibliometricians have previously recommended as best for evaluating the impact of published work [43].”
BackgroundGarfield, E. (1972) · Science · doi:10.1126/science.178.4060.471 · Cited by 2,788 · cited 4×
“For reasons of algorithmic parsimony we used the JCRs for the year each article in the co-citation network was published; a different choice at this step would be expected to have little if any effect, since almost all JCRs are quite stable over time ( Supporting Figure S1; Supporting Table S1 ).”
MethodInfluentialThe History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · cited 1×
“Journal impact factor (JIF) was for a time seen as a valuable indicator of scientific quality because it serves as a convenient, and not wholly inaccurate, proxy for expert opinion [8].”
BackgroundWhy the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 2×
“The results of a landmark study on the relationship between quality and success in a competitive market suggest that the ultimate winners in such contests are determined not only by the intrinsic value of the work but also by more intangible social variables [66].”
Background
Cited by
Topic choice contributes to the lower rate of NIH awards to African-American/black scientists
Hoppe, T. A., Litovitz, A., Willis, K. A., Meseroll, R. A., Perkins, M. J., Hutchins, B. I., Davis, A. F., Lauer, M. S., Valantine, H. A., Anderson, J. M., et al. (2019) · Science Advances · doi:10.1126/sciadv.aaw7238 · Cited by 625
“Furthermore, for topics in either the highest or lowest quintile of award rates, plotting the percentile score of each award against the median RCR of all papers it produced shows the complete absence of a correlation between study section assessment and future productivity (fig.”
MethodInfluentialSelf-citationSome bibliometric procedures for analyzing and evaluating research fields
Gutiérrez-Salcedo, M., Martínez, M. Á., Moral-Munoz, J. A., Herrera-Viedma, E., & Cobo, M. J. (2017) · Applied Intelligence · doi:10.1007/s10489-017-1105-y · Cited by 434
“…per paper, self citations rate, number of highly cited papers, h-index [35], g-index [25], hg-index [3], q 2 -index [10], crown-index [46], etc. – Indicators based on the impact of the journal : Impact Factor [28][30], SJR [56], normalized impact factor, relative citations ratio [38], etc.”
Assessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462
The science of science: From the perspective of complex systems
Zeng, A., Shen, Z., Zhou, J., Wu, J., Fan, Y., Wang, Y., & Stanley, H. E. (2017) · Physics Reports · doi:10.1016/j.physrep.2017.10.001 · Cited by 410
“This problem was addressed with an index called the relative citation ratio (RCR) in which a paper’s co-citation networkwas introduced to field-normalize the number of citations it has received [165].”
MethodDimensions: A competitor to Scopus and the Web of Science?
Thelwall, M. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.03.006 · Cited by 294
A scientometric review of geopolymer concrete
Zakka, W. P., Abdul Shukor Lim, N. H., & Chau Khun, M. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.124353 · Cited by 232
Dimensions: Bringing down barriers between scientometricians and data
Herzog, C., Hook, D., & Konkiel, S. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00020 · Cited by 219
Purkayastha, A., Palmaro, E., Falk-Krzesinski, H. J., & Baas, J. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.03.012 · Cited by 188
Veitch, D. P., Weiner, M. W., Aisen, P. S., Beckett, L. A., DeCarli, C., Green, R. C., Harvey, D., Jack, C. R., Jagust, W., Landau, S. M., et al. (2022) · Alzheimer's & Dementia · doi:10.1002/alz.12422 · Cited by 184
A systematic review of immersive virtual reality for industrial skills training
Radhakrishnan, U., Koumaditis, K., & Chinello, F. (2021) · Behaviour & Information Technology · doi:10.1080/0144929x.2021.1954693 · Cited by 200
Grissa, D., Junge, A., Oprea, T. I., & Jensen, L. J. (2022) · Database · doi:10.1093/database/baac019 · Cited by 144
“The latter is done by calculating an average rank score based on the number and distance of SNPs supporting the association, the P -value of the most significant SNP and the number of studies supporting the association weighted by the Relative Citation Ratio (30) of the underlying publications.”
MethodPharos 2023: an integrated resource for the understudied human proteome
Kelleher, K. J., Sheils, T. K., Mathias, S. L., Yang, J. J., Metzger, V. T., Siramshetty, V. B., Nguyen, D.-T., Jensen, L. J., Vidović, D., Schürer, S. C., et al. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkac1033 · Cited by 137
“Via TIGA, gene-trait associations are scored according to aggregated evidence from corresponding studies and a citation metric for publications based on the iCite Relative Citation Ratio, RCR (29).”
Early-career setback and future career impact
Wang, Y., Jones, B. F., & Wang, D. (2019) · Nature Communications · doi:10.1038/s41467-019-12189-3 · Cited by 107
The NIH Open Citation Collection: A public access, broad coverage resource
Hutchins, B. I., Baker, K. L., Davis, M. T., Diwersy, M. A., Haque, E., Harriman, R. M., Hoppe, T. A., Leicht, S. A., Meyer, P., & Santangelo, G. M. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000385 · Cited by 116
“While developing new science-of-science methodologies [10,11], the NIH Office of Portfolio Analysis (OPA) has carefully pursued unencumbered citation data resources that can be shared publicly.”
BackgroundMethodSelf-citationMagua, W., Zhu, X., Bhattacharya, A., Filut, A., Potvien, A., Leatherberry, R., Lee, Y.-G., Jens, M., Malikireddy, D. R., Carnes, M. L., et al. (2017) · Journal of Women s Health · doi:10.1089/jwh.2016.6021 · Cited by 130
“51”
Patterns of authors contribution in scientific manuscripts
Corrêa Jr., E. A., Silva, F. N., da F. Costa, L., & Amancio, D. R. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.03.003 · Cited by 95
Heyard, R., & Hottenrott, H. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00891-x · Cited by 129
“Wahls (2018) analyses the impact of project grants from the National Institutes of Health (NIH) in the United States and finds positive institution-level returns (in terms of publications and citation) to funding which, however, diminish at higher levels of funding.”
BackgroundMethodInfluentialOpen access research outputs receive more diverse citations
Huang, C., Neylon, C., Montgomery, L., Hosking, R., Diprose, J. P., Handcock, R. N., & Wilson, K. (2024) · Scientometrics · doi:10.1007/s11192-023-04894-0 · Cited by 121
“…potential confounders (Davis, 2011; Hua et al., 2016; Lewis, 2018; Sotudeh, 2020; Zhang & Watson, 2017) in analyses based simply on citation counts which remain with more sophisticated measures such as citation velocity, as shown in previous research (Hutchins et al., 2016; Seppänen et al., 2022).”
ResultField Normalization of Scientometric Indicators
Waltman, L., & van Eck, N. J. (2019) · Springer handbooks · doi:10.1007/978-3-030-02511-3_11 · Cited by 128
Paulus, F. M., Cruz, N., & Krach, S. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01487 · Cited by 96
Pacheco, J., Garvey, M. A., Sarampote, C. S., Cohen, E. D., Murphy, E. R., & Friedman‐Hill, S. R. (2022) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13543 · Cited by 99
“The Relative Citation Ratio (RCR) is a metric developed by NIH representing a citation-based measure of influence (Hutchins et al., 2016).”
Use of the journal impact factor for assessing individual articles: Statistically flawed or not?
Waltman, L., & Traag, V. A. (2021) · F1000Research · doi:10.12688/f1000research.23418.2 · Cited by 97
Peer review and preprint policies are unclear at most major journals
Klebel, T., Reichmann, S., Polka, J., McDowell, G. S., Penfold, N., Hindle, S. J., & Ross‐Hellauer, T. (2020) · PLoS ONE · doi:10.1371/journal.pone.0239518 · Cited by 105
“Levels of citations vary widely not only between broad categories of research, but also within specific disciplines and subdisciplines (Hutchins et al., 2016), and the number of journals sampled does not scale with the total number of journals or researchers in those areas.”
BackgroundCitation Metrics: A Primer on How (Not) to Normalize
Ioannidis, J. P. A., Boyack, K., & Wouters, P. F. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002542 · Cited by 87
“Table 1 summarizes factors that may be considered or debated during normalization and shows for comparison two systems, the RCR method [13] and the normalization used in the Leiden Ranking of universities (http://www.”
MethodPredicting translational progress in biomedical research
Hutchins, B. I., Davis, M. T., Meseroll, R. A., & Santangelo, G. M. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000416 · Cited by 96
“For this analysis, citation rate is preferable to RCR [18], because citations per year can be used immediately while a meaningful citation count must accrue before RCR can be calculated.”
BackgroundMethodInfluentialSelf-citationNew trends in bibliometric APIs: A comparative analysis
Vélez-Estévez, A., Pérez, I. J., García‐Sánchez, P., Moral-Muñoz, J. A., & Cobo, M. J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103385 · Cited by 95
Bornmann, L., & Williams, R. A. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.001 · Cited by 93
Bacterial volatile compound-based tools for crop management and quality
Cellini, A., Spinelli, F., Donati, I., Ryu, C., & Kloepper, J. W. (2021) · Trends in Plant Science · doi:10.1016/j.tplants.2021.05.006 · Cited by 81
Tahamtan, I., & Bornmann, L. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.07.005 · Cited by 81
Stagnation and Scientific Incentives
Bhattacharya, J., & Packalen, M. (2020) · National Bureau of Economic Research · doi:10.3386/w26752 · Cited by 64
Quantitative research assessment: using metrics against gamed metrics
Ioannidis, J. P. A., & Maniadis, Z. (2023) · Internal and Emergency Medicine · doi:10.1007/s11739-023-03447-w · Cited by 75
The NIH Common Fund/Roadmap Epigenomics Program: Successes of a comprehensive consortium
Satterlee, J. S., Chadwick, L. H., Tyson, F. L., McAllister, K., Beaver, J. M., Birnbaum, L. S., Volkow, N. D., Wilder, E. L., Anderson, J. M., & Roy, A. L. (2019) · Science Advances · doi:10.1126/sciadv.aaw6507 · Cited by 64
“All 857 articles were analyzed for their influence using iCite (26), which provides a Relative Citation Ratio (RCR) for each publication.”
MethodSelf-citationBiomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61
Carlson, J., & Harris, K. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000860 · Cited by 61
Bornmann, L., & Marx, W. (2016) · EMBO reports · doi:10.15252/embr.201642823 · Cited by 59
Scientific productivity: An exploratory study of metrics and incentives
Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195321 · Cited by 68
Deciphering the olfactory repertoire of the tiger mosquito Aedes albopictus
Lombardo, F., Salvemini, M., Fiorillo, C., Nolan, T., Zwiebel, L. J., Ribeiro, J. M. C., & Arcà, B. (2017) · BMC Genomics · doi:10.1186/s12864-017-4144-1 · Cited by 68
Essential Ingredients and Innovations in the Design and Analysis of Group-Randomized Trials
Murray, D. M., Taljaard, M., Turner, E. L., & George, S. M. (2019) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094027 · Cited by 64
“The RCR was available for 791 items; for the remaining 135 items, the Web of Science, Scopus, and Google Scholar were used to obtain citation counts.”
BackgroundMethodInfluentialStructure, function, and productivity from the National Dental Practice-Based Research Network.
Gilbert, G. H., Fellows, J. L., Allareddy, V., Cochran, D. L., Cunha-Cruz, J., Gordan, V. V., McBurnie, M. A., Meyerowitz, C., Mungia, R., Rindal, D. B., et al. (2022) · Journal of Clinical and Translational Science · doi:10.1017/cts.2022.421 · Cited by 54
“that is article level and independent of the scientific field [11].”
BackgroundReddy, V. P., Gupta, A., White, M. D., Gupta, R., Agarwal, P., Prabhu, A. V., Lieber, B. A., Chang, Y., & Agarwal, N. (2020) · Journal of neurosurgery · doi:10.3171/2019.11.jns192679 · Cited by 58
Five years post-DORA: promoting best practices for research assessment
Schmid, S. L. (2017) · Molecular Biology of the Cell · doi:10.1091/mbc.e17-08-0534 · Cited by 51
“• Researchers are developing new article-level and field-independent metrics that provide a more accurate and transparent approach to research assessment (Hutchins et al., 2016; Santangelo, 2017).”
BackgroundTwo decades of neuroscience publication trends in Africa
Maina, M. B., Ahmad, U., Arrashid, I. H., Hamidu, S. K., Nasr, F. E., Salihu, A. T., Abushouk, A. I., Abdurrazak, M., Awadelkareem, M. A., Amin, A., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-23784-8 · Cited by 56
Torres‐Salinas, D., Valderrama, P., & Arroyo‐Machado, W. (2022) · Journal of Informetrics · doi:10.1016/j.joi.2022.101315 · Cited by 53
Requiem for impact factors and high publication charges
Triggle, C. R., MacDonald, R. S., Triggle, D. J., & Grierson, D. S. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1909481 · Cited by 56
“It has been argued that the Relative Citation Ratio (RCR) proposed by Hutchins et al. (2016) offers an improvement and comes closest as in applying “field normalization” it provides a measure of where an author ranks relative to others in the same field of research (Surkis and Spore 2018).”
BackgroundEmbodied Conversational Agents in Extended Reality: A Systematic Review
Yang, F., Acevedo, P., Guo, S., Choi, M., & Mousas, C. (2025) · IEEE Access · doi:10.1109/access.2025.3566698 · Cited by 74
“The ACR (see Equation 1) indirectly measures the study’s impact and recognition in the field, as suggested by Hutchins [37].”
A Framework for Improving the Quality of Research in the Biological Sciences
Casadevall, A., Ellis, L. M., Davies, E. W., McFall-Ngai, M., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.01256-16 · Cited by 46
Surkis, A., Hogle, J. A., DiazGranados, D., Hunt, J. D., Mazmanian, P. E., Connors, E. R., Westaby, K., Whipple, E. C., Adamus, T., Mueller, M., et al. (2016) · Journal of Translational Medicine · doi:10.1186/s12967-016-0992-8 · Cited by 52
“Current CTSA efforts around common metrics have focused on the Relative Citation Ratio [24], an article-level, field-normalized citation metric.”
BackgroundLoet Leydesdorff, A., Comins, J. A., Sorensen, A. A., Bornmann, L., & Hellsten, I. (2016) · Scientometrics · doi:10.1007/s11192-016-2119-7 · Cited by 50
“In response to these changes, bibliometricians have begun to cluster the database at the level of documents instead of journals (e.g., Waltman and van Eck 2012; cf. Hutchins et al. 2016).”
BackgroundThe relative citation ratio: what is it and why should medical librarians care?
Surkis, A., & Spore, S. (2018) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2018.499 · Cited by 52
“a preprint in 2015 [8] and then as a peer-reviewed article in PLOS Biology in 2016 [9].”
BackgroundRobustness of evidence reported in preprints during peer review
Nelson, L., Ye, H., Schwenn, A., Lee, S., Arabi, S., & Hutchins, B. I. (2022) · The Lancet Global Health · doi:10.1016/s2214-109x(22)00368-0 · Cited by 54
“Articles that the NIH iSearch COVID-19 Portfolio identified had a later version, but the later version was also a preprint and no published version existed, were excluded from analysis.”
MethodInfluentialSelf-citationYang, T.-Y., Chien, T., & Lai, F. (2022) · Medicine · doi:10.1097/md.0000000000031144 · Cited by 44
“) to display the article impact from 0 to 100 by years (based on normalized citation percentiles for each article).[36,41] The overall hT index[42,43] and h-index[44] were compared with the median score on the IBP using the online technique.”
MethodRock, C. B., Prabhu, A. V., Fuller, C. D., Thomas, C. R., & Holliday, E. B. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.11.006 · Cited by 43
“The proponents of the RCR validated this metric in a data set including >88,000 publications and demonstrated that the RCR tracks well with expert opinion of research quality [13].”
BackgroundInfluentialA critical evaluation of the algorithm behind the Relative Citation Ratio (RCR)
Janssens, A. C. J. W., Goodman, M., Powell, K. R., & Gwinn, M. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002536 · Cited by 42
Citations are a good way to determine the quality of research
Caon, M., Trapp, J., & Baldock, C. (2020) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-020-00941-9 · Cited by 50
Critical Rationalism and the Search for Standard (Field-Normalized) Indicators in Bibliometrics
Bornmann, L., & Marx, W. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.05.002 · Cited by 41
Evaniew, N., Adili, A. F., Ghert, M. A., Khan, M., Madden, K., Smith, C., & Bhandari, M. (2017) · JBJS Reviews · doi:10.2106/jbjs.rvw.16.00059 · Cited by 40
Statistical fragility of randomized clinical trials in shoulder arthroplasty
McCormick, K. L., Tedesco, L. J., Swindell, H. W., Forrester, L. A., Jobin, C. M., & Levine, W. N. (2020) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2020.10.028 · Cited by 40
Statistical Fragility of Surgical and Procedural Clinical Trials in Orthopaedic Oncology
Forrester, L. A., Jang, E. S., Lawson, M. M., Çapi, A., & Tyler, W. K. (2020) · JAAOS Global Research and Reviews · doi:10.5435/jaaosglobal-d-19-00152 · Cited by 39
Publish or perish, but do not forget your software artifacts
Heumüller, R., Nielebock, S., Krüger, J., & Ortmeier, F. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09851-6 · Cited by 42
“To address this issue, we adapted the average relative citation (arc) metric (Hutchins et al. 2016; Piwowar et al. 2018).”
MethodGlynatsi, N. E., & Knight, V. A. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00718-9 · Cited by 37
Human Odour Coding in the Yellow Fever Mosquito, Aedes aegypti
Chen, Z., Liu, F., & Liu, N. (2019) · Scientific Reports · doi:10.1038/s41598-019-49753-2 · Cited by 45
Bioinformatic approaches for studying the microbiome of fermented food
Walsh, L. H., Coakley, M., Walsh, A. M., O’Toole, P. W., & Cotter, P. D. (2022) · Critical Reviews in Microbiology · doi:10.1080/1040841x.2022.2132850 · Cited by 40
“Publications deemed appropriate were selected for citation analysis using the relative citation ratio (RCR) metric (Hutchins et al. 2016, 2019) as calculated by iCite (https://icite.od.nih.gov/).”
MethodThe NIH must reduce disparities in funding to maximize its return on investments from taxpayers
Wahls, W. P. (2018) · eLife · doi:10.7554/elife.34965 · Cited by 37
“…metric encompasses the influence of the publications, as measured by how frequently other scientists cite the published work in their own articles, taking into account that article-level citation impact factors follow a log-normal distribution (see Hutchins et al., 2016 and references therein).”
BackgroundIn defense of quantitative metrics in researcher assessments
Ioannidis, J. P. A., & Maniadis, Z. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002408 · Cited by 45
Reliable novelty: New should not trump true
Brembs, B. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000117 · Cited by 44
Small research teams ‘disrupt’ science more radically than large ones
Azoulay, P. (2019) · Nature · doi:10.1038/d41586-019-00350-3 · Cited by 41
Additional support for RCR: A validated article-level measure of scientific influence
Hutchins, B. I., Hoppe, T. A., Meseroll, R. A., Anderson, J. M., & Santangelo, G. M. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003552 · Cited by 36
Self-citationBibliometric analysis of the 100 most cited articles on bone grafting in dentistry
dos Anjos, L. M., Rocha, A. D. O., Magrin, G. L., Kammer, P. V., Benfatti, C. A. M., Souza, J. C. M., Sanz, M., & Henriques, B. A. P. D. C. (2023) · Clinical Oral Implants Research · doi:10.1111/clr.14152 · Cited by 35
Article-level assessment of influence and translation in biomedical research
Santangelo, G. M. (2017) · Molecular Biology of the Cell · doi:10.1091/mbc.e16-01-0037 · Cited by 36
“…tool for selecting winners in the intense competition for status and resources is highlighted by the observation that everything published in journals with a high JIF (≥28), taken together, accounts for only a small fraction of the most influential articles (Bertuzzi, 2015; Hutchins et al., 2016).”
BackgroundMethodInfluentialSelf-citationGasparyan, A. Y., Yessirkepov, M., Voronov, A. A., АНАТОЛЬЕВИЧ, М. А., & Kitas, G. D. (2021) · Journal of Korean Medical Science · doi:10.3346/jkms.2021.36.e74 · Cited by 43
A Bibliometric Analysis of Research Trends in Geopolymer
Matsimbe, J. J., Dinka, M. O., Olukanni, D. O., & Musonda, I. (2022) · Materials · doi:10.3390/ma15196979 · Cited by 40
Taylor, M. (2023) · Scientometrics · doi:10.1007/s11192-023-04653-1 · Cited by 39
“5 Slow, slow, quick, quick, slow – a longitudinal observation of altmetric data observed citations to expected citations, for publications of a particular type, publication year and subject area (Hutchins et al., 2016; Zanotto & Carvalho, 2021).”
BackgroundA Bibliometric Analysis: Metaverse in Education Concept
Bizel, G. (2023) · Journal of Metaverse · doi:10.57019/jmv.1310768 · Cited by 42
“It shows how well a publication performs in terms of citations when compared to similarly aged articles in its field [54].”
BackgroundThe Top 100 Most-Cited Papers in Erosive Tooth Wear: A Bibliometric Analysis
Rocha, A. D. O., Santos, P. S., Machado, B. A., Bolan, M., Cardoso, M., Martins‐Júnior, P. A., & Santana, C. M. (2022) · Caries Research · doi:10.1159/000521877 · Cited by 40
“A high number of citations may indicate a paper’s potential to influence clinical decisions and future research [Hutchins et al., 2016; Perazzo et al., 2019; Ahmad et al., 2020].”
BackgroundYan, S., Zhang, H., & Wang, J. (2022) · Japanese Journal of Radiology · doi:10.1007/s11604-022-01268-z · Cited by 38
“To identify the most influential papers, we ranked all the HCPs by the Relative Citation Rate (RCR), a new metric that uses citation rates to measure influence at the paper level [27].”
MethodLi, M., Chien, T., Liao, K., & Lai, F. (2022) · Medicine · doi:10.1097/md.0000000000030682 · Cited by 34
“The hT index based on the RCR [37] is computed by equation 1, where j is the j-th article in descending order and n is the RCRs of an article for an entity.”
BackgroundMethodInfluentialTowards an open science publishing platform
Tracz, V., & Lawrence, R. N. (2016) · F1000Research · doi:10.12688/f1000research.7968.1 · Cited by 39
“A more appropriate use of citation-based indicators should also be included in measuring quality, such as the Relative Citation Ratio (RCR) recently proposed by the NIH 31 , remembering that citation-based measures take considerable time, which may be an issue particularly for younger researchers.”
BackgroundProdução científica sobre assédio moral e enfermagem: estudo bibliométrico
Lucena, P. L. C., da Costa, S. . G., Batista, J. B. V., Lucena, C. M. F., Morais, G. S. D. N., & Costa, B. H. S. (2018) · Revista da Escola de Enfermagem da USP · doi:10.1590/s1980-220x2017029103354 · Cited by 32
Impact-Based Ranking of Scientific Publications: A Survey and Experimental Evaluation
Kanellos, I., Vergoulis, T., Sacharidis, D., Dalamagas, T., & Vassiliou, Y. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2941206 · Cited by 38
“S-RCR [46] is a simplified version of the relative citation ratio RCR [58].”
BackgroundMitra, A. N., Aurora, N., Grover, S., Ananth, C. V., & Brandt, J. S. (2020) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2020.100293 · Cited by 34
From symbiont to parasite: the evolution of for-profit science publishing
Walter, P., & Mullins, R. D. (2019) · Molecular Biology of the Cell · doi:10.1091/mbc.e19-03-0147 · Cited by 31
“, not journal-based) metrics are an improvement and are gaining acceptance, although even this is not enough (Hutchins et al., 2016; Santangelo, 2017).”
BackgroundSCIENTIFIC PRODUCTION IN ONLINE JOURNALS ABOUT THE NEW CORONAVIRUS (COVID-19): BIBLIOMETRIC RESEARCH
Costa, I. C. P., Sampaio, R. S., de Souza, F. A. C., Dias, T. K. C., Costa, B. H. S., & Chaves, É. D. C. L. (2020) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2020-0235 · Cited by 34
Feasibility of common bibliometrics in evaluating translational science
Schneider, M. L., Kane, C., Rainwater, J. A., Guerrero, L. R., Tong, G., Desai, S. R., & Trochim, W. M. K. (2017) · Journal of Clinical and Translational Science · doi:10.1017/cts.2016.8 · Cited by 30
“gov/), which calculates a comparative influence measure, the Relative Citation Ratio (RCR), with NIH-supported publications as the benchmark [23].”
MethodTrends in Academic Spine Neurosurgeon Productivity as Measured by the Relative Citation Ratio
Grogan, D. P., Reddy, V. P., Gupta, A., Chang, Y., Fields, D. P., & Agarwal, N. (2020) · World Neurosurgery · doi:10.1016/j.wneu.2020.11.097 · Cited by 32
Grover, S., Elwood, A. D., Patel, J. M., Ananth, C. V., & Brandt, J. S. (2022) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2022.03.013 · Cited by 30
Lee, Y.-L., Chien, T., & Wang, J.-C. (2022) · Medicine · doi:10.1097/md.0000000000030217 · Cited by 29
“Second, based on the article contents, we extracted 5 topical entities from abstracts, including (1) affiliated countries/regions, (2) journals, (3) medical subject headings (MeSH terms), (4) article identity numbers, and (5) individual authors, based on the article relative citation ratios (RCRs)[38] over the years.”
MethodGarstka, M., Randolph, G. W., Haddad, A. B., Nathan, C. O., Ibraheem, K., Farag, M., Deot, N., Adib, H., Hoof, M. A., French, K., et al. (2019) · Head & Neck · doi:10.1002/hed.25913 · Cited by 29
The Journal Impact Factor of Orthopaedic Journals Does not Predict Individual Paper Citation Rate
Bozzo, A., Oitment, C. T. A., Evaniew, N., & Ghert, M. A. (2017) · JAAOS Global Research and Reviews · doi:10.5435/jaaosglobal-d-17-00007 · Cited by 30
Trends in the Authorship of Peer Reviewed Publications in the Urology Literature
An, J. Y., Baiocco, J. A., & Rais‐Bahrami, S. (2017) · Urology Practice · doi:10.1016/j.urpr.2017.03.008 · Cited by 27
“RCR is a field normalized measure of citation impact that accounts for discipline and time of publication.(10) We examined trends in authorship count during the last decade in all 4 journals combined as well as trends between each journal.”
Evaluating scientific impact of publications: combining citation polarity and purpose
Huang, H., Zhu, D., & Wang, X. (2021) · Scientometrics · doi:10.1007/s11192-021-04183-8 · Cited by 29
Volumetric Muscle Loss: A Bibliometric Analysis of a Decade of Progress
Kulwatno, J., Goldman, S. M., & Dearth, C. L. (2022) · Tissue Engineering Part B Reviews · doi:10.1089/ten.teb.2022.0150 · Cited by 27
“14 As per the Translation module, articles are scored based on the number of MeSH terms that fall into Human, Animal, and Molecular/Cellular Biology categories.”
InfluentialLlewellyn, N. M., Carter, D. R., Rollins, L. S., & Nehl, E. J. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002119 · Cited by 27
Kilicoglu, H., Peng, Z., Tafreshi, S., Tran, T., Rosemblat, G., & Schneider, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103123 · Cited by 33
Cited text span identification for scientific summarisation using pre-trained encoders
Zerva, C., Nghiem, M.-Q., Nguyen, N. T. H., & Ananiadou, S. (2020) · Scientometrics · doi:10.1007/s11192-020-03455-z · Cited by 30
Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study
Yu, F., Van, A. A., Patel, T., Mani, N., Carnegie, A., Corbie-Smith, G. M., Carey, T., Buse, J., & Dave, G. (2020) · Journal of Clinical and Translational Science · doi:10.1017/cts.2020.29 · Cited by 26
“It is a validated method to quantify the influence of a research article [31].”
BackgroundMethodFostering a research integrity culture: Actionable advice for institutions
Peuter, S. D., & Conix, S. (2022) · Science and Public Policy · doi:10.1093/scipol/scac059 · Cited by 29
Authorship of Publications Supported by NCI-Funded Grants Involving Low- and Middle-Income Countries
Eldridge, L., Garton, E. M., Duncan, K., & Gopal, S. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.3215 · Cited by 26
Bornmann, L., & Haunschild, R. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23729 · Cited by 27
“Hutchins et al. (2015) also studied the correlation of RCR with F1000 scores and reported a correlation coefficient of r 5 0.44 (in the supplemental information).”
BackgroundMethodInfluentialDante Israel León de la, O., Thorsteinsdóttir, H., & Calderón-Salinas, J. V. (2018) · PLoS ONE · doi:10.1371/journal.pone.0191267 · Cited by 26
A detailed open access model of the PubMed literature
Boyack, K. W., Smith, C., & Klavans, R. (2020) · Scientific Data · doi:10.1038/s41597-020-00749-y · Cited by 33
“0 database 11,12 .”
Metadata
- Authors
- Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Hutchins, B. Ian
- Yuan, Xin
- Anderson, James M.
- Santangelo, George M.
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-09-06
- Volume
- 14
- Issue
- 9
- Pages
- e1002541
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5012559
- OpenAlex
- W2953304564
- Cited by
- 500
- References
- 91
- Citations by year
- 2026: 43 · 2025: 51 · 2024: 52 · 2023: 62 · 2022: 68 · 2021: 48 · 2020: 44 · 2019: 45 · 2018: 36 · 2017: 32 · 2016: 17
- FWCI
- 40.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Biology (publishedVersion, public-domain)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (573871/2008-6)
- Fundação de Amparo à Pesquisa do Estado de Minas Gerais (573871/2008-6)
- National Institutes of Health (Z99 OD999999)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- relative citation ratio, citation analysis, co-citation networks, field normalization, iCite
- MeSH terms
- Authorship, Humans, National Institutes of Health (U.S.), United States, Bibliometrics, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.