Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level
Open accessPublished version of a preprint
The DOI matches its Crossref record; 87 of the 91 references it cites were found in a registry or an index; 1 of the 30 sources its claims cite were checked for support.
Abstract
Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.
Verification finished with gaps
Some checks didn’t finish. The results below cover the steps that did.
- Registry recordVerified
- Full textRead the full text
- Reference list87 references found
- Reference check87 of 87 references checked
- StatisticsRecomputed
- Claims86 claims anchored to a quote
- Cited sourcesChecked
- Self-checkDidn’t finish — the rest of the report still applies.
- VerificationFindings verified
Overview
CoChat Summary
## TL;DR
The paper introduces Relative Citation Ratio (RCR), an article-level, field-normalized citation metric built from co-citation networks and benchmarked to a peer group. The authors report that RCR correlates strongly with expert judgments, behaves better than several existing bibliometrics in their tests, and is available through the iCite web tool.
## Key findings
- Co-citation networks were used to define an article’s field; in a validation study, diagnostic words were more likely to be shared between an article and papers in its co-citation network than between that article and papers in the same journal, based on 1,397 articles and 249,981 pairwise comparisons within co-citation networks versus 28,516,576 comparisons with same-journal articles.
- In a benchmark set of 35,837 papers published in 2009 by NIH R01 recipients, FCRs based on co-citation networks had lower variance than FCRs based on cited or citing networks, and lower variance than ACRs or JIFs (p < 10−4 for the variance comparisons reported).
- RCR correlated well with subject matter expert ratings across three validation datasets: 2,193 Faculty of 1000-rated articles, 430 STPI-rated articles, and 290 NIH-rated articles.
- In an NIH case study of 88,835 articles from 3,089 continuously funded R01 principal investigators, 53% of investigators in the top RCR quintile stayed there across two 4-year periods, and 53% of those in the bottom quintile stayed at the bottom; among more than 200,000 articles analyzed for FCR stability, only 0.2% had an RCR drop of 0.1 or more between 2012 and 2014, and less than 2% dropped at all.
## Methods
- The metric was built from an article’s co-citation network: papers citing the reference article were used to assemble the set of co-cited papers, whose journal citation rates were averaged to form the field citation rate (FCR); the article’s article citation rate (ACR) was then normalized by FCR and benchmarked to a peer group to yield RCR.
- Citation analyses used the Thomson Reuters Web of Science dataset from 2002–2012 (extended to 2014 for FCR stability analyses), focusing on journals in which NIH R01-funded researchers published; grant and PI data came from NIH RePORTER, SPIRES, and IMPAC-II.
- Validation included cosine-similarity content analysis with IN-SPIRE on 1,397 articles from six journals, expert-comparison analyses, and regression/variance tests; RCR benchmarking used OLS for large-scale analyses and quantile regression as an alternative for smaller, skewed settings.
## Limitations
- The paper states that RCR cannot be calculated immediately after publication; enough citations must accrue first, and the authors note that 93% of co-citation-network-based FCRs stabilize after a paper has been cited five times.
- The authors explicitly say they have not determined whether small differences in RCR, such as 1.0 versus 1.2, reliably reflect meaningful differences in influence.
- The paper also states that citation metrics cannot fully capture article value, are not a substitute for reading the paper, and RCR is not designed as a measure of long-term impact or as a metric for applied research; these are stated limitations of scope.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig 4 RCRs correspond with expert reviewer scores. (A–C) Bubble plots of reviewer scores versus RCR for three different datasets. Articles are binned by reviewer score; bubble area is proportionate to the number of articles in that bin. (A) F1000 scores for 2,193 R01-funded papers published in 2009. Faculty reviewers rated the articles on a scale of one to three (“good,” “very good,” and “exceptional”, respectively); those scores were summed into a composite F1000 score for each article (S3 Fig). (B) Reviewer scores of 430 HHMI and NIH-funded papers collected by STPI. (C) Scores of 290 R01-funded articles reviewed by experts from the NIH Intramural Research Program. Black line, linear regression. (D) Box-and-whisker plots illustrating the distribution of journal impact factors (JIFs) citations per year (CPY) and RCRs for two areas of NIH-funded research from 2007–2011. Cell biology, n = 5,936; neurological function, n = 5,417. *** p < 0.001, Kruskal-Wallis with Dunn’s multiple comparison test. n.s., not significant. Mean represented by a “+.” (E) Comparison of RCRs (orange) and Thompson Reuters ratios (blue) [17,28] for the same 544 articles with a low denominator. Data points are partially transparent to allow coordinates with multiple papers (darker) to be more clearly identified.](https://figures.cochat.ai/paper-figures/e792b94297fa4418b200b0579a2ea0b9/relative-citation-ratio-rcr-a-new-metric-that-uses-citation-rates-to-measure-figure-4.jpg)


![Fig 7 Properties of RCRs at the article and investigator level. (A, B) Frequency distribution of article-level RCRs (A) and JIFs (B), from 88,835 papers (authored by 3,089 R01-funded principal investigators [PIs]) for which co-citation networks were generated. Article RCRs are well fit by a log-normal distribution (R2 = 0.99), and JIFs less so (R2 = 0.79). (C) Box-and-whisker plots summarizing JIFs for the same papers, binned by impact factor quintile (line, median; box, 25th–75th percentiles; whiskers, 10th–90th percentiles). (D) RCR for the same papers using the same bins by JIF quintile (same scale as C). Although the median RCR for each bin generally corresponds to the impact factor quintile, there is a wide range of article RCRs in each category. (E) Box-and-whisker plots summarizing RCRs of these same papers published in selected journals. In each journal, there are papers with article RCRs surpassing the median RCR of the highest impact factor journals (left three). The impact factor of each journal is shown above. (F, G) Frequency distribution of investigator-level RCRs (F) and JIFs (G), representing the mean values for papers authored by each of 3,089 R01-funded PIs. Dashed line in (F), mode of RCR for PIs.](https://figures.cochat.ai/paper-figures/7111239ea73541aeace7efe3f230790e/relative-citation-ratio-rcr-a-new-metric-that-uses-citation-rates-to-measure-figure-7.jpg)

Verification
Claims the paper makes
1 Valid · 1 Not valid · 18 Not checked · 66 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Strikingly, this analysis showed that diagnostic words are much more likely to be shared between an article and the papers in its co-citation network than between that same article and the papers that appear alongside it in its journal of publication (Fig 2).”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Co-citation Networks Represent an Article’s Area of Influence
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Abstract
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though journals with the highest impact factors have the highest median RCR”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Quantifying How Past Influence Predicts Future Performance We next undertook a large case study of all 88,835 articles published by NIH investigators who maintained continuous R01 funding from fiscal year (FY) 2003 through FY2010 to ask how the RCR of publications from individual investigators changed over this 8-y interval. Each of these investigators had succeeded at least once in renewing one or more of their projects through the NIH competitive peer review process. In aggregate, the RCR values for these articles are well matched to a log-normal distribution (Fig 7); in contrast, as noted previously by others, the distribution of impact factors of the journals in which they were published is non-normal (Fig 7A and 7B) [53,54]. Sorting into quintiles based on JIF demonstrates that, though journals with the highest impact factors have the highest median RCR, influential publications can be found in virtually all journals (Fig 7C and 7D); these data call into question the assertion that journal of publication is the strongest quality signal for researchers who are young or otherwise unknown to a field [55]. Focusing on a dozen representative journals with a wide range of JIFs further substantiates the finding that influential science appears in many venues and reveals noteworthy departures from the correlation between JIF and median RCR (see S1 Text). For example, NIH-funded articles in both Organic Letters (JIF = 4.7) and the Journal of the Acoustical Society of America (JIF = 1.6) have a higher median RCR than those in Nucleic Acids Research (JIF = 7.1; Fig 7E). As part of this case study, we also calculated the average RCR and average JIF for papers published by each of the 3,089 NIH R01 principal investigators (PIs) represented in the dataset of 88,835 articles. In aggregate, the average RCR and JIF values for NIH R01 PIs exhibited log-normal distributions (Fig 7F and 7G) with substantially different hierarchical ordering (S8 Fig). This raised a further question concerning PIs with RCR values near the mode of the log-normal distribution (dashed line in Fig 7F): as measured by the ability to publish work that influences their respective fields, to what extent does their performance fluctuate? We addressed this question by dividing the 8-y window (FY2003 through FY2010) in half. Average RCRs in the first time period (FY2003 through FY2006) were sorted into quintiles, and the percentage of PIs in the second time period (FY2007 through FY2010) that remained in the same quintile, or moved to a higher or lower quintile, was calculated (Fig 8). The position of PIs in these quintiles appeared to be relatively immobile; 53% of PIs in the top quintile remained at the top, and 53% of those in the bottom quintile remained at the bottom (Fig 8A). For each PI, we also calculated a weighted RCR (the number of articles multiplied by their average RCR); comparing on this basis yielded almost identical results (Fig 8B). It is worth noting that average FCRs for investigators were extremely stable from one 4-y period to the next (Pearson r = 0.92, Table 2), Since FCRs are the quantitative representation of co-citation networks, this further suggests that each co-citation network is successfully capturing the corresponding investigator’s field of research. Another possible interpretation of the above data is that PI RCRs perform an unbiased random walk from their initial state with a large diffusion rate. Considered from this frame, it could be said that 47% of PIs who started in the top quintile moved out of it during the second 4-y period we analyzed. To test this hypothesis directly, we performed a mean reversion test, which determines whether or not the set of values under consideration will return to an average, or mean, value over time. If drift in PI RCR were simply a random walk, then the change in RCR should by definition be independent of starting RCR, and plotting these two values against each other should result in a straight line with a slope of zero. However, the results show that change in RCR is dependent on starting RCR value (p < 0.001, linear regression analysis, n = 3,089, Fig 8C). Furthermore, randomly shuffling PI RCRs from the second 4-y period gives a slope that is significantly different than that observed for the real data (p < 0.001, extra sum-of-squares F-test, n = 3,089, Fig 8C), ruling out the possibility that these values are randomly sampled from the same distribution in each time interval.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The RCR metric is ranking invariant when the same number of uncited articles is added to two groups of equal size (Supporting Equations S7–S9 in S1 Text).”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Expert Validation of RCR as a Measure of Influence For the work presented here, we chose as a benchmark the full set of 311,497 RAs published from 2002 through 2012 by NIH R01 awardees. To measure the degree of correspondence between our method and expert opinion, we compared RCRs generated by OLS benchmarking of ACR/FCR values with three independent sets of postpublication evaluations by subject matter experts (details in S1 Text). We compared RCR with expert rankings (Fig 4) for 2,193 articles published in 2009 and evaluated by Faculty of 1000 members (Fig 4A and S3 Fig), as well as rankings of 430 Howard Hughes Medical Institute- or NIH-funded articles published between 2005 and 2011 and evaluated in a study conducted by the Science and Technology Policy Institute (STPI, Fig 4B and S4 Fig), and finally, 290 articles published in 2009 by extramurally funded NIH investigators and evaluated by NIH intramural investigators in a study of our own design (Fig 4C; S5, S6 and S7 Figs). All three approaches demonstrate that RCR values are well correlated with reviewers’ judgments. We asked experts in the latter study to provide, in addition to an overall score, scores for several independent subcriteria: likely impact of the research, importance of the question being addressed, robustness of the study, appropriateness of the methods, and human health relevance. Random Forest analysis [38] indicated that their scores for likely impact were weighted most heavily in determining their overall evaluation (S6 Fig). In addition to correlating with expert opinion, RCR is ranking invariant, which is considered to be a desirable property of bibliometric indicators [39,40]. In short, an indicator is ranking invariant when it is used to place two groups of articles in hierarchical order, and the relative positions in that order do not change when uncited articles are added to each group. The RCR metric is ranking invariant when the same number of uncited articles is added to two groups of equal size (Supporting Equations S7–S9 in S1 Text). RCR is also ranking invariant when the same proportion of uncited articles is added to two groups of unequal size (Supporting Equations S10–S11 in S1 Text). This demonstrates that the RCR method can be used effectively and safely in evaluating the relative influence of large groups of publications.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average TR ratio for these papers is 1.67, and the average RCR is 0.67.”
§ Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level > Results > Comparison of RCR to Existing Bibliometrics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 86 claims and to verify the 29 references still open.
- Probably not a substantial claim. 18
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The RCR methodology was validated using citation data from about 90,000 published papers emanating from NIH-funded research and comparing calculated RCRs to the opinions on manuscript reach of recognized experts in selected fields.”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As one of several methods to quantify publication quality or its impact on the field, the RCR has performed better than journal impact factor, citations per year, or the Thompson–Reuters ratio [11,26,27].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The wRCR was considered a measure of influence, with higher values representing the most cited publications 22 .”
From Global mapping of randomized controlled trials in dentistry (2024) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“the Relative Citation Ratio (RCR) was developed to quantify the influence of a research article that is article level and independent of the scientific field [11].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“That the median RCR of Network publications exceeded 1.0 and the weighted RCR substantially exceeded the total number of publications upon which the analysis was based, demonstrate that the network’s publications have had a greater-than-average influence in their fields [11].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [11]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
This check didn’t finish.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
(1999) · Cited by 12,854
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · cited 4×
“In recent years, decisionmakers have increasingly turned to numerical approaches such as counting first or corresponding author publications, using the impact factor of the journals in which those publications appear, and computing Hirsch or H-index values (2).”
BackgroundMethodInfluentialCo‐citation in the scientific literature: A new measure of the relationship between two documents
Small, H. (1973) · Journal of the American Society for Information Science · doi:10.1002/asi.4630240406 · Cited by 5,345 · cited 1×
“There are a priori three types of article-linked citation networks [35].”
BackgroundKoenker, R. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511754098 · Cited by 4,551
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828 · cited 2×
“These authors [41] also found that RCR is better correlated with expert opinion than citation percentiles, which is the measure bibliometricians have previously recommended as best for evaluating the impact of published work [43].”
BackgroundGarfield, E. (1972) · Science · doi:10.1126/science.178.4060.471 · Cited by 2,788 · cited 4×
“For reasons of algorithmic parsimony we used the JCRs for the year each article in the co-citation network was published; a different choice at this step would be expected to have little if any effect, since almost all JCRs are quite stable over time ( Supporting Figure S1; Supporting Table S1 ).”
MethodInfluentialThe History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · cited 1×
“Journal impact factor (JIF) was for a time seen as a valuable indicator of scientific quality because it serves as a convenient, and not wholly inaccurate, proxy for expert opinion [8].”
BackgroundWhy the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 2×
“The results of a landmark study on the relationship between quality and success in a competitive market suggest that the ultimate winners in such contests are determined not only by the intrinsic value of the work but also by more intangible social variables [66].”
Background
Cited by
Items 401–500 of 515
論文評価指標の要点整理と動向紹介 (特集 医学図書館員が知っておきたい最新情報)
千葉, 浩. (2020) · Institutional Repositories DataBase (IRDB)
Costa, I. C. P., de Lima, L. M., de Oliveira, T. C., Garcia, A. C. M., Costa, C. B. A., & Costa, B. H. S. (2020) · Revista de Pesquisa Cuidado é Fundamental Online · doi:10.9789/2175-5361.rpcfo.v12.9457
Assessing the academic influence of otolaryngologists on sinus and allergy research
Grace, Z. T., Magdich, A. R., Barinsky, G. L., Chen, T., Karim, M., Benson, B. E., Folbe, A. J., & Svider, P. F. (2021) · International Forum of Allergy & Rhinology · doi:10.1002/alr.22773
TeamTree analysis: a new approach to evaluate scientific productionPreprint
Pfrieger, F. W. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.01.128355
Comparison of Data Analytics Software Usage in Biomedical and Health Sciences Research: A Case Study
Yu, F., & Mani, N. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-71292-1_11
Characteristics of the least-cited and most-cited articles in ophthalmology journals: A pilot study
Patel, P. A., Gopali, R., Reddy, A. P., & Patel, K. K. (2021) · European Journal of Ophthalmology · doi:10.1177/11206721211042556
Maurya, R. K., Datana, S., Verma, S., Londhe, S., & Bhandari, S. K. (2021) · Journal of Dentistry Defence Section · doi:10.4103/jodd.jodd_42_20
Ambrosini, A., Baldessari, D., Pozzi, S., Battaglia, M., Beltrami, E., Merico, A., Rasconi, M., & Monaco, L. (2021) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-021-02047-1
Desafios da enfermagem em epidemiologia da Covid-19: um estudo bibliométrico
de Assunção, M. C. C., Sá, A. M. M., Valois, R. C., Peixoto, I. V. P., Teles, A. P. D. C., Marcena, J. C., & Pamplona, M. C. D. C. A. (2021) · Research Society and Development · doi:10.33448/rsd-v10i13.20988
Scientometric analysis of the knowledge domain in nanotoxicologyPreprint
Chrishtop, V. V., Prilepskii, A. Y., Nikonorova, V. G., & Mironov, V. (2021) · ChemRxiv · doi:10.26434/chemrxiv-2021-350tk
Meunier, J. (2019) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.736643922.793566636
Asfaw, Z. K., Kalagara, R., Schüpper, A. J., & Choudhri, T. F. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2021.12.120
Megalla, M., Grace, Z. T., Omari, A. M., Sanders, A., Imam, N., Koerner, J. D., Alberta, F. G., & Klein, G. R. (2022) · Cureus · doi:10.7759/cureus.23415 · Cited by 2
“The RCR is calculated as the article citation rate divided by the expected citation rate, which is based on each article’s co-citation network [3].”
MethodReflections following the ASAPbio Peer-Review conferencePreprint
Lidsky, P. V. (2018) · doi:10.31229/osf.io/9sq7t
Boero, F. (2019) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.736643922.793568043
van Osch, G. J. V. M. (2019) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.736643922.793569064
Endow, S. A. (2019) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.736643922.793568662
Grace, Z. T., Patel, H. P., Omari, A. M., Sanders, A., Imam, N., & Koerner, J. D. (2022) · Cureus · doi:10.7759/cureus.25147
“This approach is distinct from traditional bibliometric values in that RCR is based on the co-citation network of the article [1].”
BackgroundMethodZanini, C. P., Manoel, M. B. D., Santo, M. L. D. F. S. D., Moreira, T. O. D. S., Santos, P. C. B., & de Oliveira, E. C. (2022) · Brazilian Applied Science Review · doi:10.34115/basrv6n3-024
Is an Academic Title an Aim or a Device? Publication Productivity of Urologists in Turkiye
Söğütdelen, E., & Akın, G. (2022) · Journal of Urological Surgery · doi:10.4274/jus.galenos.2022.2021.0130
Embracing diversity during researcher evaluation in the academic scientific environment
Riera, R., Quesada-Cabrera, R., Martínez, A., & Dupont, S. T. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1098319
“Recently, a number of newmetrics, frameworks, and proposals have been developed to fill this gap (e.g., Mazumdar et al., 2015; Hutchins et al., 2016).”
BackgroundHaunschild, R., & Bornmann, L. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/mv5fz
Optometry and Vision Science's Top Articles: The First 100 Years
Twa, M. D. (2023) · Optometry and Vision Science · doi:10.1097/opx.0000000000002000
ANSIEDADE NO PERÍODO PRÉ-OPERATÓRIO: UMA ANÁLISE BIBLIOMÉTRICA
Barbosa, L. R., Chaves, É. D. C. L., Assunção, M. R. S., Freitas, P. S., Freire, B. S. D. M., Costa, I. C. P., & Costa, A. C. B. (2023) · Arquivos de Ciências da Saúde da UNIPAR · doi:10.25110/arqsaude.v27i3.2023-004
Noguchi, Y., & Narukawa, M. (2022) · Iryo Yakugaku (Japanese Journal of Pharmaceutical Health Care and Sciences) · doi:10.5649/jjphcs.48.161
How do Cooperation and Scientific Research Influence Drug Development? The Case of Cancer Disease
Li, S. (2023) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/196380
Bezerra, A. G., Steckelberg, R. M. D. B., Steckelberg, D. B., de Paula, J. R., & dos Santos, P. A. (2023) · Revista Virtual de Química · doi:10.21577/1984-6835.20230049
“O Manifesto de Leiden reconhece como boa prática bibliométrica o percentual de citações.(3,70) Os gráficos da Figura 6 que mostram o número de artigos e número de citações e número de artigos e citações normalizadas por ano de publicação na WoS e Scopus permitem uma visualização que dá uma ideia de percentual de citaç…”
O Astrônomo Turco e o papel científico da pesquisa em Arquitetura e Urbanismo
Duarte, C. R., & Santana, E. P. (2023) · Seminário Internacional Projetar (PROJETAR) · doi:10.29327/projetar2023.688774
Analyzing the Science Footprint of Richard P. Baum
Prior, J. O., Lalonde, M. N., Schaefer, N. G., & Schottelius, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-33533-4_25 · Cited by 1
Mendiratta, D., Herzog, I., Liggio, D. F., Ahn, D. B., Vosbikian, M. M., Kaushal, N. K., & Chu, A. (2024) · Journal of Orthopaedic Research® · doi:10.1002/jor.25826
Analysing Research Patterns In Employee Motivation and Performance A Bibliometric Perspective
Zani, R. M., Anuar, A., Ali, S., Akbar, Y. A. A., & Abdullah, F. N. (2024) · Insight Journal · doi:10.24191/ij.v0i0.24311
Zapata, M., Kuramoto, J. R., & Hernández, C. (2024) · Agroindustrial science · doi:10.17268/agroind.sci.2024.02.08
Mahmoud, A., Falcon, D. J., Bustos, V. P., Escobar‐Domingo, M. J., & Lee, B. T. (2024) · Archives of Plastic Surgery · doi:10.1055/a-2380-4278
Climbing the Research Ladder: A 25-year Analysis of K-to-R Grant Conversion among Plastic Surgeons
Arcelona, C. N., Hallman, T. G., Qureshi, U., Gutowski, K. S., Donaldson, R., Figueroa, A. E., & Gosain, A. K. (2024) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000006233
“26,27 In summary, one should not only work productively throughout their K grant project but focus on publication of work once it has reached a level that it will have maximal impact to other researchers within their field.”
Richard S. Cooper, MD in Memoriam
(2024) · Ethnicity & Disease · doi:10.18865/971853
Scientific Journals and the Dynamic Publication Landscape
Fontanarosa, P., Bauchner, H., & Flanagin, A. (2024) · Oxford University Press eBooks · doi:10.1093/med/9780197783030.003.0002
Pal, K., Weiss, C. R., Gade, T. P. F., Glasgow, E., Gill, K., Palacios, V., Doshi, P., Öklü, R., Woodrum, D. A., & Sheth, R. A. (2024) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2024.10.030
deMello, A. J. (2024) · ACS Sensors · doi:10.1021/acssensors.4c03076
The NIH BRAIN Initiative’s Impacts in Systems and Computational Neuroscience, 2014-2023Preprint
Bader, F., Bingham, C. S., David, K. K., Gebrehiwet, H., Lantz, C. L., Peng, G. C. Y., Rangel‐Gomez, M., & Gnadt, J. (2025) · eLife · doi:10.7554/elife.106137.1
The NIH BRAIN Initiative’s Experiment in Team ResearchPreprint
Bader, F., David, K. K., Lantz, C. L., Rangel‐Gomez, M., Gebrehiwet, H., Peng, G. C. Y., & Gnadt, J. (2025) · eLife · doi:10.7554/elife.106136.1
The NIH BRAIN Initiative’s Impacts in Systems and Computational Neuroscience, 2014-2023Preprint
Bader, F., Bingham, C. S., David, K. K., Gebrehiwet, H., Lantz, C. L., Peng, G. C. Y., Rangel‐Gomez, M., & Gnadt, J. (2025) · eLife · doi:10.7554/elife.106137
Author response: The NIH BRAIN Initiative’s Experiment in Team Research
Bader, F., David, K. K., Lantz, C. L., Rangel‐Gomez, M., Gebrehiwet, H., Peng, G. C. Y., & Gnadt, J. (2025) · eLife · doi:10.7554/elife.106136.1.sa0
eLife Assessment: The NIH BRAIN Initiative’s Experiment in Team Research
Nelson, S. B. (2025) · eLife · doi:10.7554/elife.106136.1.sa3
Relative Citation Ratio as an Alternative to H-Index in Academic UrologyPreprint
Shukla, J. J., Patel, R., Rao, M., Gurushankar, M., Cohen, E., Bhandari, V., Davis, M., Jain, K., Alwaal, A., Weiss, R. E., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5211150
Investigation of the publication rate of recent research projects using non-human primates in France
Cash, R., & Courtot, L. (2025) · ALTEX · doi:10.14573/altex.2501292 · Cited by 1
Bader, F., Bingham, C. S., David, K. K., Gebrehiwet, H., Lantz, C. L., Peng, G. C. Y., Rangel‐Gomez, M., & Gnadt, J. (2025) · eLife · doi:10.7554/elife.106136.2
Pandey, A., Kuznetsov, A. S., & Mukhopadhyay, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.29.667503
“On the citation-analysis front, the Relative Citation Ratio (RCR), introduced by Hutchins et al. (2016), provides a field-normalized, article-level impact metric specifically designed to mitigate biases arising from varying citation practices across scientific disciplines.”
Qureshi, U., Hallman, T. G., Arcelona, C. N., Reisner, K. R., Rodriguez, G. C., Bajaj, A., Rai, P., Gutowski, K. S., Donaldson, R., & Gosain, A. K. (2025) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000012427
Puentes, C., Petroff, R., Leeds, H. S., Jacobs, C., Kasperbauer, T. J., Wann, E. G., Winchester, D., Hafez, A. Y., & Paine, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5465015
Górski, B., Nijakowski, K., Mounssif, I., Stefanini, M., & Skurska, A. (2025) · Journal of Functional Biomaterials · doi:10.3390/jfb16100364 · Cited by 1
Senft, N., Acevedo, A. M., Gaysynsky, A., Chou, W. S., D’Angelo, H. M., & Vanderpool, R. C. (2025) · Health Communication · doi:10.1080/10410236.2025.2567513
“…to describe the number of citations and the relative citation ratio, which measures the scientific influence of an article by using its co-citation network (i.e., the papers that it is cited with in reference lists) to field-normalize the number of times it has been cited (Hutchins et al., 2016).”
Letter: Predictive Value of Neurosurgery Applicant Metrics on Resident Academic Productivity
Hemais, M., Borges, P., & Chandra, A. (2025) · Neurosurgery · doi:10.1227/neu.0000000000003848 · Cited by 1
The Use of Bibliometrics for Evaluating Research Impact
Palestine, A. G., Johnson, R. E., Conrady, C. D., Parrish, R. K., II, Federer, L., Rethlefsen, M. L., Coleman, A. L., & Holland, G. N. (2025) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2025.11.025
The Global Landscape of Spine Neuronavigation: From Its Inception to Present
Young, T., Asfaw, Z. K., Gilfillan, L., Lai, S., Choudhri, T. F., & Germano, I. M. (2025) · Clinical Spine Surgery A Spine Publication · doi:10.1097/bsd.0000000000001964
Productivity of National Cancer Institute K Awardees and Their Transition to Independent Funding
Bilalaga, M. M., Almidani, L., Mousa, J., Gaddipati, G. N., Vasireddy, R., Kancharla, P., & Murray, J. C. (2025) · Journal of Cancer Education · doi:10.1007/s13187-025-02799-5
iCite Database Snapshot 2025-02
ICite, Hutchins, B. I., Santangelo, G. M., & Haque, E. (2025) · Figshare · doi:10.35092/yhjc28555016
Self-citationiCite Database Snapshot 2024-10
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc27632943
Self-citationiCite Database Snapshot 2024-04
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc25765794
Self-citationiCite Database Snapshot 2023-08
ICite, Hutchins, B. I., & Santangelo, G. M. (2023) · Figshare · doi:10.35092/yhjc24100917
Self-citationiCite Database Snapshot 2025-03
ICite, Hutchins, B. I., Santangelo, G. M., & Haque, E. (2025) · Figshare · doi:10.35092/yhjc28789178
Self-citationiCite Database Snapshot 2023-07
ICite, Hutchins, B. I., & Santangelo, G. M. (2023) · Figshare · doi:10.35092/yhjc23867964
Self-citationiCite Database Snapshot 2024-03
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc25587399
Self-citationiCite Database Snapshot 2025-04
ICite, Hutchins, B. I., & Santangelo, G. M. (2025) · Figshare · doi:10.35092/yhjc28968326
Self-citationiCite Database Snapshot 2025-01
ICite, Hutchins, B. I., Santangelo, G. M., & Haque, E. (2025) · Figshare · doi:10.35092/yhjc28360103
Self-citationiCite Database Snapshot 2023-06
ICite, Hutchins, B. I., & Santangelo, G. M. (2023) · Figshare · doi:10.35092/yhjc23643690
Self-citationiCite Database Snapshot 2024-07
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc26496313
Self-citationiCite Database Snapshot 2023-05
ICite, Hutchins, B. I., & Santangelo, G. M. (2023) · Figshare · doi:10.35092/yhjc23289794
Self-citationiCite Database Snapshot 2024-12
ICite, Hutchins, B. I., Santangelo, G. M., & Haque, E. (2025) · Figshare · doi:10.35092/yhjc28147979
Self-citationiCite Database Snapshot 2024-05
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc25991908
Self-citationiCite Database Snapshot 2024-08
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc26955625
Self-citationiCite Database Snapshot 2025-06
iCite, Hutchins, B. I., & Santangelo, G. M. (2025) · Figshare · doi:10.35092/yhjc29533829
Self-citationiCite Database Snapshot 2024-09
ICite, Hutchins, B. I., & Santangelo, G. M. (2024) · Figshare · doi:10.35092/yhjc27165129
Self-citationiCite Database Snapshot 2025-08
iCite, Hutchins, B. I., Santangelo, G. M., & Haque, E. (2025) · Figshare · doi:10.35092/yhjc30086656
Self-citationEU-Africa partnerships in health research from 2014 to 2023: Outputs and lessons learnt
Bade, D., Coelho, A., Jajkowicz, D., Pinheiro, H., Nyirenda, T., Makanga, M., & Beattie, P. (2025) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2025.108331 · Cited by 1
Choi, J., Horan, M. R., Nusloch, M. G., Park, Y., Gt, A., Hudson, M. M., Ness, K. K., & Huang, I. (2026) · BMC Public Health · doi:10.1186/s12889-025-26003-x
“Additionally, we calculated the relative citation rate (RCR), defined as an average annual citation rate of the article dividing by the expected citation rate of NIH-funded articles in the same co-citation network published in the same year [29].”
Measuring Research Impact Through Citation Patterns: A Decade of Bariatric Surgery LiteraturePreprint
Wills, M. V., Zhan, K., Barajas-Gamboa, J. S., Mocanu, V., Virk, M. K., Bhogal, M., Kroh, M., Corcelles, R., Navarrete, S., Strong, A. T., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-8523910/v1
Funkhouser, E. M., Lyu, W., Cunha‐Cruz, J., Mungia, R., Gilbert, G. H., & National Dental PBRN Collaborative Group (2026) · Journal of Clinical and Translational Science · doi:10.1017/cts.2026.10721
“Analysis of this scholarly output has demonstrated that its publications have been influential, as judged by their Relative Citation Ratios [7,10,11].”
Shibah, S. R. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6088971
Uppal, S., Khan, H. A., Cruickshank, M. H., & Ghert, M. A. (2026) · PLoS ONE · doi:10.1371/journal.pone.0343827
“While traditional citation counts and h-index have been widely adopted, newer, more nuanced metrics like the Relative Citation Ratio (RCR) have emerged to provide field-and time-normalized assessments of article influence, offering a more robust measure of scientific impact [2,3].”
Siano, F., Segreti, M., Pani, P., Brunamonti, E., & Genovesio, A. (2026) · European Journal of Neuroscience · doi:10.1111/ejn.70464
“This network reflects the article's field, and they used the co-citation network to normalize the citation counts of the evaluated article (Hutchins et al. 2016).”
Nilius, H., Kuster, L., Boss, R., Boschetti, L., Mihalek, N., Ferreira, A. S., Naas, S., Jegerlehner, S. N., Largiadèr, C. R., NAKAS, C., et al. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112270
Allareddy, V., Law, A., Cunha‐Cruz, J., & Gilbert, G. H. (2026) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2026.01.020
Academic Productivity and Demographic Analysis of Adult Reconstruction Fellowship Program Directors
Ganz, M. P., & Scuderi, G. R. (2025) · Archives of Orthopaedics · doi:10.33696/orthopaedics.5.033
Impact Indicators of Scholars Related to Research Topics
Zhou, H., Pazos, J. F. M., Zhang, B., González, J. G., & Chen, Y. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2025-0247
“These metrics are primarily established at: the journal level (e.g., the Journal Impact Factor), the article-level (see, for example, reference (Hutchins et al. 2016)), and the author level (e.g., h-index).”
A Data-Driven Framework for Identifying Influential ResearchersPreprint
Hasan, S. M., Paul, A. K., Chatterjee, K., Dey, P., & Ray, R. S. (2026) · Research Square · doi:10.21203/rs.3.rs-9446863/v1
Crawford, D. C., Vazquez-Hidalgo, E., & Lou, H. (2026) · Journal of Clinical and Translational Science · doi:10.1017/cts.2026.10758
Navigating the transition from basic science researcher to medical educator
Bergemann, A. D., & Haudek, S. B. (2026) · AJP Advances in Physiology Education · doi:10.1152/advan.00012.2026
Merle, J. L., Golden, C. A., Simmons, R., Sloss, E. A., Sanuade, O. A., Verry, T. W., Nixon, A. J., McFarland, M., Lengnick‐Hall, R., Meza, R. D., et al. (2026) · Implementation Science · doi:10.1186/s13012-026-01514-2
“Furthermore, the relative citation ratio, which represents a citation-based measurement of research influence [18], is 28, which is in the 99.7 th percentile compared to all other NIH funded papers [19].”
InfluentialH-index and academic rank in psychiatry: A systematic review and meta-analysis
Maguire, P. A., Allison, S. C., Bastiampillai, T. J., Wilkes, F., & Looi, J. C. (2026) · Australasian Psychiatry · doi:10.1177/10398562261457051
Viswanathan, J., Karamacoska, D., Thakur, L. S., Sawan, M. J., Gómez-Arboledas, Á., Moreno‐González, I., Shaaban, C. E., Milani, S. A., Milinkeviciute, G., Loi, S. M., et al. (2026) · Alzheimer s & Dementia · doi:10.1002/alz.71597
Beyond impact factor: citation skew and article-level influence in sleep medicine journals
Hussain, S., Chowdhury, R., de Oliveira, P. G., Almansouri, A., Alterki, A., & Capasso, R. (2026) · Sleep And Breathing · doi:10.1007/s11325-026-03735-7
“While the present study does not propose a novel metric, article-level indicators such as the Relative Citation Ratio (RCR) provide field-normalized assessment of influ - ence, and author-level measures such as the h-index offer complementary perspectives [21, 22].”
NIH Support for Palliative Care Research: A Scoping Review of Awards and Publications 2014–2024
Dy, S. M., Keller, C., Fazio, E. M., Eldadah, B. A., & Bakos, A. (2026) · Journal of Palliative Medicine · doi:10.1177/10966218261455969
Growing implementation scientists: NHLBI’s K12 programs in heart, lung, blood, and sleep research
Khan, S. B., Lewis, C. C., Mudakala, P. T., Mintzer, K. A., Curry, J. S., & Mensah, G. A. (2026) · Implementation Science Communications · doi:10.1186/s43058-026-01027-5
Acampora, C. D., Munch‐Jurišić, D. M., Denne, S., & Smith, J. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1791225
Thenkabail, P. S., & Wang, D. (2026) · Remote Sensing · doi:10.3390/rs18142267
Is “Top 100 most cited” reliable to identify influential articles? Example of OTSR
Devos, P., Maroteau, G., & Dаrtus, J. (2026) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2026.104795
Metadata
- Authors
- Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Hutchins, B. Ian
- Yuan, Xin
- Anderson, James M.
- Santangelo, George M.
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-09-06
- Volume
- 14
- Issue
- 9
- Pages
- e1002541
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5012559
- OpenAlex
- W2953304564
- Cited by
- 500
- References
- 91
- Citations by year
- 2026: 43 · 2025: 51 · 2024: 52 · 2023: 62 · 2022: 68 · 2021: 48 · 2020: 44 · 2019: 45 · 2018: 36 · 2017: 32 · 2016: 17
- FWCI
- 40.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS Biology (publishedVersion, public-domain)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (573871/2008-6)
- Fundação de Amparo à Pesquisa do Estado de Minas Gerais (573871/2008-6)
- National Institutes of Health (Z99 OD999999)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- relative citation ratio, citation analysis, co-citation networks, field normalization, iCite
- MeSH terms
- Authorship, Humans, National Institutes of Health (U.S.), United States, Bibliometrics, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.