A critical evaluation of the algorithm behind the Relative Citation Ratio (RCR)
Open accessAbstract
The influence of scientific publications is increasingly assessed using quantitative approaches, but most available metrics have limitations that hamper their utility [1].Hutchins and colleagues recently proposed the Relative Citation Ratio (RCR) [2], which compares the citation rate of an article against the citation rate that is expected for its field.The metric is an attractive and intuitive solution to indicate whether an article is cited more or less frequently than its "peer" publications.As a ratio of rates, RCR is an article-level metric that is field and time independent and strongly correlated with peer review evaluations [2]; the metric was central in the proposed (and withdrawn) grant management policy of the National Institutes of Health (NIH) [3] even though the RCR has been criticized for lacking a theoretical model, having insufficient transparency, and having poor correlation with other peer evaluations [4,5].We analyzed the algorithm behind the RCR and report several concerns about the calculation of the metric that may limit its utility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The RCR has performed better than journal impact factor, citations per year, or the Thompson-Reuters ratio as a method to quantify publication quality or its impact on the field.
“As one of several methods to quantify publication quality or its impact on the field, the RCR has performed better than journal impact factor, citations per year, or the Thompson–Reuters ratio [11,26,27].”
From Structure, function, and productivity from the National Dental Practice-Based Research Network. (2022) · cites this paper as [27]
The introduction (page not provided) describes RCR as an article-level metric for assessing a publication’s influence and says it is strongly correlated with peer-review evaluations. It does not report that RCR performed better than journal impact factor, citations per year, or the Thompson–Reuters ratio. The Concluding remarks instead raise doubts about RCR’s validity and call for further research. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Co‐citation in the scientific literature: A new measure of the relationship between two documents
Small, H. (1973) · Journal of the American Society for Information Science · doi:10.1002/asi.4630240406 · Cited by 5,345 · see citing works
Garfield, E. (1972) · Science · doi:10.1126/science.178.4060.471 · Cited by 2,788 · cited 1× · see citing works
“The first citation, Rousseau and Leydesdorff [10], presents a formula for the “n-year synchronous journal impact factor” and the authors write that this formula “when n = 2. . .obtains the classical Garfield (1972) journal impact factor.””
BackgroundA review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · see citing works
Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500 · see citing works
Parolo, P. D. B., Pan, R. K., Ghosh, R., Huberman, B. A., Kaski, K. K., & Fortunato, S. (2015) · Journal of Informetrics · doi:10.1016/j.joi.2015.07.006 · Cited by 180 · see citing works
The effect of “open access” on citation impact: An analysis of ArXiv's condensed matter section
Moed, H. F. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20663 · Cited by 164 · see citing works
Novel citation-based search method for scientific literature: application to meta-analyses
Janssens, A. C. J. W., & GWINN, M. L. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0077-z · Cited by 56 · see citing works
Self-citationBornmann, L., & Haunschild, R. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23729 · Cited by 26 · see citing works
The quiet rise of the NIH’s hot new metric
Naik, G. (2016) · Nature · doi:10.1038/539150a · Cited by 15 · see citing works
Flagged first, then by citations
2 of the 11 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [11]Garfield E. Citation analysis as a tool in journal evaluation. Science. 1972; 178(4060): 471– 9.. 5079701
Cited by
Sorted by citations
Assessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462 · see citing works
Dimensions: A competitor to Scopus and the Web of Science?
Thelwall, M. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.03.006 · Cited by 294 · see citing works
Purkayastha, A., Palmaro, E., Falk-Krzesinski, H. J., & Baas, J. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.03.012 · Cited by 195 · see citing works
Field Normalization of Scientometric Indicators
Waltman, L., & van Eck, N. J. (2019) · Springer handbooks · doi:10.1007/978-3-030-02511-3_11 · Cited by 103 · see citing works
Scientific productivity: An exploratory study of metrics and incentives
Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195321 · Cited by 58 · see citing works
Structure, function, and productivity from the National Dental Practice-Based Research Network.
Gilbert, G. H., Fellows, J. L., Allareddy, V., Cochran, D. L., Cunha-Cruz, J., Gordan, V. V., McBurnie, M. A., Meyerowitz, C., Mungia, R., Rindal, D. B., et al. (2022) · Journal of Clinical and Translational Science · doi:10.1017/cts.2022.421 · Cited by 54 · see citing works
Requiem for impact factors and high publication charges
Triggle, C. R., MacDonald, R. S., Triggle, D. J., & Grierson, D. S. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1909481 · Cited by 49 · see citing works
The relative citation ratio: what is it and why should medical librarians care?
Surkis, A., & Spore, S. (2018) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2018.499 · Cited by 45 · see citing works
Critical Rationalism and the Search for Standard (Field-Normalized) Indicators in Bibliometrics
Bornmann, L., & Marx, W. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.05.002 · Cited by 41 · see citing works
Additional support for RCR: A validated article-level measure of scientific influence
Hutchins, B. I., Hoppe, T. A., Meseroll, R. A., Anderson, J. M., & Santangelo, G. M. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003552 · Cited by 36 · see citing works
Garstka, M., Randolph, G. W., Haddad, A. B., Nathan, C. O., Ibraheem, K., Farag, M., Deot, N., Adib, H., Hoof, M. A., French, K., et al. (2019) · Head & Neck · doi:10.1002/hed.25913 · Cited by 29 · see citing works
Current Trends in Subspecialty Fellowship Training for 1691 Academic Neurological Surgeons
Gupta, A., Reddy, V. P., Barpujari, A., Lavadi, R. S., Agarwal, P., Chang, Y., Mooney, J. H., Elsayed, G., & Agarwal, N. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2022.11.074 · Cited by 22 · see citing works
Impacting Capabilities: A Conceptual Framework for the Social Value of Research
Hicks, D., Stahmer, C. G., & Smith, M. (2018) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2018.00024 · Cited by 18 · see citing works
On the interplay between normalisation, bias, and performance of paper impact metrics
Dunaiski, M., Geldenhuys, J., & Visser, W. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.01.003 · Cited by 18 · see citing works
A brief review on article-, author-, and journal-level scientometric indices
Mondal, H., & Mondal, S. (2022) · Indian Dermatology Online Journal · doi:10.4103/idoj.idoj_729_21 · Cited by 20 · see citing works
A bibliometric and visual analysis of obesity and polycystic ovary syndrome from 2012 to 2022
Luo, P., Li, J., Li, P., Wang, G., Li, W., Song, Z. J., Sun, X., Fu, Z., Zhou, H., Yi, X., et al. (2022) · Frontiers in Endocrinology · doi:10.3389/fendo.2022.1011105 · Cited by 12 · see citing works
Pizzo, D. M., Imam, N., Sudah, S. Y., Faccone, R. D., Akinrodoye, M. A., Manzi, J. E., Menendez, M. E., & Gesell, M. W. (2022) · The Journal of Arthroplasty · doi:10.1016/j.arth.2022.07.022 · Cited by 12 · see citing works
Davis, F. M., Obi, A. T., Gallagher, K. A., & Henke, P. K. (2019) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2019.09.036 · Cited by 11 · see citing works
Interdisciplinarity Metric Based on the Co-Citation Network
Hernández, J. M., & Dorta–González, P. (2020) · Mathematics · doi:10.3390/math8040544 · Cited by 10 · see citing works
Development of science and education in the Western Balkan countries: competitiveness with the EU
Živković, Ž., & Panić, M. (2020) · Scientometrics · doi:10.1007/s11192-020-03554-x · Cited by 10 · see citing works
Globalised vs averaged: Bias and ranking performance on the author level
Dunaiski, M., Geldenhuys, J., & Visser, W. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.01.006 · Cited by 8 · see citing works
Decoding the nuances of scholarly output and publication metrics in orthodontics
Gandedkar, N. H., Allareddy, V., & Vaiid, N. (2024) · Seminars in Orthodontics · doi:10.1053/j.sodo.2024.05.013 · Cited by 6 · see citing works
Jiang, T., Zhang, X., Wang, H., & Yi, X. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-089233 · Cited by 6 · see citing works
Levin, G., Pareja, R., Harrison, R. F., Ramirez, P. T., & Meyer, R. (2023) · International Journal of Gynecological Cancer · doi:10.1136/ijgc-2023-004380 · Cited by 7 · see citing works
The relative citation ratio (RCR) as a novel bibliometric among 2511 academic orthopedic surgeons
Gupta, A., Meeter, A. M., Norin, J., Ippolito, J. A., & Beebe, K. S. (2022) · Journal of Orthopaedic Research® · doi:10.1002/jor.25490 · Cited by 5 · see citing works
Citation Metrics: A Philosophy of Science Perspective
Lisciandra, C. (2025) · Episteme · doi:10.1017/epi.2024.46 · Cited by 4 · see citing works
Revercomb, L., Patel, A. M., Tripathi, O. B., & Filimonov, A. (2024) · The Laryngoscope · doi:10.1002/lary.31408 · Cited by 4 · see citing works
Seppänen, J., Värri, H., & Ylönen, I. (2022) · Scientometrics · doi:10.1007/s11192-022-04393-8 · Cited by 2 · see citing works
Patel, P. A., Keane, C. A., Akhter, M. F., Fang, A. H., Soto, E., & Boyd, C. J. (2023) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000003567 · Cited by 2 · see citing works
Use of the journal impact factor for assessing individual articles: Statistically flawed or not?
Waltman, L., & Traag, V. A. (2021) · F1000Research · doi:10.12688/f1000research.23418.2 · Cited by 97 · see citing works
Assessing the Research Impact of U.S. Academic Uveitis Specialists Using the Relative Citation Ratio
Franco, J. J., Shapiro, J. N., Saukkonen, D., Azer, D. P., Roth, A. K., & Kodati, S. (2025) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2025.11.029 · Cited by 1 · see citing works
Conway, J., Ricardo, J. W., & Lipner, S. R. (2023) · Dermatology Practical & Conceptual · doi:10.5826/dpc.1303a162 · Cited by 1 · see citing works
fCite : a fractional citation tool to quantify an individual’s scientific research outputPreprint
Kozlowski, L. P. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/771485 · Cited by 2 · see citing works
Rogers, J. L., Barpujari, A., & Reddy, V. P. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2021.11.132 · Cited by 1 · see citing works
As vascular surgeons, we need to punch above our weight
Schanzer, A. S. (2020) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2019.10.023
Asfaw, Z. K., Kalagara, R., Schüpper, A. J., & Choudhri, T. F. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2021.12.120
Reflections following the ASAPbio Peer-Review conferencePreprint
Lidsky, P. V. (2018) · doi:10.31229/osf.io/9sq7t
Grace, Z. T., Patel, H. P., Omari, A. M., Sanders, A., Imam, N., & Koerner, J. D. (2022) · Cureus · doi:10.7759/cureus.25147
A Bibliometric and Visual Analysis of Obesity and Gastroesophageal Reflux DiseasePreprint
Tan, H., Luo, P., & Wang, G. (2023) · Research Square · doi:10.21203/rs.3.rs-3707082/v1
Al-Kattan, W., Terkawi, A., Bakir, M. A., Mufti, M. A., & Hamweyah, K. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1829439
- Authors
- Janssens, A. C. J. W., Goodman, M., Powell, K. R., & Gwinn, M.
- Affiliations
- Emory University (US)
- Corresponding author
- Janssens, A. Cecile J. W.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-10-02
- Volume
- 15
- Issue
- 10
- Pages
- e2002536
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC5624566
- OpenAlex
- W2760963112
- Cited by
- 42
- References
- 11
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 3 · 2023: 4 · 2022: 10 · 2021: 3 · 2020: 4 · 2019: 7 · 2018: 7 · 2017: 1
- FWCI
- 4.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (R01LM012372)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- relative citation ratio, citation metrics, bibliometric analysis, peer review, research evaluation, publication impact
- MeSH terms
- Algorithms, Bibliometrics, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
