Citation Metrics: A Primer on How (Not) to Normalize
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Citation metrics are increasingly used to appraise published research. One challenge is whether and how to normalize these metrics to account for differences across scientific fields, age (year of publication), type of document, database coverage, and other factors. We discuss the pros and cons for normalizations using different approaches. Additional challenges emerge when citation metrics need to be combined across multiple papers to appraise the corpus of scientists, institutions, journals, or countries, as well as when trying to attribute credit in multiauthored papers. Different citation metrics may offer complementary insights, but one should carefully consider the assumptions that underlie their calculation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Increasing Dominance of Teams in Production of Knowledge
Wuchty, S., Jones, B. F., & Uzzi, B. (2007) · Science · doi:10.1126/science.1136099 · Cited by 3,615
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828
Self-citationUniversality of citation distributions: Toward an objective measure of scientific impact
Radicchi, F., Fortunato, S., & Castellano, C. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0806977105 · Cited by 828
Measuring Contextual Citation Impact of Scientific Journals
Moed, H. F. (2010) · Journal of Informetrics · doi:10.1016/j.joi.2010.01.002 · Cited by 628 · cited 1×
“, based on citing or cited papers or on the references of citing or cited papers) [11].”
BackgroundHutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500 · cited 2×
“Table 1 summarizes factors that may be considered or debated during normalization and shows for comparison two systems, the RCR method [13] and the normalization used in the Leiden Ranking of universities (http://www.”
MethodDifferences in impact factor across fields and over time
Althouse, B. M., West, J. D., Bergstrom, C. T., & Bergstrom, T. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20936 · Cited by 270 · cited 1×
“Other things being equal, an influential paper in a theoretical mathematics field where reference lists are traditionally short may attract fewer citations than influential papers in fields where papers typically have much longer reference lists [6].”
BackgroundThe incidence and role of negative citations in science
Catalini, C., Lacetera, N., & Oettl, A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1502280112 · Cited by 189 · cited 1×
“Text mining approaches may also allow giving different weight to citations based on the context in which they are made, how often they are mentioned in the citing paper, and whether they are favorable or critical of the work being cited [21,22].”
BackgroundA Systematic Empirical Comparison of Different Approaches for Normalizing Citation Impact Indicators
Waltman, L., & van Eck, N. J. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.08.002 · Cited by 153
Cited by
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257
“Ideally, one wants citation indicators to measure impact in a monotonic fashion: the higher the scores, the “better” the paper (Ioannidis et al., 2016).”
BackgroundSelf-citationAssessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462
“However, the value of different normalisations needs further study [50] and there has been criticism of the proposed RCR [51].”
BackgroundSelf-citationLeydesdorff, L., Wouters, P., & Bornmann, L. (2016) · Scientometrics · doi:10.1007/s11192-016-2150-8 · Cited by 173
Self-citationLearning the local landscape of protein structures with convolutional neural networks
(2021) · doi:10.1007/s10867-021-09593-6 · Cited by 23
Campanario, J. M. (2018) · Scientometrics · doi:10.1007/s11192-018-2660-7 · Cited by 9
Metadata
- Authors
- Ioannidis, J. P. A., Boyack, K., & Wouters, P. F.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-09-06
- Volume
- 14
- Issue
- 9
- Pages
- e1002542
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- OpenAlex
- W2513297789
- Cited by
- 87
- References
- 8
- FWCI
- 9.17
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Citation Metrics: A Primer on How (Not) to Normalize. https://app.cochat.ai/papers/10.1371/journal.pbio.1002542
- The paper
- Ioannidis, J. P. A., Boyack, K., & Wouters, P. F. (2016). Citation Metrics: A Primer on How (Not) to Normalize. PLOS Biology. https://doi.org/10.1371/journal.pbio.1002542
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.