Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
The Center for Science and Technology Studies at Leiden University advocates the use of specific normalizations for assessing research performance with reference to a world average. The Journal Citation Score (JCS) and Field Citation Score (FCS) are averaged for the research group or individual researcher under study, and then these values are used as denominators of the (mean) Citations per publication (CPP). Thus, this normalization is based on dividing two averages. This procedure only generates a legitimate indicator in the case of underlying normal distributions. Given the skewed distributions under study, one should average the observed versus expected values which are to be divided first for each publication. We show the effects of the Leiden normalization for a recent evaluation where we happened to have access to the underlying data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Garfield, E. (1972) · Science · doi:10.1126/science.178.4060.471 · Cited by 2,788
Universality of citation distributions: Toward an objective measure of scientific impact
Radicchi, F., Fortunato, S., & Castellano, C. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0806977105 · Cited by 828 · cited 2×
“Furthermore, our proposal accords with recent suggestions to collapse citation distributions in different fields by normalizing to the z-score (Bornmann & Daniel, 2009) or the relative indicator cf = c / c0, as proposed by Radicchi et al. (2008). In the latter formula, c is the citation rate for a single paper and c0…”
MethodNigel Gilbert, G. (1977) · Social Studies of Science · doi:10.1177/030631277700700112 · Cited by 691
The use of bibliometric data for the measurement of university research performance
Moed, H. F., Burger, W. J. M., Frankfort, J. G., & Van Raan, A. F. J. (1985) · Research Policy · doi:10.1016/0048-7333(85)90012-5 · Cited by 472 · cited 2×
“…Cardiology Group, Heart Failure Research Center, Academic Medical Center AMC, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands. b Department of Medical Physiology, University Medical Center Utrecht, Utrecht, The Netherlands. c Amsterdam School of Communications Research (ASCoR), University of…”
BackgroundResult
Cited by
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257
A Review of Theory and Practice in Scientometrics
Mingers, J., & Leydesdorff, L. (2015) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.04.002 · Cited by 1,002
“This is the traditional “crown indicator”, but this approach to normalisation has been criticised (Leydesdorff & Opthof, 2011; Lundberg, 2007; Opthof & Leydesdorff, 2010) and an alternative has been used in several cases (Cambell et al., 2008; Rehn & Kronman, 2008; Van Veller et al., 2009).”
MethodHutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500
Towards a new crown indicator: Some theoretical considerations
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011) · Journal of Informetrics · doi:10.1016/j.joi.2010.08.001 · Cited by 424
Metadata
- Authors
- Opthof, T., & Leydesdorff, L.
- Venue
- Journal of Informetrics
- Publisher
- Elsevier BV
- Published
- 2010-07
- Volume
- 4
- Issue
- 3
- Pages
- 423-430
- ISSN
- 1751-1577 (ISSN-L)
- OpenAlex
- W2105806663
- Cited by
- 190
- References
- 4
- FWCI
- 32.75
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.