Use of the journal impact factor for assessing individual articles: Statistically flawed or not?
Open accessAbstract
Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. Using computer simulations, we demonstrate that under certain conditions the number of citations an article has received is a more accurate indicator of the value of the article than the impact factor. However, under other conditions, the impact factor is a more accurate indicator. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · cited 9× · see citing works
“technical and statistical issues in the calculation of the IF (e.g., Glänzel & Moed, 2002; Seglen, 1997), such as the definition of so-called ‘citable items’ in the denominator of the IF (e.”
InfluentialWhat do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · cited 1× · see citing works
“For further discussion on the conceptual foundation of citation analysis, we refer to Bornmann & Daniel (2008), De Bellis (2009, Chapter 7), Moed (2005, Chapters 15–17), Nicolaisen (2007), Tahamtan & Bornmann (2019).”
Editors, T. P. M. (2006) · PLoS Medicine · doi:10.1371/journal.pmed.0030291 · Cited by 373 · see citing works
Tahamtan, I., & Bornmann, L. (2019) · Scientometrics · doi:10.1007/s11192-019-03243-4 · Cited by 236 · see citing works
Editors’ JIF-boosting stratagems – Which are appropriate and which not?
Martin, B. R. (2016) · Research Policy · doi:10.1016/j.respol.2015.09.001 · Cited by 128 · see citing works
CiteScore Metrics: Creating Journal Metrics from the Scopus Citation Index
James, C., Colledge, L., Meester, W., Azoulay, N., & Plume, A. (2019) · Learned Publishing · doi:10.1002/leap.1246 · Cited by 63 · see citing works
A critical evaluation of the algorithm behind the Relative Citation Ratio (RCR)
Janssens, A. C. J. W., Goodman, M., Powell, K. R., & Gwinn, M. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002536 · Cited by 42 · see citing works
Flagged first, then by citations
- Authors
- Waltman, L., & Traag, V. A.
- Affiliations
- Leiden University (NL)
- Corresponding author
- Waltman, Ludo
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2021-03-01
- Volume
- 9
- Pages
- 366
- ISSN
- 2046-1402 (ISSN-L)
- PMC
- PMC7974631
- OpenAlex
- W3015411418
- Cited by
- 97
- References
- 7
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 19 · 2023: 12 · 2022: 17 · 2021: 13 · 2020: 6 · 2019: 6 · 2018: 1 · 2017: 2
- FWCI
- 6.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- F1000Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Universiteit Leiden
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- journal impact factor, citation analysis, San Francisco Declaration on Research Assessment, research evaluation, citation counts, bibliometric analysis
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Bibliometrics, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
