Citation rates and journal impact factors are not suitable for evaluation of research
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
(1998). Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthopaedica Scandinavica: Vol. 69, No. 3, pp. 224-229.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, American authors are particularly prone to cite each other, raising the citation rate of American science 30% above the world average [6].”
From Citation indexes do not reflect methodological quality in lung cancer randomised trials. (2003) · cites this paper as [6]
In the citing paper
“In the ISI database, more than half of all citations concern American scientists (language bias).”
From Citation indexes do not reflect methodological quality in lung cancer randomised trials. (2003) · cites this paper as [6]
In the citing paper
“Also, the value of citation factors is dependent on the research field, which requires similar units to be applied when IFs are used for comparison.”
From Citation indexes do not reflect methodological quality in lung cancer randomised trials. (2003) · cites this paper as [6]
In the citing paper
“But, in the ISI database, there is a bias towards journals published in English; for example, only two German social science journals are included in the ISI database in comparison with 542 in a German database [6].”
From Citation indexes do not reflect methodological quality in lung cancer randomised trials. (2003) · cites this paper as [6]
In the citing paper
“Nevertheless, citation factors do not represent the individual citation rate of a paper given that only a few articles account for the majority of citations [2,6], and uncited articles have the same IF as frequently cited ones.”
From Citation indexes do not reflect methodological quality in lung cancer randomised trials. (2003) · cites this paper as [6]
Citations
Cites
Garfield, E. (1972) · Science · doi:10.1126/science.178.4060.471 · Cited by 2,788
Problems of citation analysis: A critical review
MacRoberts, M. H., & MacRoberts, B. R. (1989) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198909)40:5<342::aid-asi7>3.0.co;2-u · Cited by 803
Seglen, P. O. (1992) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199210)43:9<628::aid-asi5>3.0.co;2-0 · Cited by 693
Chance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623
Sense and nonsense about the impact factor
Opthof, T. (1997) · Cardiovascular Research · doi:10.1016/s0008-6363(96)00215-5 · Cited by 286
Structure of the Biomedical Literature
Narin, F., Pinski, G., & Gee, H. H. (1976) · Journal of the American Society for Information Science · doi:10.1002/asi.4630270104 · Cited by 244
Causal relationship between article citedness and journal impact
Seglen, P. O. (1994) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199401)45:1<1::aid-asi1>3.0.co;2-y · Cited by 239
Self-citationBibliometric performance measures
Narin, F., & Hamilton, K. S. (1996) · Scientometrics · doi:10.1007/bf02129596 · Cited by 232
Moed, H. F., & van Leeuwen, T. N. (1996) · Nature · doi:10.1038/381186a0 · Cited by 226
Research Papers: Who's Uncited Now?
Hamilton, D. P. (1991) · Science · doi:10.1126/science.1986409 · Cited by 218
Evaluation of some methods for the relative assessment of scientific publications
Vinkler, P. (1986) · Scientometrics · doi:10.1007/bf02026039 · Cited by 158
Citation Analysis: A New Tool for Science Administrators
Wade, N. (1975) · Science · doi:10.1126/science.188.4187.429 · Cited by 144
Quotational and Reference Accuracy in Surgical Journals: A Continuing Peer Review Problem
Evans, J. T. (1990) · JAMA · doi:10.1001/jama.1990.03440100059009 · Cited by 129
Moed, H. F., Burger, W. J. M., Frankfort, J. G., & van Raan, A. F. J. (1985) · Scientometrics · doi:10.1007/bf02016935 · Cited by 124
The scientometric weight of 50 nations in 27 science areas, 1989–1993. Part II. Life sciences
Braun, T., Glänzel, W., & Grupp, H. (1995) · Scientometrics · doi:10.1007/bf02020421 · Cited by 81
Validation of bibliometric indicators in the field of mathematics
Korevaar, J. C. (1996) · Scientometrics · doi:10.1007/bf02093488 · Cited by 73
Moed, H. F., van Leeuwen, T. N., & Reedijk, J. (1996) · Scientometrics · doi:10.1007/bf02093487 · Cited by 69
The SCI Journal Citation Reports: A potential tool for studying journals?
Magri, M.-H., & SOLARI, A. (1996) · Scientometrics · doi:10.1007/bf02018235 · Cited by 58
Some methodological problems in ranking scientists by citation analysis
Folly, G., Hajtman, B., Nagy, J. I., & Ruff, I. (1981) · Scientometrics · doi:10.1007/bf02025636 · Cited by 53
How Authors Select Journals: A Test of the Reward Maximization Model of Submission Behaviour
Gordon, M. D. (1984) · Social Studies of Science · doi:10.1177/030631284014001003 · Cited by 47
Structural Determinants of Stratification in Science
Hargens, L. L., & Felmlee, D. (1984) · American Sociological Review · doi:10.2307/2095425 · Cited by 46
Moed, H. F., Burger, W. J. M., Frankfort, J. G., & van Raan, A. F. J. (1987) · 研究 技術 計画 · Cited by 44
Ernst, E., Saradeth, T., & Resch, K. L. (1993) · Nature · doi:10.1038/363296a0 · Cited by 44
Model for quantitative selection of relative scientometric impact indicators
Vinkler, P. (1996) · Scientometrics · doi:10.1007/bf02017315 · Cited by 39
Metcalfe, N. B. (1995) · Nature · doi:10.1038/376720b0 · Cited by 37
Lange, L. L. (1985) · Scientometrics · doi:10.1007/bf02016936 · Cited by 36
Standards issues in a national bibliometric database: The Australian case
Bourke, P., & Butler, L. (1996) · Scientometrics · doi:10.1007/bf02018478 · Cited by 28
Sengupta, I. N. (1986) · Scientometrics · doi:10.1007/bf02016772 · Cited by 24
Møller, A. P. (1990) · Nature · doi:10.1038/348480c0 · Cited by 22
Vinkler, P. (1996) · Scientometrics · doi:10.1007/bf02016908 · Cited by 21
THE LAW OF CONSTANT CITATION FOR SCIENTIFIC LITERATURE
Gomperts, M. C. (1968) · Journal of Documentation · doi:10.1108/eb026448 · Cited by 19
Lewison, G., Anderson, J. L., & Jack, J. (1995) · Nature · doi:10.1038/377671a0 · Cited by 18
When more is less: Quantity versus quality of publications in the evaluation of academic vitae.
Hayes, S. C. (1983) · American Psychologist · doi:10.1037/0003-066x.38.12.1398 · Cited by 18
Science indicators derived from databases: The case of the social sciences
Artus, H. M. (1996) · Scientometrics · doi:10.1007/bf02093626 · Cited by 16
When more is less: Quantity versus quality of publications in the evaluation of academic vitae.
Hayes, S. C. (1983) · American Psychologist · doi:10.1037//0003-066x.38.12.1398 · Cited by 15
Impact factors of Russian journals
Sorokin, N. I. (1996) · Nature · doi:10.1038/380578c0 · Cited by 9
Bruk av siteringsanalyse og andre bibliometriske metoder i evaluering av forskningskvalitet.
Seglen, P. O. (1989) · Tidsskrift for Den Norske Laegeforening · Cited by 9
Self-citationPedersen, M. N. (1996) · Politica · doi:10.7146/politica.v28i1.68000 · Cited by 9
Tidsskriftspublicering og forskningsevaluering
Andersen, H. (1996) · Biblioteksarbejde · doi:10.7146/bibarb.v0i48.102743 · Cited by 4
Semenza, G. (1996) · FEBS Letters · doi:10.1016/s0014-5793(96)90932-1 · Cited by 3
Quinn, T. (1990) · Nature · doi:10.1038/348480a0 · Cited by 2
Research works printed but not linked: 6
- Hansen H F, 1995, Frederiksberg, Denmark, Samfundslitteratur;, 1
- Andersen H, 1996, Dansk Sociologi, 2, 72
- Seglen P O, 1989, NOP-Nytt (Helsingfors), 15, 2
- Seglen P O, 1989, Tidsskr Nor Laegeforen, 31, 3229
- Seglen P O, 1992, Representations of Science and Technology, 240
- Metcalfe N B, 1995, Nature, 377, 260
Cited by
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257
“For example, Seglen (1998) examined problems attached to citation analyses and concluded that “. . . citation rates are determined by so many technical factors that it is doubtful whether pure scientific quality has any detectible effect at all . . .” (p. 226) Broadly speaking, while extensive…”
BackgroundScholarly communication and bibliometrics
Borgman, C. L., & Furner, J. (2002) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440360102 · Cited by 1,121
“…and MacRoberts (1987) and MacRoberts and MacRoberts (1996)); Peritz (1992) offered a succinct summary of “problems of theory and method;” and recently Seglen has assumed the MacRoberts’ mantle as perhaps the most prolific contributor to the critical viewpoint (see, for example, Seglen (1998)).”
BackgroundMeho, L. I., & Yang, K. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20677 · Cited by 974
“Critics, on the other hand, claim that citation counting has serious problems or limitations that affect its validity (MacRoberts & MacRoberts, 1996; Seglen, 1998).”
Domain analysis in information science: Eleven approaches – traditional as well as innovative
Hjørland, B. (2002) · Journal of Documentation · doi:10.1108/00220410210431136 · Cited by 481
The rise and rise of citation analysis
Meho, L. I. (2007) · Physics World · doi:10.1088/2058-7058/20/1/33 · Cited by 371
Metadata
- Authors
- Seglen, P. O.
- Affiliations
- Nordic Institute for Studies in Innovation, Research and Education (NO)
- Corresponding author
- Seglen, Per O
- Venue
- Acta Orthopaedica Scandinavica
- Published
- 1998-01
- Volume
- 69
- Issue
- 3
- Pages
- 224-229
- ISSN
- 0001-6470 (ISSN-L)
- OpenAlex
- W2014147337
- Semantic Scholar
- ce5741bb0be1ba3fd9ad2c2f94496538cbd470e6
- Cited by
- 240
- References
- 42
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 11 · 2023: 8 · 2022: 13 · 2021: 13 · 2020: 18 · 2019: 10 · 2018: 18 · 2017: 17 · 2016: 13 · 2015: 12 · 2014: 2 · 2013: 11 · 2012: 7
- FWCI
- 0.52
- Type
- journal article
- Language
- en
- Open access
- gold
- actaorthop.org
- Licence
- cc-by-nc
- Locations
- Acta Orthopaedica Scandinavica (publishedVersion)
- Acta Orthopaedica (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- citation counts, journal impact factor, research evaluation, bibliometric analysis, orthopaedic research
- MeSH terms
- Evaluation Studies as Topic, Periodicals as Topic, Research, Bibliometrics, Bias, Journalism, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.