The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Do Rankings Reflect Research Quality?
Open accessAbstract
Publication and citation rankings have become major indicators of the scientific worth of universities and determine to a large extent the career of individual scholars. Such rankings do not effectively measure research quality, which should be the essence of any evaluation. These quantity rankings are not objective; two citation rankings, based on different samples, produce entirely different results. For that reason, an alternative ranking is developed as a quality indicator, based on membership on academic editorial boards of professional journals. It turns out that the ranking of individual scholars based on that measure is far from objective. Furthermore, the results differ markedly, depending on whether research quantity or quality is considered. Thus, career decisions based on rankings are dominated by chance and do not reflect research quality. We suggest that evaluations should rely on multiple criteria. Public management should return to approved methods such as engaging independent experts who in turn provide measurements of research quality for their research communities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720 · cited 1×
“This has been called the “Matthew Effect” in science (Merton, 1968).”
Multitask principal – agent analyses: Incentive contracts, asset ownership, and job design
Holmstrom, B., & Milgrom, P. (1991) · The Journal of Law, Economics, and Organization · doi:10.1093/jleo/7.special_issue.24 · Cited by 6,046
Toward a new economics of science
Partha, D., & David, P. A. (1994) · Research Policy · doi:10.1016/0048-7333(94)01002-1 · Cited by 2,448
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
The influence of management journals in the 1980s and 1990s
Podsakoff, P. M., MacKenzie, S. B., Bachrach, D. G., & Podsakoff, N. P. (2005) · Strategic Management Journal · doi:10.1002/smj.454 · Cited by 912 · cited 1×
“6 Many journal rankings according to citations have been undertaken (e.g. Cheng et al., 1995; Diamond, 1989; Laband and Piette, 1994; Liebowitz and Palmer, 1984; Paul, 2004; Podsakoff et al., 2005; Sombatsompop et al., 2004).”
The use of bibliometric data for the measurement of university research performance
Moed, H. F., Burger, W. J. M., Frankfort, J. G., & Van Raan, A. F. J. (1985) · Research Policy · doi:10.1016/0048-7333(85)90012-5 · Cited by 472
The rise and rise of citation analysis
Meho, L. I. (2007) · Physics World · doi:10.1088/2058-7058/20/1/33 · Cited by 371
Stochastic modeling of citation slips
Simkin, M. V., & Roychowdhury, V. P. (2005) · Scientometrics · doi:10.1007/s11192-005-0028-2 · Cited by 135
A mathematical theory of citing
Simkin, M. V., & Roychowdhury, V. P. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20653 · Cited by 93
Cited by
The strength of strong ties in the creation of innovation
Rost, K. (2011) · Research Policy · doi:10.1016/j.respol.2010.12.001 · Cited by 430
Self-citationCeilings or floors? Gender wage gaps by education in Spain
de la Rica, S., Dolado, J. J., & Llorens, V. (2008) · Journal of Population Economics · doi:10.1007/s00148-006-0128-1 · Cited by 281
Metadata
- Authors
- Frey, B. S., & Rost, K.
- Venue
- Journal of Applied Economics
- Publisher
- Informa UK Limited
- Published
- 2010-05
- Volume
- 13
- Issue
- 1
- Pages
- 1-38
- ISSN
- 1514-0326
- 1667-6726
- OpenAlex
- W3125468813
- Cited by
- 170
- References
- 9
- FWCI
- 15.55
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do Rankings Reflect Research Quality? https://app.cochat.ai/papers/10.1016/s1514-0326(10)60002-5
- The paper
- Frey, B. S., & Rost, K. (2010). Do Rankings Reflect Research Quality? Journal of Applied Economics. https://doi.org/10.1016/s1514-0326(10)60002-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.