Effectiveness of Journal Ranking Schemes as a Tool for Locating Information
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The rise of electronic publishing, preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer immediate, reliable information on the value of that research. METHODOLOGY/PRINCIPAL FINDINGS: Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data. CONCLUSIONS/SIGNIFICANCE: Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276 · cited 1×
“For a set of journals F~fJxg and a journal ranking R, we define the multi-class AUC statistic M(F,R) as [27]”
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 1×
“For example, because social influence may affect the citations to a paper, small differences in quality may lead to large differences in the number of citations [18].”
Background
Cited by
Atypical Combinations and Scientific Impact
Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013) · Science · doi:10.1126/science.1240474 · Cited by 1,457
Self-citationFortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497
Quantifying Long-Term Scientific Impact
Wang, D., Song, C., & Barabási, A.-L. (2013) · Science · doi:10.1126/science.1237825 · Cited by 753
Measuring Contextual Citation Impact of Scientific Journals
Moed, H. F. (2010) · Journal of Informetrics · doi:10.1016/j.joi.2010.01.002 · Cited by 628
Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500
“An important point to keep in mind when interpreting RCR values, though, is that citations follow a power law or log-normal distribution [9], wherein one researcher’s choice of a particular RA is at least partly informed by the choices that other researchers have previously made.”
BackgroundOpen Access and Global Participation in Science
Evans, J. A., & Reimer, J. (2009) · Science · doi:10.1126/science.1154562 · Cited by 258
Duch, J., Zeng, X. H. T., Sales-Pardo, M., Radicchi, F., Otis, S., Woodruff, T. K., & Nunes Amaral, L. A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051332 · Cited by 255
Self-citationThe use and misuse of journal metrics and other citation indicators
Pendlebury, D. A. (2009) · Archivum Immunologiae et Therapiae Experimentalis · doi:10.1007/s00005-009-0008-y · Cited by 227
“Nunes Amaral, published their journal ranking methodology in the online publication PLoS One (Stringer et al. 2008).”
BackgroundJacques, T. S., & Sebire, N. J. (2010) · JRSM Short Reports · doi:10.1258/shorts.2009.100020 · Cited by 220
Power laws in citation distributions: evidence from Scopus
Brzezinski, M. (2015) · Scientometrics · doi:10.1007/s11192-014-1524-z · Cited by 196
Use of the journal impact factor for assessing individual articles: Statistically flawed or not?
Waltman, L., & Traag, V. A. (2021) · F1000Research · doi:10.12688/f1000research.23418.2 · Cited by 97
Determinants of the citation rate of medical research publications from a developing country
Annalingam, A., Damayanthi, H., Jayawardena, R., & Ranasinghe, P. (2014) · SpringerPlus · doi:10.1186/2193-1801-3-140 · Cited by 64
“The number of peer-reviewed scientific publications in the medical literature has dramatically increased during the last few decades (Stringer et al. 2008).”
BackgroundCitation-Based Journal Rankings: Key Questions, Metrics, and Data Sources
Walters, W. H. (2017) · IEEE Access · doi:10.1109/access.2017.2761400 · Cited by 56
“is a significant predictor of article impact [167]–[170], it is a mistake to draw conclusions about particular articles based on the characteristics of the journals in which they appeared.”
BackgroundHigh-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information
Stern, D. I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112520 · Cited by 42
Shi, X., Leskovec, J., & McFarland, D. A. (2010) · Proceedings of the 10th annual joint conference on Digital libraries · doi:10.1145/1816123.1816131 · Cited by 36
Citation Success Index − An Intuitive Pair-Wise Journal Comparison Metric
Milojević, S., Radicchi, F., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2016.12.006 · Cited by 33
Metadata
- Authors
- Stringer, M. J., Sales-Pardo, M., & Nunes Amaral, L. A.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2008-02-27
- Volume
- 3
- Issue
- 2
- Pages
- e1683
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2085692706
- Cited by
- 177
- References
- 5
- FWCI
- 18.73
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.