Critical Rationalism and the Search for Standard (Field-Normalized) Indicators in Bibliometrics
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Different bibliographic DBs sort their indexed content based on their own individual subject classification schemes.
“different bibliographic DBs sort their indexed content based on their own individual subject classification schemes”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [178]
Journal impact indicators normalized by subject field have been developed to address these limitations.
“Journal impact indicators that are normalized by subject field have been developed to address these limitations [178,233].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [178]
A problem in citation-based evaluations and analysis is mainly caused by skewed citation distributions.
“apart from differences in citation culture between disciplines, the other obvious problem in citation based evaluations and analysis is mainly caused by skewed citation distributions [178,274].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [178]
More reliable results are obtained at higher aggregation levels, such as institutions or countries.
“generally, more reliable results are obtained at higher aggregation levels (e.g., institutions or a countries) [178].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [178]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · see citing works
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828 · see citing works
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · see citing works
Phillips, D. P. (1974) · American Sociological Review · doi:10.2307/2094294 · Cited by 1,063 · see citing works
Universality of citation distributions: Toward an objective measure of scientific impact
Radicchi, F., Fortunato, S., & Castellano, C. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0806977105 · Cited by 828 · see citing works
Journal impact factor: a brief review.
Garfield, E. (1999) · PubMed · Cited by 744 · see citing works
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567 · see citing works
Self-citationMartin, B. R., & Irvine, J. M. (1983) · Research Policy · doi:10.1016/0048-7333(83)90005-7 · Cited by 529 · see citing works
Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500 · see citing works
Evaluation Practices and Effects of Indicator Use—A Literature Review
Rijcke, S. D., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016) · Research Evaluation · doi:10.1093/reseval/rvv038 · Cited by 467 · see citing works
Towards a new crown indicator: Some theoretical considerations
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2010) · Journal of Informetrics · doi:10.1016/j.joi.2010.08.001 · Cited by 438 · see citing works
Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H. (2011) · Journal of Informetrics · doi:10.1016/j.joi.2011.01.006 · Cited by 291 · see citing works
Self-citationBornmann, L., Loet Leydesdorff, A., & Mutz, R. (2012) · Journal of Informetrics · doi:10.1016/j.joi.2012.10.001 · Cited by 205 · see citing works
Self-citationAlternative metrics in scientometrics: a meta-analysis of research into three altmetrics
Bornmann, L. (2015) · Scientometrics · doi:10.1007/s11192-015-1565-y · Cited by 202 · see citing works
Self-citationLeydesdorff, L., Wouters, P., & Bornmann, L. (2016) · Scientometrics · doi:10.1007/s11192-016-2150-8 · Cited by 173 · see citing works
Self-citationA Systematic Empirical Comparison of Different Approaches for Normalizing Citation Impact Indicators
Waltman, L., & van Eck, N. J. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.08.002 · Cited by 153 · see citing works
Bornmann, L., & Marx, W. (2015) · Journal of Informetrics · doi:10.1016/j.joi.2015.01.006 · Cited by 138 · see citing works
Self-citationBornmann, L. (2014) · Research Evaluation · doi:10.1093/reseval/rvu002 · Cited by 124 · see citing works
Self-citationLoet Leydesdorff, A., & Bornmann, L. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21450 · Cited by 123 · see citing works
Self-citationThe need for standards in bibliometric research and technology
Glänzel, W. (1996) · Scientometrics · doi:10.1007/bf02018475 · Cited by 119 · see citing works
Bornmann, L. (2017) · Higher Education · doi:10.1007/s10734-016-9995-x · Cited by 117 · see citing works
Self-citationThe h-index and its alternatives: An application to the 100 most prolific economists
Tol, R. S. J. (2009) · Scientometrics · doi:10.1007/s11192-008-2079-7 · Cited by 117 · see citing works
Ruiz‐Castillo, J., & Waltman, L. (2014) · Journal of Informetrics · doi:10.1016/j.joi.2014.11.010 · Cited by 114 · see citing works
Handbook of Bibliometric Indicators: Quantitative Tools for Studying and Evaluating Research
Todeschini, R., & Baccini, A. (2016) · Cited by 112 · see citing works
Little scientometrics, big scientometrics ... and beyond?
Glänzel, W., & Schoepflin, U. (1994) · Scientometrics · doi:10.1007/bf02018107 · Cited by 110 · see citing works
Characteristic scores and scales in assessing citation impact
Glänzel, W., & Schubert, A. P. (1988) · Journal of Information Science · doi:10.1177/016555158801400208 · Cited by 99 · see citing works
Universality of scholarly impact metricsPreprint
Kaur, J., Radicchi, F., & Menczer, F. (2013) · Cited by 97 · see citing works
Bibliometrics and Research Evaluation: Uses and Abuses
Gingras, Y. (2016) · Cited by 92 · see citing works
Universality of scholarly impact metrics
Kaur, J., Radicchi, F., & Menczer, F. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.09.002 · Cited by 91 · see citing works
On the causes of subject-specific citation rates in Web of Science
Marx, W., & Bornmann, L. (2014) · Scientometrics · doi:10.1007/s11192-014-1499-9 · Cited by 85 · see citing works
Self-citationRadicchi, F., & Castellano, C. (2011) · Journal of Informetrics · doi:10.1016/j.joi.2011.09.002 · Cited by 79 · see citing works
Radicchi, F., & Castellano, C. (2012) · RePEc: Research Papers in Economics · Cited by 71 · see citing works
The scientific economy of attention: A novel approach to the collective rationality of science
Franck, G. (2002) · Scientometrics · doi:10.1023/a:1016059402618 · Cited by 66 · see citing works
Scientific productivity: An exploratory study of metrics and incentives
Lindner, M. D., Torralba, K. D., & Khan, N. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195321 · Cited by 58 · see citing works
Crespo, J. A., Li, Y., & Ruiz‐Castillo, J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0058727 · Cited by 53 · see citing works
Bornmann, L., Anegón, F. D. M., & Mutz, R. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22923 · Cited by 48 · see citing works
Self-citationWeb Indicators for Research Evaluation: A Practical Guide
Thelwall, M. A. (2016) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00733ed1v01y201609icr052 · Cited by 45 · see citing works
Bornmann, L., & Haunschild, R. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.07.002 · Cited by 43 · see citing works
Self-citationA critical evaluation of the algorithm behind the Relative Citation Ratio (RCR)
Janssens, A. C. J. W., Goodman, M., Powell, K. R., & Gwinn, M. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002536 · Cited by 42 · see citing works
Bornmann, L., & Haunschild, R. (2016) · RePEc: Research Papers in Economics · Cited by 38 · see citing works
Self-citationRethinking Critically Reflective Research Practice: Beyond Popper’s Critical Rationalism
Ulrich, W. (2006) · Cited by 35 · see citing works
Loet Leydesdorff, A., & Milojević, S. (2015) · Soziale Welt · doi:10.5771/0038-6073-2015-2-193 · Cited by 33 · see citing works
A comparison of the Web of Science and publication-level classification systems of science
Perianes‐Rodríguez, A., & Ruiz‐Castillo, J. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.10.007 · Cited by 29 · see citing works
The Normalization of Citation Counts Based on Classification Systems
Bornmann, L., Marx, W., & Barth, A. (2013) · Publications · doi:10.3390/publications1020078 · Cited by 27 · see citing works
Self-citationBornmann, L., & Haunschild, R. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23729 · Cited by 26 · see citing works
Self-citationBornmann, L., & Haunschild, R. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.04.015 · Cited by 26 · see citing works
Self-citationHaunschild, R., Schier, H., Marx, W., & Bornmann, L. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.03.004 · Cited by 24 · see citing works
Self-citationMutz, R., Bornmann, L., & Daniel, H. (2014) · Research Evaluation · doi:10.1093/reseval/rvu023 · Cited by 22 · see citing works
Self-citationSirtes, D. (2012) · Journal of Informetrics · doi:10.1016/j.joi.2012.01.008 · Cited by 20 · see citing works
The comparison of normalization procedures based on different classification systems
Li, Y., & Ruiz‐Castillo, J. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.09.005 · Cited by 16 · see citing works
The quiet rise of the NIH’s hot new metric
Naik, G. (2016) · Nature · doi:10.1038/539150a · Cited by 15 · see citing works
Haunschild, R., Schier, H., Marx, W., & Bornmann, L. (2018) · RePEc: Research Papers in Economics · Cited by 14 · see citing works
Self-citationSome practical aspects of the standardization of scientometric indicators
Vinkler, P. (1996) · Scientometrics · doi:10.1007/bf02018481 · Cited by 13 · see citing works
Bornmann, L., & Haunschild, R. (2018) · Journal of Korean Medical Science · doi:10.3346/jkms.2018.33.e138 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Bloudoff-Indelicato, M. (2015). NIH metric that assesses article impact stirs debate. The US biomedical funding agency is using the metric to evaluate funded research. Retrieved November 10, 2015, from http://www.nature.com/news/nih-metric-that-assesses- article-impact-stirs-debate-1.18734
- Bornmann, L., & Wohlrabe, K. (2017). Normalization of Citation Impact in Economics (August 01, 2017). CESifo Working Paper Series No. 6592. Available at SSRN: https://ssrn.com/abstract=3024474.
- Ziman, J. (1996). "Postacademic Science": Constructing Knowledge with Networks and Norms. Science Studies, 9(1), 67-80.
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
Lei, L., & Sun, Y. (2020) · Scientometrics · doi:10.1007/s11192-019-03338-y · Cited by 43 · see citing works
- Authors
- Bornmann, L., & Marx, W.
- Affiliations
- Max Planck Society (DE)
- Max Planck Institute for Solid State Research (DE)
- Corresponding author
- Bornmann, Lutz
- Venue
- Journal of Informetrics
- Publisher
- Elsevier BV
- Published
- 2018-08
- Volume
- 12
- Issue
- 3
- Pages
- 598-604
- ISSN
- 1751-1577 (ISSN-L)
- OpenAlex
- W2803543889
- Cited by
- 41
- References
- 54
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 2 · 2023: 3 · 2022: 6 · 2021: 7 · 2020: 7 · 2019: 7
- FWCI
- 4.00
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Journal of Informetrics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- critical rationalism, research evaluation, bibliometric indicators, Karl Popper, ISSI, indicator standardization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
