CoChat
Sign Up
Cited by 117scientometrics and bibliometrics researchWeb visibility and informetricsPsychology Research and Bibliometrics

Measuring Impact in Research Evaluations: A Thorough Discussion of Methods for, Effects of and Problems with Impact Measurements

Open access
Bornmann, Lutz·2017·Higher Education
DOI 10.1007/s10734-016-9995-x
Open this paper in CoChat — free accountOpen PDF

Abstract

Impact of science is one of the most important topics in scientometrics. Recent developments show a fundamental change in impact measurements from impact on science to impact on society. Since impact measurement is currently in a state of far reaching changes, this paper describes recent developments and facing problems in this area. For that, the results of key publications (dealing with impact measurement) are discussed. The paper discusses how impact is generally measured within science and beyond, which effects impact measurements have on the science system and which problems are associated with impact measurement. The problems associated with impact measurement constitute the focus of this paper: Science is marked by inequality, random chance, anomalies, the right to make mistakes, unpredictability and a high significance of extreme events, which might distort impact measurements. Scientometricians as the producer of impact scores and decision makers as their consumers should be aware of these problems and should consider them in the generation and interpretation of bibliometric results, respectively.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10734-016-9995-x

Cites

86 of 86 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Use of publication and citation counts and derived indicators is only valid when applied at similar levels of aggregation and in appropriate context.

    “the use of publication and citation counts and, accordingly, the indicators that are derived from them, is only valid when they are applied at the similar levels of aggregation and in the appropriate context [260].”

    From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [260]

  • Citations may be appropriate for assessing scientific impact of research, but they do not show impact outside the scientific community.

    “Citations may be appropriate in assessing the scientific impact of the research, but they do not show the impact of the research outside the scientific community [1,3,5,6,260,264,265].”

    From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [260]

  • Scientific impact of any evaluated entity cannot be determined solely by citations due to different citing behaviors and incompatible citation counts between disciplines.

    “It is often argued that the scientific impact of any evaluated entity cannot be determined solely by citations due to the different citing behaviors and resulting incompatible citation counts between disciplines [1-3,5,260,267].”

    From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [260]

  • These studies are often targeted at general public and published in national language, which narrows potential audience and reduces likelihood of citation.

    “These studies are also often targeted at the general public and, therefore, are often published in the national language, which narrows the size of potential audience and, consequently, reduces the likelihood of citation [87,96,260,265,294].”

    From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [260]

  • The 'publish or perish' attitude stimulates productivity of researchers but creates a bias towards quantity, often overlooking quality.

    “this 'publish or perish' attitude, on the one hand, stimulates the productivity of researchers, but, on the other, creates a bias towards quantity, often overlooking quality [2,3,6-8,44,81,163,260,263,284-288].”

    From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [260]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Bornmann, L., de Moya-Anego ´n, F., & Leydesdorff, L. (2010). Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE, 5(10), e11344.
  2. Lanham, MD: Scarecrow Press.
  3. Luhmann, N. (2012a). Theory of society (Vol. 1). Stanford, CA: Stanford University Press.
  4. Merton, R. K., & Barber, E. G. (2004). The travels and adventures of serendipity: A study in historical semantics and the sociology of science. Princeton: Princeton University Press.
  5. Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628-638.

Cited by

Sorted by citations

Related

Authors
Bornmann, L.
Affiliations
Max Planck Society (DE)
Corresponding author
Bornmann, Lutz
ORCID
Bornmann, Lutz
Venue
Higher Education
Publisher
Springer Science and Business Media LLC
Published
2016-03-03
Volume
73
Issue
5
Pages
775-787
ISSN
  • 0018-1560 (ISSN-L)
  • 1573-174X
DOI
10.1007/s10734-016-9995-x
OpenAlex
W792777101
Cited by
117
References
86
Citations by year
2026: 14 · 2025: 11 · 2024: 12 · 2023: 9 · 2022: 11 · 2021: 15 · 2020: 8 · 2019: 13 · 2018: 16 · 2017: 4 · 2016: 4
FWCI
11.51
Type
journal article
Language
en
Open access
Licence
Locations
  • Higher Education (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Max-Planck-Gesellschaft
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
research impact assessment, research evaluation, bibliometric analysis, impact score, social impact of science, measurement bias, scientific inequality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review