The DOI matches its Crossref record; 143 of the 144 references it cites were found in a registry or an index; none of the 86 sources its claims cite is checked for support yet.
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Open accessAbstract
Citations are increasingly used as performance indicators in research policy and within the research system. Usually, citations are assumed to reflect the impact of the research or its quality. What is the justification for these assumptions and how do citations relate to research quality? These and similar issues have been addressed through several decades of scientometric research. This article provides an overview of some of the main issues at stake, including theories of citation and the interpretation and validity of citations as performance measures. Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics. The article investigates how citations may relate to these various research quality dimensions. It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations. On the contrary, there is no evidence that citations reflect other key dimensions of research quality. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This review argues that citations can serve as limited indicators of scientific impact and relevance but not of other key research-quality dimensions such as solidity/plausibility, originality, or societal value. It synthesizes citation theories, validation studies, and quality dimensions to caution against using citation indicators as substitutes for peer review.
Key findings
- Citations are argued to reflect aspects of scientific impact and relevance, but with important limitations; there is no evidence that they reflect other key dimensions of research quality—solidity/plausibility, originality, and societal value.
- Comparative validation studies generally find moderately positive correspondence between citation indicators and peer review, but correlations are far from perfect; in REF 2014, field-weighted citation impact had a Spearman correlation of .28 at the overall level, with significant field variation and decreases for more recent outputs, leading to the conclusion that metrics cannot provide a like-for-like replacement for peer review.
- Citation distributions are highly skewed: in many humanities disciplines an average paper receives less than one citation over 10 years, versus more than 40 in some biomedical fields (Web of Science 2005–2015); the Lowry et al. (1951) protein-measurement paper has more than 305,000 Web of Science citations, and the 4S statin trial paper more than 7,700.
- The h-index is popular but not field-normalized and not corrected for career length, disadvantaging younger researchers; field, publication-year, and document-type normalization is standard, but there is no general agreement on the most appropriate normalization method.
Methods
- Design: narrative/conceptual overview and selective literature review, not a systematic review or new empirical study.
- Data or sample: no new dataset; the article draws on prior scientometric literature, citation theories, validation studies, and selected examples.
- Main analysis: conceptual synthesis of citation indicators, referencing behavior, peer-review comparisons, and dimensions of research quality.
Limitations
- Stated: coverage of topics and literature is selective and does not discuss all details; the literature on interaction between citing practices and evaluation processes is only referred to in passing; constructivist and semiotic theories of quality and citation are not discussed.
- Stated/evident: few previous studies have systematically addressed how citations relate to each research-quality dimension, and the final part of the article is described as more explorative and discursive.
- Evident: the paper presents no systematic search strategy or empirical test of its central claims, so its conclusions rely on narrative synthesis and selected examples.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
76 Not checked yet · 44 Own interpretation · 16 No basis given
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the contrary, there is no evidence that citations reflect other key dimensions of research quality.”
- Also stated in: § Concluding Remarks · p. 12 · plain
“there is no evidence that citations reflect other key dimensions of research quality”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.
- Also stated in: § Concluding Remarks · p. 12 · plain
In the paper
“The larger part of all scientific papers are never cited or cited a few times only in the subsequent scientific literature”
What the sources say
- Not checked yetcites Aksnes, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Citation distributions are very skewed.”
What the sources say
- Not checked yetcites Price (1965)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“There are large variations in average citation rates across different subject areas.”
What the sources say
- Not checked yetcites data from Web of Science 2005-2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“but empirical studies have shown that two different methods for normalization, AoR and RoA, did not produce very different results, particular at the level of countries and institutions”
What the sources say
- Not checked yetcites Lariviere & Gingras, 2011
- Not checked yetcites Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the contrary, some articles have an extremely large number of citations, reaching into the hundreds and even thousands.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“There is no general agreement on what is the most appropriate method”
What the sources say
- Not checked yetcites Ioannidis, Boyack, & Wouters, 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Although important insights on the role of references in the scientific article have been obtained, the accumulation of knowledge at the same time has been hampered by the fact that different classification systems have been applied in previous studies”
What the sources say
- Not checked yetcites Liu, 1993
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“References to highly cited publications are more often present in the introduction than in other parts of the publications”
What the sources say
- Not checked yetcites Voos & Dagaev, 1976
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Because of this, articles describing commonly used methods may receive a very large number of citations.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 42
- Probably not a substantial claim. 58
- The quote only partly states this claim. 1
In the paper
“Most importantly, it is not field-normalized and no corrections are made for career length, which means that the indicator disfavors younger researchers”
What the sources say
- Not checked yetcites Alonso, Cabrerizo, Herrera-Viedma, & Herrera, 2009
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The views of the MacRoberts's previously led to much debate, but their conclusions are generally seen as too sweeping”
What the sources say
- Not checked yetcites Aksnes, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Even now, in spite of detailed studies of referencing behavior, there is no unified theory.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As indicated above, previous studies of the citation process have not provided any simple answer to the question of what citations stand for.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While empirical studies have revealed a multitude of factors involved in the citation process, the issue has also been approached from another angle: by comparing citation indicators with the outcome of peer review. During recent decades, many such studies have been carried out.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nevertheless, some overall findings remain: the references have a multitude of functions in the scientific article, only a small proportion of the relevant literature is cited, and the authors have a multitude of motives for including particular studies as references.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Studies undertaken have revealed that the role of the reference, both in the citing text and with respect to the cited text, is complex.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Moreover, the citing is biased: some sources are cited essentially every time they are used, while other research is never cited even though it may be used more often than the highly cited work.”
What the sources say
- Not checked yetcites MacRoberts & MacRoberts, 1989, 1996
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These and later studies have revealed that the references have a multitude of functions in the scientific article.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, the studies are often based on rather small samples of papers from selected scientific fields, and the results may not have general validity.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Based on empirical case studies, they showed that a very small proportion of the knowledge basis of an article (consisting of hundreds or thousands of former publications) actually are cited.”
What the sources say
- Not checked yetcites MacRoberts & MacRoberts, 1989, 1996
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Therefore, it will be more persuasive to cite an authoritative paper, and authors tend to select references that will be regarded as authoritative by the intended audience.”
What the sources say
- Not checked yetcites Gilbert, 1977
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This article has now been cited more than 305,000 times in the Web of Science database”
What the sources say
- Not checked yetcites Van Noorden, Maher, & Nuzzo, 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In a scientific article, some of the references will represent works that are crucial or significant antecedents to the present work; others may represent more general background literature”
What the sources say
- Not checked yetcites Aksnes, 2005
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Hence, the textual functions of citations vary considerably.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“For the citation process, this implies that other incentives shape the citing patterns, like creating visibility for one's work through self-citations or citing a journal editor's work as an attempt to enhance the chances of acceptance for publication.”
What the sources say
- Not checked yetcites Bornmann & Daniel, 2008
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“There is a growing concern about this issue, as well as on potential negative impact of research metrics on the scientific community.”
What the sources say
- Not checked yetcites Hicks et al., 2015
- Not checked yetcites Cagan, 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Citation measures are also core indicators in several university rankings, such as the Leiden ranking and Academic Ranking of World Universities (ARWU)”
What the sources say
- Not checked yetcites Piro & Sivertsen, 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In recent years, bibliometric indicators have increasingly been applied in the context of research evaluation as well as research policy more generally.”
What the sources say
- Not checked yetcites Moed, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For example, Levitt and Thelwall (2011) have argued that short citation windows have the problem that articles published earlier in a year have a significant advantage (i.e., are on average more highly cited) compared with publications appearing later in a year.”
What the sources say
- Not checked yetcites Levitt and Thelwall (2011)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Only a small fraction of the scholarly literature in the humanities is represented in the Web of Science, and most of the references and citations will not be captured by the database.”
What the sources say
- Not checked yetcites Marx and Bornmann (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The coverage of the scientific and scholarly literature varies across these databases, and the results of citation studies are thus dependent upon the particular characteristics of the databases and their coverage.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In the scientific paper, the references have various purposes. Authors are not including references merely because of their scientific quality.”
What the sources say
- Not checked yetcites Bornmann & Daniel, 2008
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It has since been the standard to adjust for field, publication year, and publication type when calculating citation indicators.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Moreover, the application of citation indicators has also been criticized more generally, with respect to their validity as performance measures and their potentially negative impact upon the research system”
What the sources say
- Not checked yetcites MacRoberts & MacRoberts, 1989
- Not checked yetcites Osterloh & Frey, 2015
- Not checked yetcites Weingart, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When one article is cited by many subsequent publications, even more people become aware of this article. Thus, its visibility, and thereby the chances of getting even more citations, increases.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the studies, assessments by peers have been typically considered as a kind of standard to which citation indicators can be validated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some of these problems have a fundamental character and are inherent in any use of citations as indicators, others may be resolved by the construction of more advanced indicators, while others again may be of less importance in practice.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, panels increasingly are considering citation measures as part of the evaluation procedure, which means that the two cannot be considered as completely independent of each other.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If authors cite the works they find useful, frequently cited publications may be assumed to have been more useful than papers which are hardly cited at all.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, it may be concluded that most of the comparative studies seem to have found a moderately positive correspondence, but the correlations identified have been far from perfect and have varied among the studies.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For the field-weighted citation impact, a Spearman correlation coefficient of .28 was identified at an overall level, albeit with significant variations across fields. Moreover, there were significant decreases in correlation for more recent outputs.”
What the sources say
- Not checked yetcites Higher Education Funding Council for England, 2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, a large number of different citation measures exist and the outcome would also depend on which indicators are selected for the comparative analysis.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“A significant degree of concordance was found '. . . supporting the choice of using both techniques in order to assess the quality of Italian research institutions' (Ancaiani et al., 2015, p. 254).”
What the sources say
- Not checked yetcites Ancaiani et al., 2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Although most citation analysts seem to agree that citing or referencing is biased, it has been argued that this bias is not fatal for the use of citation as performance indicators-to a certain extent, the biases are averaged out at aggregated levels.”
hedgedconditional- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“According to Luukkonen (1990), the presence of different cognitive meanings of citations and motivations for citing does not necessarily invalidate the use of citations as (imperfect) performance measures.”
What the sources say
- Not checked yetcites Luukkonen (1990)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Problems related to more technical issues, such as discrepancies between target articles and cited references (misspellings of journal names, author names, errors in the reference lists, etc.), and mistakes in the indexing procedures conducted by Clarivate Analytics (previously Thomson Reuters) or Elsevier (Leydesdorff et al., 2016; Moed, 2002) may confuse citation analyses. Such errors affect in”
What the sources say
- Not checked yetcites Leydesdorff et al., 2016
- Not checked yetcites Moed, 2002
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Aksnes and Taxt (2004) analyzed the relationship between bibliometric indicators and the outcomes of a peer review of Norwegian research groups at a mathematical and natural science faculty, reporting positive but weak correlations.”
What the sources say
- Not checked yetcites Aksnes and Taxt (2004)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that the different functions the references may have in a text are much more complex than merely providing documentation and support for particular claims.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Gilbert (1977) argued that citing ('referencing') is essentially a device for persuasion.”
What the sources say
- Not checked yetcites Gilbert (1977)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Small (1982) identified five distinctions: A cited work may be (a) refuted, (b) noted only, (c) reviewed, (d) applied, or (e) supported by the citing work.”
What the sources say
- Not checked yetcites Small (1982)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Typically, a scientific article is structured as a progression from the general to the particular”
What the sources say
- Not checked yetcites Law, 1986
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Although the literature coverage of citation databases has improved (Web of Science and Scopus), the coverage of the humanities and several social science disciplines remains limited (Waltman, 2016).”
What the sources say
- Not checked yetcites Waltman, 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As is evident from the overview above, there is no simple answer to the question what citation indicators measure or indicate.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, both the methodological basis for comparing peer assessments and citation indicators and the assumption that the two may be expected to correlate may be questionable.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, citation frequencies are determined by other mechanisms and are not a simple reflection of the quality dynamics.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“According to Merton's view, the norms of science oblige researchers to cite the work upon which they draw, and in this way acknowledge or credit contributions by others”
What the sources say
- Not checked yetcites Merton, 1979
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“He found that the h-index and the normalized citation impact indicator both correlated quite well with peer judgments.”
What the sources say
- Not checked yetcites van Raan (2006)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“According to Bornmann and Daniel (2008), many studies have methodological weaknesses and have provided findings with little reliability.”
What the sources say
- Not checked yetcites Bornmann and Daniel (2008)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“However, the methodological fundament for this conclusion has been contested by Baccini and De Nicolao (2016), who argue that the analysis is flawed and that informed peer review and bibliometrics do not produce similar results.”
What the sources say
- Not checked yetcites Baccini and De Nicolao (2016)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Rinia, van Leeuwen, van Vuren, and van Raan (1998) showed that various citation indicators correlated significantly with peer ratings of research programs in condensed matter physics.”
What the sources say
- Not checked yetcites Rinia, van Leeuwen, van Vuren, and van Raan (1998)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“According to the founder of the SCI, the fact that authors do not cite all their influences does not invalidate the use of citations as performance measures when enough literature is taken into account”
What the sources say
- Not checked yetcites Garfield, 1997
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
In the paper
“The fact that citation distributions are extremely skewed also raises questions concerning the use of mean as indicator, and Bornmann and Mutz (2011) have proposed to use percentile ranks as a non-parametric alternative to the means of citation distributions for the normalization.”
What the sources say
- Not checked yetcites Bornmann and Mutz (2011)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“According to Welljams-Dorof (1997), this has important implications: In general, the larger the citation data set being used, the higher the confidence level of the results. Analyses involving entire fields of research, nations, regions and large universities are virtually unaffected by the concerns and caveats about citation data . . . The confidence level at these large aggregate levels is quite”
What the sources say
- Not checked yetcites Welljams-Dorof (1997)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the use of impact as the most appropriate concept has usually been justified by theoretical considerations, and there are few attempts to address the issue empirically or relate it to previous findings on citation behavior.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Still, a basic limitation is that many innovations are not patented and patents are not suitable to assess societal relevance or impact in a broader context.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown above, the question on the relation between citations and research quality is complex and will arise differently depending on the field analyzed, the database used, the timeframe and indicators applied, and so forth.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“According to Waltman, van Eck, and Wouters (2013), citation impact should be distinguished from scientific impact, as an influential researcher sometimes has a lower performance in terms of highly cited publications than some of their less influential colleagues.”
What the sources say
- Not checked yetcites Waltman, van Eck, and Wouters (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the viewpoint described in the introduction that bibliometric assessment is superior compared to the traditional peer-review method is not justified in our opinion.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the referee system does not fully ensure the quality dimension related to solidity and plausibility, and there are no indications that high citation counts reflect solidity.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“However, considering the various limitations attached to citations as performance measures, most bibliometricians have argued that a bibliometric analysis should not function as a substitute for a peer review (Moed, 2005).”
What the sources say
- Not checked yetcites Moed, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Citation indicators may easily be misused or applied in contexts where they lack justification or validity.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition, research quality is a multidimensional concept, and in this section, we will look further into this issue.”
What the sources say
- Not checked yetcites Polanyi (1962)
- Not checked yetcites Gulbrandsen, 2000
- Not checked yetcites Lamont, 2009
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As an example, even a 'mistaken' publication can have a large impact by stimulating further research.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because of the sociological and aggregational processes involved. In the beginning, an article may be cited for substantive reasons (e.g., its content has been used). Later, when the article is widely known and has received many citations, sociological mechanisms will be of increasing importance (authors citing authoritative papers, the bandwagon effect, etc.). Some papers will benefit”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The view generally held among experts within bibliometrics seem to be that citations represent a good but not perfect impact measure.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As is evident from the discussion of this paper, this is an important point, as citations are not able to capture all aspects of the quality concept. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As a starting point, we can take the three dimensions distinguished by Polanyi (1962): plausibility, originality, and scientific value.”
What the sources say
- Not checked yetcites Polanyi (1962)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Impact is the most commonly used concept for what citations reflect, although other concepts such as influence, importance, significance, and utility occasionally also are used (Moed, 2005).”
What the sources say
- Not checked yetcites Moed, 2005
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“There seems to be a widespread assumption that this issue cannot be adequately assessed through standard citation indicators, and in recent years, increasing attention has been devoted to developing methodologies for assessing and measuring societal relevance and impact (Bornmann, 2012, 2013).”
What the sources say
- Not checked yetcites Bornmann, 2012, 2013
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, there is a lack of empirical studies confirming that this is actually the case, and possibly some of the biases is of a fundamental nature attached to all citations measures, while the effect of others may tend to level out when aggregated levels are considered.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Societal relevance is often considered to be something which is much harder to measure than scientific relevance or impact (B. R. Martin, 2011).”
What the sources say
- Not checked yetcites B. R. Martin, 2011
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, all assessments of research quality may be context-dependent, in terms of, for example, the time of assessment and the time/field/sector perspectives of the evaluators.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Using these definitions, it is also evident that impact would be a more adequate interpretation of citations than quality.”
What the sources say
- Not checked yetcites B. R. Martin and Irvine, 1983
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite these and numerous similar examples, it is not possible to identify societal relevance from citation counts per se, and uncited or little cited publication may have contributed with results of great societal relevance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Because of this, it has long been emphasized by bibliometricians that more than one indicator should be used in research evaluation contexts (van Raan, 1993).”
What the sources say
- Not checked yetcites van Raan, 1993
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“citations reflect-with important limitations-aspects related to scientific impact and relevance”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The literature contains numerous publications of which the solidity is poor, the results unreliable or even involving misconduct or scientific fraud (Fanelli, 2009).”
What the sources say
- Not checked yetcites Fanelli, 2009
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For example, the mean normalized citation score is size-independent and does not take into account the number of publications. According to Abramo and D'Angelo (2016), this is a major problem with this indicator because it does not truly represent productivity.”
What the sources say
- Not checked yetcites Abramo and D'Angelo (2016)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, this topic has rarely been addressed specifically in the literature and there are few studies analyzing the issue empirically.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“On this basis, it is often argued that bibliometric analysis can counterbalance shortcomings and mistakes in the peers' judgments (Aksnes, 2005). Thus, a bibliometric study should be considered as complementary to a peer evaluation (Council of Canadian Academies, 2012).”
What the sources say
- Not checked yetcites Aksnes, 2005
- Not checked yetcites Council of Canadian Academies, 2012
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In sum, whereas plausibility/soundness, scientific value, and societal value and originality seem commonly perceived key characteristics of research quality, each of these dimensions include a variety of aspects; they may be contextdependent and may also conflict with each other.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“At the same time, there are also problems with the peer-review system.”
What the sources say
- Not checked yetcites Eyre-Walker and Stoletzki (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.
“Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics.”
The authors state this as an assumption.
There should be a correlation between citation indicators and peer assessments if citations legitimately can be used as indicators of scientific performance.
“The basic assumption is that there should be a correlation if citations legitimately can be used as indicators of scientific performance.”
The paper relies on this without support.
Peer reviews are applied in many different contexts, of which peer assessments of manuscripts submitted to journals and publishers is the most fundamental one.
“Peer reviews are applied in many different contexts, of which peer assessments of manuscripts submitted to journals and publishers probably is the most fundamental one.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is often argued that the scientific impact of any evaluated entity cannot be determined solely by citations due to the different citing behaviors and resulting incompatible citation counts between disciplines [1-3,5,260,267].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
In the citing paper
“Citations may be appropriate in assessing the scientific impact of the research, but they do not show the impact of the research outside the scientific community [1,3,5,6,260,264,265].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
In the citing paper
“peer-reviewing has its own characteristic biases and limitations, which is one of the reasons why, nowadays, peer-review has been increasingly extensively replaced with bibliometric measures of impact [3,44,276].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
In the citing paper
“this 'publish or perish' attitude, on the one hand, stimulates the productivity of researchers, but, on the other, creates a bias towards quantity, often overlooking quality [2,3,6-8,44,81,163,260,263,284-288].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Corresponding Author: · p. 2
“Many previous studies have compared citation indicators with the outcome of peer review, and in the 'Understanding Citations' part, this issue is examined.”
§ Validation Studies · p. 6
“## Validation Studies While empirical studies have revealed a multitude of factors involved in the citation process, the issue has also been approached from another angle: by comparing citation indicators with the outcome of peer review.”
Citations
Cites
PROTEIN MEASUREMENT WITH THE FOLIN PHENOL REAGENT
Lowry, O. H., Rosebrough, N., Farr, A. L., & Randall, R. (1951) · Journal of Biological Chemistry · doi:10.1016/s0021-9258(19)52451-6 · Cited by 316,394
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720
(1968) · PubMed · Cited by 4,803
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“However, the recent attention to the lack of replicable results in biomedical, clinical, and psychological studies (Ioannidis, 2005) may lead to a higher social status of replications studies.”
BackgroundPrice, D. J. D. S. (1965) · Science · doi:10.1126/science.149.3683.510 · Cited by 3,205
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828
Self-citationFanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“The literature contains numerous publications of which the solidity is poor, the results unreliable or even involving misconduct or scientific fraud (Fanelli, 2009).”
BackgroundGoogle Scholar, Scopus and the Web of Science: A Longitudinal and Cross-Disciplinary Comparison
Harzing, A.-W., & Alakangas, S. (2016) · Scientometrics · doi:10.1007/s11192-015-1798-9 · Cited by 1,784
Research works printed but not linked: 15
- Biagioli M., Gaming Metrics. Beyond Publish or Perish: Metrics and the new Ecologies of Academic Misconduct
- Carlsson H., 2009, InfoTrend, 64, 82
- Dymond L. H., 2015, A recent history of recognized economic thought—Contributions of the Nobel laureates to economic science
- Garfield E., 1977, Essay of an information scientist, 1, 84
- Gulbrandsen J. M., 2000, Research quality and organisational factors: An investigation of the relationship
- Higher Education Funding Council for England. (2015). The metric tide: Correlation analysis of REF2014 scores and metrics: Supplementary report II to the independent review of the role of metrics in research assessment and management. Retrieved from www.dcscience.net/2015_metrictideS2.pdf
- Langfeldt L., Scordato L. (2015, April). Assessing the broader impacts of research: A review of methods and practices (NIFU Working Paper 8/2015). Oslo.
- le Pair C, 1995, International Forum on Information and Documentation, 20, 16
- Luukkonen T., 1990, Citations in the rhetorical, reward, and communication systems of science
- Merton R. K., 1979, Citation indexing—Its theory and application in science, technology, and humanities
- Smith L. C., 1981, Library Trends, 30, 83
- van Raan A. F. J, 2000, The web of knowledge: A Festschrift in honor of Eugene Garfield, 301
- Welljams-Dorof A., 1997, Evaluating science and scientists: An East-West dialogue on research evaluation in post-communist Europe, 202
- Wouters P. (1999b). The citation culture (Doctoral thesis). University of Amsterdam, The Netherlands. Retrieved from http://garfield.library.upenn.edu/wouters/wouters.pdf
- Wouters P., Handbook of science policy studies
Cited by
Sorted by citations
Items 201–300 of 1,000
A Bibliometric Analysis of Research Trends in Biodegradation of Plastics
Akinpelu, E. A., & Nchu, F. (2022) · Polymers · doi:10.3390/polym14132642 · Cited by 21
Dardas, L. A., Sallam, M., Woodward, A. L., Sweis, N. S., Sweis, N., & Sawair, F. (2023) · Publications · doi:10.3390/publications11010005 · Cited by 21
Research Mapping of Trauma Experiences in Autism Spectrum Disorders: A Bibliometric Analysis
González, O. H., Fresno, A., Spencer, R., Mínguez, R. T., Fernández, D. G., & Sepúlveda-Opazo, F. (2023) · Healthcare · doi:10.3390/healthcare11091267 · Cited by 21
Integrated Sensing and Communication in Next‐Generation Wireless Networks: Insights and Trends
Singh, S., & Samal, U. (2025) · International Journal of Communication Systems · doi:10.1002/dac.70014 · Cited by 21
Su, S., Wang, T., Wei, R., Jia, X., Lin, Q., & Bai, M. (2022) · Frontiers in Surgery · doi:10.3389/fsurg.2022.811023 · Cited by 21
Mapping the research trends of third language acquisition: A bibliometric analysis based on Scopus
Liu, Z. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1021517 · Cited by 21
Quantification 2.0? Bibliometric Infrastructures in Academic Evaluation
Krüger, A. K. (2020) · Politics and Governance · doi:10.17645/pag.v8i2.2575 · Cited by 20
Ending publication bias: A values-based approach to surface null and negative results
Curry, S., Mercado-Lara, E., Arechavala‐Gomeza, V., Begley, C. G., Bernard, C., Bernard, R., Bertuzzi, S., Bhalla, N., Bowers, D., Brod, S., et al. (2025) · PLoS Biology · doi:10.1371/journal.pbio.3003368 · Cited by 26
Metrics and epistemic injustice
Ma, L. (2022) · Journal of Documentation · doi:10.1108/jd-12-2021-0240 · Cited by 20
Tahamtan, I., & Bornmann, L. (2022) · El Profesional de la Informacion · doi:10.3145/epi.2022.jul.11 · Cited by 26
The inconsistency of h-index: A mathematical analysis
Brito, R., & Navarro, A. R. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101106 · Cited by 20
Kunze, K. N., Richardson, M., Bernstein, D. N., Premkumar, A., Piuzzi, N. S., & McLawhorn, A. S. (2020) · JAAOS Global Research and Reviews · doi:10.5435/jaaosglobal-d-20-00187 · Cited by 20
Ponting, S. S.-A. (2021) · Journal of Hospitality and Tourism Insights · doi:10.1108/jhti-09-2020-0164 · Cited by 20
Jubilee, R. V. W., Kamarudin, F., Latiff, A. R. A., Hussain, H. I., & Anwar, N. A. M. (2021) · Asia-Pacific Journal of Business Administration · doi:10.1108/apjba-09-2020-0343 · Cited by 20
Twitter Mentions Influence Academic Citation Count of Shoulder and Elbow Surgery Publications
Sudah, S. Y., Faccone, R. D., Nasra, M. H., Constantinescu, D. S., Menendez, M. E., & Nicholson, A. D. (2022) · Cureus · doi:10.7759/cureus.21762 · Cited by 20
Petrovich, E. (2022) · Synthese · doi:10.1007/s11229-022-03515-2 · Cited by 20
Meta-Research: How problematic citing practices distort science
Horbach, S. P. J. M., Aagaard, K., & Schneider, J. W. (2021) · doi:10.31222/osf.io/aqyhg · Cited by 21
Is OpenAlex suitable for research quality evaluation and which citation indicator is best?
Thelwall, M. A., & Jiang, X. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70020 · Cited by 30
Quoted speech in linguistics research article titles: patterns of use and effects on citations
Pearson, W. S. (2021) · Scientometrics · doi:10.1007/s11192-020-03827-5 · Cited by 19
Lone Geniuses or One among Many? An Explorative Study of Contemporary Highly Cited Researchers
Aksnes, D. W., & Aagaard, K. (2021) · Journal of Data and Information Science · doi:10.2478/jdis-2021-0019 · Cited by 19
Self-citationA bibliometric analysis of executive functions in autism spectrum disorder
Shekarro, M., Fazeli-Varzaneh, M., & Kuravackel, G. M. (2021) · Current Psychology · doi:10.1007/s12144-021-01947-6 · Cited by 19
Demartini, M., & Taticchi, P. (2021) · International Journal of Productivity and Performance Management · doi:10.1108/ijppm-02-2021-0063 · Cited by 19
Vaishya, R., Gupta, B. M., Misra, A., Mamdapur, G. M. N., Walke, R., & Vaish, A. (2023) · Diabetes & Metabolic Syndrome Clinical Research & Reviews · doi:10.1016/j.dsx.2023.102898 · Cited by 19
Seraj, P. M. I., Klímová, B., & Khan, R. (2024) · SAGE Open · doi:10.1177/21582440241251998 · Cited by 19
Griffin, P. A., & Sun, E. Y. (2024) · The International Journal of Accounting · doi:10.1142/s1094406024500070 · Cited by 19
Linkov, V., O’Doherty, K. C., Choi, E.-S., & Han, G. Q. (2021) · SAGE Open · doi:10.1177/21582440211009191 · Cited by 19
Research articles promoted in embargo e-mails receive higher citations and altmetrics
Lemke, S., Brede, M., Rotgeri, S., & Peters, I. (2021) · Scientometrics · doi:10.1007/s11192-021-04217-1 · Cited by 19
Olejniczak, A. J., Savage, W. E., & Wheeler, R. (2022) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2022.812312 · Cited by 18
Weitzel, S. D. R. (2022) · doi:10.21452/abec.2022.isbn.978-65-89167-68-6.012 · Cited by 20
Steingard, D. S., Balduccini, M., & Sinha, A. (2022) · AI & Society · doi:10.1007/s00146-022-01459-2 · Cited by 18
Determining factors of participation and success rates in research funding competitions: Case study
Rusu, V. D., Mocanu, M., & Bibiri, A.-D. (2022) · PLoS ONE · doi:10.1371/journal.pone.0272292 · Cited by 18
Link prediction and feature relevance in knowledge networks: A machine learning approach
Zinilli, A., & Cerulli, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290018 · Cited by 20
Half a century of Omega – The International Journal of Management Science: A bibliometric analysis
Hussain, W., Merigó, J. M., Rahimi, I., & Lev, B. (2024) · Omega · doi:10.1016/j.omega.2024.103226 · Cited by 18
University sustainability performance as a catalyst for societal change
de Villiers, C., Dimes, R., Houqe, M. N., Hu, N., & Molinari, M. (2025) · Accounting Auditing & Accountability Journal · doi:10.1108/aaaj-06-2024-7133 · Cited by 18
Suclupe-Navarro, P., Limaymanta, C. H., Ramírez, N., & Guillén, H. (2021) · Revista española de Documentación Científica · doi:10.3989/redc.2021.2.1753 · Cited by 18
Fleming, C. L., Golzan, M., Gunawan, C., & McGrath, K. C. Y. (2022) · Global Challenges · doi:10.1002/gch2.202200009 · Cited by 18
Challenges in building scholarly knowledge graphs for research assessment in open science
Manghi, P. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00322 · Cited by 20
From Integrity to Inflation: Ethical and Unethical Citation Practices in Academic Publishing
Plevris, V. (2025) · Journal of Academic Ethics · doi:10.1007/s10805-025-09631-1 · Cited by 17
Grit and L2 grit research in SLA (2013–2025)
González, C. F., & Ledo, M. (2025) · Studies in Second Language Acquisition · doi:10.1017/s0272263125100909 · Cited by 17
Evaluating Learners’ Satisfaction with a Distance Online CLIL Lesson During the Pandemic
Yang, W., & Yang, L.-Z. (2021) · English Teaching & Learning · doi:10.1007/s42321-021-00091-5 · Cited by 20
Malkawi, R., Daradkeh, M. K., Elhassan, A., & Petrov, P. (2022) · Information · doi:10.3390/info13110546 · Cited by 17
Zhang, Y., Wu, M., Zhang, G., & Lü, J. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24754 · Cited by 17
Investigating hybridity in artificial intelligence research
Williams, K., Berman, G., & Michalska, S. (2023) · Big Data & Society · doi:10.1177/20539517231180577 · Cited by 17
Hadiyawarman, Wisely, N., Iqbal, M., Timuda, G. E., Darsono, N., Yuliarto, B., & Khaerudini, D. S. (2023) · Clean Technologies and Environmental Policy · doi:10.1007/s10098-023-02686-x · Cited by 17
Journal of Language and Linguistic Studies: A fifteen-year bibliometric quest for a bigger impact
Syahid, A., & Qodir, A. (2021) · Journal of Language and Linguistic Studies · doi:10.17263/jlls.903415 · Cited by 17
Mejía, C., & Kajikawa, Y. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18073370 · Cited by 17
The Most Disruptive Publications in Craniofacial Surgery (1954–2014)
Horen, S. R., Hansdorfer, M. A., Kronshtal, R., Dorafshar, A. H., & Becerra, A. Z. (2021) · Journal of Craniofacial Surgery · doi:10.1097/scs.0000000000007804 · Cited by 17
Ojemaye, M. O., & Okoh, A. I. (2021) · International Journal of Energy Research · doi:10.1002/er.6907 · Cited by 17
Leader knowledge hiding and employee organizational identification in the Egyptian service industry
Abdelmotaleb, M., Mouri, N., & Saha, S. K. (2021) · Journal of Knowledge Management · doi:10.1108/jkm-09-2020-0722 · Cited by 17
Why employee engagement matters? Evidences from Delhi Metro Rail Corporation in India
Shrotryia, V. K., Saroha, K., & Dhanda, U. (2021) · Personnel Review · doi:10.1108/pr-07-2020-0513 · Cited by 19
Kunze, K. N., Manzi, J. E., Polce, E. M., Vadhera, A. S., Bhandari, M., & Piuzzi, N. S. (2022) · Internal and Emergency Medicine · doi:10.1007/s11739-022-02939-5 · Cited by 17
Gender disparities in Russian academia: a bibliometric analysis
Pilkina, M., & Lovakov, A. V. (2022) · Scientometrics · doi:10.1007/s11192-022-04383-w · Cited by 17
The Microbiome–Gut–Brain Axis and Dementia: A Bibliometric Analysis
Sun, H., Feng, Y., Zhang, Q., Li, J., Wang, Y., Su, Z., Cheung, T. C. C., Jackson, T., Sha, S., & Xiang, Y. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192416549 · Cited by 18
Mapping Co-creation Research in Tourism: A Bibliometric Visualisation Review
Orts-Cardador, J. J., Gálvez, J. C. P., Fuentes, G. M. G.-C., & Jara-Alba, C. (2025) · SAGE Open · doi:10.1177/21582440251335473 · Cited by 18
Badiuzzaman, M. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1619897 · Cited by 16
Aria, M., Cuccurullo, C., D’Aniello, L., & Spano, M. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101837 · Cited by 16
Bordoloi, T., Philip Shapira, A., & Mativenga, P. T. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.121347 · Cited by 16
Altmetrics and their relationship with citation counts: a case of journal articles in physics
Shrivastava, R., & Mahajan, P. (2022) · Global Knowledge, Memory and Communication · doi:10.1108/gkmc-07-2021-0122 · Cited by 16
Луговий, В. І., Драч, І. І., Петроє, О., Zinchenko, V. V., Mielkov, Y., Жиляєв, І., Регейло, І. Ю., & БАЗЕЛЮК, Н. В. (2021) · doi:10.31874/978-617-7644-53-7-2021 · Cited by 16
Política editorial de um periódico científico
Rode, S. D. M., Sales, D. P., & Alves, M. D. S. (2022) · doi:10.21452/abec.2022.isbn.978-65-89167-68-6.011 · Cited by 16
Carballo-Costa, L., Quintela-del-Rı́o, A., Costa, J. V., & Costas, R. (2022) · Physiotherapy Theory and Practice · doi:10.1080/09593985.2022.2073927 · Cited by 17
Hou, L., Wu, Q., & Xie, Y. (2022) · Scientometrics · doi:10.1007/s11192-022-04509-0 · Cited by 16
Tran, H. N., Bollinger, J., Salvestrini, S., Chu, K. H., & Juang, R. (2022) · Journal of Environmental Engineering · doi:10.1061/joeedu.eeeng-7074 · Cited by 16
Citation metrics and evaluation of journals and conferences
Gupta, S., Kumar, N., & Bhalla, S. (2023) · Journal of Information Science · doi:10.1177/01655515231151411 · Cited by 16
Potter, R. W. K., & Pearson, B. C. (2023) · npj Ocean Sustainability · doi:10.1038/s44183-023-00020-y · Cited by 16
The rise of empirical online platform research in the new millennium
Cheng, H. K., Sokol, D. D., & Zang, X. (2023) · Journal of Economics & Management Strategy · doi:10.1111/jems.12571 · Cited by 16
Prabowo, N. K., & Irwanto, I. (2023) · Journal of Engineering Education/Journal of engineering education transformations/Journal of engineering education transformation · doi:10.16920/jeet/2023/v37i2/23154 · Cited by 16
Pamboris, G. M., Plakias, S., Tsiakiri, A., Karakitsiou, G., Bebeletsi, P., Vadikolias, Κ., Aggelousis, N., Tsiptsios, D., & Christidi, F. (2024) · Sports · doi:10.3390/sports12100276 · Cited by 17
Application of geographic information system in ecotourism: a global bibliometric analysis
Amadu, F. O., Nhamo, L., Benzougagh, B., & Turyasingura, B. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2460711 · Cited by 16
Examining the pull, the push, and their simultaneous effects on managerial turnover
Choi, Y.-H., & Park, N. K. (2020) · Management Decision · doi:10.1108/md-06-2019-0822 · Cited by 16
Ranking institutions within a discipline: The steep mountain of academic excellence
Sziklai, B. R. (2021) · Journal of Informetrics · doi:10.1016/j.joi.2021.101133 · Cited by 16
Top 100 cited articles on diabetes mellitus and Covid-19: A bibliometric analysis
Vishwanathan, K., Kambhampati, S. B. S., & Vaishya, R. (2021) · Diabetes & Metabolic Syndrome Clinical Research & Reviews · doi:10.1016/j.dsx.2021.05.033 · Cited by 16
The effect of TQM practices on INGOs' staff work-related attitudes
Karia, N. B., & Saleh, F. I. M. (2021) · Benchmarking An International Journal · doi:10.1108/bij-02-2021-0080 · Cited by 16
Tica, A., & Weißenberger, B. E. (2021) · Journal of Accounting & Organizational Change · doi:10.1108/jaoc-11-2020-0196 · Cited by 16
Are Chinese cities getting smarter in terms of knowledge and technology they produce?
van der Wouden, F. (2021) · World Development · doi:10.1016/j.worlddev.2021.105729 · Cited by 16
The Literature of Chemoinformatics: 1978–2018
Willett, P. (2020) · International Journal of Molecular Sciences · doi:10.3390/ijms21155576 · Cited by 16
Kvæl, L. A. H., & Bergland, A. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06175-z · Cited by 16
Duarte, I., dos Santos, H. G., & Rodrigues, M. L. (2025) · Anais da Academia Brasileira de Ciências · doi:10.1590/0001-3765202520241035 · Cited by 15
Ghaffari, F., Shabak, M., Norouzi, N., & Fallah, S. N. (2021) · International Journal of Building Pathology and Adaptation · doi:10.1108/ijbpa-05-2021-0071 · Cited by 15
Kassis, M., Giacaman, R., & Hashweh, M. Z. (2022) · Middle East Critique · doi:10.1080/19436149.2022.2052466 · Cited by 17
Hermanu, A. I., Sari, D., Sondari, M. C., & Dimyati, M. (2023) · Journal of Science and Technology Policy Management · doi:10.1108/jstpm-05-2022-0089 · Cited by 15
Garrido, R. H., Orts-Cardador, J. J., & Calañas, C. P. (2023) · ENLIGHTENING TOURISM A PATHMAKING JOURNAL · doi:10.33776/et.v13i2.8029 · Cited by 15
The 100 Most Cited Papers in the History of the American Surgical Association
Landreneau, J. P., Weaver, M., Delaney, C. P., Aminian, A., Dimick, J. B., Lillemoe, K. D., & Schauer, P. R. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000003633 · Cited by 15
Schulz, A. N., Lucardi, R. D., & Marsico, T. D. (2020) · Annals of the Entomological Society of America · doi:10.1093/aesa/saaa052 · Cited by 15
The rise of UK–China research collaboration: Trends, opportunities and challenges
Adams, J. R., Johnson, J., & Grant, J. (2021) · Science and Public Policy · doi:10.1093/scipol/scab069 · Cited by 15
Empfehlungen zur Bewertung wissenschaftlicher Leistungen bei Berufungsverfahren in der Psychologie
Gärtner, A., Leising, D., & Schönbrodt, F. D. (2023) · Psychologische Rundschau · doi:10.1026/0033-3042/a000630 · Cited by 15
Wu, J. F., Nivargi, R., Lanka, S. S. T., Menon, A. M., Modukuri, S. A., Nakshatri, N. S., Wei, X., Wang, Z., Caverlee, J., Rajtmajer, S. M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04580 · Cited by 22
Scelles, N., & da Silva, J. A. T. (2025) · Scientometrics · doi:10.1007/s11192-025-05268-4 · Cited by 14
Trust in the academy: a conceptual framework for understanding trust on academic web profiles
Francke, H. (2021) · Journal of Documentation · doi:10.1108/jd-01-2021-0010 · Cited by 14
An Assessment of the U.S. and Chinese Industrial Bases in Quantum Technology
Parker, E., Gonzales, D., Kochhar, A., Litterer, S., O'Connor, K., Schmid, J., Scholl, K., Silberglitt, R., Chang, J., Eusebi, C. A., et al. (2022) · RAND Corporation eBooks · doi:10.7249/rra869-1 · Cited by 14
Las revistas científicas-académicas y el proceso editorial hacia la ciencia abierta en Uruguay
Jordi, M. D. C. L., Valenzuela, C., Morelli, T., Caubet, Y. F., & Suárez, S. L. P. (2022) · doi:10.21452/abec.2022.isbn.978-65-89167-68-6.014 · Cited by 14
Adeniji, A. O., Okaiyeto, K., George, M. J., Tanor, E. B., Semerjian, L., & Okoh, A. I. (2022) · Environmental Geochemistry and Health · doi:10.1007/s10653-022-01353-2 · Cited by 14
A longitudinal analysis of highly cited papers in four CALL journals
Choubsaz, Y., Jalilifar, A., & Boulton, A. (2023) · ReCALL · doi:10.1017/s0958344023000137 · Cited by 14
Authorship and citation patterns of highly cited biomedical researchers: a cross-sectional study
Perneger, T. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00137-1 · Cited by 14
The Emergence of the Open Research University Through International Research Collaboration
Heiser, R. E. (2023) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v24i3.7328 · Cited by 14
Citation Ethics: An Exploratory Survey of Norms and Behaviors
Bruton, S. V., Macchione, A. L., Brown, M., & Hosseini, M. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-024-09539-2 · Cited by 14
Mapping Scientific Knowledge of Renewable Energy and Tourism
Jiménez-Islas, D., Pérez-Romero, M. E., Álvarez‐García, J., & del Río‐Rama, M. D. L. C. (2024) · Sustainability · doi:10.3390/su16156356 · Cited by 14
Noaman, A. Y. M., Gad-Elrab, A. A. A., & Baabdullah, A. M. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100516 · Cited by 16
Abildinova, G. Z., Benberin, V. V., Vochshenkova, T. A., Afshar, A., Mussin, N. M., Kaliyev, A. A., Zhussupova, Z., & Tamadon, A. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1452227 · Cited by 15
Metadata
- Authors
- Aksnes, D. W., Langfeldt, L., & Wouters, P.
- Affiliations
- Nordic Institute for Studies in Innovation, Research and Education (NO)
- Leiden University (NL)
- Corresponding author
- Aksnes, Dag W.
- Venue
- Sage Open
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 9
- Issue
- 1
- ISSN
- 2158-2440 (ISSN-L)
- OpenAlex
- W2913788248
- Semantic Scholar
- 59660bbee2813e0ddc65cd8e7fa35ea51f58b6e7
- Cited by
- 1,257
- References
- 132
- Citations by year
- 2026: 153 · 2025: 207 · 2024: 223 · 2023: 202 · 2022: 181 · 2021: 177 · 2020: 89 · 2019: 23
- FWCI
- 78.79
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- SAGE Open (publishedVersion, cc-by)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Norges Forskningsråd (256223)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- citation, citation metrics, research quality, research evaluation, research funding, scientific impact, originality, social values, plausibility, science policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
