The DOI matches its Crossref record; 143 of the 144 references it cites were found in a registry or an index; none of the 86 sources its claims cite is checked for support yet.
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Open accessAbstract
Citations are increasingly used as performance indicators in research policy and within the research system. Usually, citations are assumed to reflect the impact of the research or its quality. What is the justification for these assumptions and how do citations relate to research quality? These and similar issues have been addressed through several decades of scientometric research. This article provides an overview of some of the main issues at stake, including theories of citation and the interpretation and validity of citations as performance measures. Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics. The article investigates how citations may relate to these various research quality dimensions. It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations. On the contrary, there is no evidence that citations reflect other key dimensions of research quality. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This review argues that citations can serve as limited indicators of scientific impact and relevance but not of other key research-quality dimensions such as solidity/plausibility, originality, or societal value. It synthesizes citation theories, validation studies, and quality dimensions to caution against using citation indicators as substitutes for peer review.
Key findings
- Citations are argued to reflect aspects of scientific impact and relevance, but with important limitations; there is no evidence that they reflect other key dimensions of research quality—solidity/plausibility, originality, and societal value.
- Comparative validation studies generally find moderately positive correspondence between citation indicators and peer review, but correlations are far from perfect; in REF 2014, field-weighted citation impact had a Spearman correlation of .28 at the overall level, with significant field variation and decreases for more recent outputs, leading to the conclusion that metrics cannot provide a like-for-like replacement for peer review.
- Citation distributions are highly skewed: in many humanities disciplines an average paper receives less than one citation over 10 years, versus more than 40 in some biomedical fields (Web of Science 2005–2015); the Lowry et al. (1951) protein-measurement paper has more than 305,000 Web of Science citations, and the 4S statin trial paper more than 7,700.
- The h-index is popular but not field-normalized and not corrected for career length, disadvantaging younger researchers; field, publication-year, and document-type normalization is standard, but there is no general agreement on the most appropriate normalization method.
Methods
- Design: narrative/conceptual overview and selective literature review, not a systematic review or new empirical study.
- Data or sample: no new dataset; the article draws on prior scientometric literature, citation theories, validation studies, and selected examples.
- Main analysis: conceptual synthesis of citation indicators, referencing behavior, peer-review comparisons, and dimensions of research quality.
Limitations
- Stated: coverage of topics and literature is selective and does not discuss all details; the literature on interaction between citing practices and evaluation processes is only referred to in passing; constructivist and semiotic theories of quality and citation are not discussed.
- Stated/evident: few previous studies have systematically addressed how citations relate to each research-quality dimension, and the final part of the article is described as more explorative and discursive.
- Evident: the paper presents no systematic search strategy or empirical test of its central claims, so its conclusions rely on narrative synthesis and selected examples.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
76 Not checked yet · 44 Own interpretation · 16 No basis given
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the contrary, there is no evidence that citations reflect other key dimensions of research quality.”
- Also stated in: § Concluding Remarks · p. 12 · plain
“there is no evidence that citations reflect other key dimensions of research quality”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.
- Also stated in: § Concluding Remarks · p. 12 · plain
In the paper
“The larger part of all scientific papers are never cited or cited a few times only in the subsequent scientific literature”
What the sources say
- Not checked yetcites Aksnes, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Citation distributions are very skewed.”
What the sources say
- Not checked yetcites Price (1965)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“There are large variations in average citation rates across different subject areas.”
What the sources say
- Not checked yetcites data from Web of Science 2005-2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“but empirical studies have shown that two different methods for normalization, AoR and RoA, did not produce very different results, particular at the level of countries and institutions”
What the sources say
- Not checked yetcites Lariviere & Gingras, 2011
- Not checked yetcites Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the contrary, some articles have an extremely large number of citations, reaching into the hundreds and even thousands.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“There is no general agreement on what is the most appropriate method”
What the sources say
- Not checked yetcites Ioannidis, Boyack, & Wouters, 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Although important insights on the role of references in the scientific article have been obtained, the accumulation of knowledge at the same time has been hampered by the fact that different classification systems have been applied in previous studies”
What the sources say
- Not checked yetcites Liu, 1993
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“References to highly cited publications are more often present in the introduction than in other parts of the publications”
What the sources say
- Not checked yetcites Voos & Dagaev, 1976
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Because of this, articles describing commonly used methods may receive a very large number of citations.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 42
- Probably not a substantial claim. 58
- The quote only partly states this claim. 1
In the paper
“Most importantly, it is not field-normalized and no corrections are made for career length, which means that the indicator disfavors younger researchers”
What the sources say
- Not checked yetcites Alonso, Cabrerizo, Herrera-Viedma, & Herrera, 2009
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The views of the MacRoberts's previously led to much debate, but their conclusions are generally seen as too sweeping”
What the sources say
- Not checked yetcites Aksnes, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Even now, in spite of detailed studies of referencing behavior, there is no unified theory.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As indicated above, previous studies of the citation process have not provided any simple answer to the question of what citations stand for.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While empirical studies have revealed a multitude of factors involved in the citation process, the issue has also been approached from another angle: by comparing citation indicators with the outcome of peer review. During recent decades, many such studies have been carried out.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nevertheless, some overall findings remain: the references have a multitude of functions in the scientific article, only a small proportion of the relevant literature is cited, and the authors have a multitude of motives for including particular studies as references.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Studies undertaken have revealed that the role of the reference, both in the citing text and with respect to the cited text, is complex.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Moreover, the citing is biased: some sources are cited essentially every time they are used, while other research is never cited even though it may be used more often than the highly cited work.”
What the sources say
- Not checked yetcites MacRoberts & MacRoberts, 1989, 1996
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These and later studies have revealed that the references have a multitude of functions in the scientific article.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, the studies are often based on rather small samples of papers from selected scientific fields, and the results may not have general validity.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Based on empirical case studies, they showed that a very small proportion of the knowledge basis of an article (consisting of hundreds or thousands of former publications) actually are cited.”
What the sources say
- Not checked yetcites MacRoberts & MacRoberts, 1989, 1996
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Therefore, it will be more persuasive to cite an authoritative paper, and authors tend to select references that will be regarded as authoritative by the intended audience.”
What the sources say
- Not checked yetcites Gilbert, 1977
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This article has now been cited more than 305,000 times in the Web of Science database”
What the sources say
- Not checked yetcites Van Noorden, Maher, & Nuzzo, 2014
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In a scientific article, some of the references will represent works that are crucial or significant antecedents to the present work; others may represent more general background literature”
What the sources say
- Not checked yetcites Aksnes, 2005
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Hence, the textual functions of citations vary considerably.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“For the citation process, this implies that other incentives shape the citing patterns, like creating visibility for one's work through self-citations or citing a journal editor's work as an attempt to enhance the chances of acceptance for publication.”
What the sources say
- Not checked yetcites Bornmann & Daniel, 2008
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“There is a growing concern about this issue, as well as on potential negative impact of research metrics on the scientific community.”
What the sources say
- Not checked yetcites Hicks et al., 2015
- Not checked yetcites Cagan, 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Citation measures are also core indicators in several university rankings, such as the Leiden ranking and Academic Ranking of World Universities (ARWU)”
What the sources say
- Not checked yetcites Piro & Sivertsen, 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In recent years, bibliometric indicators have increasingly been applied in the context of research evaluation as well as research policy more generally.”
What the sources say
- Not checked yetcites Moed, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For example, Levitt and Thelwall (2011) have argued that short citation windows have the problem that articles published earlier in a year have a significant advantage (i.e., are on average more highly cited) compared with publications appearing later in a year.”
What the sources say
- Not checked yetcites Levitt and Thelwall (2011)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Only a small fraction of the scholarly literature in the humanities is represented in the Web of Science, and most of the references and citations will not be captured by the database.”
What the sources say
- Not checked yetcites Marx and Bornmann (2015)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The coverage of the scientific and scholarly literature varies across these databases, and the results of citation studies are thus dependent upon the particular characteristics of the databases and their coverage.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In the scientific paper, the references have various purposes. Authors are not including references merely because of their scientific quality.”
What the sources say
- Not checked yetcites Bornmann & Daniel, 2008
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It has since been the standard to adjust for field, publication year, and publication type when calculating citation indicators.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Moreover, the application of citation indicators has also been criticized more generally, with respect to their validity as performance measures and their potentially negative impact upon the research system”
What the sources say
- Not checked yetcites MacRoberts & MacRoberts, 1989
- Not checked yetcites Osterloh & Frey, 2015
- Not checked yetcites Weingart, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When one article is cited by many subsequent publications, even more people become aware of this article. Thus, its visibility, and thereby the chances of getting even more citations, increases.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the studies, assessments by peers have been typically considered as a kind of standard to which citation indicators can be validated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some of these problems have a fundamental character and are inherent in any use of citations as indicators, others may be resolved by the construction of more advanced indicators, while others again may be of less importance in practice.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, panels increasingly are considering citation measures as part of the evaluation procedure, which means that the two cannot be considered as completely independent of each other.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If authors cite the works they find useful, frequently cited publications may be assumed to have been more useful than papers which are hardly cited at all.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, it may be concluded that most of the comparative studies seem to have found a moderately positive correspondence, but the correlations identified have been far from perfect and have varied among the studies.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For the field-weighted citation impact, a Spearman correlation coefficient of .28 was identified at an overall level, albeit with significant variations across fields. Moreover, there were significant decreases in correlation for more recent outputs.”
What the sources say
- Not checked yetcites Higher Education Funding Council for England, 2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, a large number of different citation measures exist and the outcome would also depend on which indicators are selected for the comparative analysis.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
In the paper
“A significant degree of concordance was found '. . . supporting the choice of using both techniques in order to assess the quality of Italian research institutions' (Ancaiani et al., 2015, p. 254).”
What the sources say
- Not checked yetcites Ancaiani et al., 2015
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Although most citation analysts seem to agree that citing or referencing is biased, it has been argued that this bias is not fatal for the use of citation as performance indicators-to a certain extent, the biases are averaged out at aggregated levels.”
hedgedconditional- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“According to Luukkonen (1990), the presence of different cognitive meanings of citations and motivations for citing does not necessarily invalidate the use of citations as (imperfect) performance measures.”
What the sources say
- Not checked yetcites Luukkonen (1990)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Problems related to more technical issues, such as discrepancies between target articles and cited references (misspellings of journal names, author names, errors in the reference lists, etc.), and mistakes in the indexing procedures conducted by Clarivate Analytics (previously Thomson Reuters) or Elsevier (Leydesdorff et al., 2016; Moed, 2002) may confuse citation analyses. Such errors affect in”
What the sources say
- Not checked yetcites Leydesdorff et al., 2016
- Not checked yetcites Moed, 2002
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Aksnes and Taxt (2004) analyzed the relationship between bibliometric indicators and the outcomes of a peer review of Norwegian research groups at a mathematical and natural science faculty, reporting positive but weak correlations.”
What the sources say
- Not checked yetcites Aksnes and Taxt (2004)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that the different functions the references may have in a text are much more complex than merely providing documentation and support for particular claims.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Gilbert (1977) argued that citing ('referencing') is essentially a device for persuasion.”
What the sources say
- Not checked yetcites Gilbert (1977)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Small (1982) identified five distinctions: A cited work may be (a) refuted, (b) noted only, (c) reviewed, (d) applied, or (e) supported by the citing work.”
What the sources say
- Not checked yetcites Small (1982)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Typically, a scientific article is structured as a progression from the general to the particular”
What the sources say
- Not checked yetcites Law, 1986
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Although the literature coverage of citation databases has improved (Web of Science and Scopus), the coverage of the humanities and several social science disciplines remains limited (Waltman, 2016).”
What the sources say
- Not checked yetcites Waltman, 2016
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As is evident from the overview above, there is no simple answer to the question what citation indicators measure or indicate.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, both the methodological basis for comparing peer assessments and citation indicators and the assumption that the two may be expected to correlate may be questionable.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, citation frequencies are determined by other mechanisms and are not a simple reflection of the quality dynamics.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“According to Merton's view, the norms of science oblige researchers to cite the work upon which they draw, and in this way acknowledge or credit contributions by others”
What the sources say
- Not checked yetcites Merton, 1979
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“He found that the h-index and the normalized citation impact indicator both correlated quite well with peer judgments.”
What the sources say
- Not checked yetcites van Raan (2006)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“According to Bornmann and Daniel (2008), many studies have methodological weaknesses and have provided findings with little reliability.”
What the sources say
- Not checked yetcites Bornmann and Daniel (2008)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“However, the methodological fundament for this conclusion has been contested by Baccini and De Nicolao (2016), who argue that the analysis is flawed and that informed peer review and bibliometrics do not produce similar results.”
What the sources say
- Not checked yetcites Baccini and De Nicolao (2016)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Rinia, van Leeuwen, van Vuren, and van Raan (1998) showed that various citation indicators correlated significantly with peer ratings of research programs in condensed matter physics.”
What the sources say
- Not checked yetcites Rinia, van Leeuwen, van Vuren, and van Raan (1998)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“According to the founder of the SCI, the fact that authors do not cite all their influences does not invalidate the use of citations as performance measures when enough literature is taken into account”
What the sources say
- Not checked yetcites Garfield, 1997
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
In the paper
“The fact that citation distributions are extremely skewed also raises questions concerning the use of mean as indicator, and Bornmann and Mutz (2011) have proposed to use percentile ranks as a non-parametric alternative to the means of citation distributions for the normalization.”
What the sources say
- Not checked yetcites Bornmann and Mutz (2011)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“According to Welljams-Dorof (1997), this has important implications: In general, the larger the citation data set being used, the higher the confidence level of the results. Analyses involving entire fields of research, nations, regions and large universities are virtually unaffected by the concerns and caveats about citation data . . . The confidence level at these large aggregate levels is quite”
What the sources say
- Not checked yetcites Welljams-Dorof (1997)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the use of impact as the most appropriate concept has usually been justified by theoretical considerations, and there are few attempts to address the issue empirically or relate it to previous findings on citation behavior.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Still, a basic limitation is that many innovations are not patented and patents are not suitable to assess societal relevance or impact in a broader context.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown above, the question on the relation between citations and research quality is complex and will arise differently depending on the field analyzed, the database used, the timeframe and indicators applied, and so forth.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“According to Waltman, van Eck, and Wouters (2013), citation impact should be distinguished from scientific impact, as an influential researcher sometimes has a lower performance in terms of highly cited publications than some of their less influential colleagues.”
What the sources say
- Not checked yetcites Waltman, van Eck, and Wouters (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the viewpoint described in the introduction that bibliometric assessment is superior compared to the traditional peer-review method is not justified in our opinion.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the referee system does not fully ensure the quality dimension related to solidity and plausibility, and there are no indications that high citation counts reflect solidity.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“However, considering the various limitations attached to citations as performance measures, most bibliometricians have argued that a bibliometric analysis should not function as a substitute for a peer review (Moed, 2005).”
What the sources say
- Not checked yetcites Moed, 2005
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Citation indicators may easily be misused or applied in contexts where they lack justification or validity.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition, research quality is a multidimensional concept, and in this section, we will look further into this issue.”
What the sources say
- Not checked yetcites Polanyi (1962)
- Not checked yetcites Gulbrandsen, 2000
- Not checked yetcites Lamont, 2009
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As an example, even a 'mistaken' publication can have a large impact by stimulating further research.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because of the sociological and aggregational processes involved. In the beginning, an article may be cited for substantive reasons (e.g., its content has been used). Later, when the article is widely known and has received many citations, sociological mechanisms will be of increasing importance (authors citing authoritative papers, the bandwagon effect, etc.). Some papers will benefit”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The view generally held among experts within bibliometrics seem to be that citations represent a good but not perfect impact measure.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As is evident from the discussion of this paper, this is an important point, as citations are not able to capture all aspects of the quality concept. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As a starting point, we can take the three dimensions distinguished by Polanyi (1962): plausibility, originality, and scientific value.”
What the sources say
- Not checked yetcites Polanyi (1962)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Impact is the most commonly used concept for what citations reflect, although other concepts such as influence, importance, significance, and utility occasionally also are used (Moed, 2005).”
What the sources say
- Not checked yetcites Moed, 2005
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“There seems to be a widespread assumption that this issue cannot be adequately assessed through standard citation indicators, and in recent years, increasing attention has been devoted to developing methodologies for assessing and measuring societal relevance and impact (Bornmann, 2012, 2013).”
What the sources say
- Not checked yetcites Bornmann, 2012, 2013
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, there is a lack of empirical studies confirming that this is actually the case, and possibly some of the biases is of a fundamental nature attached to all citations measures, while the effect of others may tend to level out when aggregated levels are considered.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Societal relevance is often considered to be something which is much harder to measure than scientific relevance or impact (B. R. Martin, 2011).”
What the sources say
- Not checked yetcites B. R. Martin, 2011
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, all assessments of research quality may be context-dependent, in terms of, for example, the time of assessment and the time/field/sector perspectives of the evaluators.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Using these definitions, it is also evident that impact would be a more adequate interpretation of citations than quality.”
What the sources say
- Not checked yetcites B. R. Martin and Irvine, 1983
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite these and numerous similar examples, it is not possible to identify societal relevance from citation counts per se, and uncited or little cited publication may have contributed with results of great societal relevance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Because of this, it has long been emphasized by bibliometricians that more than one indicator should be used in research evaluation contexts (van Raan, 1993).”
What the sources say
- Not checked yetcites van Raan, 1993
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“citations reflect-with important limitations-aspects related to scientific impact and relevance”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The literature contains numerous publications of which the solidity is poor, the results unreliable or even involving misconduct or scientific fraud (Fanelli, 2009).”
What the sources say
- Not checked yetcites Fanelli, 2009
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For example, the mean normalized citation score is size-independent and does not take into account the number of publications. According to Abramo and D'Angelo (2016), this is a major problem with this indicator because it does not truly represent productivity.”
What the sources say
- Not checked yetcites Abramo and D'Angelo (2016)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, this topic has rarely been addressed specifically in the literature and there are few studies analyzing the issue empirically.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“On this basis, it is often argued that bibliometric analysis can counterbalance shortcomings and mistakes in the peers' judgments (Aksnes, 2005). Thus, a bibliometric study should be considered as complementary to a peer evaluation (Council of Canadian Academies, 2012).”
What the sources say
- Not checked yetcites Aksnes, 2005
- Not checked yetcites Council of Canadian Academies, 2012
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In sum, whereas plausibility/soundness, scientific value, and societal value and originality seem commonly perceived key characteristics of research quality, each of these dimensions include a variety of aspects; they may be contextdependent and may also conflict with each other.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“At the same time, there are also problems with the peer-review system.”
What the sources say
- Not checked yetcites Eyre-Walker and Stoletzki (2013)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper relies on this without support.
Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.
“Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics.”
The authors state this as an assumption.
There should be a correlation between citation indicators and peer assessments if citations legitimately can be used as indicators of scientific performance.
“The basic assumption is that there should be a correlation if citations legitimately can be used as indicators of scientific performance.”
The paper relies on this without support.
Peer reviews are applied in many different contexts, of which peer assessments of manuscripts submitted to journals and publishers is the most fundamental one.
“Peer reviews are applied in many different contexts, of which peer assessments of manuscripts submitted to journals and publishers probably is the most fundamental one.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is often argued that the scientific impact of any evaluated entity cannot be determined solely by citations due to the different citing behaviors and resulting incompatible citation counts between disciplines [1-3,5,260,267].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
In the citing paper
“Citations may be appropriate in assessing the scientific impact of the research, but they do not show the impact of the research outside the scientific community [1,3,5,6,260,264,265].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
In the citing paper
“peer-reviewing has its own characteristic biases and limitations, which is one of the reasons why, nowadays, peer-review has been increasingly extensively replaced with bibliometric measures of impact [3,44,276].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
In the citing paper
“this 'publish or perish' attitude, on the one hand, stimulates the productivity of researchers, but, on the other, creates a bias towards quantity, often overlooking quality [2,3,6-8,44,81,163,260,263,284-288].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Corresponding Author: · p. 2
“Many previous studies have compared citation indicators with the outcome of peer review, and in the 'Understanding Citations' part, this issue is examined.”
§ Validation Studies · p. 6
“## Validation Studies While empirical studies have revealed a multitude of factors involved in the citation process, the issue has also been approached from another angle: by comparing citation indicators with the outcome of peer review.”
Citations
Cites
PROTEIN MEASUREMENT WITH THE FOLIN PHENOL REAGENT
Lowry, O. H., Rosebrough, N., Farr, A. L., & Randall, R. (1951) · Journal of Biological Chemistry · doi:10.1016/s0021-9258(19)52451-6 · Cited by 316,394
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720
(1968) · PubMed · Cited by 4,803
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“However, the recent attention to the lack of replicable results in biomedical, clinical, and psychological studies (Ioannidis, 2005) may lead to a higher social status of replications studies.”
BackgroundPrice, D. J. D. S. (1965) · Science · doi:10.1126/science.149.3683.510 · Cited by 3,205
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828
Self-citationFanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“The literature contains numerous publications of which the solidity is poor, the results unreliable or even involving misconduct or scientific fraud (Fanelli, 2009).”
BackgroundGoogle Scholar, Scopus and the Web of Science: A Longitudinal and Cross-Disciplinary Comparison
Harzing, A.-W., & Alakangas, S. (2016) · Scientometrics · doi:10.1007/s11192-015-1798-9 · Cited by 1,784
Research works printed but not linked: 15
- Biagioli M., Gaming Metrics. Beyond Publish or Perish: Metrics and the new Ecologies of Academic Misconduct
- Carlsson H., 2009, InfoTrend, 64, 82
- Dymond L. H., 2015, A recent history of recognized economic thought—Contributions of the Nobel laureates to economic science
- Garfield E., 1977, Essay of an information scientist, 1, 84
- Gulbrandsen J. M., 2000, Research quality and organisational factors: An investigation of the relationship
- Higher Education Funding Council for England. (2015). The metric tide: Correlation analysis of REF2014 scores and metrics: Supplementary report II to the independent review of the role of metrics in research assessment and management. Retrieved from www.dcscience.net/2015_metrictideS2.pdf
- Langfeldt L., Scordato L. (2015, April). Assessing the broader impacts of research: A review of methods and practices (NIFU Working Paper 8/2015). Oslo.
- le Pair C, 1995, International Forum on Information and Documentation, 20, 16
- Luukkonen T., 1990, Citations in the rhetorical, reward, and communication systems of science
- Merton R. K., 1979, Citation indexing—Its theory and application in science, technology, and humanities
- Smith L. C., 1981, Library Trends, 30, 83
- van Raan A. F. J, 2000, The web of knowledge: A Festschrift in honor of Eugene Garfield, 301
- Welljams-Dorof A., 1997, Evaluating science and scientists: An East-West dialogue on research evaluation in post-communist Europe, 202
- Wouters P. (1999b). The citation culture (Doctoral thesis). University of Amsterdam, The Netherlands. Retrieved from http://garfield.library.upenn.edu/wouters/wouters.pdf
- Wouters P., Handbook of science policy studies
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696
The extent and drivers of gender imbalance in neuroscience reference lists
Dworkin, J. D., Linn, K. A., Teich, E. G., Zurn, P., Shinohara, R. T., & Bassett, D. S. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0658-y · Cited by 543
Leadership styles and sustainable performance: A systematic literature review
Piwowar‐Sulej, K., & Iqbal, Q. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.134600 · Cited by 372
Langham-Putrow, A., Bakker, C., & Riegelman, A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253129 · Cited by 291
Social companionship with artificial intelligence: Recent trends and future avenues
Chaturvedi, R., Verma, S., Das, R., & Dwivedi, Y. K. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122634 · Cited by 244
de la Poza, E., Merello, P., Barberá, A., & Celani, A. (2021) · Sustainability · doi:10.3390/su13042038 · Cited by 217
Al‐Mamary, Y. H., Abdulrab, M., Alwaheeb, M. A., & Alshammari, N. G. M. (2020) · Education + Training · doi:10.1108/et-04-2020-0096 · Cited by 229
The Newcastle–Ottawa Scale for Assessing the Quality of Studies in Systematic Reviews
Gualdi-Russo, E., & Zaccagni, L. (2026) · Publications · doi:10.3390/publications14010004 · Cited by 233
Diverse approaches to learning with immersive Virtual Reality identified from a systematic review
Won, M., Ungu, D. A. K., Matovu, H., Treagust, D. F., Tsai, C., Park, J., Mocerino, M., & Tasker, R. (2022) · Computers & Education · doi:10.1016/j.compedu.2022.104701 · Cited by 204
Delcker, J., Heil, J., Ifenthaler, D., Seufert, S., & Spirgi, L. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00452-7 · Cited by 198
Interpreting Bibliometric Data
Szomszor, M., Adams, J. R., Fry, R., Gebert, C., Pendlebury, D., Potter, R. W. K., & Rogers, G. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2020.628703 · Cited by 203
Knowledge-hiding behaviors and employees’ silence: mediating role of psychological contract breach
Bari, M. W., Ghaffar, M., & Ahmad, B. (2020) · Journal of Knowledge Management · doi:10.1108/jkm-02-2020-0149 · Cited by 182
Trade credit research before and after the global financial crisis of 2008 – A bibliometric overview
Pattnaik, D., Hassan, M. K., Kumar, S., & Paul, J. (2020) · Research in International Business and Finance · doi:10.1016/j.ribaf.2020.101287 · Cited by 174
Piwowar‐Sulej, K. (2021) · Sustainable Development · doi:10.1002/sd.2166 · Cited by 165
Dougherty, M. R. P., & Horne, Z. (2022) · Royal Society Open Science · doi:10.1098/rsos.220334 · Cited by 141
Review of Organizational Ambidexterity Research
Kassotaki, O. A. (2022) · SAGE Open · doi:10.1177/21582440221082127 · Cited by 126
How much is too much? The difference between research influence and self-citation excess
Szomszor, M., Pendlebury, D., & Adams, J. R. (2020) · Scientometrics · doi:10.1007/s11192-020-03417-5 · Cited by 133
Worldwide research trends on tumor burden and immunotherapy: a bibliometric analysis
Zhang, L., Zheng, H., Jiang, S., Liu, Y.-G., Zhang, T., Zhang, J., Lü, X., Zhao, H., Sang, X., & Xu, Y. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001022 · Cited by 120
The pace of artificial intelligence innovations: Speed, talent, and trial-and-error
Tang, X., Li, X., Ding, Y., Song, M., & Bu, Y. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101094 · Cited by 131
Ahmad, S., Omar, R., & Quoquab, F. (2020) · Journal of Family Business Management · doi:10.1108/jfbm-12-2019-0081 · Cited by 111
Bibliometrics and systematic reviews: A comparison between the Proknow-C and the Methodi Ordinatio
de Carvalho, G. D. G., Sokulski, C. C., da Silva, W. V., de Carvalho, H. G., de Moura, R. V., de Francisco, A. C., & da Veiga, C. P. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101043 · Cited by 122
From the store to omnichannel retail: looking back over three decades of research
Hänninen, M., Kwan, S. K., & Mitronen, L. (2020) · The International Review of Retail Distribution and Consumer Research · doi:10.1080/09593969.2020.1833961 · Cited by 106
Aljuwaiber, A. (2020) · Journal of Entrepreneurship in Emerging Economies · doi:10.1108/jeee-08-2019-0123 · Cited by 105
Vélez-Estévez, A., García‐Sánchez, P., Moral-Muñoz, J. A., & Cobo, M. J. (2022) · Scientometrics · doi:10.1007/s11192-022-04486-4 · Cited by 117
Proliferation and Apoptosis Pathways and Factors in Oral Squamous Cell Carcinoma
He, S., Chakraborty, R., & Ranganathan, S. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms23031562 · Cited by 102
Panda, D. K., & Ray, S. (2022) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00698-5 · Cited by 98
Kunze, K. N., Polce, E. M., Vadhera, A. S., Williams, B. T., Nwachukwu, B. U., Nho, S. J., & Chahla, J. A. (2020) · The American Journal of Sports Medicine · doi:10.1177/0363546520903703 · Cited by 97
Ecosystem services and life cycle assessment: A bibliometric review
VanderWilde, C. P., & Newell, J. P. (2021) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2021.105461 · Cited by 93
Scientific laws of research funding to support citations and diffusion of knowledge in life science
Mosleh, M., Roshani, S., & Coccia, M. (2022) · Scientometrics · doi:10.1007/s11192-022-04300-1 · Cited by 89
Isager, P. M., van Aert, R. C. M., Bahník, Š., Brandt, M. J., DeSoto, K. A., Giner‐Sorolla, R., Krueger, J. I., Perugini, M., Ropovik, I., Veer, A. V. '., et al. (2021) · Psychological Methods · doi:10.1037/met0000438 · Cited by 94
Research on Small and Medium-Sized Towns: Framing a New Field of Inquiry
Wagner, M., & Growe, A. (2021) · World · doi:10.3390/world2010008 · Cited by 85
Umeokafor, N. I., Umar, T., & Evangelinos, K. I. (2022) · Safety Science · doi:10.1016/j.ssci.2022.105897 · Cited by 84
What is going on in entrepreneurship research? A bibliometric and SNA analysis
Lillo, F. G., Seva‐Larrosa, P., & Sánchez‐García, E. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2022.113624 · Cited by 83
Shen, K., & Shen, H. J. (2020) · Tourism Review · doi:10.1108/tr-11-2019-0451 · Cited by 82
Loorbach, D. A., & Wittmayer, J. M. (2023) · Sustainability Science · doi:10.1007/s11625-023-01335-y · Cited by 93
Bibliometric analysis methods for the medical literature
Ganti, L., Persaud, N. A., & Stead, T. S. (2025) · Academic Medicine & Surgery · doi:10.62186/001c.129134 · Cited by 75
Drug-Drug Interactions and Synergy: From Pharmacological Models to Clinical Application
Calzetta, L., Page, C. P., Matera, M. G., Cazzola, M., & Rogliani, P. (2024) · Pharmacological Reviews · doi:10.1124/pharmrev.124.000951 · Cited by 78
Development of socially sustainable transport research: A bibliometric and visualization analysis
Bao, L., Kusadokoro, M., Chitose, A., & Chen, C. (2022) · Travel Behaviour and Society · doi:10.1016/j.tbs.2022.08.012 · Cited by 72
Pinto, T. D., & Teixeira, A. A. C. (2020) · Scientometrics · doi:10.1007/s11192-020-03419-3 · Cited by 71
Irwanto, I. (2021) · European Journal of Educational Research · doi:10.12973/eu-jer.10.4.2045 · Cited by 71
Raza, S. A., Qazi, W., & Yousufi, S. Q. (2021) · Journal of Applied Research in Higher Education · doi:10.1108/jarhe-03-2020-0065 · Cited by 70
Shareefa, M., & Moosa, V. (2020) · European Journal of Educational Research · doi:10.12973/eu-jer.9.1.331 · Cited by 68
In which fields are citations indicators of research quality?
Thelwall, M. A., Kousha, K., Stuart, E., Makita, M., Abdoli, M., Wilson, P., & Levitt, J. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24767 · Cited by 79
Bibliometrics in Academic Recruitment: A Screening Tool Rather than a Game Changer
Reymert, I. (2020) · Minerva · doi:10.1007/s11024-020-09419-0 · Cited by 70
Cheng, S., Chang‐Chien, G., Huang, Q., Zhang, Y., Yan, P., Zhang, J., Wang, Y., Zhang, D., & Teng, G. (2019) · Aerosol and Air Quality Research · doi:10.4209/aaqr.2019.06.0327 · Cited by 65
Pölönen, J., Guns, R., Kulczycki, E., Sivertsen, G., & Engels, T. (2020) · Journal of Data and Information Science · doi:10.2478/jdis-2021-0004 · Cited by 71
Kousha, K., & Thelwall, M. A. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24810 · Cited by 74
Irwanto, I., Wahyudiati, D., Saputro, A. D., & Laksana, S. D. (2023) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v18i05.37021 · Cited by 74
A long-term bibliometric analysis of journals influencing management accounting and control research
Balstad, M. T., & Berg, T. (2019) · Journal of Management Control · doi:10.1007/s00187-019-00287-8 · Cited by 64
Añazco, D., Nicolalde, B., Espinosa, I., Camacho, J., Mushtaq, M., Gimenez, J., & Teran, E. (2021) · PeerJ · doi:10.7717/peerj.10927 · Cited by 63
Aftab, J., Sarwar, H., Amin, A., & Kiran, A. (2021) · Social Responsibility Journal · doi:10.1108/srj-09-2020-0393 · Cited by 63
Leising, D., Thielmann, I., Glöckner, A., Gärtner, A., & Schönbrodt, F. D. (2022) · Personality Science · doi:10.5964/ps.6029 · Cited by 75
Adetunji, T. L., Olawale, F., Olisah, C., Adetunji, A. E., & Aremu, A. O. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.908487 · Cited by 62
A bibliometric reflection on the history of green human resource management research
Farrukh, M. H. M., Raza, A., Ansari, N. Y., & Bhutta, U. S. (2021) · Management Research Review · doi:10.1108/mrr-09-2020-0585 · Cited by 61
A critical review of HR analytics: visualization and bibliometric analysis approach
Arora, M., Prakash, A., Dixit, S., Mittal, A., & Singh, S. (2022) · Information Discovery and Delivery · doi:10.1108/idd-05-2022-0038 · Cited by 70
Richardson, R. J., Fink, J. K., Glynn, P., Hufnagel, R. B., Махаева, Г. Ф., & Wijeyesakere, S. J. (2020) · Advances in neurotoxicology · doi:10.1016/bs.ant.2020.01.001 · Cited by 61
Criteria for assessing grant applications: a systematic review
Hug, S. E., & Aeschbach, M. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-0412-9 · Cited by 60
Knowledge hiding in academia: an empirical study of Indian higher education students
Garg, N., Talukdar, A., Ganguly, A., & Kumar, C. (2021) · Journal of Knowledge Management · doi:10.1108/jkm-10-2020-0783 · Cited by 60
Examining the developments in scheduling algorithms research: A bibliometric approach
Omotehinwa, T. O. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e09510 · Cited by 59
Sawicki, J., Ganzha, M., & Paprzycki, M. (2023) · Data Intelligence · doi:10.1162/dint_a_00213 · Cited by 63
Living Labs: From Niche to Mainstream Innovation Management
Greve, K., Vita, R. D., Leminen, S., & Westerlund, M. (2021) · Sustainability · doi:10.3390/su13020791 · Cited by 59
Pitafi, A. H., & Ren, M. (2021) · Internet Research · doi:10.1108/intr-11-2019-0469 · Cited by 58
The past, present, and future of China-related accounting research
Cheng, Q., Hail, L., & Yu, G. (2022) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2022.101544 · Cited by 58
Shea, T. P., Usman, S. A., Sengottuvel, A., & Parayitam, S. (2021) · VINE Journal of Information and Knowledge Management Systems · doi:10.1108/vjikms-12-2020-0232 · Cited by 57
Abusive supervision and its relationship with knowledge hiding: the mediating role of distrust
Farooq, R., & Sultana, A. (2021) · International Journal of Innovation Science · doi:10.1108/ijis-08-2020-0121 · Cited by 57
Menidjel, C., Benhabib, A., Bilgihan, A., & Madanoglu, M. (2019) · International Journal of Retail & Distribution Management · doi:10.1108/ijrdm-01-2019-0020 · Cited by 57
A bibliometric review of barriers to circular economy implementation in solid waste management
Akomea-Frimpong, I., Tetteh, P. A., Ofori, J. N. A., Tumpa, R. J., Pariafsai, F., Tenakwah, E. S., Asogwa, I. E., Vanapalli, K. R., Adu‐Gyamfi, G., Kukah, A. S. K., et al. (2024) · Discover Environment · doi:10.1007/s44274-024-00050-4 · Cited by 54
Bonkra, A., Pathak, S. K., Kaur, A., & Shah, M. A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10628-8 · Cited by 57
A journey of social sustainability in organization during MDG & SDG period: A bibliometric analysis
Kumari, A., & Singh, M. P. (2023) · Socio-Economic Planning Sciences · doi:10.1016/j.seps.2023.101668 · Cited by 52
How faculty define quality, prestige, and impact of academic journals
Morales, E., McKiernan, E. C., Niles, M. T., Schimanski, L. A., & Alperín, J. P. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257340 · Cited by 53
Kokol, P. (2023) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2023.1513 · Cited by 57
Rushforth, A., & Hammarfelt, B. (2023) · Quantitative Science Studies · doi:10.1162/qss_a_00280 · Cited by 61
Peer Review versus Bibliometrics: Which Method Better Predicts the Scholarly Impact of Publications?
Abramo, G., D’Angelo, C. A., & Reale, E. (2019) · Scientometrics · doi:10.1007/s11192-019-03184-y · Cited by 50
Applications of natural language processing in software traceability: A systematic mapping study
Pauzi, Z., & Capiluppi, A. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111616 · Cited by 55
López-López, E., Tobón, S., & Herting, D. C. (2024) · Alteridad · doi:10.17163/alt.v19n2.2024.03 · Cited by 49
Analyzing digital communication: a comprehensive literature review
Strauss, C., Harr, M. D., & Pieper, T. M. (2024) · Management Review Quarterly · doi:10.1007/s11301-024-00455-8 · Cited by 64
The relation between intellectual capital and digital transformation: a bibliometric analysis
Yılmaz, A. A., & Tuzlukaya, Ş. E. (2023) · International Journal of Innovation Science · doi:10.1108/ijis-08-2022-0145 · Cited by 47
Delayed citation impact of interdisciplinary research
Zhang, Y., Wang, Y., Du, H., & Havlin, S. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101468 · Cited by 47
Charting the Growth and Structure of Early ChatGPT-Education Research: A Bibliometric Study
Watrianthos, R., Ahmad, S. T., & Muskhir, M. (2023) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/5221 · Cited by 47
A gender- and class-sensitive explanatory model for rural women entrepreneurship in Turkey
Sefer, B. K. (2020) · International Journal of Gender and Entrepreneurship · doi:10.1108/ijge-07-2019-0113 · Cited by 47
Li, H., & Yin, Z. (2022) · Journal of Librarianship and Information Science · doi:10.1177/09610006221106178 · Cited by 46
Soliman, M. S. A., Fatnassi, T., Elgammal, I., & Figueiredo, R. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7010012 · Cited by 47
The role of metrics in peer assessments
Langfeldt, L., Reymert, I., & Aksnes, D. W. (2020) · Research Evaluation · doi:10.1093/reseval/rvaa032 · Cited by 50
Self-citationGlobal Trends of Green Pesticide Research from 1994 to 2019: A Bibliometric Analysis
Smith, H. H., Idris, O. A., & Maboeta, M. S. (2021) · Journal of Toxicology · doi:10.1155/2021/6637516 · Cited by 46
Bornmann, L., & Tekles, A. (2021) · Journal of Informetrics · doi:10.1016/j.joi.2021.101159 · Cited by 45
Aryadoust, V., Zakaria, A., Lim, M. H., & Chen, C. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01941 · Cited by 48
Α systematic meta-review analysis of review papers in the marine plastic pollution literature
Aretoulaki, E., Ponis, S. T., Plakas, G., & Agalianos, K. (2020) · Marine Pollution Bulletin · doi:10.1016/j.marpolbul.2020.111690 · Cited by 45
An analysis of the effects of sharing research data, code, and preprints on citations
Colavizza, G., Cadwallader, L., LaFlamme, M., Dozot, G., Lecorney, S., Rappo, D., & Hrynaszkiewicz, I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0311493 · Cited by 44
A meta-analysis of semantic classification of citations
Kunnath, S. N., Herrmannova, D., Pride, D., & Knoth, P. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00159 · Cited by 43
Measuring the influence of non-scientific features on citations
Mammola, S., Piano, E., Doretto, A., Caprio, E., & Chamberlain, D. (2022) · Scientometrics · doi:10.1007/s11192-022-04421-7 · Cited by 44
A review of air transport service quality studies: current status and future research agenda
Zhang, Y., Lee, S.-Y., & Gu, Y. (2023) · Journal of the Air Transport Research Society · doi:10.59521/ef52bb6324bd7035 · Cited by 43
Ecological restoration research progress and prospects: A bibliometric analysis
Shen, Z., Tian, Y., Yao, Y., Jiang, W., Dong, J., Huang, X., Wu, X., Farooq, T. H., & Yan, W. (2023) · Ecological Indicators · doi:10.1016/j.ecolind.2023.110968 · Cited by 46
Zhu, S., Liu, Y., Gu, Z., & Zhao, Y. (2021) · Advanced Healthcare Materials · doi:10.1002/adhm.202002222 · Cited by 43
Cerezo, R., Lara, J. A., Azevedo, R., & Romero, C. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108155 · Cited by 52
A Bibliometric Analysis of Child Language During 1900–2021
Guo, X. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.862042 · Cited by 42
Quality assessment of meta-analyses on soil organic carbon
Fohrafellner, J., Zechmeister‐Boltenstern, S., Murugan, R., & Valkama, E. (2023) · SOIL · doi:10.5194/soil-9-117-2023 · Cited by 41
Citation differences across research funding and access modalities
Dorta–González, P., & Dorta-González, M. I. (2023) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2023.102734 · Cited by 44
A critical review of the advancements in acid-activated metakaolin geopolymers
Alvi, I. H., Li, Q., Hu, H., Onyekwena, C. C., Hou, Y., Hakuzweyezu, T., Iftikhar, S., & Chen, B. (2024) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2024.135609 · Cited by 41
Kaltenbrunner, W., Pinfield, S., Waltman, L., Woods, H. B., & Brumberg, J. (2022) · Journal of Documentation · doi:10.1108/jd-01-2022-0022 · Cited by 44
Citation Data and Analysis: Limitations and Shortcomings
WORRALL, J. L., & Cohn, E. G. (2023) · Journal of Contemporary Criminal Justice · doi:10.1177/10439862231170972 · Cited by 41
Metadata
- Authors
- Aksnes, D. W., Langfeldt, L., & Wouters, P.
- Affiliations
- Nordic Institute for Studies in Innovation, Research and Education (NO)
- Leiden University (NL)
- Corresponding author
- Aksnes, Dag W.
- Venue
- Sage Open
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 9
- Issue
- 1
- ISSN
- 2158-2440 (ISSN-L)
- OpenAlex
- W2913788248
- Semantic Scholar
- 59660bbee2813e0ddc65cd8e7fa35ea51f58b6e7
- Cited by
- 1,257
- References
- 132
- Citations by year
- 2026: 153 · 2025: 207 · 2024: 223 · 2023: 202 · 2022: 181 · 2021: 177 · 2020: 89 · 2019: 23
- FWCI
- 78.79
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- SAGE Open (publishedVersion, cc-by)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Norges Forskningsråd (256223)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- citation, citation metrics, research quality, research evaluation, research funding, scientific impact, originality, social values, plausibility, science policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
