Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories

May combine two claims.

The h-index is not field-normalized and no corrections are made for career length, which means that the indicator disfavors younger researchers.

“Most importantly, it is not field-normalized and no corrections are made for career length, which means that the indicator disfavors younger researchers”

• cites Alonso, Cabrerizo, Herrera-Viedma, & Herrera, 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The conclusions of MacRoberts and MacRoberts are generally seen as too sweeping.

“The views of the MacRoberts's previously led to much debate, but their conclusions are generally seen as too sweeping”

• cites Aksnes, 2005Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

There is no unified theory of citation.

“Even now, in spite of detailed studies of referencing behavior, there is no unified theory.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Previous studies of the citation process have not provided any simple answer to the question of what citations stand for.

“As indicated above, previous studies of the citation process have not provided any simple answer to the question of what citations stand for.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many studies have compared citation indicators with the outcome of peer review.

“While empirical studies have revealed a multitude of factors involved in the citation process, the issue has also been approached from another angle: by comparing citation indicators with the outcome of peer review. During recent decades, many such studies have been carried out.”

May combine two claims.

The references have a multitude of functions in the scientific article, only a small proportion of the relevant literature is cited, and the authors have a multitude of motives for including particular studies as references.

“Nevertheless, some overall findings remain: the references have a multitude of functions in the scientific article, only a small proportion of the relevant literature is cited, and the authors have a multitude of motives for including particular studies as references.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Studies have revealed that the role of the reference, both in the citing text and with respect to the cited text, is complex.

“Studies undertaken have revealed that the role of the reference, both in the citing text and with respect to the cited text, is complex.”

Classified as probably not a substantial claim. May combine two claims.

The citing is biased: some sources are cited essentially every time they are used, while other research is never cited even though it may be used more often than the highly cited work.

“Moreover, the citing is biased: some sources are cited essentially every time they are used, while other research is never cited even though it may be used more often than the highly cited work.”

• cites MacRoberts & MacRoberts, 1989, 1996Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

References have a multitude of functions in the scientific article.

“These and later studies have revealed that the references have a multitude of functions in the scientific article.”

May combine two claims.

Studies on the role of references are often based on rather small samples of papers from selected scientific fields, and the results may not have general validity.

“Moreover, the studies are often based on rather small samples of papers from selected scientific fields, and the results may not have general validity.”

Classified as probably not a substantial claim.

Based on empirical case studies, MacRoberts and MacRoberts showed that a very small proportion of the knowledge basis of an article actually are cited.

“Based on empirical case studies, they showed that a very small proportion of the knowledge basis of an article (consisting of hundreds or thousands of former publications) actually are cited.”

• cites MacRoberts & MacRoberts, 1989, 1996Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Authors tend to select references that will be regarded as authoritative by the intended audience.

“Therefore, it will be more persuasive to cite an authoritative paper, and authors tend to select references that will be regarded as authoritative by the intended audience.”

• cites Gilbert, 1977Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Lowry et al. (1951) article has now been cited more than 305,000 times in the Web of Science database.

“This article has now been cited more than 305,000 times in the Web of Science database”

• cites Van Noorden, Maher, & Nuzzo, 2014Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

In a scientific article, some references represent works that are crucial or significant antecedents to the present work; others may represent more general background literature.

“In a scientific article, some of the references will represent works that are crucial or significant antecedents to the present work; others may represent more general background literature”

• cites Aksnes, 2005Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The textual functions of citations vary considerably.

“Hence, the textual functions of citations vary considerably.”

May combine two claims.

Other incentives shape the citing patterns, like creating visibility for one's work through self-citations or citing a journal editor's work as an attempt to enhance the chances of acceptance for publication.

“For the citation process, this implies that other incentives shape the citing patterns, like creating visibility for one's work through self-citations or citing a journal editor's work as an attempt to enhance the chances of acceptance for publication.”

• cites Bornmann & Daniel, 2008Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There is growing concern about the misuse of citation indicators and the potential negative impact of research metrics on the scientific community.

“There is a growing concern about this issue, as well as on potential negative impact of research metrics on the scientific community.”

• cites Hicks et al., 2015Not checked yet

  Not checked in the free run.

• cites Cagan, 2013Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Citation measures are core indicators in several university rankings, such as the Leiden ranking and the Academic Ranking of World Universities.

“Citation measures are also core indicators in several university rankings, such as the Leiden ranking and Academic Ranking of World Universities (ARWU)”

• cites Piro & Sivertsen, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Bibliometric indicators have increasingly been applied in research evaluation and research policy more generally.

“In recent years, bibliometric indicators have increasingly been applied in the context of research evaluation as well as research policy more generally.”

• cites Moed, 2005Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Levitt and Thelwall (2011) have argued that short citation windows have the problem that articles published earlier in a year have a significant advantage compared with publications appearing later in a year.

“For example, Levitt and Thelwall (2011) have argued that short citation windows have the problem that articles published earlier in a year have a significant advantage (i.e., are on average more highly cited) compared with publications appearing later in a year.”

• cites Levitt and Thelwall (2011)Not checked yet

  Not checked in the free run.

May combine two claims.

Only a small fraction of the scholarly literature in the humanities is represented in the Web of Science, and most of the references and citations will not be captured by the database.

“Only a small fraction of the scholarly literature in the humanities is represented in the Web of Science, and most of the references and citations will not be captured by the database.”

• cites Marx and Bornmann (2015)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The coverage of scientific and scholarly literature varies across databases, and the results of citation studies are dependent upon the particular characteristics of the databases and their coverage.

“The coverage of the scientific and scholarly literature varies across these databases, and the results of citation studies are thus dependent upon the particular characteristics of the databases and their coverage.”

May combine two claims.

In the scientific paper, references have various purposes, and authors are not including references merely because of their scientific quality.

“In the scientific paper, the references have various purposes. Authors are not including references merely because of their scientific quality.”

• cites Bornmann & Daniel, 2008Not checked yet

  Not checked in the free run.

May combine two claims.

Citations reflect aspects related to scientific impact and relevance, although with important limitations.

“It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations.”

• AssumptionRests on an assumption: Research quality is a multidimensional concept comprising plausibility/soundness, originality, scientific value, and societal value as key characteristics.

Probably not a substantial claim: may not pass S3 (load-bearing).

It has since been the standard to adjust for field, publication year, and publication type when calculating citation indicators.

“It has since been the standard to adjust for field, publication year, and publication type when calculating citation indicators.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The application of citation indicators has been criticized with respect to their validity as performance measures and their potentially negative impact upon the research system.

“Moreover, the application of citation indicators has also been criticized more generally, with respect to their validity as performance measures and their potentially negative impact upon the research system”

• cites MacRoberts & MacRoberts, 1989Not checked yet

  Not checked in the free run.

• cites Osterloh & Frey, 2015Not checked yet

  Not checked in the free run.

• cites Weingart, 2004Not checked yet

  Not checked in the free run.

May combine two claims.

When one article is cited by many subsequent publications, even more people become aware of this article, increasing its visibility and chances of getting even more citations.

“When one article is cited by many subsequent publications, even more people become aware of this article. Thus, its visibility, and thereby the chances of getting even more citations, increases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In validation studies, assessments by peers have been typically considered as a kind of standard to which citation indicators can be validated.

“In the studies, assessments by peers have been typically considered as a kind of standard to which citation indicators can be validated.”

May combine two claims.

Some problems with citations as indicators are fundamental and inherent in any use of citations, others may be resolved by more advanced indicators, and others may be of less importance in practice.

“Some of these problems have a fundamental character and are inherent in any use of citations as indicators, others may be resolved by the construction of more advanced indicators, while others again may be of less importance in practice.”

May combine two claims.

Panels increasingly consider citation measures as part of the evaluation procedure, so the two cannot be considered as completely independent of each other.

“Moreover, panels increasingly are considering citation measures as part of the evaluation procedure, which means that the two cannot be considered as completely independent of each other.”

Classified as probably not a substantial claim.

If authors cite the works they find useful, frequently cited publications may be assumed to have been more useful than papers which are hardly cited at all.

“If authors cite the works they find useful, frequently cited publications may be assumed to have been more useful than papers which are hardly cited at all.”

May combine two claims.

Overall, most comparative studies seem to have found a moderately positive correspondence between citation indicators and peer review, but the correlations identified have been far from perfect and have varied among the studies.

“Overall, it may be concluded that most of the comparative studies seem to have found a moderately positive correspondence, but the correlations identified have been far from perfect and have varied among the studies.”

Classified as probably not a substantial claim. May combine two claims.

For field-weighted citation impact, a Spearman correlation coefficient of .28 was identified at an overall level, with significant variations across fields, and significant decreases in correlation for more recent outputs.

“For the field-weighted citation impact, a Spearman correlation coefficient of .28 was identified at an overall level, albeit with significant variations across fields. Moreover, there were significant decreases in correlation for more recent outputs.”

• cites Higher Education Funding Council for England, 2015Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

The outcome of comparative analyses of citation indicators and peer review depends on which indicators are selected.

“Finally, a large number of different citation measures exist and the outcome would also depend on which indicators are selected for the comparative analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Italian RAE, a significant degree of concordance was found between grades attributed to journal articles by informed peer review and by bibliometric indicators.

“A significant degree of concordance was found '. . . supporting the choice of using both techniques in order to assess the quality of Italian research institutions' (Ancaiani et al., 2015, p. 254).”

• cites Ancaiani et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

Although most citation analysts seem to agree that citing or referencing is biased, it has been argued that this bias is not fatal for the use of citation as performance indicators-to a certain extent, the biases are averaged out at aggregated levels.

“Although most citation analysts seem to agree that citing or referencing is biased, it has been argued that this bias is not fatal for the use of citation as performance indicators-to a certain extent, the biases are averaged out at aggregated levels.”

Probably not a substantial claim: may not pass S3 (load-bearing).

According to Luukkonen (1990), the presence of different cognitive meanings of citations and motivations for citing does not necessarily invalidate the use of citations as (imperfect) performance measures.

“According to Luukkonen (1990), the presence of different cognitive meanings of citations and motivations for citing does not necessarily invalidate the use of citations as (imperfect) performance measures.”

• cites Luukkonen (1990)Not checked yet

  Not checked in the free run.

May combine two claims.

Problems related to technical issues such as discrepancies between target articles and cited references and mistakes in indexing procedures may confuse citation analyses and affect the accuracy of citation counts to individual articles.

“Problems related to more technical issues, such as discrepancies between target articles and cited references (misspellings of journal names, author names, errors in the reference lists, etc.), and mistakes in the indexing procedures conducted by Clarivate Analytics (previously Thomson Reuters) or Elsevier (Leydesdorff et al., 2016; Moed, 2002) may confuse citation analyses. Such errors affect in”

• cites Leydesdorff et al., 2016Not checked yet

  Not checked in the free run.

• cites Moed, 2002Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Bibliometric indicators and outcomes of peer review of Norwegian research groups at a mathematical and natural science faculty show positive but weak correlations.

“Aksnes and Taxt (2004) analyzed the relationship between bibliometric indicators and the outcomes of a peer review of Norwegian research groups at a mathematical and natural science faculty, reporting positive but weak correlations.”

• cites Aksnes and Taxt (2004)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The different functions the references may have in a text are much more complex than merely providing documentation and support for particular claims.

“This means that the different functions the references may have in a text are much more complex than merely providing documentation and support for particular claims.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gilbert (1977) argued that citing is essentially a device for persuasion.

“Gilbert (1977) argued that citing ('referencing') is essentially a device for persuasion.”

• cites Gilbert (1977)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Small (1982) identified five distinctions: A cited work may be refuted, noted only, reviewed, applied, or supported by the citing work.

“Small (1982) identified five distinctions: A cited work may be (a) refuted, (b) noted only, (c) reviewed, (d) applied, or (e) supported by the citing work.”

• cites Small (1982)Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Typically, a scientific article is structured as a progression from the general to the particular.

“Typically, a scientific article is structured as a progression from the general to the particular”

• cites Law, 1986Not checked yet

  Not checked in the free run.

May combine two claims.

Although the literature coverage of citation databases has improved, the coverage of the humanities and several social science disciplines remains limited.

“Although the literature coverage of citation databases has improved (Web of Science and Scopus), the coverage of the humanities and several social science disciplines remains limited (Waltman, 2016).”

• cites Waltman, 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

There is no simple answer to the question what citation indicators measure or indicate.

“As is evident from the overview above, there is no simple answer to the question what citation indicators measure or indicate.”

May combine two claims.

Both the methodological basis for comparing peer assessments and citation indicators and the assumption that the two may be expected to correlate may be questionable.

“Thus, both the methodological basis for comparing peer assessments and citation indicators and the assumption that the two may be expected to correlate may be questionable.”

May combine two claims.

Citation frequencies are determined by other mechanisms and are not a simple reflection of the quality dynamics.

“At the same time, citation frequencies are determined by other mechanisms and are not a simple reflection of the quality dynamics.”

Classified as probably not a substantial claim.

According to Merton's view, the norms of science oblige researchers to cite the work upon which they draw, and in this way acknowledge or credit contributions by others.

“According to Merton's view, the norms of science oblige researchers to cite the work upon which they draw, and in this way acknowledge or credit contributions by others”

• cites Merton, 1979Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The h-index and the normalized citation impact indicator both correlated quite well with peer judgments for research groups within chemistry in the Netherlands.

“He found that the h-index and the normalized citation impact indicator both correlated quite well with peer judgments.”

• cites van Raan (2006)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

According to Bornmann and Daniel (2008), many studies have methodological weaknesses and have provided findings with little reliability.

“According to Bornmann and Daniel (2008), many studies have methodological weaknesses and have provided findings with little reliability.”

• cites Bornmann and Daniel (2008)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The methodological fundament for the conclusion that informed peer review and bibliometrics produce similar results has been contested.

“However, the methodological fundament for this conclusion has been contested by Baccini and De Nicolao (2016), who argue that the analysis is flawed and that informed peer review and bibliometrics do not produce similar results.”

• cites Baccini and De Nicolao (2016)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Various citation indicators correlated significantly with peer ratings of research programs in condensed matter physics.

“Rinia, van Leeuwen, van Vuren, and van Raan (1998) showed that various citation indicators correlated significantly with peer ratings of research programs in condensed matter physics.”

• cites Rinia, van Leeuwen, van Vuren, and van Raan (1998)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

According to Garfield, the fact that authors do not cite all their influences does not invalidate the use of citations as performance measures when enough literature is taken into account.

“According to the founder of the SCI, the fact that authors do not cite all their influences does not invalidate the use of citations as performance measures when enough literature is taken into account”

• cites Garfield, 1997Not checked yet

  Not checked in the free run.

May combine two claims.

Because citation distributions are extremely skewed, the use of mean as indicator is questionable, and percentile ranks have been proposed as a non-parametric alternative.

“The fact that citation distributions are extremely skewed also raises questions concerning the use of mean as indicator, and Bornmann and Mutz (2011) have proposed to use percentile ranks as a non-parametric alternative to the means of citation distributions for the normalization.”

• cites Bornmann and Mutz (2011)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

According to Welljams-Dorof, the larger the citation data set being used, the higher the confidence level of the results, and analyses involving entire fields, nations, regions and large universities are virtually unaffected by concerns about citation data.

“According to Welljams-Dorof (1997), this has important implications: In general, the larger the citation data set being used, the higher the confidence level of the results. Analyses involving entire fields of research, nations, regions and large universities are virtually unaffected by the concerns and caveats about citation data . . . The confidence level at these large aggregate levels is quite”

• cites Welljams-Dorof (1997)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The use of impact as the most appropriate concept has usually been justified by theoretical considerations, and there are few attempts to address the issue empirically or relate it to previous findings on citation behavior.

“However, the use of impact as the most appropriate concept has usually been justified by theoretical considerations, and there are few attempts to address the issue empirically or relate it to previous findings on citation behavior.”

May combine two claims.

A basic limitation of using patents to assess societal relevance is that many innovations are not patented and patents are not suitable to assess societal relevance or impact in a broader context.

“Still, a basic limitation is that many innovations are not patented and patents are not suitable to assess societal relevance or impact in a broader context.”

Classified as probably not a substantial claim. May combine two claims.

The relation between citations and research quality is complex and arises differently depending on the field analyzed, the database used, the timeframe and indicators applied.

“As shown above, the question on the relation between citations and research quality is complex and will arise differently depending on the field analyzed, the database used, the timeframe and indicators applied, and so forth.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Citation impact should be distinguished from scientific impact, as an influential researcher sometimes has a lower performance in terms of highly cited publications than some of their less influential colleagues.

“According to Waltman, van Eck, and Wouters (2013), citation impact should be distinguished from scientific impact, as an influential researcher sometimes has a lower performance in terms of highly cited publications than some of their less influential colleagues.”

• cites Waltman, van Eck, and Wouters (2013)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

The viewpoint that bibliometric assessment is superior to traditional peer review is not justified.

“However, the viewpoint described in the introduction that bibliometric assessment is superior compared to the traditional peer-review method is not justified in our opinion.”

May combine two claims.

The referee system does not fully ensure the quality dimension related to solidity and plausibility, and there are no indications that high citation counts reflect solidity.

“Thus, the referee system does not fully ensure the quality dimension related to solidity and plausibility, and there are no indications that high citation counts reflect solidity.”

Classified as probably not a substantial claim.

Most bibliometricians have argued that a bibliometric analysis should not function as a substitute for a peer review.

“However, considering the various limitations attached to citations as performance measures, most bibliometricians have argued that a bibliometric analysis should not function as a substitute for a peer review (Moed, 2005).”

• cites Moed, 2005Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Citation indicators may easily be misused or applied in contexts where they lack justification or validity.

“Citation indicators may easily be misused or applied in contexts where they lack justification or validity.”

Classified as probably not a substantial claim.

Research quality is a multidimensional concept.

“In addition, research quality is a multidimensional concept, and in this section, we will look further into this issue.”

• cites Polanyi (1962)Not checked yet

  Not checked in the free run.

• cites Gulbrandsen, 2000Not checked yet

  Not checked in the free run.

• cites Lamont, 2009Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Even a 'mistaken' publication can have a large impact by stimulating further research.

“As an example, even a 'mistaken' publication can have a large impact by stimulating further research.”

May combine two claims.

The skewness in citation distribution is due to sociological and aggregational processes: initially articles are cited for substantive reasons, but later sociological mechanisms become increasingly important, benefiting some papers greatly.

“This is because of the sociological and aggregational processes involved. In the beginning, an article may be cited for substantive reasons (e.g., its content has been used). Later, when the article is widely known and has received many citations, sociological mechanisms will be of increasing importance (authors citing authoritative papers, the bandwagon effect, etc.). Some papers will benefit”

Probably not a substantial claim: may not pass S2 (checkable).

The view generally held among experts within bibliometrics seems to be that citations represent a good but not perfect impact measure.

“The view generally held among experts within bibliometrics seem to be that citations represent a good but not perfect impact measure.”

May combine two claims.

Citations are not able to capture all aspects of the quality concept, and an increased use of citation indicators in research evaluation and funding may imply less attention to other research quality dimensions such as solidity/plausibility, originality, and societal value.

“As is evident from the discussion of this paper, this is an important point, as citations are not able to capture all aspects of the quality concept. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Polanyi distinguished three dimensions of research quality: plausibility, originality, and scientific value.

“As a starting point, we can take the three dimensions distinguished by Polanyi (1962): plausibility, originality, and scientific value.”

• cites Polanyi (1962)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Impact is the most commonly used concept for what citations reflect, although other concepts such as influence, importance, significance, and utility occasionally also are used.

“Impact is the most commonly used concept for what citations reflect, although other concepts such as influence, importance, significance, and utility occasionally also are used (Moed, 2005).”

• cites Moed, 2005Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There is a widespread assumption that societal relevance cannot be adequately assessed through standard citation indicators, and increasing attention has been devoted to developing methodologies for assessing and measuring societal relevance and impact.

“There seems to be a widespread assumption that this issue cannot be adequately assessed through standard citation indicators, and in recent years, increasing attention has been devoted to developing methodologies for assessing and measuring societal relevance and impact (Bornmann, 2012, 2013).”

• cites Bornmann, 2012, 2013Not checked yet

  Not checked in the free run.

May combine two claims.

There is a lack of empirical studies confirming that larger citation data sets have higher confidence levels, and some biases may be fundamental to all citation measures while others may level out when aggregated.

“Nevertheless, there is a lack of empirical studies confirming that this is actually the case, and possibly some of the biases is of a fundamental nature attached to all citations measures, while the effect of others may tend to level out when aggregated levels are considered.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Societal relevance is often considered to be much harder to measure than scientific relevance or impact.

“Societal relevance is often considered to be something which is much harder to measure than scientific relevance or impact (B. R. Martin, 2011).”

• cites B. R. Martin, 2011Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

All assessments of research quality may be context-dependent in terms of time of assessment and time/field/sector perspectives of evaluators.

“Moreover, all assessments of research quality may be context-dependent, in terms of, for example, the time of assessment and the time/field/sector perspectives of the evaluators.”

Probably not a substantial claim: may not pass S2 (checkable).

Impact would be a more adequate interpretation of citations than quality.

“Using these definitions, it is also evident that impact would be a more adequate interpretation of citations than quality.”

• cites B. R. Martin and Irvine, 1983Not checked yet

  Not checked in the free run.

May combine two claims.

It is not possible to identify societal relevance from citation counts per se, and uncited or little cited publications may have contributed results of great societal relevance.

“Despite these and numerous similar examples, it is not possible to identify societal relevance from citation counts per se, and uncited or little cited publication may have contributed with results of great societal relevance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It has long been emphasized by bibliometricians that more than one indicator should be used in research evaluation contexts.

“Because of this, it has long been emphasized by bibliometricians that more than one indicator should be used in research evaluation contexts (van Raan, 1993).”

• cites van Raan, 1993Not checked yet

  Not checked in the free run.

May combine two claims.

Citations reflect aspects related to scientific impact and relevance, with important limitations.

“citations reflect-with important limitations-aspects related to scientific impact and relevance”

May combine two claims.

The literature contains numerous publications of which the solidity is poor, the results unreliable, or even involving misconduct or scientific fraud.

“The literature contains numerous publications of which the solidity is poor, the results unreliable or even involving misconduct or scientific fraud (Fanelli, 2009).”

• cites Fanelli, 2009Not checked yet

  Not checked in the free run.

May combine two claims.

The mean normalized citation score is size-independent and does not take into account the number of publications, which is a major problem because it does not truly represent productivity.

“For example, the mean normalized citation score is size-independent and does not take into account the number of publications. According to Abramo and D'Angelo (2016), this is a major problem with this indicator because it does not truly represent productivity.”

• cites Abramo and D'Angelo (2016)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The topic of how citations relate to each dimension of research quality has rarely been addressed specifically in the literature, and there are few empirical studies analyzing the issue.

“Surprisingly, this topic has rarely been addressed specifically in the literature and there are few studies analyzing the issue empirically.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Bibliometric analysis can counterbalance shortcomings and mistakes in peers' judgments, and a bibliometric study should be considered as complementary to a peer evaluation.

“On this basis, it is often argued that bibliometric analysis can counterbalance shortcomings and mistakes in the peers' judgments (Aksnes, 2005). Thus, a bibliometric study should be considered as complementary to a peer evaluation (Council of Canadian Academies, 2012).”

• cites Aksnes, 2005Not checked yet

  Not checked in the free run.

• cites Council of Canadian Academies, 2012Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Plausibility/soundness, scientific value, societal value, and originality seem commonly perceived key characteristics of research quality, but each includes a variety of aspects, may be context-dependent, and may conflict with each other.

“In sum, whereas plausibility/soundness, scientific value, and societal value and originality seem commonly perceived key characteristics of research quality, each of these dimensions include a variety of aspects; they may be contextdependent and may also conflict with each other.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

There are problems with the peer-review system.

“At the same time, there are also problems with the peer-review system.”

• cites Eyre-Walker and Stoletzki (2013)Not checked yet

  Not checked in the free run.