Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories

In the paper

“Most importantly, it is not field-normalized and no corrections are made for career length, which means that the indicator disfavors younger researchers”

What the sources say

• Not checked yetcites Alonso, Cabrerizo, Herrera-Viedma, & Herrera, 2009

In the paper

“The views of the MacRoberts's previously led to much debate, but their conclusions are generally seen as too sweeping”

What the sources say

• Not checked yetcites Aksnes, 2005

In the paper

“Even now, in spite of detailed studies of referencing behavior, there is no unified theory.”

In the paper

“As indicated above, previous studies of the citation process have not provided any simple answer to the question of what citations stand for.”

In the paper

“While empirical studies have revealed a multitude of factors involved in the citation process, the issue has also been approached from another angle: by comparing citation indicators with the outcome of peer review. During recent decades, many such studies have been carried out.”

In the paper

“Nevertheless, some overall findings remain: the references have a multitude of functions in the scientific article, only a small proportion of the relevant literature is cited, and the authors have a multitude of motives for including particular studies as references.”

In the paper

“Studies undertaken have revealed that the role of the reference, both in the citing text and with respect to the cited text, is complex.”

In the paper

“Moreover, the citing is biased: some sources are cited essentially every time they are used, while other research is never cited even though it may be used more often than the highly cited work.”

What the sources say

• Not checked yetcites MacRoberts & MacRoberts, 1989, 1996

In the paper

“These and later studies have revealed that the references have a multitude of functions in the scientific article.”

In the paper

“Moreover, the studies are often based on rather small samples of papers from selected scientific fields, and the results may not have general validity.”

In the paper

“Based on empirical case studies, they showed that a very small proportion of the knowledge basis of an article (consisting of hundreds or thousands of former publications) actually are cited.”

What the sources say

• Not checked yetcites MacRoberts & MacRoberts, 1989, 1996

In the paper

“Therefore, it will be more persuasive to cite an authoritative paper, and authors tend to select references that will be regarded as authoritative by the intended audience.”

What the sources say

• Not checked yetcites Gilbert, 1977

In the paper

“This article has now been cited more than 305,000 times in the Web of Science database”

What the sources say

• Not checked yetcites Van Noorden, Maher, & Nuzzo, 2014

In the paper

“In a scientific article, some of the references will represent works that are crucial or significant antecedents to the present work; others may represent more general background literature”

What the sources say

• Not checked yetcites Aksnes, 2005

In the paper

“Hence, the textual functions of citations vary considerably.”

In the paper

“For the citation process, this implies that other incentives shape the citing patterns, like creating visibility for one's work through self-citations or citing a journal editor's work as an attempt to enhance the chances of acceptance for publication.”

What the sources say

• Not checked yetcites Bornmann & Daniel, 2008

In the paper

“There is a growing concern about this issue, as well as on potential negative impact of research metrics on the scientific community.”

What the sources say

• Not checked yetcites Hicks et al., 2015
• Not checked yetcites Cagan, 2013

In the paper

“Citation measures are also core indicators in several university rankings, such as the Leiden ranking and Academic Ranking of World Universities (ARWU)”

What the sources say

• Not checked yetcites Piro & Sivertsen, 2016

In the paper

“In recent years, bibliometric indicators have increasingly been applied in the context of research evaluation as well as research policy more generally.”

What the sources say

• Not checked yetcites Moed, 2005

In the paper

“For example, Levitt and Thelwall (2011) have argued that short citation windows have the problem that articles published earlier in a year have a significant advantage (i.e., are on average more highly cited) compared with publications appearing later in a year.”

What the sources say

• Not checked yetcites Levitt and Thelwall (2011)

In the paper

“Only a small fraction of the scholarly literature in the humanities is represented in the Web of Science, and most of the references and citations will not be captured by the database.”

What the sources say

• Not checked yetcites Marx and Bornmann (2015)

In the paper

“The coverage of the scientific and scholarly literature varies across these databases, and the results of citation studies are thus dependent upon the particular characteristics of the databases and their coverage.”

In the paper

“In the scientific paper, the references have various purposes. Authors are not including references merely because of their scientific quality.”

What the sources say

• Not checked yetcites Bornmann & Daniel, 2008

In the paper

“It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations.”

In the paper

“It has since been the standard to adjust for field, publication year, and publication type when calculating citation indicators.”

In the paper

“Moreover, the application of citation indicators has also been criticized more generally, with respect to their validity as performance measures and their potentially negative impact upon the research system”

What the sources say

• Not checked yetcites MacRoberts & MacRoberts, 1989
• Not checked yetcites Osterloh & Frey, 2015
• Not checked yetcites Weingart, 2004

In the paper

“When one article is cited by many subsequent publications, even more people become aware of this article. Thus, its visibility, and thereby the chances of getting even more citations, increases.”

In the paper

“In the studies, assessments by peers have been typically considered as a kind of standard to which citation indicators can be validated.”

In the paper

“Some of these problems have a fundamental character and are inherent in any use of citations as indicators, others may be resolved by the construction of more advanced indicators, while others again may be of less importance in practice.”

In the paper

“Moreover, panels increasingly are considering citation measures as part of the evaluation procedure, which means that the two cannot be considered as completely independent of each other.”

In the paper

“If authors cite the works they find useful, frequently cited publications may be assumed to have been more useful than papers which are hardly cited at all.”

In the paper

“Overall, it may be concluded that most of the comparative studies seem to have found a moderately positive correspondence, but the correlations identified have been far from perfect and have varied among the studies.”

In the paper

“For the field-weighted citation impact, a Spearman correlation coefficient of .28 was identified at an overall level, albeit with significant variations across fields. Moreover, there were significant decreases in correlation for more recent outputs.”

What the sources say

• Not checked yetcites Higher Education Funding Council for England, 2015

In the paper

“Finally, a large number of different citation measures exist and the outcome would also depend on which indicators are selected for the comparative analysis.”

In the paper

“A significant degree of concordance was found '. . . supporting the choice of using both techniques in order to assess the quality of Italian research institutions' (Ancaiani et al., 2015, p. 254).”

What the sources say

• Not checked yetcites Ancaiani et al., 2015

In the paper

“Although most citation analysts seem to agree that citing or referencing is biased, it has been argued that this bias is not fatal for the use of citation as performance indicators-to a certain extent, the biases are averaged out at aggregated levels.”

In the paper

“According to Luukkonen (1990), the presence of different cognitive meanings of citations and motivations for citing does not necessarily invalidate the use of citations as (imperfect) performance measures.”

What the sources say

• Not checked yetcites Luukkonen (1990)

In the paper

“Problems related to more technical issues, such as discrepancies between target articles and cited references (misspellings of journal names, author names, errors in the reference lists, etc.), and mistakes in the indexing procedures conducted by Clarivate Analytics (previously Thomson Reuters) or Elsevier (Leydesdorff et al., 2016; Moed, 2002) may confuse citation analyses. Such errors affect in”

What the sources say

• Not checked yetcites Leydesdorff et al., 2016
• Not checked yetcites Moed, 2002

In the paper

“Aksnes and Taxt (2004) analyzed the relationship between bibliometric indicators and the outcomes of a peer review of Norwegian research groups at a mathematical and natural science faculty, reporting positive but weak correlations.”

What the sources say

• Not checked yetcites Aksnes and Taxt (2004)

In the paper

“This means that the different functions the references may have in a text are much more complex than merely providing documentation and support for particular claims.”

In the paper

“Gilbert (1977) argued that citing ('referencing') is essentially a device for persuasion.”

What the sources say

• Not checked yetcites Gilbert (1977)

In the paper

“Small (1982) identified five distinctions: A cited work may be (a) refuted, (b) noted only, (c) reviewed, (d) applied, or (e) supported by the citing work.”

What the sources say

• Not checked yetcites Small (1982)

In the paper

“Typically, a scientific article is structured as a progression from the general to the particular”

What the sources say

• Not checked yetcites Law, 1986

In the paper

“Although the literature coverage of citation databases has improved (Web of Science and Scopus), the coverage of the humanities and several social science disciplines remains limited (Waltman, 2016).”

What the sources say

• Not checked yetcites Waltman, 2016

In the paper

“As is evident from the overview above, there is no simple answer to the question what citation indicators measure or indicate.”

In the paper

“Thus, both the methodological basis for comparing peer assessments and citation indicators and the assumption that the two may be expected to correlate may be questionable.”

In the paper

“At the same time, citation frequencies are determined by other mechanisms and are not a simple reflection of the quality dynamics.”

In the paper

“According to Merton's view, the norms of science oblige researchers to cite the work upon which they draw, and in this way acknowledge or credit contributions by others”

What the sources say

• Not checked yetcites Merton, 1979

In the paper

“He found that the h-index and the normalized citation impact indicator both correlated quite well with peer judgments.”

What the sources say

• Not checked yetcites van Raan (2006)

In the paper

“According to Bornmann and Daniel (2008), many studies have methodological weaknesses and have provided findings with little reliability.”

What the sources say

• Not checked yetcites Bornmann and Daniel (2008)

In the paper

“However, the methodological fundament for this conclusion has been contested by Baccini and De Nicolao (2016), who argue that the analysis is flawed and that informed peer review and bibliometrics do not produce similar results.”

What the sources say

• Not checked yetcites Baccini and De Nicolao (2016)

In the paper

“Rinia, van Leeuwen, van Vuren, and van Raan (1998) showed that various citation indicators correlated significantly with peer ratings of research programs in condensed matter physics.”

What the sources say

• Not checked yetcites Rinia, van Leeuwen, van Vuren, and van Raan (1998)

In the paper

“According to the founder of the SCI, the fact that authors do not cite all their influences does not invalidate the use of citations as performance measures when enough literature is taken into account”

What the sources say

• Not checked yetcites Garfield, 1997

In the paper

“The fact that citation distributions are extremely skewed also raises questions concerning the use of mean as indicator, and Bornmann and Mutz (2011) have proposed to use percentile ranks as a non-parametric alternative to the means of citation distributions for the normalization.”

What the sources say

• Not checked yetcites Bornmann and Mutz (2011)

In the paper

“According to Welljams-Dorof (1997), this has important implications: In general, the larger the citation data set being used, the higher the confidence level of the results. Analyses involving entire fields of research, nations, regions and large universities are virtually unaffected by the concerns and caveats about citation data . . . The confidence level at these large aggregate levels is quite”

What the sources say

• Not checked yetcites Welljams-Dorof (1997)

In the paper

“However, the use of impact as the most appropriate concept has usually been justified by theoretical considerations, and there are few attempts to address the issue empirically or relate it to previous findings on citation behavior.”

In the paper

“Still, a basic limitation is that many innovations are not patented and patents are not suitable to assess societal relevance or impact in a broader context.”

In the paper

“As shown above, the question on the relation between citations and research quality is complex and will arise differently depending on the field analyzed, the database used, the timeframe and indicators applied, and so forth.”

In the paper

“According to Waltman, van Eck, and Wouters (2013), citation impact should be distinguished from scientific impact, as an influential researcher sometimes has a lower performance in terms of highly cited publications than some of their less influential colleagues.”

What the sources say

• Not checked yetcites Waltman, van Eck, and Wouters (2013)

In the paper

“However, the viewpoint described in the introduction that bibliometric assessment is superior compared to the traditional peer-review method is not justified in our opinion.”

In the paper

“Thus, the referee system does not fully ensure the quality dimension related to solidity and plausibility, and there are no indications that high citation counts reflect solidity.”

In the paper

“However, considering the various limitations attached to citations as performance measures, most bibliometricians have argued that a bibliometric analysis should not function as a substitute for a peer review (Moed, 2005).”

What the sources say

• Not checked yetcites Moed, 2005

In the paper

“Citation indicators may easily be misused or applied in contexts where they lack justification or validity.”

In the paper

“In addition, research quality is a multidimensional concept, and in this section, we will look further into this issue.”

What the sources say

• Not checked yetcites Polanyi (1962)
• Not checked yetcites Gulbrandsen, 2000
• Not checked yetcites Lamont, 2009

In the paper

“As an example, even a 'mistaken' publication can have a large impact by stimulating further research.”

In the paper

“This is because of the sociological and aggregational processes involved. In the beginning, an article may be cited for substantive reasons (e.g., its content has been used). Later, when the article is widely known and has received many citations, sociological mechanisms will be of increasing importance (authors citing authoritative papers, the bandwagon effect, etc.). Some papers will benefit”

In the paper

“The view generally held among experts within bibliometrics seem to be that citations represent a good but not perfect impact measure.”

In the paper

“As is evident from the discussion of this paper, this is an important point, as citations are not able to capture all aspects of the quality concept. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.”

In the paper

“As a starting point, we can take the three dimensions distinguished by Polanyi (1962): plausibility, originality, and scientific value.”

What the sources say

• Not checked yetcites Polanyi (1962)

In the paper

“Impact is the most commonly used concept for what citations reflect, although other concepts such as influence, importance, significance, and utility occasionally also are used (Moed, 2005).”

What the sources say

• Not checked yetcites Moed, 2005

In the paper

“There seems to be a widespread assumption that this issue cannot be adequately assessed through standard citation indicators, and in recent years, increasing attention has been devoted to developing methodologies for assessing and measuring societal relevance and impact (Bornmann, 2012, 2013).”

What the sources say

• Not checked yetcites Bornmann, 2012, 2013

In the paper

“Nevertheless, there is a lack of empirical studies confirming that this is actually the case, and possibly some of the biases is of a fundamental nature attached to all citations measures, while the effect of others may tend to level out when aggregated levels are considered.”

In the paper

“Societal relevance is often considered to be something which is much harder to measure than scientific relevance or impact (B. R. Martin, 2011).”

What the sources say

• Not checked yetcites B. R. Martin, 2011

In the paper

“Moreover, all assessments of research quality may be context-dependent, in terms of, for example, the time of assessment and the time/field/sector perspectives of the evaluators.”

In the paper

“Using these definitions, it is also evident that impact would be a more adequate interpretation of citations than quality.”

What the sources say

• Not checked yetcites B. R. Martin and Irvine, 1983

In the paper

“Despite these and numerous similar examples, it is not possible to identify societal relevance from citation counts per se, and uncited or little cited publication may have contributed with results of great societal relevance.”

In the paper

“Because of this, it has long been emphasized by bibliometricians that more than one indicator should be used in research evaluation contexts (van Raan, 1993).”

What the sources say

• Not checked yetcites van Raan, 1993

In the paper

“citations reflect-with important limitations-aspects related to scientific impact and relevance”

In the paper

“The literature contains numerous publications of which the solidity is poor, the results unreliable or even involving misconduct or scientific fraud (Fanelli, 2009).”

What the sources say

• Not checked yetcites Fanelli, 2009

In the paper

“For example, the mean normalized citation score is size-independent and does not take into account the number of publications. According to Abramo and D'Angelo (2016), this is a major problem with this indicator because it does not truly represent productivity.”

What the sources say

• Not checked yetcites Abramo and D'Angelo (2016)

In the paper

“Surprisingly, this topic has rarely been addressed specifically in the literature and there are few studies analyzing the issue empirically.”

In the paper

“On this basis, it is often argued that bibliometric analysis can counterbalance shortcomings and mistakes in the peers' judgments (Aksnes, 2005). Thus, a bibliometric study should be considered as complementary to a peer evaluation (Council of Canadian Academies, 2012).”

What the sources say

• Not checked yetcites Aksnes, 2005
• Not checked yetcites Council of Canadian Academies, 2012

In the paper

“In sum, whereas plausibility/soundness, scientific value, and societal value and originality seem commonly perceived key characteristics of research quality, each of these dimensions include a variety of aspects; they may be contextdependent and may also conflict with each other.”

In the paper

“At the same time, there are also problems with the peer-review system.”

What the sources say

• Not checked yetcites Eyre-Walker and Stoletzki (2013)