Evaluation Practices and Effects of Indicator Use—A Literature Review

Rijcke, Sarah de; Wouters, Paul F.; Rushforth, Alex D.; Franssen, Thomas P.; Hammarfelt, Björn

doi:10.1093/reseval/rvv038

Cited by 467scientometrics and bibliometrics researchHigher Education Governance and DevelopmentResearch, Science, and Academia

Evaluation Practices and Effects of Indicator Use—A Literature Review

Rijcke, Sarah de, Wouters, Paul F., Rushforth, Alex D., Franssen, Thomas P., Hammarfelt, Björn·2016·Research Evaluation

DOI 10.1093/reseval/rvv038

Open this paper in CoChat — free account

Abstract

This review of the international literature on evaluation systems, evaluation practices, and metrics (mis)uses was written as part of a larger review commissioned by the Higher Education Funding Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014–5). The literature on evaluation systems, practices, and effects of indicator uses is extremely heterogeneous: it comprises hundreds of sources published in different media, spread over disciplines, and with considerable variation in the nature of the evidence. A condensation of the state-of-the-art in relevant research is therefore highly timely. Our review presents the main strands in the literature, with a focus on empirical materials about possible effects of evaluation exercises, ‘gaming’ of indicators, and strategic responses by scientific communities and others to requirements in research assessments. In order to increase visibility and availability, an adapted and updated review is presented here as a stand-alone—after authorization by HEFCE.

Integrity signals

Registry

Crossref · 10.1093/reseval/rvv038

Cites

117 of 117 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Only a minority of stakeholders employing bibliometric indicators are adequately familiar with their meaning and appropriate use.
“Yet, only a minority of stakeholders employing bibliometric indicators are adequately familiar with their meaning and appropriate use [2,4,5,7].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [7]
The 'publish or perish' attitude stimulates productivity of researchers but creates a bias towards quantity, often overlooking quality.
“this 'publish or perish' attitude, on the one hand, stimulates the productivity of researchers, but, on the other, creates a bias towards quantity, often overlooking quality [2,3,6-8,44,81,163,260,263,284-288].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [7]
Publication data and citation metrics continue to play the most important role in research assessments worldwide, so indicator-based research assessments remain at the center of global debate about their validity and influence.
“because publication data and citation metrics continue to play the most important role in research assessments worldwide, indicator-based research assessments also remain at the center of the global debate, not only about their validity, but also about their influence on research activities, scientific output, and its quality [1,5-7,44,260,265,289].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [7]

The Audit Society: Rituals of VerificationPreprint
Power, M. (1999) · RePEc: Research Papers in Economics · Cited by 3,597 · see citing works
The History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · see citing works
On Justification: Economies of Worth
Boltanski, L., & Thévenot, L. (2006) · doi:10.1515/9781400827145 · Cited by 2,709 · see citing works
Audit Cultures
Strathern, M. (2003) · Routledge eBooks · doi:10.4324/9780203449721 · Cited by 2,665 · see citing works
Trust in numbers: the pursuit of objectivity in science and public life
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1499 · Cited by 2,621 · see citing works
Rankings and Reactivity: How Public Measures Recreate Social Worlds
Espeland, W. N., & Sauder, M. (2007) · American Journal of Sociology · doi:10.1086/517897 · Cited by 2,453 · see citing works
Commensuration as a Social Process
Espeland, W. N., & Stevens, M. L. (1998) · Annual Review of Sociology · doi:10.1146/annurev.soc.24.1.313 · Cited by 2,113 · see citing works
Toward a Comparative Sociology of Valuation and Evaluation
Lamont, M. (2012) · Annual Review of Sociology · doi:10.1146/annurev-soc-070308-120022 · Cited by 1,492 · see citing works
Trust in Numbers: The Pursuit of Objectivity in Science and Public Life
Porter, T. M. (1995) · doi:10.1515/9781400821617 · Cited by 1,431 · see citing works
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · see citing works
On Justification: Economies of Worth
Boltanski, L., & Thévenot, L. (2006) · Cited by 1,236 · see citing works
The Discipline of Rankings: Tight Coupling and Organizational Change
Sauder, M., & Espeland, W. N. (2009) · American Sociological Review · doi:10.1177/000312240907400104 · Cited by 1,055 · see citing works
Performance-based university research funding systems
Hicks, D. M. (2011) · Research Policy · doi:10.1016/j.respol.2011.09.007 · Cited by 1,025 · see citing works
How Economics Shapes Science
Stephan, P. E. (2012) · Harvard University Press eBooks · doi:10.4159/harvard.9780674062757 · Cited by 958 · see citing works
Problems of citation analysis: A critical review
MacRoberts, M. H., & MacRoberts, B. R. (1989) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198909)40:5 · Cited by 807 · see citing works
Theorizing the Micro-politics of Resistance: New Public Management and Managerial Identities in the UK Public Services
Thomas, R., & Davies, A. (2005) · Organization Studies · doi:10.1177/0170840605051821 · Cited by 793 · see citing works
Knowledge, Higher Education, and the New Managerialism
Deem, R., Hillyard, S., & Reed, M. (2007) · doi:10.1093/acprof:oso/9780199265909.001.0001 · Cited by 715 · see citing works
Impact of bibliometrics upon the science system: Inadvertent consequences?
Weingart, P. (2005) · Scientometrics · doi:10.1007/s11192-005-0007-7 · Cited by 603 · see citing works
How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management
Ràfols, I., Loet Leydesdorff, A., O’Hare, A., Nightingale, P., & Stirling, A. (2012) · Research Policy · doi:10.1016/j.respol.2012.03.015 · Cited by 581 · see citing works
Audit culture and Illiberal governance
Shore, C. (2008) · Anthropological Theory · doi:10.1177/1463499608093815 · Cited by 580 · see citing works
Living with the H-Index? Metric Assemblages in the Contemporary Academy
Burrows, R. (2012) · The Sociological Review · doi:10.1111/j.1467-954x.2012.02077.x · Cited by 523 · see citing works
Knowledge, Higher Education, and the New Managerialism: The Changing Management of UK Universities
Deem, R., Hillyard, S., & Reed, M. I. (2007) · Durham Research Online (Durham University) · Cited by 516 · see citing works
Breaking the silence: The hidden injuries of neo-liberal academia
Gill, R. (2016) · Feministische Studien · doi:10.1515/fs-2016-0105 · Cited by 460 · see citing works
Journal list fetishism and the perversion of scholarship: reactivity and the ABS list
Willmott, H. C. (2011) · Organization · doi:10.1177/1350508411403532 · Cited by 456 · see citing works
Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts
Butler, L. (2002) · Research Policy · doi:10.1016/s0048-7333(02)00007-0 · Cited by 450 · see citing works
On the Way towards New Public Management? The Governance of University Systems in England, the Netherlands, Austria, and Germany
de Boer, H., Enders, J., & Schimank, U. (2007) · Springer eBooks · doi:10.1007/978-1-4020-5831-8_5 · Cited by 403 · see citing works
PUBLIC SECTOR REFORM IN DUTCH HIGHER EDUCATION: THE ORGANIZATIONAL TRANSFORMATION OF THE UNIVERSITY
de Boer, H. F., Enders, J., & Leišytė, L. (2007) · Public Administration · doi:10.1111/j.1467-9299.2007.00632.x · Cited by 371 · see citing works
Intended and unintended consequences of a publish‐or‐perish culture: A worldwide survey
van Dalen, H. P., & Henkens, K. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22636 · Cited by 355 · see citing works
Performance Measurement and Theory
(2017) · Routledge eBooks · doi:10.4324/9781315211947 · Cited by 349 · see citing works
History of the journal impact factor: Contingencies and consequences
Archambault, É., & Larivière, V. (2009) · Scientometrics · doi:10.1007/s11192-007-2036-x · Cited by 326 · see citing works
Beyond the multiversity: neoliberalism and the rise of the schizophrenic university
Shore, C. (2010) · Social Anthropology · doi:10.1111/j.1469-8676.2009.00094.x · Cited by 319 · see citing works
Academic Institutions in Search of Quality: Local Orders and Global Standards
Paradeise, C., & Thœnig, J.-C. (2013) · Organization Studies · doi:10.1177/0170840612473550 · Cited by 277 · see citing works
Taylorizing business school research: On the ‘one best way’ performative effects of journal ranking lists
Mingers, J., & Willmott, H. C. (2012) · Human Relations · doi:10.1177/0018726712467048 · Cited by 274 · see citing works
Metrics: A profusion of measures
Noorden, R. V. (2010) · Nature · doi:10.1038/465864a · Cited by 268 · see citing works
Quality Assurance in UK Higher Education: Issues of Trust, Control, Professional Autonomy and Accountability
Hoecht, A. (2006) · Higher Education · doi:10.1007/s10734-004-2533-2 · Cited by 255 · see citing works
Stressing Academe: The Wear and Tear of the New Public Management
Chandler, J., Barry, J., & Clark, H. (2002) · Human Relations · doi:10.1177/0018726702055009019 · Cited by 251 · see citing works
The Evaluation Society
Dahler‐Larsen, P. (2011) · Stanford University Press eBooks · doi:10.11126/stanford/9780804776929.001.0001 · Cited by 250 · see citing works
The evolution of research activity in Spain
Jiménez‐Contreras, E., Anegón, F. D. M., & López‐Cózar, E. D. (2002) · Research Policy · doi:10.1016/s0048-7333(02)00008-2 · Cited by 226 · see citing works
Constitutive Effects of Performance Indicators: Getting beyond unintended consequences
Dahler‐Larsen, P. (2013) · Public Management Review · doi:10.1080/14719037.2013.770058 · Cited by 215 · see citing works
Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven DisciplinesPreprint
Harley, D., Acord, S. K., Earl‐Novell, S., Lawrence, S., & King, C. J. (2010) · RePEc: Research Papers in Economics · Cited by 213 · see citing works
Beyond bibliometrics: harnessing multidimensional indicators of scholarly impact
(2014) · Choice Reviews Online · doi:10.5860/choice.186316 · Cited by 211 · see citing works
In search of relevance: the changing contract between science and society
Hessels, L. K., van Lente, H., & Smits, R. (2009) · Science and Public Policy · doi:10.3152/030234209x442034 · Cited by 195 · see citing works
UK Research Assessment Exercises: Informed judgments on research quality or quantity?
Moed, H. F. (2007) · Scientometrics · doi:10.1007/s11192-008-0108-1 · Cited by 186 · see citing works
Conservatism and risk-taking in peer review: Emerging ERC practices
Luukkonen, T. (2012) · Research Evaluation · doi:10.1093/reseval/rvs001 · Cited by 185 · see citing works
Irresponsible Authorship and Wasteful Publication
Huth, E. J. (1986) · Annals of Internal Medicine · doi:10.7326/0003-4819-104-2-257 · Cited by 184 · see citing works
Accountability in context: effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University
Hammarfelt, B., & de Rijcke, S. (2014) · Research Evaluation · doi:10.1093/reseval/rvu029 · Cited by 184 · see citing works
Self-citation
Modifying publication practices in response to funding formulas
Butler, L. (2003) · Research Evaluation · doi:10.3152/147154403781776780 · Cited by 179 · see citing works
Perverse Audit Culture and Accountability of the Modern Public University
Craig, R., AMERNIC, J. H., & Tourish, D. (2014) · Financial Accountability and Management · doi:10.1111/faam.12025 · Cited by 174 · see citing works
The Impact of Research Selectivity on Academic Work and Identity in UK Universities
Harley, S. (2002) · Studies in Higher Education · doi:10.1080/03075070220119986b · Cited by 173 · see citing works
Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994-2005) and their Editors' views
Chew, M., Villanueva, E., & Weyden, M. B. V. D. (2007) · Journal of the Royal Society of Medicine · doi:10.1177/014107680710000313 · Cited by 164 · see citing works
The Changing Governance of the Sciences : The Advent of Research Evaluation Systems
Whitley, R. D., & Gläser, J. (2007) · Research Explorer (The University of Manchester) · doi:10.1007/978-1-4020-6746-4 · Cited by 156 · see citing works
Beyond breakthrough research: Epistemic properties of research and their consequences for research funding
Laudel, G., & Gläser, J. (2014) · Research Policy · doi:10.1016/j.respol.2014.02.006 · Cited by 148 · see citing works
Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation
Whitley, R. D., Glaeser, J., & Engwall, L. (2010) · Research Explorer (The University of Manchester) · Cited by 145 · see citing works
Beyond Bibliometrics
De Bellis, N. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9445.001.0001 · Cited by 143 · see citing works
How incentives trickle down: Local use of a national bibliometric indicator system
Aagaard, K. (2015) · Science and Public Policy · doi:10.1093/scipol/scu087 · Cited by 141 · see citing works
Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator
Aagaard, K., Bloch, C. W., & Schneider, J. W. (2015) · Research Evaluation · doi:10.1093/reseval/rvv003 · Cited by 138 · see citing works
The representation of the social sciences and humanities in the Web of Science--a comparison of publication patterns and incentive structures in Flanders and Norway (2005-9)
Ossenblok, T. L. B., Engels, T., & Sivertsen, G. (2012) · Research Evaluation · doi:10.1093/reseval/rvs019 · Cited by 134 · see citing works
Researchers’ perceptions of citations
Aksnes, D. W., & Rip, A. (2009) · Research Policy · doi:10.1016/j.respol.2009.02.001 · Cited by 133 · see citing works
Accounting for Impact? The Journal Impact Factor and the Making of Biomedical Research in the Netherlands
Rushforth, A., & de Rijcke, S. (2015) · Minerva · doi:10.1007/s11024-015-9274-5 · Cited by 132 · see citing works
Self-citation
Changing Governance of the Public Sciences
Whitley, R. D. (2007) · Sociology of the sciences/Sociology of the sciences yearbook · doi:10.1007/978-1-4020-6746-4_1 · Cited by 130 · see citing works
The Assessment of Research Quality in UK Universities: Peer Review or Metrics?
Taylor, J. (2010) · British Journal of Management · doi:10.1111/j.1467-8551.2010.00722.x · Cited by 130 · see citing works
How peer review empowers the academic profession and university managers: Changes in relationships between the state, universities and the professoriate
Musselin, C. (2013) · Research Policy · doi:10.1016/j.respol.2013.02.002 · Cited by 127 · see citing works
Assessing university research: a plea for a balanced approach
Butler, L. (2007) · Science and Public Policy · doi:10.3152/030234207x254404 · Cited by 123 · see citing works
Made to Measure: Taming Practices with Results-based Accountability
Keevers, L., Treleaven, L., Sykes, C., & Darcy, M. D. (2011) · Organization Studies · doi:10.1177/0170840611430597 · Cited by 116 · see citing works
What do the scientists think about the impact factor?
Buela‐Casal, G., & Zych, I. (2012) · Scientometrics · doi:10.1007/s11192-012-0676-y · Cited by 114 · see citing works
The UK and Italian research assessment exercises face to face
Rebora, G., & Turri, M. (2013) · Research Policy · doi:10.1016/j.respol.2013.06.009 · Cited by 113 · see citing works
Understanding Academic Work in a Changing Institutional Environment
Leišytė, L., & Dee, J. R. (2012) · Higher education · doi:10.1007/978-94-007-2950-6_3 · Cited by 112 · see citing works
Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994-2005) and their Editors' views
Chew, M. M., Villanueva, E., & Weyden, M. B. V. D. (2007) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.100.3.142 · Cited by 108 · see citing works
An Outline of the Bibliometric Indicator Used for Performance-Based Funding of Research Institutions in Norway
Schneider, J. W. (2009) · European Political Science · doi:10.1057/eps.2009.19 · Cited by 107 · see citing works
Changing Governance and Authority Relations in the Public Sciences
Whitley, R. D. (2011) · Minerva · doi:10.1007/s11024-011-9182-2 · Cited by 107 · see citing works
The controversial policies of journal ratings: evaluating social sciences and humanities
Pontille, D., & Torny, D. (2010) · Research Evaluation · doi:10.3152/095820210x12809191250889 · Cited by 103 · see citing works
The history and meaning of the term "phoneme"
Jones, D. C. (1957) · Medical Entomology and Zoology · Cited by 94 · see citing works
The Science of science policy: a handbook
(2011) · Choice Reviews Online · doi:10.5860/choice.49-0822 · Cited by 88 · see citing works
What Happens When Funding is Linked to Publication Counts?
Butler, L. (2006) · Kluwer Academic Publishers eBooks · doi:10.1007/1-4020-2755-9_18 · Cited by 79 · see citing works
The Citation: From Culture to Infrastructure
Wouters, P. F. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9445.003.0006 · Cited by 76 · see citing works
Self-citation
The heterogeneous landscape of bibliometric indicators: Evaluating models for allocating resources at Swedish universities
Hammarfelt, B., Nelhans, G., Eklund, P., & Åström, F. (2016) · Research Evaluation · doi:10.1093/reseval/rvv040 · Cited by 74 · see citing works
Self-citation
A critical analysis of the journal impact factors ofAngewandte Chemie and the journal of the American Chemical Society inaccuracies in published impact factors based on overall citations only
Moed, H. F., van Leeuwen, T. N., & Reedijk, J. (1996) · Scientometrics · doi:10.1007/bf02093487 · Cited by 71 · see citing works
How implementation of bibliometric practice affects the role of academic libraries
Åström, F., & Hansson, J. (2012) · Journal of Librarianship and Information Science · doi:10.1177/0961000612456867 · Cited by 69 · see citing works
The UK Research Assessment Exercise
Martin, B. R., & Whitley, R. D. (2010) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199590193.003.0002 · Cited by 68 · see citing works
Citation counts and social comparisons: Scientists' use and evaluation of citation index data
Hargens, L. L., & Schuman, H. (1990) · Social Science Research · doi:10.1016/0049-089x(90)90006-5 · Cited by 63 · see citing works
Overturning some assumptions about the effects of evaluation systems on publication performance
Osuna, C., Castro, L. C., & Menéndez, L. S. (2010) · Scientometrics · doi:10.1007/s11192-010-0312-7 · Cited by 61 · see citing works
Journal List Fetishism and the Perversion of Scholarship: Reactivity and the ABS ListPreprint
Willmott, H. C. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1753627 · Cited by 59 · see citing works
The UK Research Assessment Exercise: A Case of Regulatory Capture?
Whitley, R., Martin, B. R., Glaeser, J., et al. (2010) · Research Explorer (The University of Manchester) · Cited by 59 · see citing works
Influence of a performance indicator on Danish research production and citation impact 2000–12
Ingwersen, P., & Larsen, B. (2014) · Scientometrics · doi:10.1007/s11192-014-1291-x · Cited by 54 · see citing works
Democratising research evaluation: Achieving greater public engagement with bibliometrics-informed peer review
Derrick, G., & Pavone, V. (2013) · Science and Public Policy · doi:10.1093/scipol/sct007 · Cited by 47 · see citing works
Impacts of performance-based research funding systems: A review of the concerns and the evidence
OECD, & Butler, L. (2010) · OECD eBooks · doi:10.1787/9789264094611-7-en · Cited by 46 · see citing works
Instruments as empirical evidence for the analysis of Higher Education policies
Reale, E., & Seeber, M. (2012) · Higher Education · doi:10.1007/s10734-012-9585-5 · Cited by 46 · see citing works
Rank and File: Managing Individual Performance in University Research
Nedeva, M., Boden, R., & Nugroho, Y. (2012) · Higher Education Policy · doi:10.1057/hep.2012.12 · Cited by 44 · see citing works
Unable to resist: Researchers’ responses to research assessment in the Czech Republic
Linková, M. (2013) · Human Affairs · doi:10.2478/s13374-014-0207-z · Cited by 43 · see citing works
To intervene or not to intervene; is that the question? On the role of scientometrics in research evaluation
de Rijcke, S., & Rushforth, A. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23382 · Cited by 42 · see citing works
Self-citation
The United Kingdom's Research Assessment Exercise
HARE, P. G. (2003) · Higher Education Management · doi:10.1787/hemp-v15-art12-en · Cited by 39 · see citing works
The Limits of Universality: How Field-Specific Epistemic Conditions Affect Authority Relations and their Consequences.
Gläser, J., Lange, S., Laudel, G., et al. (2010) · University of Twente Research Information · Cited by 39 · see citing works
Integrating Scientometric Indicators into Sociological Studies: Methodical and Methodological Problems
Gläser, J., & Laudel, G. (2001) · Scientometrics · doi:10.1023/a:1014243832084 · Cited by 38 · see citing works
The Limits of Universality
Gläser, J., Lange, S., Laudel, G., et al. (2010) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199590193.003.0010 · Cited by 38 · see citing works
Performativity, work-related emotions and collective research identities in UK university education departments: an exploratory study
Wilson, M. H., & Holligan, C. (2013) · Cambridge Journal of Education · doi:10.1080/0305764x.2013.774321 · Cited by 37 · see citing works
Performance Measurement and the Governance of American Academic Science
Feller, I. (2009) · Minerva · doi:10.1007/s11024-009-9129-z · Cited by 37 · see citing works
Creative knowledge environments for research groups in biotechnology. The influence of leadership and organizational support in universities and business companies
Hemlin, S. (2006) · Scientometrics · doi:10.1007/s11192-006-0053-9 · Cited by 35 · see citing works
Tensions between evaluations and communication practices
Laudel, G., & Gläser, J. (2006) · Journal of Higher Education Policy and Management · doi:10.1080/13600800600980130 · Cited by 35 · see citing works
Facing Inwards and Outwards at Once: The liminal temporalities of academic perfomativity
Keenoy, T. (2005) · Time & Society · doi:10.1177/0961463x05055140 · Cited by 31 · see citing works
Interactions between journal attributes and authors' willingness to wait for editorial decisions
Rousseau, S., & Rousseau, R. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22637 · Cited by 29 · see citing works
Accountability, performance assessment, and evaluation: Policy pressures and responses from research councils
M., C., Kretz, A. J., & Sigurdson, K. (2013) · Research Evaluation · doi:10.1093/reseval/rvs041 · Cited by 28 · see citing works
Constitutive Effects of Performance Indicator Systems
Dahler‐Larsen, P. (2007) · Advances in program evaluation · doi:10.1016/s1474-7863(07)10002-8 · Cited by 27 · see citing works
Use and Usefulness of Performance Measurement in State Science and Technology Programs
Cozzens, S. E., & Melkers, J. E. (1997) · Policy Studies Journal · doi:10.1111/j.1541-0072.1997.tb00032.x · Cited by 26 · see citing works
The UK Research Excellence Framework and the Transformation of Research Production
Sousa, S., & Brennan, J. (2013) · Higher education dynamics · doi:10.1007/978-94-007-7028-7_4 · Cited by 25 · see citing works
The ‘Economy of Memory’: Publications, Citations, and the Paradox of Effective Research Governance
Woelert, P. (2013) · Minerva · doi:10.1007/s11024-013-9232-z · Cited by 25 · see citing works
Defending universities: Argument and persuasion
Collini, S. (2015) · Power and Education · doi:10.1177/1757743814567383 · Cited by 25 · see citing works
Interfering Governance and Emerging Centres of Control
van der Meulen, B. (2007) · Sociology of the sciences/Sociology of the sciences yearbook · doi:10.1007/978-1-4020-6746-4_9 · Cited by 18 · see citing works
Transparency of research policy and the role of librarians
Demšar, F., & Južnič, P. (2013) · Journal of Librarianship and Information Science · doi:10.1177/0961000613503002 · Cited by 18 · see citing works
The Research Assessment Exercise (RAE) and after: 'You never know how it will all turn out'
McNay, I. (1998) · Perspectives Policy and Practice in Higher Education · doi:10.1080/713847899 · Cited by 17 · see citing works
Quality Assurance in UK Higher Education
Shrimplin, V. (2002) · Astrophysics and space science library · doi:10.1007/978-94-010-0606-4_6 · Cited by 16 · see citing works
Editorial
Braun, T. (2012) · Scientometrics · doi:10.1007/s11192-012-0754-1 · Cited by 16 · see citing works
Productivity Effects of Research Assessment ExercisesPreprint
Moore, W. J., Newman, R. J., Sloane, P. J., et al. (2002) · RePEc: Research Papers in Economics · Cited by 15 · see citing works
Power at the Interfaces: The Contested Orderings of Academic Presents and Futures in a Social Science Department
Stöckelová, T. (2014) · Higher Education Policy · doi:10.1057/hep.2014.20 · Cited by 15 · see citing works
Measuring What’s Valued or Valuing What’s Measured? Knowledge Production and the Research Assessment Exercise
Alldred, P., & Miller, T. (2007) · Palgrave Macmillan UK eBooks · doi:10.1057/9780230287013_9 · Cited by 15 · see citing works
Rank and file [Web site design]
Thomas, B. (1998) · IEEE Internet Computing · doi:10.1109/4236.707812 · Cited by 14 · see citing works
Bibliometric Services in Research Evaluation: A New Task Area Strengthening the Jurisdiction of Academic Librarians
Petersohn, S. (2014) · Purdue e-Pubs (Purdue University System) · doi:10.13140/2.1.5179.1365 · Cited by 11 · see citing works
A review of the literature on citation impact indicatorsPreprint
Waltman, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02099 · Cited by 10 · see citing works

Flagged first, then by citations

Sorted by citations

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · see citing works
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257 · see citing works
Self-citation
A billion-dollar donation: estimating the cost of researchers’ time spent on peer review
Aczel, B., Szaszi, B., & Holcombe, A. O. (2021) · Research Integrity and Peer Review · doi:10.1186/s41073-021-00118-2 · Cited by 306 · see citing works
A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research
Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm, P., Larkin, C., Lepori, B., Mosoni-Fried, J., Oliver, E., et al. (2018) · Research Evaluation · doi:10.1093/reseval/rvx025 · Cited by 258 · see citing works
Systematic Analysis of Agreement between Metrics and Peer Review in the UK REF
Traag, V. A., & Waltman, L. (2019) · Palgrave Communications · doi:10.1057/s41599-019-0233-x · Cited by 65 · see citing works
Critical Rationalism and the Search for Standard (Field-Normalized) Indicators in Bibliometrics
Bornmann, L., & Marx, W. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.05.002 · Cited by 41 · see citing works
Citation Success Index − An Intuitive Pair-Wise Journal Comparison Metric
Milojević, S., Radicchi, F., & Bar-Ilan, J. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2016.12.006 · Cited by 33 · see citing works

Authors: Rijcke, S. D., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B.
Affiliations: Leiden University (NL)
Corresponding author: Rijcke, Sarah de
ORCID: Rijcke, Sarah de
Wouters, Paul F.
Franssen, Thomas P.
Hammarfelt, Björn
Venue: Research Evaluation
Publisher: Oxford University Press (OUP)
Published: 2015-12-29
Volume: 25
Issue: 2
Pages: 161-169
ISSN: 0958-2029 (ISSN-L)
1471-5449
DOI: 10.1093/reseval/rvv038
OpenAlex: W2235214564
Cited by: 467
References: 117
Citations by year: 2026: 41 · 2025: 35 · 2024: 52 · 2023: 43 · 2022: 58 · 2021: 56 · 2020: 56 · 2019: 44 · 2018: 41 · 2017: 28 · 2016: 12 · 2015: 1
FWCI: 4.49
Type: journal article
Language: en
Open access: closed
Locations: Research Evaluation (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Leiden Repository (Leiden University) (submittedVersion)
HighWire Press Open Archive (submittedVersion)
Data Archiving and Networked Services (DANS) (submittedVersion)
Topics: Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Social Sciences › Political Science and International Relations › Higher Education Governance and Development
Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
Keywords: evaluation metrics, assessment practices, evaluation system, research evaluation, higher education research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Evaluation Practices and Effects of Indicator Use—A Literature Review

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks