Systematic Analysis of Agreement between Metrics and Peer Review in the UK REF
Open accessAbstract
When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found. However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review. We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review. In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF. The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review. To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree relatively well with peer review and may offer an alternative to peer review.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Correlations between metrics and expert judgements vary greatly across disciplines, application contexts, and between metrics used.
“it was also observed that the correlations between metrics and expert judgements vary greatly across disciplines, application contexts, and between metrics used [44,276,280,282].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [282]
Bibliometric indicators were generally shown to be positively correlated with peer-review evaluations.
“Although bibliometric indicators were generally shown to be positively correlated with peer-review evaluations”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [282]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 42 · Cited by 65
Research works printed but not linked: 17
- Allen N, Heath O (2013) Reputations and research quality in British political science: the importance of journal and publisher rankings in the 2008 RAE. Br J Polit Int Relat 15:147-162. https://doi.org/10.1111/1467-856X.12006 · DOI
- Bence V, Oppenheim CT (2005) The evolution of the UK's Research Assessment Exercise: publications, performance and perceptions. J Educ Adm Hist 37:137-155. https://doi.org/10.1080/00220620500211189 · DOI
- Carpenter MP, Gibb F, Harris M, Irvine J, Martin BR, Narin F (1988) Bibliometric profiles for British academic institutions: an experiment to develop research output indicators. Scientometrics 14:213-233. https://doi.org/10.1007/ BF02020076 · DOI
- Clerides S, Pashardes P, Polycarpou A (2011) Peer review vs metric-based assessment: testing for bias in the RAE ratings of UK economics departments. Economica 78:565-583. https://doi.org/10.1111/j.1468-0335.2009.00837.x · DOI
- Colman AM, Dhillon D, Coulthard B (1995) A bibliometric evaluation of the research performance of British university politics departments: publications in leading journals. Scientometrics 32:49-66. https://doi.org/10.1007/ BF02020188 · DOI
- Crewe I (1988) Reputation, research and reality: the publication records of UK departments of politics, 1978-1984. Scientometrics 14:235-250. https://doi. org/10.1007/BF02020077 · DOI
- Farla K, Simmonds P (2015) REF accountability review: costs, benefits and burden -report by Technopolis to the four UK higher education funding bodies, Technopolis
- Harzing A-W (2017) Running the REF on a rainy Sunday afternoon: do metrics match peer review? https://harzing.com/publications/white-papers/running- the-ref-on-a-rainy-sunday-afternoon-do-metrics-match-peer-review Acces- sed 21 Nov 2018.
- Johnes J, Taylor J, Francis B (1993) The research performance of UK universities: a statistical analysis of the results of the 1989 Research Selectivity Exercise. J R Stat Soc A156:271-286. https://doi.org/10.2307/2982732 · DOI
- Kelly A, Burrows R (2011) Measuring the value of sociology? Some notes on performative metricization in the contemporary academy. Sociol Rev 59:130-150. https://doi.org/10.1111/j.1467-954X.2012.02053.x · DOI
- Lim Ban Seng, Willett P (1995) The citedness of publications by United Kingdom library schools. J Inf Sci 21:68-71. https://doi.org/10.1177/ 016555159502100109 · DOI
- Mckay S (2012) Social policy excellence-peer review or metrics? Analyzing the 2008 Research Assessment Exercise in social work and social policy and administration. Soc Policy Adm 46:526-543. https://doi.org/10.1111/j.1467- 9515.2011.00824.x · DOI
- Mryglod O, Kenna R, Holovatch Y, Berche B (2013a) Absolute and specific measures of research group excellence. Scientometrics 95:115-127. https:// doi.org/10.1007/s11192-012-0874-7 · DOI
- Pollitt A, Potoglou D, Patil S, Burge P, Guthrie S, King S, Wooding S, Wooding S, Grant J (2016) Understanding the relative valuation of research impact: a best-worst scaling experiment of the general public and biomedical and health researchers. BMJ Open 6:e010916. https://doi.org/10.1136/bmjopen- 2015-010916 · DOI
- Taylor J (1994) Measuring research performance in business and management studies in the United Kingdom: the 1992 Research Assessment Exercise. Br J Manag 5:275-288. https://doi.org/10.1111/j.1467-8551.1994.tb00079.x · DOI
- West J, Bergstrom T (2010) Big Macs and Eigenfactor scores: don't let correlation coefficients fool you. J Am Soc Inf Sci Technol 61:1-25. https://doi.org/ 10.1002/ASI.V61:9 · DOI
- Zacharewicz T, Lepori B, Reale E, Jonkers K (2018) Performance-based research funding in EU member states-a comparative assessment. Sci Public Policy scy041. https://doi.org/10.1093/scipol/scy041 · DOI
- Authors
- Traag, V. A., & Waltman, L.
- Affiliations
- Leiden University (NL)
- Corresponding author
- Traag, V. A.
- Venue
- Palgrave Communications
- Published
- 2019-03-12
- Volume
- 5
- Issue
- 1
- ISSN
- 2055-1045 (ISSN-L)
- OpenAlex
- W2886446017
- Cited by
- 65
- References
- 42
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 8 · 2023: 5 · 2022: 3 · 2021: 11 · 2020: 13 · 2019: 10
- FWCI
- 6.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Humanities and Social Sciences Communications (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- citation metrics, peer review, Research Excellence Framework, disciplinary variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
