Evaluating scientific research in Italy: The 2004–10 research evaluation exercise
Abstract
The Italian Research Evaluation assessment for the period 2004–10 (VQR 2004–10) has analyzed almost 185,000 articles, books, patents, and other scientific outcomes submitted for evaluation by Italian universities and other public research bodies. This article describes the main features of this exercise, introducing its legal framework and the criteria used for evaluation. The innovative methodology that has been used for evaluation, based on a combination of peer review and bibliometric methods, is discussed and indicators for assessing the quality of participating research bodies are derived accordingly. The article also presents the main results obtained at the University level, trying to understand the existing relationship among research quality and University characteristics such as location, dimension, age, scientific specialization, and funding.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For the whole sample and VQR-weights, a value of 0.3441 is reported in Ancaiani et al. 2015.
“For the whole sample and VQR-weights, a value of 0.3441 is reported in (Ancaiani et al. 2015).”
From Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise (2016) · cites this paper as Ancaiani et al. 2015
For Area 13 and VQR-weights, a value of 0.6104 is reported in the Appendix B of the ANVUR Final Report and in Cicero et al. 2013 and Ancaiani et al. 2015.
“For Area 13 and VQR-weights, a value of 0.6104 is reported in the Appendix B of the ANVUR Final Report and also in (Cicero et al. 2013) and (Ancaiani et al. 2015).”
From Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise (2016) · cites this paper as Ancaiani et al. 2015
Related papers described the same data as indicating significant concordance, remarkable agreement, similar evaluations, or fair to good agreement.
“Results of the experiment are also presented and commented as 'giving evidence of a significant degree of concordance among peer review and bibliometric evaluations' (Ancaiani et al. 2015).”
From Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise (2016) · cites this paper as Ancaiani et al. 2015
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Henderson, R. M., Jaffe, A. B., & Trajtenberg, M. (1998) · The Review of Economics and Statistics · doi:10.1162/003465398557221 · Cited by 1,336 · see citing works
Citation Analysis in Research Evaluation
Moed, H. F. (2005) · Springer eBooks · doi:10.1007/1-4020-3714-7 · Cited by 1,334 · see citing works
Academic research and industrial innovation
Mansfield, E. (1991) · Research Policy · doi:10.1016/0048-7333(91)90080-a · Cited by 1,277 · see citing works
Polanyi, M. F. D. (1962) · Minerva · doi:10.1007/bf01101453 · Cited by 1,006 · see citing works
Comparing bibliometric statistics obtained from the Web of Science and Scopus
Archambault, É., Campbell, D., Gingras, Y., & Larivière, V. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21062 · Cited by 833 · see citing works
Three options for citation tracking: Google Scholar, Scopus and Web of Science
Bakkalbasi, N., Bauer, K., Glover, J., & Wang, L. (2006) · Biomedical Digital Libraries · doi:10.1186/1742-5581-3-7 · Cited by 752 · see citing works
University Research Evaluation and Funding: An International Comparison
Geuna, A., & Martin, B. R. (2003) · Minerva · doi:10.1023/b:mine.0000005155.70870.bd · Cited by 641 · see citing works
The Republic of Science: Its Political and Economic Theory
Polanyi, M. F. D. (2000) · Minerva · doi:10.1023/a:1026591624255 · Cited by 631 · see citing works
Griliches, Z. (1998) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226308906.001.0001 · Cited by 609 · see citing works
van Raan, A. F. J. (1996) · Scientometrics · doi:10.1007/bf02129602 · Cited by 390 · see citing works
Impact factor: outdated artefact or stepping-stone to journal certification?
Vanclay, J. K. (2011) · Scientometrics · doi:10.1007/s11192-011-0561-0 · Cited by 370 · see citing works
The Four Literatures of Social Science
Hicks, D. M. (2006) · Kluwer Academic Publishers eBooks · doi:10.1007/1-4020-2755-9_22 · Cited by 337 · see citing works
Publishing as Prostitution? – Choosing Between One's Own Ideas and Academic Success
Frey, B. S. (2003) · Public Choice · doi:10.1023/a:1024208701874 · Cited by 296 · see citing works
Citation Analysis: Comparison of Web of Science®, Scopus™, SciFinder®, and Google Scholar
Li, J., Burnham, J. F., Lemley, T., & Britton, R. M. (2010) · Journal of Electronic Resources in Medical Libraries · doi:10.1080/15424065.2010.505518 · Cited by 267 · see citing works
R&D and Productivity: The Econometric Evidence.
(1999) · The Economic Journal · doi:10.1111/1468-0297.00446 · Cited by 259 · see citing works
Huang, M., & Chang, Y. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20885 · Cited by 193 · see citing works
The UK Research Assessment Exercise: the evolution of a national research evaluation system
Barker, K. E. (2007) · Research Evaluation · doi:10.3152/095820207x190674 · Cited by 146 · see citing works
Baxt, W. G., Waeckerle, J. F., Berlin, J. A., & Callaham, M. L. (1998) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(98)70006-x · Cited by 141 · see citing works
Exploring size and agglomeration effects on public research productivity
Bonaccorsi, A. J., & Daraio, C. C. (2005) · Scientometrics · doi:10.1007/s11192-005-0205-3 · Cited by 133 · see citing works
Evaluating R&D Impacts: Methods and Practice
(1993) · Springer eBooks · doi:10.1007/978-1-4757-5182-6 · Cited by 86 · see citing works
Citations versus journal impact factor as proxy of quality: could the latter ever be preferable?
Abramo, G., D’Angelo, C. A., & Costa, F. D. (2010) · Scientometrics · doi:10.1007/s11192-010-0200-1 · Cited by 80 · see citing works
Self-citationHunt, G. E., Cleary, M. L., & Walter, G. (2010) · Harvard Review of Psychiatry · doi:10.3109/10673229.2010.493742 · Cited by 80 · see citing works
Horta, H., & Lacy, T. A. (2011) · Science and Public Policy · doi:10.3152/030234211x12960315267813 · Cited by 73 · see citing works
Metrics or Peer Review? Evaluating the 2001 UK Research Assessment Exercise in Political Science
Butler, L., & McAllister, I. (2008) · Political Studies Review · doi:10.1111/j.1478-9299.2008.00167.x · Cited by 59 · see citing works
Quantitative and qualitative methods and measures in the evaluation of research
Roessner, D. (2000) · Research Evaluation · doi:10.3152/147154400781777296 · Cited by 53 · see citing works
Research on research evaluation
Hemlin, S. (1996) · Social Epistemology · doi:10.1080/02691729608578815 · Cited by 48 · see citing works
Revisiting size effects in higher education research productivity
Abramo, G., Cicero, T., & D’Angelo, C. A. (2011) · Higher Education · doi:10.1007/s10734-011-9471-6 · Cited by 44 · see citing works
Self-citationBenchmarking Google Scholar with the New Zealand PBRF research assessment exercise
Smith, A. G. (2007) · Scientometrics · doi:10.1007/s11192-008-0219-8 · Cited by 42 · see citing works
Research evaluation in European national science and technology systems
Georghiou, L. (1995) · Research Evaluation · doi:10.1093/rev/5.1.3 · Cited by 37 · see citing works
Bornmann, L., & Leydesdorff, L. (2012) · Journal of Informetrics · doi:10.1016/j.joi.2011.11.002 · Cited by 31 · see citing works
Is the university model an organizational necessity? Scale and agglomeration effects in science
Brandt, T., & Schubert, T. (2012) · Scientometrics · doi:10.1007/s11192-012-0834-2 · Cited by 30 · see citing works
Rigby, J., & Julian, K. (2014) · Scientometrics · doi:10.1007/s11192-014-1259-x · Cited by 13 · see citing works
The Performance of Peer Review and a Beauty Contest of Referee Processes of Economics Journals
Seidl, C., Schmidt, U., & Grösche, P. (2005) · Studies of Applied Economics · Cited by 11 · see citing works
Toivanen, O., & Waterson, M. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2239617 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257 · see citing works
Baccini, A., & De Nicolao, G. (2016) · Scientometrics · doi:10.1007/s11192-016-1929-y · Cited by 68 · see citing works
Peer Review versus Bibliometrics: Which Method Better Predicts the Scholarly Impact of Publications?
Abramo, G., D’Angelo, C. A., & Reale, E. (2019) · Scientometrics · doi:10.1007/s11192-019-03184-y · Cited by 50 · see citing works
- Authors
- Ancaiani, A., Anfossi, A. F., Barbara, A., Benedetto, S., Blasi, B., Carletti, V., Cicero, T., Ciolfi, A., Costa, F., Colizza, G., Costantini, M., di Cristina, F., Ferrara, A., Lacatena, R. M., Malgarini, M., Mazzotta, I., Nappi, C. A., Romagnosi, S., & Sileoni, S.
- Affiliations
- University of Pisa (IT)
- Corresponding author
- Malgarini, Marco
- ORCID
- Venue
- Research Evaluation
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-05-03
- Volume
- 24
- Issue
- 3
- Pages
- 242-255
- ISSN
- 0958-2029 (ISSN-L)
- 1471-5449
- OpenAlex
- W2138780415
- Cited by
- 75
- References
- 36
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 6 · 2023: 2 · 2022: 5 · 2021: 5 · 2020: 15 · 2019: 10 · 2018: 5 · 2017: 10 · 2016: 8 · 2015: 4
- FWCI
- 8.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Research Evaluation (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Political Science and International Relations › Higher Education Governance and Development
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Keywords
- Italian universities, peer review, bibliometric indicators, research quality assessment, research funding, university characteristics, scientific specialization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
