Bibliometric evaluation vs. informed peer review: Evidence from Italy
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bertocchi et al. 2015 introduced references to the relevant literature which were not presented in the working paper versions.
“(Bertocchi et al. 2015) introduced references to the relevant literature which were not presented in the working paper versions of the paper.”
From Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise (2016) · cites this paper as Bertocchi et al. 2015
The value 0.54 used in this paper is drawn directly from the Area 13 report and reproduced in Bertocchi et al. 2015.
“The value of 0.54 that we used in this paper is drawn directly from the Area 13 report and reproduced also in (Bertocchi et al. 2015).”
From Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise (2016) · cites this paper as Bertocchi et al. 2015
Related papers described the same data as indicating significant concordance, remarkable agreement, similar evaluations, or fair to good agreement.
“Results of the experiment are also presented and commented as 'giving evidence of a significant degree of concordance among peer review and bibliometric evaluations' (Ancaiani et al. 2015).”
From Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise (2016) · cites this paper as Bertocchi et al. 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
The Calculation of Posterior Distributions by Data Augmentation
Tanner, M. A., & Wong, W. H. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478458 · Cited by 3,929 · see citing works
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · see citing works
A multivariate technique for multiply imputing missing values using a sequence of regression models
Raghunathan, T. E., Lepkowski, J. M., Hoewyk, J. V., & Solenberger, P. W. (2001) · Survey methodology · Cited by 1,981 · see citing works
What do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · see citing works
Fully conditional specification in multivariate imputation
Van Buuren, S., Brand, J. P. L., Groothuis-Oudshoorn, C. G. M., & Rubin, D. B. (2006) · Journal of Statistical Computation and Simulation · doi:10.1080/10629360600810434 · Cited by 1,217 · see citing works
Performance-based university research funding systems
Hicks, D. M. (2011) · Research Policy · doi:10.1016/j.respol.2011.09.007 · Cited by 1,025 · see citing works
University Research Evaluation and Funding: An International Comparison
Geuna, A., & Martin, B. R. (2003) · Minerva · doi:10.1023/b:mine.0000005155.70870.bd · Cited by 641 · see citing works
Ràfols, I., Loet Leydesdorff, A., O’Hare, A., Nightingale, P., & Stirling, A. (2012) · Research Policy · doi:10.1016/j.respol.2012.03.015 · Cited by 581 · see citing works
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567 · see citing works
The use of bibliometric data for the measurement of university research performance
Moed, H. F., Burger, W. J. M., Frankfort, J. G., & Van Raan, A. F. J. (1985) · Research Policy · doi:10.1016/0048-7333(85)90012-5 · Cited by 472 · see citing works
The Research Excellence Framework and the 'impact agenda': are we creating a Frankenstein monster?
Martin, B. R. (2011) · Research Evaluation · doi:10.3152/095820211x13118583635693 · Cited by 470 · see citing works
Psychological relevance and information science
Harter, S. P. (1992) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199210)43:9 · Cited by 415 · see citing works
Hicks, D. M. (1999) · Scientometrics · doi:10.1007/bf02457380 · Cited by 391 · see citing works
Eyre-Walker, A., & Stoletzki, N. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001675 · Cited by 297 · see citing works
Nicolaisen, J. (2007) · Annual Review of Information Science and Technology · doi:10.1002/aris.2007.1440410120 · Cited by 295 · see citing works
Oswald, A. J. (2007) · Economica · doi:10.1111/j.1468-0335.2006.00575.x · Cited by 214 · see citing works
Comparative analysis of a set of bibliometric indicators and central peer review criteria
Rinia, E. J., van Leeuwen, T. N., van Vuren, H. G., & van Raan, A. F. J. (1998) · Research Policy · doi:10.1016/s0048-7333(98)00026-2 · Cited by 193 · see citing works
Allen, L., Jones, C., Dolby, K., LYNN, D. W., & Walport, M. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005910 · Cited by 169 · see citing works
Assessing university research: a plea for a balanced approach
Butler, L. (2007) · Science and Public Policy · doi:10.3152/030234207x254404 · Cited by 123 · see citing works
Using the h-index to measure the quality of journals in the field of business and management
Mingers, J., Macri, F., & Petrovici, D. A. (2011) · Information Processing & Management · doi:10.1016/j.ipm.2011.03.009 · Cited by 117 · see citing works
Exploring the emerging knowledge base of ‘the knowledge society’
Fagerberg, J., Landström, H., & Martin, B. R. (2012) · Research Policy · doi:10.1016/j.respol.2012.03.007 · Cited by 116 · see citing works
Moed, H. F. (2007) · Science and Public Policy · doi:10.3152/030234207x255179 · Cited by 114 · see citing works
The UK and Italian research assessment exercises face to face
Rebora, G., & Turri, M. (2013) · Research Policy · doi:10.1016/j.respol.2013.06.009 · Cited by 113 · see citing works
Coverage and overlap of the new social sciences and humanities journal lists
Hicks, D. M., & Wang, J. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21458 · Cited by 95 · see citing works
Uncertainty Measures for Economics Journal Impact Factors
Stern, D. I. (2013) · Journal of Economic Literature · doi:10.1257/jel.51.1.173 · Cited by 76 · see citing works
Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in EconomicsPreprint
Combes, P., & Linnemer, L. (2010) · RePEc: Research Papers in Economics · Cited by 69 · see citing works
How Should Peer‐review Panels Behave?
Sgroi, D., & Oswald, A. J. (2013) · The Economic Journal · doi:10.1111/ecoj.12070 · Cited by 26 · see citing works
F1000 recommendations as a new data source for research evaluation: A comparison with citationsPreprint
Waltman, L., & Costas, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1303.3875 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Fleiss, J.L., 1981. Statistical Methods for Rates and Proportions, second ed. John Wiley, New York, NY.
- Harzing, A.-W., van der Wal, R., 2008. Comparing the Google Scholar h-index with the ISI Journal Impact Factor, Available at http://www.harzing.com/h indexjournals.htm .
- Little, R.E., Rubin, D.B., 2002. Statistical Analysis of Missing Data, second ed. John Wiley & Sons, New York, NY.
- Martin, B.R., Whitley, R., 2010. The UK Research Assessment Exercise: a case of regulatory capture? In: Whitley, R., Gläser, J., Engwall, L. (Eds.), Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation. Oxford University Press, Oxford, pp. 51-80.
- Moed, H.F., 2005. Citation Analysis in Research Evaluation. Springer, Dordrecht.
- OECD,2010. Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. OECD Publishing, Paris.
- Rubin, D.B., 1987. Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons, New York, NY.
- Seglen, P.O., 1992. The skewness of science. Journal of the American Society for Information Science 43, 628-638.
Cited by
Sorted by citations
Revisiting the Scientometric Conceptualization of Impact and Its Measurement
Abramo, G. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2018.05.001 · Cited by 80 · see citing works
Baccini, A., & De Nicolao, G. (2016) · Scientometrics · doi:10.1007/s11192-016-1929-y · Cited by 68 · see citing works
Systematic Analysis of Agreement between Metrics and Peer Review in the UK REF
Traag, V. A., & Waltman, L. (2019) · Palgrave Communications · doi:10.1057/s41599-019-0233-x · Cited by 65 · see citing works
Peer Review versus Bibliometrics: Which Method Better Predicts the Scholarly Impact of Publications?
Abramo, G., D’Angelo, C. A., & Reale, E. (2019) · Scientometrics · doi:10.1007/s11192-019-03184-y · Cited by 50 · see citing works
Haddawy, P., Hassan, S.-U., Asghar, A., & Amin, S. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2015.12.005 · Cited by 45 · see citing works
- Authors
- Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F.
- Affiliations
- University of Modena and Reggio Emilia (IT)
- Bocconi University (IT)
- Centre for Studies in Economics and Finance (IT)
- Italian National Agency for the Evaluation of Universities and Research Institutes (IT)
- University of Rome Tor Vergata (IT)
- Corresponding author
- Jappelli, Tullio
- Venue
- Research Policy
- Publisher
- Elsevier BV
- Published
- 2015-03
- Volume
- 44
- Issue
- 2
- Pages
- 451-466
- ISSN
- 0048-7333 (ISSN-L)
- OpenAlex
- W3121265048
- Cited by
- 86
- References
- 29
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 8 · 2023: 5 · 2022: 5 · 2021: 8 · 2020: 8 · 2019: 6 · 2018: 8 · 2017: 15 · 2016: 9 · 2015: 7 · 2014: 1
- FWCI
- 9.72
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Research Policy (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- Cineca Institutional Research Information System (Tor Vergata University) (submittedVersion, other-oa)
- Econstor (Econstor) (submittedVersion)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- Università di Modena e Reggio Emilia
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Keywords
- bibliometric analysis, research evaluation, economic research, management research, statistics research, journal articles, statistical bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
