Do They Agree? Bibliometric Evaluation versus Informed Peer Review in the Italian Research Assessment Exercise
Open accessAI snapshot
Written from the paper’s full text.
TL;DR
This paper re-examines ANVUR’s Italian VQR experiment comparing informed peer review (IR) and bibliometric evaluation, arguing that ANVUR’s claim of “good” or “more than adequate” agreement is not supported by standard kappa guidelines. It concludes that IR and bibliometrics generally do not produce similar results, and that the apparent exception in Area 13 (economics and statistics) is fatally flawed by protocol changes.
Key findings
- In the experiment, 9,199 articles were evaluated by both IR and bibliometrics, representing 9.3% of the 99,005 submitted articles. Overall weighted kappa was 0.32 with linear weights and 0.38 with VQR weights.
- For Areas 1–9, kappas were mostly in the range 0.1–0.35 (linear weights) or 0.15–0.35 (VQR weights), with Area 13 at 0.54. Sub-area kappas ranged from 0.095 for nuclear physics to 0.56 for economics; only four of 43 sub-areas had kappa values greater than 0.40.
- Under available guidelines, the paper interprets nearly all area and sub-area agreements as “unacceptable,” “poor,” or at most “fair,” except Area 13 and some of its sub-areas, which showed moderate agreement.
- A meta-analysis found Area 13 significantly different from the other areas (p = 0.0036 linear; p = 0.0086 VQR), and 3 of its 4 sub-areas fell outside the 95% prediction funnel. The paper estimates at least 326 of 590 Area 13 articles (55.3%) were evaluated directly by GEV consensus groups rather than by referees.
Methods
- Re-analysis of ANVUR’s VQR 2004–2010 experiment using data published in the Final Report and Area reports, because raw data were not made available.
- Weighted Cohen’s kappa was used with two weighting schemes: linear weights and ANVUR’s VQR weights, applied to four ordered merit classes.
- Statistical meta-analysis with funnel plots compared Area 13 and its sub-areas against the other areas/sub-areas, using kappa values and sample sizes.
Limitations
- The paper states that raw data were not made available by ANVUR, so the authors could not control the experiment in detail or reproduce the published results; the analysis relies only on published reports.
- The estimate that at least 55.3% of Area 13 IR evaluations were decided by consensus groups depends on stated hypotheses and is described as a lower-bound estimate.
- The paper notes that results for the economic history sub-area should be treated with particular caution, and the text makes evident that protocol differences in Area 13 are inferred from published reports rather than directly observed raw data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 43 cited sources checked · 0 supported · 0 flagged
- Findingcitation§ ABSTRACT · p. 1
The degree of agreement between informed peer review and bibliometrics in the experiment, always in the range 0.09-0.42, has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, at most fair.
“This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement (always in the range 0.09-0.42) has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, as, at most, fair.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Sheskin 2003 source
Not checked in the free run.
- Findingreasoning§ 5. A self-fulfilling experiment · p. 14
Under weak assumptions, at least 326 articles out of 590 (55.3%) considered for the experiment were evaluated not by referees but by the consensus groups, i.e., directly by GEV members.
“Under weak assumptions, it is possible to demonstrate that at least 326 articles out of 590 (55,3%) considered for the experiment were evaluated not by referees, but by the consensus groups, that is directly by the members of the GEV.”
Quote in the paperSubstantial claim - Findingown data§ 4. A meta-analysis of the experiment · p. 11
The kappa of the Economic history sub-area lies within the prediction limits for both linear- and VQR-weighted cases.
“Conversely, the kappa of the Economic history sub-area lies within the limits for both the linear- and VQR-weighted cases.”
Quote in the paperSubstantial claim - Findingnone given§ 5. A self-fulfilling experiment · p. 13
A careful scrutiny of ANVUR's Final Report and area reports highlights at least five profiles by which the Area 13 experiment differs from the experiments in other areas.
“A careful scrutiny of ANVUR's Final Report and area reports permit to highlight at least five profiles by which Area 13 experiment is different from the ones performed in the other areas.”
Quote in the paperSubstantial claim - Findingnone given§ 5. A self-fulfilling experiment · p. 14
In Area 13, the protocol was different: when the two referee reports were communicated to the two GEV members in charge, they formed a consensus group which directly decided the final evaluation, considering the referee reports as simple information.
“In Area 13 the protocol was completely different. When the two referee's reports were comunicated to the two GEV members in charge of the considered article, they formed a 'consensus group' which directly decided the final evaluation of the article, by considering the referee's reports as simple information for their final evaluation.”
Quote in the paperSubstantial claim
- Findingown data§ 3. Comparing IR and bibliometrics · p. 8
When VQR-weights are considered, kappas for all areas are in the range 0.15-0.35, except Area 13 with 0.54.
“When VQR-weights are considered, kappas for all the areas are again in the range 0.15-0.35 with the exception of 0.54 for Area 13 (Economics and statistics).”
Quote in the paperSubstantial claim - Findingown data§ 3. Comparing IR and bibliometrics · p. 8
When linear weights are considered, kappa values for all areas are in the range 0.1-0.35, except Area 13 (Economics and statistics) with 0.54.
“When linear weights are considered, kappa values for all the areas are in the range 0.1-0.35, with the only exception of a value of 0.54 for Area 13 (Economics and statistics).”
Quote in the paperSubstantial claim - Findingreasoning§ 5. A self-fulfilling experiment · p. 13
In Area 13, bibliometric classification of an article was very easy because it was based on the journal ranking developed and published by the GEV, which classified journals into the four VQR categories and each article was evaluated according to its journal's classification.
“In Area 13 the bibliometric classification of article was instead very easy because it was based on the journal ranking developed and published by the GEV. This ranking classified journal in the four VQR categories and each article was evaluated according to the classification of the journal in which it had been published.”
Quote in the paperSubstantial claim - Findingown data§ Appendix · p. 18
In the other areas, the percent of articles that received a concordant D score by two referees is 21.1% (705/3441).
“Consider that in the other areas, the percent of articles that received a concordant D score by two referees is 21.1% (705/3441).”
Quote in the paperSubstantial claim - Findingnone given§ 6. Concluding remarks · p. 16
For the remaining Areas 1 to 9, the ANVUR experiment resulted in degrees of agreement between informed peer review and bibliometrics that can be described as unacceptable, or poor to fair.
“it appears that the ANVUR experiment resulted in degrees of agreement between IR and bibliometrics that can be described as unacceptable, or poor to fair.”
Quote in the paperSubstantial claim - Findingreasoning§ Appendix · p. 18
The evaluation of at least 326 articles, that is 55.3% of the total, was decided by Consensus Groups composed by GEV members.
“This means that the evaluation of, at least, 326 articles, that is 55.3% of the total, was decided by Consensus Groups composed by GEV members.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: None of the articles was evaluated at first sight as D (limited) by both GEV members, so that an article was not submitted to the informed peer review process, allowing the 590 articles to be treated as evaluated by two referees.
- The authors state this as an assumption: The Consensus Group formed by the two GEV members has never modified a concordant judgement, i.e. a judgment for which both referees were in agreement.
- Findingown data§ Appendix · p. 18
In Area 13 this percentage is more than doubled: 44.3% (117/264).
“In Area 13 this percentage is more than doubled: 44.3% (117/264).”
Quote in the paperSubstantial claim - Findingown data§ 6. Concluding remarks · p. 16
The only notable exception is Area 13 (economics and statistics) and its sub-areas, where the degree of agreement was in the range 0.29-0.56.
“The only notable exception is represented by Area 13 (economics and statistics) and its sub-areas, where the degree of agreement was in the range 0.29-0.56.”
Quote in the paperSubstantial claim - Findingreasoning§ 6. Concluding remarks · p. 16
As a consequence of some of the Area 13 protocol modifications, it can be estimated that at least 53% of the informed peer review evaluations was not expressed by referees, but directly by the Area 13 panelists.
“As a consequence of some of these modifications, it can be estimated that at least 53% of the IR evaluations was not expressed by referees, but directly by the Area 13 panelists.”
Quote in the paperSubstantial claim- The authors state this as an assumption: None of the articles was evaluated at first sight as D (limited) by both GEV members, so that an article was not submitted to the informed peer review process, allowing the 590 articles to be treated as evaluated by two referees.
- The authors state this as an assumption: The Consensus Group formed by the two GEV members has never modified a concordant judgement, i.e. a judgment for which both referees were in agreement.
- Findingnone given§ 6. Concluding remarks · p. 16
A careful scrutiny of the experiment protocol adopted in Area 13 highlighted that the panel in charge of the evaluation introduced substantial modifications to the protocol adopted in all the other areas involved in the experiment.
“A careful scrutiny of the experiment protocol adopted in Area 13 highlighted that the panel in charge of the evaluation introduced substantial modifications to the protocol adopted in all the other areas involved in the experiment.”
Quote in the paperSubstantial claim - Findingcitation§ Appendix · p. 18
The referees gave concordant judgments for 264 articles.
“We know (ANVUR 2013) that the referees gave concordant judgments for 264 articles.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
- Findingnone given§ 5. A self-fulfilling experiment · p. 15
Economic history is one of the few sub-areas where bibliometric evaluations were lower than those obtained through informed peer review.
“It is in fact one of the few sub-areas where bibliometric evaluations were lower than the ones obtained through IR.”
Quote in the paperNot a substantial claim - Findingown data§ 6. Concluding remarks · p. 16
The statistical meta-analysis performed confirms that results for Area 13 are significantly different from those reached for the other areas.
“The statistical meta-analysis performed, confirms that results for Area 13 are significantly different from those reached for the other areas.”
Quote in the paperSubstantial claim - Findingown data§ 6. Concluding remarks · p. 16
When available guidelines for interpreting kappa values are considered, for nearly all research areas and sub-areas considered, the degree of agreement (range 0.09-0.42) has to be interpreted as unacceptable, poor, or at most fair.
“for nearly all research areas and sub-areas considered , the degree of agreement (always in the range 0.09-0.42) has to be interpreted as 'unacceptable', 'poor' or in a few cases as, at most, 'fair'.”
Quote in the paperSubstantial claim - Findingreasoning§ Appendix · p. 19
At least 64 articles out of the 311 (21%) for which informed peer review and bibliometrics evaluation coincided were directly evaluated by the Consensus Group.
“At least 64 articles out of the 311 (21%) for which IR and bibliometrics evaluation coincided were directly evaluated by the Consensus Group.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Each time peers expressed a concordant evaluation, this coincided with the bibliometric evaluation.
- Premisecitation§ 1. Introduction · p. 2
More than nine thousand journal articles, about 10% of the total, submitted for the assessment were evaluated by ANVUR using both bibliometrics and peer review.
“Indeed, more than nine thousands journal articles (the 10% of total) submitted for the assessment were evaluated by ANVUR by using both bibliometrics and peer review.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
- Interpretationreasoning§ ABSTRACT · p. 1
The adoption of both informed peer review and bibliometrics in the Italian research assessment possibly introduced systematic and unknown biases in its final results.
“the adoption of both methods in the Italian research assessment possibly introduced systematic and unknown biases in its final results.”
Quote in the paperSubstantial claim - Premisecitation§ 2004-2008 · p. 6
Area 13 (Economic Science and Statistics) adopted a different system of bibliometric evaluation, scoring each article according to the ranking of the journal in which it had been published.
“Area 13 (Economic Science and Statistics) adopted a different system of bibliometric evaluation, by scoring each article according to the ranking of the journal in which it had been published.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
- Interpretationown data§ 5. A self-fulfilling experiment · p. 14
The relatively high kappas calculated for Area 13 can be interpreted as indicating fair to good agreement between the evaluation based on the journal ranking developed by the panelists and the informed peer review performed by the panelists on the basis of the referee reports.
“Indeed the relatively high kappas calculated for Area 13 can be interpreted as indicating a 'fair to good agreement' between the evaluation based on the journal ranking developed by the panelists, and the IR performed by the panelists on the basis of the referee reports.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3. Comparing IR and bibliometrics · p. 9
According to existing guidelines, almost all kappa values in Table 2 indicate agreement that can be described as unacceptable, poor, or fair for nearly all Areas and sub-areas.
“According to these guidelines, results of the experiment, that is almost all the values for kappas listed in Table 2, are indicative of agreement that can be described as 'unacceptable', or alternatively as 'poor' or 'fair', for nearly all the Areas and sub-areas.”
Quote in the paperSubstantial claim - Premisecitation§ 2004-2008 · p. 6
In Area 13, journal articles were not subjected to informed peer review.
“In this area, journal articles were not subjected to IR.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
- Interpretationreasoning§ 5. A self-fulfilling experiment · p. 13
In Area 13, a referee requested to evaluate an article published in a journal ranked by the GEV immediately knew the article's bibliometric evaluation by checking the journal ranking.
“Therefore in Area 13 a referee requested to evaluate an article published in a journal ranked by the GEV, immediately knew the article's bibliometric evaluation: he just had to check the journal ranking.”
Quote in the paperSubstantial claim - Interpretationcitation§ 5. A self-fulfilling experiment · p. 13
The random sampling of articles in Area 13 was possibly affected by authors' requests to be evaluated through peer review.
“The random sampling of articles in Area 13 was possibly affected by authors's requests of being evaluated through peer review.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
- Interpretationcitation§ 4. A meta-analysis of the experiment · p. 13
The hypothesis that in economics and statistics there are shared evaluations which do not exist in other disciplinary areas can be rejected, given the worldwide discussion about research evaluation in economics, including the British RAE.
“A first hypothesis is that in economics and statistics there are shared evaluations which do not exist in the other disciplinary areas. This hypothesis can be promptly rejected, given that a harsh discussion about research evaluation in economics have been developed worldwide also in reference to the British RAE (Lee 2007).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lee 2007
Not checked in the free run.
- Interpretationreasoning§ 2004-2008 · p. 6
The adoption of the evaluative mix created many problems when evaluation results were used for comparing institutions and research fields.
“The adoption of this evaluative mix created many problems when evaluation results were used for comparing institutions and research fields.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5. A self-fulfilling experiment · p. 14
The modifications of the protocol specific to Area 13 have possibly introduced a substantial bias toward agreement between bibliometrics and informed peer review.
“These modifications of the protocol, specific of the experiment performed in Area 13, have possibly introduced a substantial bias toward agreement between bibliometrics and IR.”
Quote in the paperSubstantial claim- Rests on an assumption: Area 13 panelists who decided the journal ranking used for bibliometric evaluation were the same panelists who decided the final score of the informed peer review process.
- Premisecitation§ 4. A meta-analysis of the experiment · p. 11
Each (sub-)area can be considered as performing a parallel experiment on different subjects.
“Indeed each (sub-)area can be considered as performing a parallel experiment on different subjects (Sun 2011; Spiegelhalter 2005).”
Quote in the paperSubstantial claim0 of 2 sources checked - Interpretationcitation§ 3. Comparing IR and bibliometrics · p. 8
ANVUR stressed the importance of significance tests for Cohen's kappa by improperly interpreting statistical significance as agreement.
“ANVUR stressed the importance of significance test for Cohen's kappa, by improperly interpreting statistical significance as agreement.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3. Comparing IR and bibliometrics · p. 9
The value 0.61 appears inconsistent when the other kappas calculated for the sub-areas of Area 13, reported in Table 2, are considered.
“The value of 0.61 appears inconsistent when the other kappas calculated for the sub-areas of Area 13, reported in Table 2, are considered.”
Quote in the paperSubstantial claim - Premisecitation§ 3. Comparing IR and bibliometrics · p. 7
The statistical technique adopted by ANVUR was Cohen's kappa.
“The statistical technique adopted by ANVUR was Cohen's kappa (Cohen 1960), the most popular index of interrater agreement for nominal categories (Sheskin 2003).”
Quote in the paperSubstantial claim0 of 2 sources checked - Interpretationown data§ 3. Comparing IR and bibliometrics · p. 9
The only exceptions are Area 13 and four sub-areas (three within Area 13), whose agreement can be described as acceptable, fair to good, or moderate.
“The only exceptions are Area 13 (Economics and statistics), and four sub-areas, three of which are defined inside Area 13, with an agreement that can be described as 'acceptable' or alternatively as 'fair to good' or 'moderate'.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5. A self-fulfilling experiment · p. 15
By considering the five profiles, the experiment performed in Area 13 was radically different from those performed in the other areas.
“By considering these five profiles it appears that the experiment performed in Area 13 was radically different from those performed in the other areas.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3. Comparing IR and bibliometrics · p. 9
The use by ANVUR of expressions such as 'good degree of agreement' and 'more than adequate' appears misleading when existing guidelines in Table 1 are considered.
“The use made by ANVUR of expressions such as 'good degree of agreement' and 'more than adequate' appears to be misleading when existing guidelines reported in Table 1 are considered.”
Quote in the paperSubstantial claim - Interpretationown data§ 4. A meta-analysis of the experiment · p. 13
Area 13 does not conform to the distribution of the other areas.
“As we can easily understand, Area 13 does not conform to the distribution of the other areas.”
Quote in the paperSubstantial claim- Rests on an assumption: Cohen's kappa is asymptotically normal and its sample variance is inversely proportional to the number of assessed papers in an area.
- The authors state this as an assumption: The kappas for the nine areas other than Area 13 are observed.
- Interpretationown data§ 4. A meta-analysis of the experiment · p. 12
The meta-analysis further confirms rejection of the hypothesis that the experiment within Area 13 had the same statistical properties as the experiments in the other 9 areas.
“This further confirms the rejection of the hypothesis that the experiment within Area 13 had the same statistical properties as the experiments in the other 9 areas.”
Quote in the paperSubstantial claim- Rests on an assumption: Cohen's kappa is asymptotically normal and its sample variance is inversely proportional to the number of assessed papers in an area.
- The authors state this as an assumption: The kappas for the nine areas other than Area 13 are observed.
- Interpretationreasoning§ 3. Comparing IR and bibliometrics · p. 8
In the whole sample, just a little less or a little more than one third of evaluations reached through bibliometrics and IR are agreements.
“That is in the whole sample just a little less or a little more than one third of evaluations reached through bibliometrics and IR are agreements.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6. Concluding remarks · p. 16
For these reasons, results reached for Area 13 have to be considered as fatally flawed by virtue of the protocol modifications introduced by the area panel.
“For these reasons, results reached for Area 13 have to be considered as fatally flawed by virtue of the protocol modifications introduced by the area panel.”
Quote in the paperSubstantial claim - Interpretationcitation§ 6. Concluding remarks · p. 16
Ceteris paribus, areas, disciplinary sectors or institutions with a higher percentage of research outputs evaluated through informed peer review are more likely to have obtained lower average scores than those with a lower percentage of informed peer review.
“Ceteris paribus , areas, disciplinary sectors or institution with a higher percentage of research outputs evaluated through IR, are more likely to have obtained lower average scores than areas, disciplinary sectors and institutions with lower percentage of IR.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6. Concluding remarks · p. 16
It is impossible to affirm, in a comparison between two departments, whether a department has a higher score because it produced better research or because it had a lower percentage of research outputs evaluated by informed peer review.
“It is impossible to affirm, for example, in a comparison between two departments, if a department has a higher score because it had produced better research or because it had a lower percentage of research outputs evaluated by IR.”
Quote in the paperSubstantial claim - Premisenone given§ 6. Concluding remarks · p. 16
All research outputs different from articles in indexed journals were evaluated through peer review.
“Indeed, as we have seen, all research outputs different from articles in indexed journals were evaluated through peer review.”
Quote in the paperSubstantial claim - Interpretationcitation§ 6. Concluding remarks · p. 16
VQR final results reached for areas, subareas, disciplinary sectors and institutions depend on the mix of instruments used to evaluate research outputs.
“As a consequence, VQR final results reached for Areas, subareas, disciplinary sectors and institutions depend on the mix of instruments used to evaluate research outputs.”
Quote in the paperSubstantial claim - Study design and conductnone given§ 6. Concluding remarks · p. 16
In Area 13, the synthesis of the two referees' judgments was defined by a Consensus Group composed of at least two panel members, who considered the referees' judgments as mere information for deciding the final score.
“(iv) the synthesis of the two referees's judgments was defined by a Consensus Group composed by (at least) two panel members, who considered the referees judgments as mere information for deciding the final score;”
Quote in the paperSubstantial claim - Interpretationreasoning§ Appendix · p. 18
The number 54 is just an underestimate of the number of articles evaluated as D by the Consensus Group.
“On this basis, we can presumptively affirm that the number 54 is just an underestimate of the number of articles evaluated as D by the Consensus Group.”
Quote in the paperNot a substantial claim - Study design and conductnone given§ 6. Concluding remarks · p. 16
In Area 13, random sampling took into account authors' requests to be evaluated by peer review.
“(i) random sampling took into account authors' requests to be evaluated by peer review;”
Quote in the paperSubstantial claim - Interpretationown data§ 6. Concluding remarks · p. 17
The conclusion reached by ANVUR, according to which the more than adequate agreement between bibliometric and informed peer review fully justifies the use of both techniques of assessment, must be reversed.
“The conclusion reached by ANVUR, according to which the 'more than adequate' agreement between bibliometric and informed peer review 'fully justifies' the use of both techniques of assessment, must be reversed.”
Quote in the paperSubstantial claim - Premisecitation§ 6. Concluding remarks · p. 16
As documented by ANVUR, articles evaluated through informed peer review had lower evaluation than articles evaluated through bibliometrics.
“As documented by ANVUR, articles evaluated through IR had lower evaluation than articles evaluated through bibliometrics (Cicero et al. 2013; ANVUR 2013).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Cicero et al. 2013
Not checked in the free run.
- cites ANVUR 2013
Not checked in the free run.
- Interpretationown data§ 6. Concluding remarks · p. 17
The Italian research assessment exercise and the experiment conducted by ANVUR seem to support that informed peer review and bibliometric analysis do not produce similar results.
“Indeed, the Italian research assessment exercise and the experiment conducted by the ANVUR seem to support that informed peer review and bibliometric analysis do not produce similar results.”
Quote in the paperSubstantial claim - Interpretationcitation§ 6. Concluding remarks · p. 17
The two techniques cannot be considered as substitute.
“A fortiori the two techniques cannot be considered as 'substitute', as sustained by (Bertocchi et al. 2013e, 2013a, 2013b, 2013c, 2013d).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Bertocchi et al. 2013e
Not checked in the free run.
- cites 2013a
Not checked in the free run.
- cites 2013b
Not checked in the free run.
- Study design and conductnone given§ 6. Concluding remarks · p. 16
In Area 13, the panel members forming the Consensus Groups knew that their final judgment would be used for the experiment.
“(v) the panel members forming the Consensus Groups knew that their final judgment would be used for the experiment.”
Quote in the paperSubstantial claim - Premisecitation§ 6. Concluding remarks · p. 16
Negative correlations between overall results and the percentage of research outputs evaluated through informed peer review were documented for two areas.
“Negative correlations between overall results and the percentage of research outputs evaluated through the informed peer review were documented for two areas (Baccini 2014a; De Nicolao 2014).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Baccini 2014a
Not checked in the free run.
- cites De Nicolao 2014
Not checked in the free run.
- Interpretationown data§ Appendix · p. 18
It is impossible to establish, by using publicly available data, how much this result is due to a higher level of agreement between the two referees or to an initial agreement of the two GEV members, both evaluating an article as D.
“It is impossible to establish, by using publicly available data, how much this result is due to a higher level of agreement between the two referees or to an initial agreement of the two GEV members, both evaluating an article as D.”
Quote in the paperSubstantial claim
- May combine two claims. 32
- The quote only partly states this claim. 1
- Classified as probably not a substantial claim. 30
May combine two claims.
Findingown data§ ABSTRACT · p. 1The only notable exception to the poor agreement was a moderate agreement for economics and statistics (Area 13) and its sub-fields, confirmed by a statistical meta-analysis.
“The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields.”
- Also stated in: § 1. Introduction · p. 2 · plain
“Section 4 contains a statistical metaanalysis of the experiment, by showing that only in a research field (Economics and statistics) there was evidence of a good agreement between results reached through peer review and bibliometric indicators.”
Quote in the paperSubstantial claim- Also stated in: § 1. Introduction · p. 2 · plain
May combine two claims. The quote only partly states this claim.
Findingown data§ 4. A meta-analysis of the experiment · p. 11For both linear- and VQR-weighted kappa, the point for Area 13 lies outside the 95% funnel, so H0 is rejected at 5% significance.
“For both the linear- and VQR-weighted kappa, it is found that the point ( m10 , ) lies outside the funnel, meaning that H0 is rejected with significance level 5% , see top panels of Fig. 1 for the funnel plots and Table 2 for the p -values.”
Quote in the paperSubstantial claim- Rests on an assumption: Cohen's kappa is asymptotically normal and its sample variance is inversely proportional to the number of assessed papers in an area.
- The authors state this as an assumption: The kappas for the nine areas other than Area 13 are observed.
May combine two claims.
Findingreasoning§ 5. A self-fulfilling experiment · p. 14GEV members directly evaluated at least 54.3% of articles classified A, 58% of B, 83.7% of C, and 31.6% of D.
“In particular it is possible to estimate that GEV members evaluated directly at least 54.3% of articles classified as A (excellent); 58% of articles classified as B (good); 83.7% of the articles classified as C (acceptable), and 31.6% of the articles classified as D(limited).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 3. Comparing IR and bibliometrics · p. 8Kappas calculated for the whole sample of articles are 0.32 with linear weights and 0.38 with VQR-weights.
“Kappas calculated for the whole sample of articles are respectively 0.32 with linear weights and 0.38 with VQR-weights.”
Quote in the paperSubstantial claimMay combine two claims.
Findingcitation§ 5. A self-fulfilling experiment · p. 13In Area 13, referees could be aware of their engagement in the experiment, while in all other areas referees did not know they were engaged in the experiment.
“In Area 13, referees were in a position to be aware of their engagement in the experiment, while in all the other areas referees ignored to be engaged in the experiment.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4. A meta-analysis of the experiment · p. 11For both linear- and VQR-weighted kappa, 3 out of 4 sub-area kappas of Area 13 lie outside the 95% funnel, so H0 is rejected at 5% in these three cases.
“For both the linear- and VQR-weighted kappa, 3 out of 4 sub-area kappas of Area 13 lie outside the funnel, meaning that in these three cases H0 is rejected with significance level 5%, see bottom panels of Figure 2 and also Table 3 for the p -values.”
Quote in the paperSubstantial claim- Rests on an assumption: Cohen's kappa is asymptotically normal and its sample variance is inversely proportional to the number of assessed papers in an area.
- The authors state this as an assumption: The kappas for the nine areas other than Area 13 are observed.
May combine two claims.
Findingown data§ 3. Comparing IR and bibliometrics · p. 8With sub-areas and VQR-weights, the minimum kappa is for electronic engineering (0.09) and the maximum for economics (0.56); again only four sub-areas have kappa values greater than 0.40.
“If sub-areas and VQR-weights are considered, the minimum kappa is calculated for electronic engineering (0.09) and the maximum one for economics (0.56); again only four sub-areas have kappa values greater than 0.40.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone given§ 3. Comparing IR and bibliometrics · p. 7Many articles were dropped by the experiment because the bibliometric evaluation resulted in an inconclusive informed peer review value; this induced a distortion in the sample that was not considered in the analysis by ANVUR.
“It is worthwhile to note that many articles were dropped by the experiment because the bibliometric evaluation resulted in an inconclusive IR value. This induced a distortion in the sample that was not considered in the analysis by ANVUR.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 3. Comparing IR and bibliometrics · p. 8With sub-areas and linear weights, the minimum kappa is for nuclear physics (0.095) and the maximum for economics (0.56); only four sub-areas out of 43 have kappa values greater than 0.40.
“When sub-areas and linear weights are considered the minimum kappa is calculated for the nuclear physics (0.095) and the maximum one for economics (0.56); only four sub-areas out of 43 have kappa values greater than 0.40.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Findingnone given§ Appendix · p. 18In Area 13, 590 journal articles were selected for the experiment.
“In Area 13, 590 journal articles were selected for the experiment.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ Appendix · p. 19For articles evaluated as B, peers were in agreement for 73 articles; but bibliometric and peer review coincided for just 56 articles.
“Indeed, for articles evaluated as B, peers were in agreement for 73 articles; but bibliometric and peer review coincided for just 56 articles.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ Appendix · p. 1854.3% of articles classified as A through informed peer review were evaluated directly by the GEV Consensus Groups; this percentage increases to 58% for articles classified as B; at least 83.7% of articles classified as C were decided by Consensus Groups; and the percentage decreases to 31.6% for products evaluated as D.
“We can see that 54.3% of articles classified as A through IR were evaluated directly by the GEV Consensus Groups; this percentage increases to 58% for articles classified as B; and at least 83.7% of articles classified as C were decided by Consensus Groups. The percentage decreases to 31.6% for products evaluated as D.”
Quote in the paperSubstantial claim- The authors state this as an assumption: None of the articles was evaluated at first sight as D (limited) by both GEV members, so that an article was not submitted to the informed peer review process, allowing the 590 articles to be treated as evaluated by two referees.
- The authors state this as an assumption: The Consensus Group formed by the two GEV members has never modified a concordant judgement, i.e. a judgment for which both referees were in agreement.
May combine two claims.
Interpretationreasoning§ ABSTRACT · p. 1The experiment protocol adopted in Area 13 was substantially modified with respect to all other research fields, to the point that results for economics and statistics have to be considered as fatally flawed.
“We show that the experiment protocol adopted in Area 13 was substantially modified with respect to all the other research fields, to the point that results for economics and statistics have to be considered as fatally flawed.”
- Also stated in: § 1. Introduction · p. 2 · plain
“Section 5 shows that the good agreement in this research field can be explained by an experimental protocol which differs substantially from the one adopted in all the other research areas.”
Quote in the paperSubstantial claim- Also stated in: § 1. Introduction · p. 2 · plain
Classified as probably not a substantial claim.
Premisecitation§ ABSTRACT · p. 1During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review and by bibliometrics.
“During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
May combine two claims.
Interpretationreasoning§ ABSTRACT · p. 1The evidence of a poor agreement supports the conclusion that informed peer review and bibliometrics do not produce similar results.
“The evidence of a poor agreement supports the conclusion that IR and bibliometrics do not produce similar results”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premiseown data§ ABSTRACT · p. 1A sample of articles was evaluated by both informed peer review and bibliometrics, and agreement was analyzed by weighted Cohen's kappas.
“A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ ABSTRACT · p. 1The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of informed peer review and bibliometrics within the same research assessment exercise.
“The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of IR and bibliometrics within the same research assessment exercise.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Premisecitation§ 2. The Italian research assessment exercise · p. 4Each article was classified by considering two bibliometric indicators: the total number of citations received in the time window starting from its publication and 31th December 2011, and the impact factor of the journal in which it was published.
“Each article was classified by considering two bibliometric indicators: (i) the total number of citations received in the time window starting from its publication and 31th December 2011, and (ii) the impact factor of the journal in which it was published (ad-hoc bibliometric indicators were also devised by some of the GEVs).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Premisecitation§ 3. Comparing IR and bibliometrics · p. 8Related papers described the same data as indicating significant concordance, remarkable agreement, similar evaluations, or fair to good agreement.
“Results of the experiment are also presented and commented as 'giving evidence of a significant degree of concordance among peer review and bibliometric evaluations' (Ancaiani et al. 2015).”
- Also stated in: § 3. Comparing IR and bibliometrics · p. 8 · boosted
“This phrase is slightly reworded in the working paper versions: 'there is remarkable agreement between bibliometric and peer review evaluation' (Bertocchi et al. 2013d, 2013e, 2013a, 2013b, 2013c).”
- Also stated in: § 3. Comparing IR and bibliometrics · p. 8 · plain
“And the same data are also interpreted in a policy oriented paper as revealing that 'informed peer review and bibliometric analysis produce similar evaluations' (Bertocchi et al. 2014).”
Quote in the paperSubstantial claim0 of 3 sources checked- Also stated in: § 3. Comparing IR and bibliometrics · p. 8 · boosted
Classified as probably not a substantial claim.
Premisecitation§ 2. The Italian research assessment exercise · p. 4In the bibliometric areas (Areas 1 to 9), a mix of peer review and bibliometrics was adopted.
“In the bibliometric areas (1 to 9), a mix of peer review and bibliometrics was adopted.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Classified as probably not a substantial claim.
Premisecitation§ 2004-2008 · p. 5When impact factor and citations gave divergent information, the article was submitted to informed peer review.
“When impact factor and citations gave divergent information -high citations and low impact factor or viceversa -the corresponding entry in the classification matrix was denoted by 'IR' indicating that the article had to be submitted to a process called 'informed peer review' (IR).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Classified as probably not a substantial claim.
Premisenone given§ 5. A self-fulfilling experiment · p. 14In the generic description of the experiment, the scores separately assigned by the two referees were summarized according to an algorithm, in many cases a simple average of the two scores.
“As we have seen, the generic description of the experiment indicates that the scores separately assigned by the two referees were summarized according to an algorithm, in many cases a simple average of the two scores.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 2004-2008 · p. 6The first major problem was that the joint distribution of article citations and journal impact factors varies among research fields; as a consequence the VQR distribution rule was not respected for all fields, and in some fields to be classified as excellent or good was simpler than in other fields.
“The first major problem was originated in bibliometric areas: the joint distribution of article citations and journal impact factors varies among research fields. As a consequence the VQR distribution rule was not respected for all fields, and because of this in some fields to be classified as excellent or good was simpler that in other fields.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3. Comparing IR and bibliometrics · p. 9For Area 13 and VQR-weights, a value of 0.6104 is reported in the Appendix B of the ANVUR Final Report and in Cicero et al. 2013 and Ancaiani et al. 2015.
“For Area 13 and VQR-weights, a value of 0.6104 is reported in the Appendix B of the ANVUR Final Report and also in (Cicero et al. 2013) and (Ancaiani et al. 2015).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites ANVUR 2013
Not checked in the free run.
- cites Cicero et al. 2013
Not checked in the free run.
- cites Ancaiani et al. 2015 source
Not checked in the free run.
May combine two claims.
Study design and conductnone given§ 1. Introduction · p. 2Raw data were not made available by ANVUR, so it was impossible for the authors to control in detail the experiment and reproduce the published results.
“Since raw data were not made available by ANVUR, 1 it was impossible for us to control in details the experiment and to reproduce the results published and by ANVUR and by its collaborators in different outlets.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premisecitation§ 3. Comparing IR and bibliometrics · p. 8ANVUR's Final Report and Area Reports described the observed agreement as good or more than adequate.
“ANVUR's Final Report described generally these data as indicating 'a good degree of agreement for the whole sample and for each GEV' [italics added]”
- Also stated in: § 3. Comparing IR and bibliometrics · p. 8 · plain
“in the conclusion of the report, results are summarized by writing that there is 'a more than adequate [in Italian 'più che adeguata'] agreement between evaluations done by adopting the peer review method and the bibliometric one' (Cicero et al. 2013; ANVUR 2013).”
- Also stated in: § 3. Comparing IR and bibliometrics · p. 8 · plain
“This phrase is repeated verbatim in the conclusions of all the Area Reports , irrespective of the range of values commented.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Cicero et al. 2013
Not checked in the free run.
- cites ANVUR 2013
Not checked in the free run.
- Also stated in: § 3. Comparing IR and bibliometrics · p. 8 · plain
May combine two claims.
Interpretationreasoning§ 2004-2008 · p. 6There was lack of comparability not only between Areas but also between research fields inside the same research Area.
“Although an explicit warning was not issued at the time of publication of results, there was lack of comparability not only between Areas but also between research fields inside the same research Area.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premisecitation§ 3. Comparing IR and bibliometrics · p. 9The value 0.54 used in this paper is drawn directly from the Area 13 report and reproduced in Bertocchi et al. 2015.
“The value of 0.54 that we used in this paper is drawn directly from the Area 13 report and reproduced also in (Bertocchi et al. 2015).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites ANVUR 2013
Not checked in the free run.
- cites Bertocchi et al. 2015 source
Not checked in the free run.
May combine two claims.
Interpretationreasoning§ 5. A self-fulfilling experiment · p. 13In areas 1-9, bibliometric classification of articles was not easily available because it required complete knowledge of the suitable matrix; a referee could foresee the bibliometric classification only with difficulty and high margin of error.
“As we have seen, in the areas 1-9 bibliometric classification of articles was not easily available because it requested the complete knowledge of the suitable matrix; a referee could have foreseen the bibliometric classification of an article only with difficulty and with a quite high margin of error.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Interpretationreasoning§ 5. A self-fulfilling experiment · p. 15The higher level of agreement between bibliometrics and informed peer review in Area 13 may be the result of the modifications introduced by the Area 13 panel to the experiment protocol adopted in the other areas.
“And that the higher livel of agreement between bibliometrics and IR in Area 13 may be the result of the modifications introduced by the Area 13 panel to the experiment protocol adopted in the other Areas.”
Quote in the paperSubstantial claim- Rests on an assumption: Area 13 panelists who decided the journal ranking used for bibliometric evaluation were the same panelists who decided the final score of the informed peer review process.
May combine two claims.
Interpretationreasoning§ 5. A self-fulfilling experiment · p. 13In Area 13, referees knew both that they were participating in the experiment and the bibliometric evaluation with which their judgement was to be compared; these two pieces of information were not available to referees in other areas.
“So in Area 13, not only referees knew that they were partecipating to the experiment, but they also knew the bibliometric evaluation with which their judgement was to be compared. These two pieces of information were not available to the referees ot the other areas.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 3. Comparing IR and bibliometrics · p. 9In the conclusion of the Area 9 report, the phrase about more than adequate agreement is followed by a contradictory statement that the degree of concordance is moderate in nearly all sub-areas while rather high for informatic engineering.
“In the conclusion of the Area 9 report, that phrase is followed by the contradictory statement that: 'The degree of concordance between peer evaluations and bibliometric evaluations is moderate (in Italian: 'moderato') in near all the sub-areas, while it results rather high (in Italian: 'piuttosto elevato') for informatic engineering' (ANVUR 2013).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Classified as probably not a substantial claim.
Premisecitation§ 3. Comparing IR and bibliometrics · p. 7In informed peer review, two referees chosen by two different members of the GEV were asked to evaluate according to their subjective perception of the quality of the product in reference to the world distribution of the research products of the scientific field.
“In this case two referees, chosen by two different members of the GEV, were asked to evaluate according 'to their subjective perception of the quality of the product in reference to the world distribution of the research products of the scientific field' (ANVUR 2013).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3. Comparing IR and bibliometrics · p. 8Different guidelines for interpreting kappa values are available in the literature, as summarized in Table 1.
“Different guidelines are available (Fleiss et al. 2003; Landis and Koch 1977; Altman 1991; George and Mallery 2003; Stemler and Tsai 2008) as summarized in Table 1.”
Quote in the paperNot a substantial claim0 of 3 sources checkedProbably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 2004-2008 · p. 6This experiment is crucial for the overall soundness of the results of the national assessment.
“This experiment is crucial for the overall soundness of the results of the national assessment.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 2. The Italian research assessment exercise · p. 4Four by four matrices with the 16 possible combinations of classes of citations and journal impact factors were defined; each GEV defined its own algorithms for translating these combinations into final scores; at least 48 different matrices were used for article classification.
“Four by four matrices with the 16 possible combinations of classes of citations and journal impact factors were then defined. Each GEV defined its own algorithms for translating these possible combinations in the final score of the articles. At least 48 different matrices were used for article classification (Baccini 2016).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Baccini 2016
Not checked in the free run.
Classified as probably not a substantial claim.
Premisecitation§ 3. Comparing IR and bibliometrics · p. 9For the whole sample and VQR-weights, a value of 0.3441 is reported in Ancaiani et al. 2015.
“For the whole sample and VQR-weights, a value of 0.3441 is reported in (Ancaiani et al. 2015).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ancaiani et al. 2015 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 3. Comparing IR and bibliometrics · p. 9Bertocchi et al. 2015 introduced references to the relevant literature which were not presented in the working paper versions.
“(Bertocchi et al. 2015) introduced references to the relevant literature which were not presented in the working paper versions of the paper.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Bertocchi et al. 2015 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone given§ 3. Comparing IR and bibliometrics · p. 9The term 'adequate' is not used in the relevant literature but is adopted by ANVUR only in its reports.
“Really, the term 'adequate' is not used in the relevant literature, but it is adopted by the ANVUR only in its reports.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 2. The Italian research assessment exercise · p. 4Normalizations were not introduced for citation windows or for research fields.
“Normalizations were not introduced neither for citation windows nor for research fields.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Premisecitation§ 2004-2008 · p. 6To address this problem, ANVUR carried out an experiment for assessing the agreement of the two evaluation methods: for Areas 1-9 and 13, a random sample of articles was submitted to both bibliometric and informed peer review evaluation.
“To address this problem, ANVUR carried out an experiment for assessing the agreement of the two evaluation methods: for Areas 1-9 and 13, a random sample of articles was submitted to both bibliometric and IR evaluation.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ 6. Concluding remarks · p. 16The negative correlation may originate from the mix of instruments used, but also from the intrinsic lower quality of research outputs which were evaluated by informed peer review.
“It is worthwhile to note that this negative correlation may originate by the mix of instruments used, but also by the intrinsic 'lower quality' of research outputs which were evaluated by IR.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 6. Concluding remarks · p. 17In the overall sample there is an unacceptable, poor, or at most fair agreement, and this result does not justify the joint use of both techniques.
“Indeed in the overall sample there is an unacceptable, a poor or at most, and in a few cases, a fair agreement, and this result all but justifies at all the joint use of both techniques.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premisecitation§ 5. A self-fulfilling experiment · p. 15In the economic history sub-area, there was higher disagreement between the two external referees compared with other areas, and there were particularly strict or generous judgments.
“there was a higher disagreement between the two external referees, compared with other areas, and there were particularly strict or generous judgments.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites ANVUR 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationreasoning§ 5. A self-fulfilling experiment · p. 15The economic history sub-area might have deserved a richer scrutiny, impossible with data publicly released by ANVUR.
“The second is about the idiosyncratic character of the economic history sub-area, which might have deserved a richer scrutiny, impossible with data publicly released by ANVUR.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 6. Concluding remarks · p. 16It is impossible to disentangle the two phenomena, and as a consequence it is impossible to give a clear interpretation of the results of the Italian research assessment exercise.
“The point to be underlined here is that it is impossible to disentangle these two phenomena, and as a consequence it is impossible to give a clear interpretation of the results of the Italian research assessment exercise.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationreasoning§ 5. A self-fulfilling experiment · p. 15It may be conjectured that the Area panel might have encountered difficulties in enacting the appropriate choice of referees because there were not enough panelists expert in economic history and the history of economic thought.
“It may be conjectured that the Area panel might have encountered difficulties in enacting the appropriate choice of referees because there was not enough panelists who were expert in the economic history and history of economic thought field.”
Quote in the paperNot a substantial claim- Rests on an assumption: Economic history is the only sub-area of the experiment for which a dedicated sub-area panel was not defined.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ Appendix · p. 19This value is strongly underestimated.
“This value is strongly underestimated.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Study design and conductnone given§ 6. Concluding remarks · p. 16In Area 13, the referees might have known the precise merit class in which each article was classified by using bibliometrics.
“(iii) the referees might have known the precise merit class in which each article was classified by using bibliometrics;”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Study design and conductnone given§ 6. Concluding remarks · p. 16In Area 13, the referees might have known that they were part of the experiment.
“(ii) the referees might have known that they were part of the experiment;”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown data§ 6. Concluding remarks · p. 17Agencies that run research assessments should feel confident that results reached by using one technique will differ from those reached using the other.
“This suggest that the agencies that run research assessments should feel confident that results that will be reached by using a technique will differ from those reached using the other.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationreasoning§ 5. A self-fulfilling experiment · p. 15The results of the experiment for the economic history sub-area have to be considered with particular caution.
“These conjectures seem to suggest that the results of the experiment for economic history sub-area have to be considered with particular caution.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Cohen's kappa is asymptotically normal and its sample variance is inversely proportional to the number of assessed papers in an area.
“Recalling that Cohen's kappa is asymptotically normal and its sample variance is inversely proportional to the number mj of the assessed papers in the j -th area, under H0 we have”
The authors state this as an assumption.
The kappas for the nine areas other than Area 13 are observed.
“We assume that 𝑘𝑗 for 𝑗 = 1,2, … ,9 are observed.”
The authors state this as an assumption.
None of the articles was evaluated at first sight as D (limited) by both GEV members, so that an article was not submitted to the informed peer review process, allowing the 590 articles to be treated as evaluated by two referees.
“If we suppose (Hypothesis 1) that none of the articles was evaluated at a first sight as D (limited) by both GEV members -in this case an article was not submitted to the IR process-, then the 590 articles products can be treated as they were evaluated by two referees.”
The authors state this as an assumption.
The Consensus Group formed by the two GEV members has never modified a concordant judgement, i.e. a judgment for which both referees were in agreement.
“Let's suppose now (Hypothesis 2) that the Consensus Group formed by the two GEV members has never modified a concordant judgement, i.e. a judgment for which both referees were in agreement.”
The authors state this as an assumption.
Each time peers expressed a concordant evaluation, this coincided with the bibliometric evaluation.
“A third hypothesis is also formulated: each time peers expressed a concordant evaluation, this coincided with the bibliometric evaluation.”
The paper relies on this without support.
Area 13 panelists who decided the journal ranking used for bibliometric evaluation were the same panelists who decided the final score of the informed peer review process.
“It is worthwhile to recall that Area 13 panelists had decided the journal ranking used for bibliometric evaluation of articles submitted to the research assessment; they were the same panelists that decided the final score of the IR process.”
The paper relies on this without support.
GEV13 members knew that all journal articles which had to be submitted to peer review were used in the context of the experiment, and they knew with certainty for each article the merit class in which it was assigned by using bibliometric evaluation.
“GEV13 members knew that all journal articles which had to be submitted to peer review were used in the context of the experiment. They also knew with certainty for each article the merit class in which it was assigned by using bibliometric evaluation.”
The paper relies on this without support.
In the other areas, the two pieces of information about articles submitted to peer review and their bibliometric merit class were probably not so easily available to GEV members.
“In the other areas these two pieces of information were probably not so easily available to GEV members, because thay had many journal articles that had been submitted to peer review, possibly without a univocal bibliometric evaluation.”
The paper relies on this without support.
Economic history is the only sub-area of the experiment for which a dedicated sub-area panel was not defined.
“Economic history is the only sub-area of the experiment for which a dedicated sub-area panel was not defined.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Conclusions stating poor agreement between metric and peer review were also obtained.
“conclusions stating poor agreement between metric and peer-review were also obtained [283].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [283]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table A1 row 7 total does not sum to 64
MajorThe four entries in row 7 sum to 47, not the reported total of 64. The paper uses 64 in its subsequent claim about articles evaluated by Consensus Groups.
Reported 64 · recomputed 47
“| (7=4-3) # papers evaluated by GEV concordant with biblio | 45 | -17 | 18 | 1 | 64 |”
Economic history does not show moderate agreement
MinorThe abstract describes Area 13 and its sub-fields as showing moderate agreement, but its economic-history sub-field has kappas of 0.32 and 0.29. Both fall below the moderate ranges in the paper's interpretation table.
“The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields.”
“| Economic history | 37 | 0,32 | 0,29 |”
Concordant D-review percentage is miscalculated
MinorThe stated fraction 705/3441 is 20.5%, not 21.1%.
Reported 21.1 · recomputed 20.48823
“21.1% (705/3441)”
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Practical Statistics for Medical Research.
Walter, S., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,665 · see citing works
Practical Statistics for Medical Research
Altman, D. G. (1990) · doi:10.1201/9780429258589 · Cited by 10,668 · see citing works
SPSS for Windows Step by Step: A Simple Guide and Reference
Mallery, P., & George, D. (1998) · Cited by 10,312 · see citing works
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · see citing works
Practical Statistics for Medical Research.
Bartko, J. J., & Altman, D. G. (1992) · Journal of the American Statistical Association · doi:10.2307/2290246 · Cited by 4,834 · see citing works
Handbook of Parametric and Nonparametric Statistical Procedures
(2004) · Technometrics · doi:10.1198/tech.2004.s209 · Cited by 4,664 · see citing works
Handbook of Parametric and Nonparametric Statistical Procedures: Third Edition
Sheskin, D. J. (2003) · doi:10.1201/9781420036268 · Cited by 1,551 · see citing works
Funnel plots for comparing institutional performance
Spiegelhalter, D. J. (2005) · Statistics in Medicine · doi:10.1002/sim.1970 · Cited by 832 · see citing works
van Raan, A. F. J. (2006) · Scientometrics · doi:10.1556/scient.67.2006.3.10 · Cited by 627 · see citing works
Martin, B. R., & Irvine, J. M. (1983) · Research Policy · doi:10.1016/0048-7333(83)90005-7 · Cited by 529 · see citing works
van Raan, A. F. J. (1996) · Scientometrics · doi:10.1007/bf02129602 · Cited by 390 · see citing works
Meta-analysis of Cohen’s kappa
Sun, S. (2011) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-011-0077-3 · Cited by 282 · see citing works
Aluja, T. B. (2006) · Rovira & Virgili repository · Cited by 197 · see citing works
Comparative analysis of a set of bibliometric indicators and central peer review criteria
Rinia, E. J., van Leeuwen, T. N., van Vuren, H. G., & van Raan, A. F. J. (1998) · Research Policy · doi:10.1016/s0048-7333(98)00026-2 · Cited by 193 · see citing works
Peer reviews and bibliometric indicators: a comparative study at a Norwegian university
Aksnes, D. W., & Taxt, R. E. (2004) · Research Evaluation · doi:10.3152/147154404781776563 · Cited by 179 · see citing works
Citation indexes in sociological and historical research
Garfield, E. (1963) · American Documentation · doi:10.1002/asi.5090140405 · Cited by 178 · see citing works
Allocative efficiency in public research funding: Can bibliometrics help?
Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2008) · Research Policy · doi:10.1016/j.respol.2008.11.001 · Cited by 171 · see citing works
Allen, L., Jones, C., Dolby, K., LYNN, D. W., & Walport, M. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005910 · Cited by 169 · see citing works
Lee, F. S. (2006) · Cambridge Journal of Economics · doi:10.1093/cje/bel021 · Cited by 142 · see citing works
Best Practices in Interrater Reliability Three Common Approaches
Stemler, S. E., & Tsai, J. (2008) · Sage eBooks · doi:10.4135/9781412995627.d5 · Cited by 131 · see citing works
Bibliometric evaluation vs. informed peer review: Evidence from Italy
Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2015) · Research Policy · doi:10.1016/j.respol.2014.08.004 · Cited by 86 · see citing works
Evaluating scientific research in Italy: The 2004–10 research evaluation exercise
Ancaiani, A., Anfossi, A. F., Barbara, A., Benedetto, S., Blasi, B., Carletti, V., Cicero, T., Ciolfi, A., Costa, F., Colizza, G., et al. (2015) · Research Evaluation · doi:10.1093/reseval/rvv008 · Cited by 75 · see citing works
Bibliometric indicators versus expert opinion in assessing research performance
Koenig, M. E. D. (1983) · Journal of the American Society for Information Science · doi:10.1002/asi.4630340207 · Cited by 64 · see citing works
Abramo, G., & D’Angelo, C. A. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23323 · Cited by 57 · see citing works
Reconceptualising evaluation in higher education : the practice turn
Saunders, M., Trowler, P., & Bamber, V. (2011) · Cited by 51 · see citing works
Peer review and the h-index: Two studies
Norris, M. W., & Oppenheim, C. (2009) · Journal of Informetrics · doi:10.1016/j.joi.2009.11.001 · Cited by 40 · see citing works
Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001Preprint
Mahdi, S., D’Este, P., & Neely, A. D. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1154053 · Cited by 39 · see citing works
Assessment of Research Performance in Biology: How Well Do Peer Review and Bibliometry Correlate?
Lovegrove, B. G., & Johnson, S. D. (2008) · BioScience · doi:10.1641/b580210 · Cited by 39 · see citing works
Citation indexes do not reflect methodological quality in lung cancer randomised trials.
Berghmans, T., Meert, A. P., Mascaux, C., Paesmans, M., Lafitte, J. J., & Sculier, J. P. (2003) · Annals of Oncology · doi:10.1093/annonc/mdg203 · Cited by 38 · cited 2× · see citing works
“…of expert peer-reviewers commenting articles at the time of their publication and some form of article-level citation metrics (Allen et al. 2009; Berghmans et al. 2003); or the analysis of the correlation between peer ratings assigned to individual researchers or research groups, and…”
BackgroundPredicting results of the Research Excellence Framework using departmental h-index
Mryglod, O., Ralph Kenna, B., Holovatch, Y. V., & Berche, B. (2014) · Scientometrics · doi:10.1007/s11192-014-1512-3 · Cited by 38 · see citing works
Wouters, P. F., Thelwall, M. A., Kousha, K., Waltman, L., de Rijcke, S., & Franssen, T. (2015) · Cited by 16 · see citing works
Bibliometric Evaluation vs. Informed Peer Review: Evidence from ItalyPreprint
Bertocchi, G., Gambardella, A., Jappelli, T., et al. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2363226 · Cited by 13 · see citing works
Baccini, A. (2016) · Project Muse (Johns Hopkins University) · Cited by 13 · see citing works
Self-citationEssentials of Mathematical Statistics
Albright, B. J. (2013) · CERN Document Server (European Organization for Nuclear Research) · Cited by 5 · see citing works
Baccini, A. (2016) · Use Siena air (University of Siena) · Cited by 5 · see citing works
Self-citationLa VQR di Area 13: una riflessione di sintesi
Baccini, A. (2014) · Use Siena air (University of Siena) · Cited by 3 · see citing works
Self-citationResearch assessment: Work in progress, or 'la lutta continua'
McNay, I. (2011) · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 43 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2013b). Bibliometric evaluation vs. informed peer review: Evidence from Italy. ReCent WP. Center for Economic Research, University of Modena and Reggio Emilia, Dept. of Economics Marco Biagi. · DOI
- Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2013c). Bibliometric evaluation vs. informed peer review: Evidence from Italy. IZA Discussion paper. Institute for the Study of Labour (IZA), Bonn. · DOI
- Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2013d). Bibliometric evaluation vs. informed peer review: Evidence from Italy. CEPR Discussion papers. · DOI
- Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2013e). Bibliometric evaluation vs. informed peer review: Evidence from Italy. CSEF working papers. Naples: Centre for Studies in Economics and Finance (CSEF). · DOI
- Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2014). Assessing Italian research quality: A comparison between bibliometric evaluation and informed peer review. In V. C. s. P. Portal (Ed.). www.voxeu.org . http://www.voxeu.org/article/research-quality-assessment-tools-lessons-italy . CEPR (Centre for Economic Policy Research).
- Cicero, T., Malgarini, M., Nappi, C. A., & Peracchi, F. (2013). Bibliometric and peer review methods for research evaluation: a methodological appraisement (in Italian). MPRA (Munich Personal REPEc Archive). Munich.
- HEFCE. (2015). The metric tide: Correlation analysis of REF2014 scores and metrics (Supplementary Report II to the Independent Review of the Role of Metrics in Research Assessment and Management).
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · see citing works
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019) · Sage Open · doi:10.1177/2158244019829575 · Cited by 1,257 · see citing works
Citation gaming induced by bibliometric evaluation: A country-level comparative analysis
Baccini, A., De Nicolao, G., & Petrovich, E. (2019) · PLOS ONE · doi:10.1371/journal.pone.0221212 · Cited by 140 · see citing works
Self-citationIn which fields are citations indicators of research quality?
Thelwall, M. A., Kousha, K., Stuart, E., Makita, M., Abdoli, M., Wilson, P., & Levitt, J. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24767 · Cited by 68 · see citing works
Critical remarks on the Italian research assessment exercise VQR 2011–2014
Franceschini, F., & Maisano, D. A. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.02.005 · Cited by 53 · see citing works
The conundrum of research productivity: a study on sociologists in Italy
Akbaritabar, A., Casnici, N., & Squazzoni, F. (2017) · Scientometrics · doi:10.1007/s11192-017-2606-5 · Cited by 51 · see citing works
Peer Review versus Bibliometrics: Which Method Better Predicts the Scholarly Impact of Publications?
Abramo, G., D’Angelo, C. A., & Reale, E. (2019) · Scientometrics · doi:10.1007/s11192-019-03184-y · Cited by 50 · see citing works
Bornmann, L., & Tekles, A. (2021) · Journal of Informetrics · doi:10.1016/j.joi.2021.101159 · Cited by 45 · see citing works
A global exploratory comparison of country self-citations 1996-2019
Baccini, A., & Petrovich, E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294669 · Cited by 42 · see citing works
Self-citationAbramo, G., & D’Angelo, C. A. (2016) · Scientometrics · doi:10.1007/s11192-016-2153-5 · Cited by 37 · see citing works
Thelwall, M. A., Kousha, K., Stuart, E., Makita, M., Abdoli, M., Wilson, P., & Levitt, J. M. (2023) · Research Policy · doi:10.1016/j.respol.2023.104829 · Cited by 35 · see citing works
The north–south divide in the Italian higher education system
Abramo, G., D’Angelo, C. A., & Rosati, F. (2016) · Scientometrics · doi:10.1007/s11192-016-2141-9 · Cited by 32 · see citing works
Many-Citednesss: Citations Measure More than Just Scientific Impact
D’Ippoliti, C. (2017) · Journal of Economic Surveys · doi:10.1111/joes.12416 · Cited by 29 · see citing works
Diversity of backgrounds and ideas: The case of research evaluation in economics
Corsi, M., D’Ippoliti, C., & Zacchia, G. (2019) · Research Policy · doi:10.1016/j.respol.2019.103820 · Cited by 24 · see citing works
Italy’s national research assessment: some unpleasant effects
Grisorio, M. J., & Prota, F. (2019) · Studies in Higher Education · doi:10.1080/03075079.2019.1693989 · Cited by 24 · see citing works
The different responses of universities to introduction of performance-based research funding
Abramo, G., & D’Angelo, C. A. (2021) · Research Evaluation · doi:10.1093/reseval/rvab022 · Cited by 20 · see citing works
Peer Review and Citation Data in Predicting University Rankings, a Large-Scale Analysis
Pride, D., & Knoth, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00066-0_17 · Cited by 20 · see citing works
Baccini, A., Barabesi, L., & Nicolao, G. D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242520 · Cited by 20 · see citing works
Self-citationThe impact of a national research assessment on the publications of sociologists in Italy
Akbaritabar, A., Bravo, G., & Squazzoni, F. (2021) · Science and Public Policy · doi:10.1093/scipol/scab013 · Cited by 20 · see citing works
A Case Study of Pluralism in Economics: The Heterodox Glass Ceiling in Italy
Corsi, M., D’Ippoliti, C., & Zacchia, G. (2018) · Review of Political Economy · doi:10.1080/09538259.2018.1423974 · Cited by 18 · see citing works
Abramo, G., D’Angelo, C. A., & Costa, F. D. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.07.006 · Cited by 18 · see citing works
Pinar, M., & Unlu, E. (2020) · Research Evaluation · doi:10.1093/reseval/rvaa003 · Cited by 17 · see citing works
Convergence or polarisation? The impact of research assessment exercises in the Italian case
Daniele Checchi, A., Mazzotta, I., Momigliano, S., & Olivanti, F. (2020) · Scientometrics · doi:10.1007/s11192-020-03517-2 · Cited by 17 · see citing works
Two Decades of Research Assessment in Italy. Addressing the Criticisms
Bonaccorsi, A. J. (2020) · Scholarly Assessment Reports · doi:10.29024/sar.28 · Cited by 16 · see citing works
Abramo, G., & D’Angelo, C. A. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2017.06.003 · Cited by 15 · see citing works
Baccini, A. (2016) · Project Muse (Johns Hopkins University) · Cited by 13 · see citing works
Self-citationDaniele Checchi, A., Ciolfi, A., Fraja, G. D., Mazzotta, I., & Verzillo, S. (2021) · Economica · doi:10.1111/ecca.12373 · Cited by 13 · see citing works
Journal ratings: a paper affiliation methodology
Docampo, D., & Safón, V. (2021) · Scientometrics · doi:10.1007/s11192-021-04045-3 · Cited by 12 · see citing works
Abramo, G., & D’Angelo, C. A. (2022) · Research Evaluation · doi:10.1093/reseval/rvac026 · Cited by 12 · see citing works
Insights from the co-authorship network of the Italian academic statisticians
Bacci, S., Bertaccini, B., & Petrucci, A. (2023) · Scientometrics · doi:10.1007/s11192-023-04761-y · Cited by 11 · see citing works
Advancing women to leadership in academia: does personal branding matter?
Heriberta, H., Gaus, N., Paramma, M. A., & Utami, N. (2024) · Qualitative Research Journal · doi:10.1108/qrj-06-2023-0091 · Cited by 10 · see citing works
On the evolution of the glass ceiling in Italian academia: the case of economics
Corsi, M., D’Ippoliti, C., & Zacchia, G. (2019) · Science in Context · doi:10.1017/s0269889720000046 · Cited by 10 · see citing works
Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2016) · Scientometrics · doi:10.1007/s11192-016-1965-7 · Cited by 8 · see citing works
Biesenbender, S. (2018) · Higher Education Quarterly · doi:10.1111/hequ.12193 · Cited by 8 · see citing works
Baccini, A., & Nicolao, G. D. (2017) · Research Evaluation · doi:10.1093/reseval/rvx013 · Cited by 8 · see citing works
Self-citationAre Italian research assessment exercises size-biased?
Demetrescu, C., Ribichini, A., & Schaerf, M. (2020) · Scientometrics · doi:10.1007/s11192-020-03643-x · Cited by 7 · see citing works
Baccini, A., & Nicolao, G. D. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00172 · Cited by 6 · see citing works
Self-citationMethods to evaluate institutional responses to performance‐based research funding systems
Buckle, R. A., & Creedy, J. (2022) · Australian Economic Papers · doi:10.1111/1467-8454.12263 · Cited by 6 · see citing works
Chang, Y., Chen, D., & Huang, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259453 · Cited by 5 · see citing works
Baccini, A., & Re, C. (2025) · Scientometrics · doi:10.1007/s11192-025-05255-9 · Cited by 5 · see citing works
Self-citationFifteen Years of a PBRFS in New Zealand: Incentives and Outcomes
Buckle, R. A., Creedy, J., & Ball, A. (2021) · Australian Economic Review · doi:10.1111/1467-8462.12415 · Cited by 5 · see citing works
Reply to the comment of Bertocchi et al.
Baccini, A., & Nicolao, G. D. (2016) · Scientometrics · doi:10.1007/s11192-016-2055-6 · Cited by 5 · see citing works
Self-citationErrors and secret data in the Italian research assessment exercise. A comment to a replyPreprint
Baccini, A., & Nicolao, G. D. (2017) · arXiv (Cornell University) · doi:10.13130/2282-5398/8872 · Cited by 5 · see citing works
Self-citationIovino, G. (2021) · Advances in Cartography and GIScience of the ICA · doi:10.5194/ica-adv-3-7-2021 · Cited by 4 · see citing works
Italian Research Assessment VQR: Framework, Achievements and Controversies
Wang, X. (2022) · International Conference on Distance Education and Learning (ICDEL) · doi:10.1145/3543321.3543364 · Cited by 4 · see citing works
Mezzetti, M., & Negri, I. (2024) · Scientometrics · doi:10.1007/s11192-024-05154-5 · Cited by 4 · see citing works
The performance based research fund in New Zealand: taking stock and looking forward
Buckle, R. A., & Creedy, J. (2022) · New Zealand Economic Papers · doi:10.1080/00779954.2022.2156382 · Cited by 4 · see citing works
Theoretical research without projects
Navascués, M., & Budroni, C. (2019) · PLoS ONE · doi:10.1371/journal.pone.0214026 · Cited by 3 · see citing works
Ranking Departments based on research quality: a statistical evaluation of the ISPD indicator
Galli, F., & Greco, F. P. (2025) · Scientometrics · doi:10.1007/s11192-025-05240-2 · Cited by 2 · see citing works
D’Angelo, C. A. (2024) · Research Evaluation · doi:10.1093/reseval/rvaf045 · Cited by 2 · see citing works
Interpreting the outcomes of research assessments: A geometrical approach
Cappelletti–Montano, B., Columbu, S., Montaldo, S., & Musio, M. (2022) · Journal of Informetrics · doi:10.1016/j.joi.2022.101254 · Cited by 2 · see citing works
Should geoscientists, employers, funding agencies and publishers be slaves to impact factors?
Pereira, D. (2017) · Episodes · doi:10.18814/epiiugs/2017/v40i4/017034 · Cited by 2 · see citing works
Metrics and peer review agreement at the institutional level
Traag, V., Malgarini, M., & Sarlo, S. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00352 · Cited by 2 · see citing works
Peer-Review, Bibliometrie, Altmetrics – gibt es einen Königsweg bei Evaluierungen?
Mittermaier, B. (2026) · Ubiquity Press eBooks · doi:10.5334/bdh.h · Cited by 1 · see citing works
Guidelines for Peer Review. A Survey of International Practices
Capaccioni, A., & Spina, G. (2018) · Springer eBooks · doi:10.1007/978-3-319-68554-0_3 · Cited by 1 · see citing works
Baccini, A., Barabesi, L., & Nicolao, G. D. (2018) · Use Siena air (University of Siena) · Cited by 1 · see citing works
Self-citationDoshi, N., Gundam, S., & Chaudhury, B. (2021) · arXiv (Cornell University) · Cited by 1 · see citing works
The assessment of research quality / La valutazione della qualità della ricerca
Masino, G. (2019) · AMS Acta (University of Bologna) · doi:10.6092/unibo/amsacta/6286
Цитируемость, показатели цитируемости и качество исследований: обзор базовых понятий и теорий
Aksnes, D. W., ЛАНГФЕЛТ, Л., & ВУТЕРС, П. (2021) · Международный форум по информации · doi:10.36535/0203-6460-2021-03-1
Cuciureanu, G., Țurcan, N., Cojocaru, I., & Cojocaru, I. (2023) · Science and innovation · doi:10.15407/scine19.02.003
Efficient Threshold-based Model for Scientific Productivity Evaluation: a ProposalPreprint
Sergioli, G., Bianchini, F., & Graziani, P. (2026) · Research Square · doi:10.21203/rs.3.rs-8397937/v1
Jie, R., Chu, C., & Wang, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-10069566/v1
Mößner, N., & Erlach, K. (2026) · Ubiquity Press eBooks · doi:10.5334/bdh
Gender, Departmental, and Interdisciplinary Biases
Thelwall, M. A. (2026) · White Rose University Press eBooks · doi:10.22599/quantitativemethods.j
Aggregate Citation Indicators for Scholars, Departments, Institutions, Funders, and Journals
Thelwall, M. A. (2026) · White Rose University Press eBooks · doi:10.22599/quantitativemethods.i
Benkendorff, A., Sina, E., Bergt, A. C., Stadelmaier, J., & Meerpohl, J. J. (2026) · BMJ Open · doi:10.1136/bmjopen-2026-117895
- Authors
- Baccini, A., & De Nicolao, G.
- Affiliations
- University of Siena (IT)
- Corresponding author
- Baccini, Alberto
- ORCID
- Baccini, Alberto
- Venue
- Scientometrics
- Published
- 2016-03-23
- Volume
- 108
- Issue
- 3
- Pages
- 1651-1671
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W3126024829
- Semantic Scholar
- 0065dd6a48a35b4fa94b0aa349e70e198fce79c8
- Cited by
- 68
- References
- 43
- Citations by year
- 2026: 5 · 2025: 3 · 2024: 3 · 2023: 5 · 2022: 6 · 2021: 14 · 2020: 6 · 2019: 9 · 2018: 5 · 2017: 7 · 2016: 5
- FWCI
- 8.16
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Scientometrics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Use Siena air (University of Siena) (submittedVersion)
- Use Siena air (University of Siena) (submittedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- ANVUR, bibliometric analysis, research evaluation policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
