The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; none of the 20 sources its claims cite is checked for support yet.
Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements
Open accessAbstract
How does scientific research affect the world around us? Being able to answer this question is of great importance in order to appropriately channel efforts and resources in science. The impact by scientists in academia is currently measured by citation based metrics such as h-index, i-index and citation counts. These academic metrics aim to represent the dissemination of knowledge among scientists rather than the impact of the research on the wider world. In this work we are interested in measuring scientific impact beyond academia, on the economy, society, health and legislation (comprehensive impact). Indeed scientists are asked to demonstrate evidence of such comprehensive impact by authoring case studies in the context of the Research Excellence Framework (REF). We first investigate the extent to which existing citation based metrics can be indicative of comprehensive impact. We have collected all recent REF impact case studies from 2014 and we have linked these to papers in citation networks that we constructed and derived from CiteSeerX, arXiv and PubMed Central using a number of text processing and information retrieval techniques. We have demonstrated that existing citation-based metrics for impact measurement do not correlate well with REF impact results. We also consider metrics of online attention surrounding scientific works, such as those provided by the Altmetric API. We argue that in order to be able to evaluate wider non-academic impact we need to mine information from a much wider set of resources, including social media posts, press releases, news articles and political debates stemming from academic work. We also provide our data as a free and reusable collection for further analysis, including the PubMed citation network and the correspondence between REF case studies, grant applications and the academic literature.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper tests whether citation-based and altmetric metrics correlate with REF impact case study scores as a proxy for scientific impact beyond academia. It finds negligible correlation for MNCS, author h-index, and Altmetric, with only a weak relationship for per-case-study h-index, and regression models fail to predict REF scores.
Key findings
- Existing citation-based metrics do not correlate well with REF impact: overall Pearson r = 0.035 for MNCS and r = −0.005 for average author h-index.
- A newly calculated per-REF-case-study h-index showed a stronger but still weak correlation, r = 0.141 overall.
- Altmetric scores also showed little relationship with REF impact, r = −0.0803, and approximately 40% of papers had no Altmetric score.
- A linear OLS regression using per-study h-index, per-author h-index, total citations, and mean Altmetric score as features had very poor predictive capability, R² = −0.113; SVR was also unsuccessful.
Methods
- Collected all 6,637 REF 2014 impact case studies and built Experimental Citation Networks from CiteSeerX/RefSeer (~5.3M papers), arXiv/Paperscape (~903K papers), and PubMed Central (1.2M papers).
- Linked REF case studies to papers using regular expressions, Freecite CRF parsing, and SOLR search with conservative exact title and author-surname matching; this produced 1,052 linked papers across 647 unique REF studies and 235 UoA-per-institution result groups.
- Calculated MNCS, author h-index, per-case-study h-index, and Altmetric scores, then plotted and correlated them against REF impact scores and trained OLS and SVR regression models with leave-one-out cross-validation.
Limitations
- The paper states that only a minority of REF studies could be linked to ECN papers because REF bibliographies were not machine readable and had varied styles; it also notes possible missing papers from the open-access ECNs and that Web of Science or Scopus might yield different results.
- The paper states that per-case-study REF scores are not available, so results are grouped by UoA-per-institution; Altmetric coverage was insufficient, with about 40% of papers missing scores, and author disambiguation remains a challenge.
- Evident from the text: the conservative matching strategy limits recall, some UoA analyses have very few data points, and the regression dataset was too small to divide into folds, using leave-one-out instead.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
27 Own interpretation · 25 Own result · 16 Not checked yet · 10 Own method · 7 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that of the 1052 papers linked to REF case studies, 63 were duplicate entries.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > ECN overlap and duplicate papers
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our search pipeline was able to identify links between papers in the available ECNs and 647 unique REF case studies.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results
plaincorrelationaln = 647- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Internally the ECNs contain no duplicate papers”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > ECN overlap and duplicate papers
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have demonstrated that existing citation-based metrics for impact measurement do not correlate well with REF impact results.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that the predictive capability of the model was very poor (R2 = −0.113).”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Regression modelling
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also determined that papers citing the 63 duplicate papers have no overlap.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > ECN overlap and duplicate papers
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately a large number of the papers under investigation (approximately 40%) had no Altmetric score at all (the API endpoint returned a 404 indicating missing paper profile).”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We describe a way of calculating h-index with respect to REF case studies which offers a stronger correlation than the other metrics investigated in this paper.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: h-index for REF case studies has not been calculated before.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, what is immediately clear from this graph is that there is a more visible correlation between the two metrics (the Pearson coefficient for the whole dataset including the 5 categories shown was r = 0.141).”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Case study h-index and REF impact score
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Coverage by Altmetric of the papers in our experiment citation networks (ECNs) was insufficient to allow us to draw conclusions regarding the potential of Altmetric scores”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When predicting REF scores from the set of aforementioned metrics as features, neither a simple regression model nor a more complex SVR model was successful.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note: *There are 647 unique studies linked across the three ECNs, some of which appear multiple times giving a total of 711.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Linking REF case studies and other data sources
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The inverse appears to be true for Physics and Allied Health Professions papers, which have some of the lower h-index and citation counts in previous graphs but have the highest Altmetric scores.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, neither of these UoAs demonstrate any kind of correlation between Altmetric scores and REF scores.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However our regular expression and CRF pipeline returns 6627 references and of these we are able to match 1052 papers from our ECNs.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Linking REF case studies and other data sources
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Perhaps surprisingly the results show that there is little in the way of correlation between Altmetric score and REF impact. The Pearson coefficient for the overall relationship is r = −0.0803.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This leaves us with 235 result groups (with an average case study population of 2.65 and Standard Deviation of 3.2) to be used in experiments and plotted.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results
plainn = 647- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We deliberately use data from close to the REF study period where possible such that we can accurately measure the 10 year window within which the REF report is valid.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The REF guidelines stipulate that all supporting works must have been published in the ten year window starting in December 2003 and ending in December 2013.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The model was trained using Leave-One-Out cross validation since the data set was too small to sensibly divide up into folds.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Regression modelling
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, we do not know exactly how many references there are in total and what proportion of them we are able to successfully match.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Linking REF case studies and other data sources
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus reports tend to focus on a small number of high impact works from institutions, penalising academics who contribute in small amounts to a large number of projects.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Comprehensive impact projects
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In addition to this, it is possible to game the Altmetric system through the generation of false likes and mentions [21], adding significant noise.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Altmetrics and social based metrics
What the sources say
- Not checked yetcites [21] L Bornmann, 2014, Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics, Journal of Informetrics source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The ECNs were comprehensive in their combined coverage of STEM subjects and the timespans covered too.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe this is due to the format of the REF studies which had very varied bibliography styles which were not machine readable.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Not all researchers in scientific domains covered by arXiv, CiteSeerX and PubMed Central publish to these repositories.
In the paper
“The assessment process is resource intensive, requiring a committee of academics to evaluate each submission individually, necessitating brief 3–5 page submissions [25, p. 51].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Comprehensive impact projects
What the sources say
- Not checked yetcites [25] REF 2014. Assessment framework and guidance on submissions; 2011. Available from: http://www.ref.ac.uk/pubs/2011-02/.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The subjective nature of human assessors, combined with flexible guidelines and varying opinions on what makes a good case study is also a potential weakness of this system.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Comprehensive impact projects
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This would support the view taken by McKeown et al. that full text features drawn from academic papers are more informative in calculating academic impact [20].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Conclusions
What the sources say
- Not checked yetcites [20] K McKeown, 2016, Predicting the impact of scientific concepts using full-text features, Journal of the Association for Information Science and Technology source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Since we require an exact match between titles normalising for case and punctuation as well as checking that one or more author surnames match between REF citation and paper metadata record, our matching strategy is very conservative and as such we estimate that there are very few errors but also limits our recall for citation matches.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Linking REF case studies and other data sources
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However we believe that these systems are far from mature.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Altmetrics and social based metrics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Since these are not as comprehensive as those provided by Google Scholar and Web of Science, we have calculated the h-indices of the most prolific authors in our study and compared these against their respective Google h-index as a probable upper limit.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Author h-index and REF impact score
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We checked that the titles of the RefSeer papers and those suggested by CrossRef matched word for word before accepting the CrossRef suggested DOI.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Data sources—altmetrics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Bornmann (2008) suggests that variation of citation behaviour across disciplines may also mean that comparisons made via h-index and JIF across scientific domains are invalid [10].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Academic impact metrics
What the sources say
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that although Computer Science papers are cited many times by academics (as previous graphs imply), they are not discussed as frequently on social media.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
hedgedcorrelationaln = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe that REF impact is more of a proxy for comprehensive impact whereas citation-based metrics and altmetrics mostly measure academic impact.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Discussion > Summary
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, it more succinctly encapsulates the academic impact of the REF case study than the per-author metric used above.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Case study h-index and REF impact score
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Bornmann (2016) suggests that the 10 principles of the Leiden Manifesto, originally written to address citation-based metrics, also apply to the use of altmetrics to measure academic impact [22].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Altmetrics and social based metrics
What the sources say
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Cronin (1984) suggests that citation behaviour may be driven by a number of motivators including the negation or correction of some part of the cited work [9, pp 35–46].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Academic impact metrics
What the sources say
- Not checked yetcites Cronin (1984)
- Not checked yetcites [9] B Cronin, 1984, The role and significance of citations in scientific communication
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“MNCS is usually normalised by year and by scientific discipline [4].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Mean normalised citation score and REF impact score
What the sources say
- Not checked yetcites [4] Waltman L. A review of the literature on citation impact indicators. 2015 jul;Available from: http://arxiv.org/abs/1507.02099.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Whilst REF does contain provisions for interdisciplinary researchers, typically submissions are only assessed by one Unit-of-Assessment sub-panel (a panel that judges work deemed to be in a similar academic discipline e.g. Physics, Mathematics, Biology and so forth) [25, p. 15] placing great importance on employing diverse assessment panels who can fairly judge interdisciplinary work.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Introduction > Comprehensive impact projects
What the sources say
- Not checked yetcites [25] REF 2014. Assessment framework and guidance on submissions; 2011. Available from: http://www.ref.ac.uk/pubs/2011-02/.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Large diverse citation networks should allow us to calculate citation metrics accurately for the REF studies under examination.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Data sources—scientific papers
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite our ECNs being large and relatively comprehensive, it is to be expected that authors featured within them also publish in journals and publications that are not present in these networks but that are present in Google’s indexing platform.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Author h-index and REF impact score
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They found that social data coverage was inadequate for drawing any conclusions.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
What the sources say
- Not checked yetcites [28] M Thelwall, 2013, Do Altmetrics Work? Twitter and Ten Other Social Web Services, PLoS ONE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This metric represents the citation-count impact or Academic impact of a REF case study as opposed to the average per-author metric which describes the average academic impact of an author’s entire work.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Case study h-index and REF impact score
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“This finding seems to support the experience of Thelwall et al [28].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
What the sources say
- Not checked yetcites [28] M Thelwall, 2013, Do Altmetrics Work? Twitter and Ten Other Social Web Services, PLoS ONE
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the existing metrics for academic impact explored above, Altmetric would seem to be the one most closely linked to socio-economic impact since it is modelled upon online social media interactions rather than citations.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since per-case-study h-index is a metric derived from the pure citation counts depicted in Fig 4, it should come as no surprise that the graph is very similar.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Case study h-index and REF impact score
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our scoped survey does not provide much insight into the extent of duplication across an author’s entire back-catalogue of work.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Author h-index and REF impact score
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Haustein et al. (2014) suggest that papers are more likely to be tweeted if they are “curious or funny, have potential health applications or refer to a catastrophe” [30].”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Altmetric
What the sources say
- Not checked yetcites [30] S Haustein, 2016, Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter, Journal of the Association for Information Science and Technology source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper relies on this without support.
Altmetric scores are a valid measure of social-web-based impact.
“we chose to use Altmetric as our source as they provide an easy to use API which takes a paper’s PubMed or DOI identifier and produces a continuous score which could be plotted against REF and used in regression without any further manipulation.”
The authors state this as an assumption.
All papers referenced by a REF case study are from within the same UoA as that REF case study.
“This decision assumes that all papers referenced by a REF case study are from within the same UoA as that REF case study.”
The paper relies on this without support.
The REF impact system currently in place is a very manual and resource intensive process.
“The REF impact system currently in place is a very manual and resource intensive process.”
The paper relies on this without support.
Existing analysis of policy documents is based on counting references in these documents to scholarly articles.
“Furthermore, existing analysis of policy documents is based on counting references in these documents to scholarly articles.”
The paper relies on this without support.
Altmetric.com and other altmetrics providers already examine some policy documents as part of their offerings.
“Altmetric.com and other altmetrics providers already examine some policy documents as part of their offerings.”
The paper relies on this without support.
Not all researchers in scientific domains covered by arXiv, CiteSeerX and PubMed Central publish to these repositories.
“Not all researchers in scientific domains covered by arXiv, CiteSeerX and PubMed Central publish to these repositories.”
The paper relies on this without support.
Papers need time to accumulate citations.
“Papers need time to accumulate citations.”
The paper relies on this without support.
h-index for REF case studies has not been calculated before.
“We do not believe h-index for REF case studies has been calculated before.”
The paper relies on this without support.
The REF guidelines stipulate that all supporting works must have been published in the ten year window starting in December 2003 and ending in December 2013.
“The REF guidelines stipulate that all supporting works must have been published in the ten year window starting in December 2003 and ending in December 2013.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Citations may be appropriate in assessing the scientific impact of the research, but they do not show the impact of the research outside the scientific community [1,3,5,6,260,264,265].”
From Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World (2021) · cites this paper as [264]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > ECN overlap and duplicate papers
“Since there are of the order of 2 million papers in our ECNs, we restricted the search and comparison to the 1052 papers that have explicit links to REF case studies”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Materials and methods > Data sources—scientific papers
“These citation networks contain approximately 5.3M and 903K papers respectively.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“This leaves us with 235 result groups (with an average case study population of 2.65 and Standard Deviation of 3.2)”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results
Reported 2.65 · recomputed 2.753191
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Case study h-index and REF impact score
“Since per-case-study h-index is a metric derived from the pure citation counts depicted in Fig 4, it should come as no surprise that the graph is very similar.”
§ Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements > Results > Mean normalised citation score and REF impact score
“Figure: Fig 4 Average MNCS per REF case study vs REF case study score.”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828
The History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770
What do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356
Do Altmetrics Work? Twitter and Ten Other Social Web Services
Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013) · PLoS ONE · doi:10.1371/journal.pone.0064841 · Cited by 907
Costas, R., Zahedi, Z., & Wouters, P. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23309 · Cited by 711
Edwards, M. A., & Roy, S. (2017) · Environmental Engineering Science · doi:10.1089/ees.2016.0223 · Cited by 688
Bornmann, L. (2014) · Journal of Informetrics · doi:10.1016/j.joi.2014.09.005 · Cited by 578
Priem, J. R., Taraborelli, D., Groth, P., & Neylon, C. (2010) · Lincoln (University of Nebraska) · doi:10.5281/zenodo.12684248 · Cited by 459
Piwowar, H. (2013) · Nature · doi:10.1038/493159a · Cited by 398
The rise and rise of citation analysis
Meho, L. I. (2007) · Physics World · doi:10.1088/2058-7058/20/1/33 · Cited by 371
The citation process: The role and significance of citations in scientific communication
Cronin, B. (1984) · International Linear Collider · Cited by 358
ORCID: a system to uniquely identify researchers
Laurel, H., Fenner, M. H., Paglione, L., Pentz, E., & Ratner, H. (2012) · Learned Publishing · doi:10.1087/20120404 · Cited by 334
Detecting h-index manipulation through self-citation analysis
Bartneck, C., & Kokkelmans, S. (2011) · Scientometrics · doi:10.1007/s11192-010-0306-5 · Cited by 301
Rossner, M., Epps, H. V., & Hill, E. (2007) · The Journal of Experimental Medicine · doi:10.1084/jem.20072544 · Cited by 278
Rossner, M., Epps, H. V., & Hill, E. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200711140 · Cited by 248
Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter
Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23456 · Cited by 231
Altmetric: enriching scholarly content with article‐level discussion and metrics
Adie, E. A., & William, R. O. E. (2012) · Learned Publishing · doi:10.1087/20130103 · Cited by 224
Identifying Meaningful Citations
Valenzuela, M., Ha, V., & Etzioni, O. (2015) · National Conference on Artificial Intelligence · Cited by 124
Haunschild, R., & Bornmann, L. (2017) · Scientometrics · doi:10.1007/s11192-016-2237-2 · Cited by 91
Predicting the impact of scientific concepts using full-text features
McKeown, K., Daume, H., Chaturvedi, S., Paparrizos, J., Thadani, K., Barrio, P., Biran, O., Bothe, S., Collins, M., Fleischmann, K. R., et al. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23612 · Cited by 73
Programmatic evaluation and comparison based on standardized citation scores
McAllister, P. R., Narin, F., & Corrigan, J. G. (1983) · IEEE Transactions on Engineering Management · doi:10.1109/tem.1983.6448622 · Cited by 65
RefSeer: A citation recommendation system
Huang, W., Wu, Z., Mitra, P., & Giles, C. L. (2014) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2014.6970192 · Cited by 58
Refseer: A citation recommendation system
Huang, W., Wu, Z., Mitra, P., & Giles, C. L. (2014) · Cited by 43
STAR METRICS and the Science of Science Policy
Largent, M. A., & Lane, J. I. (2012) · Review of Policy Research · doi:10.1111/j.1541-1338.2012.00567.x · Cited by 34
Analyzing the impact of an author’s publications
Vucovich, L. A., Blaine Baker, J., & Smith, J. T. (2008) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.96.1.63 · Cited by 24
Bornmann, L., & Haunschild, R. (2016) · Online Information Review · doi:10.1108/oir-09-2015-0314 · Cited by 21
Grimson, J. B. (2014) · Cited by 15
A review of the literature on citation impact indicatorsPreprint
Waltman, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02099 · Cited by 10
Research works printed but not linked: 1
- Lane J, Bertuzzi S. The STAR METRICS project: current and future uses for S&E workforce data. In: Science of Science Measurement Workshop, held Washington DC; 2010.
Cited by
Sorted by citations
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696
Ai for social good: unlocking the opportunity for positive impact
Tomašev, N., Cornebise, J., Hutter, F., Mohamed, S., Picciariello, A., Connelly, B., Belgrave, D. C. M., Ezer, D., Haert, F. C. V. D., Mugisha, F., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15871-z · Cited by 357
A Meta-Review of Indoor Positioning Systems
Mendoza-Silva, G. M., Torres-Sospedra, J., & Huerta, J. (2019) · Sensors · doi:10.3390/s19204507 · Cited by 268
Bornmann, L., & Haunschild, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0197133 · Cited by 143
Methods for mapping the impact of social sciences and humanities—A literature review
Pedersen, D. B., Grønvad, J. F., & Hvidtfeldt, R. (2019) · Research Evaluation · doi:10.1093/reseval/rvz033 · Cited by 147
Metadata
- Authors
- Ravenscroft, J., Liakata, M., Clare, A., & Duma, D.
- Affiliations
- University of Warwick (GB)
- Aberystwyth University (GB)
- University of Edinburgh (GB)
- Corresponding author
- Ravenscroft, James
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-03-09
- Volume
- 12
- Issue
- 3
- Pages
- e0173152
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5344357
- OpenAlex
- W2593028313
- Semantic Scholar
- 7c6dd680030a3111409c45dfc28b1d9feebc25e3
- Cited by
- 202
- References
- 29
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 21 · 2023: 18 · 2022: 21 · 2021: 19 · 2020: 30 · 2019: 28 · 2018: 18 · 2017: 4
- FWCI
- 19.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Engineering and Physical Sciences Research Council (1535875, 1643924)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- scientific impact measurement, citation metrics, Research Excellence Framework, h-index, I-index, citation networks, altmetrics, online attention, social media, media coverage, science policy, grant applications
- MeSH terms
- Achievement, Humans, Publishing, Science, Models, Statistical, Biomedical Research, Journal Impact Factor, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
