CoChat
Sign Up
Cited by 202scientometrics and bibliometrics researchWeb visibility and informetricsAcademic Publishing and Open Access

Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements

Open access
Ravenscroft, James, Liakata, Maria, Clare, Amanda, Duma, Daniel·2017·PLOS ONE
DOI 10.1371/journal.pone.0173152PMC PMC5344357
Open this paper in CoChat — free accountOpen PDF

Abstract

How does scientific research affect the world around us? Being able to answer this question is of great importance in order to appropriately channel efforts and resources in science. The impact by scientists in academia is currently measured by citation based metrics such as h-index, i-index and citation counts. These academic metrics aim to represent the dissemination of knowledge among scientists rather than the impact of the research on the wider world. In this work we are interested in measuring scientific impact beyond academia, on the economy, society, health and legislation (comprehensive impact). Indeed scientists are asked to demonstrate evidence of such comprehensive impact by authoring case studies in the context of the Research Excellence Framework (REF). We first investigate the extent to which existing citation based metrics can be indicative of comprehensive impact. We have collected all recent REF impact case studies from 2014 and we have linked these to papers in citation networks that we constructed and derived from CiteSeerX, arXiv and PubMed Central using a number of text processing and information retrieval techniques. We have demonstrated that existing citation-based metrics for impact measurement do not correlate well with REF impact results. We also consider metrics of online attention surrounding scientific works, such as those provided by the Altmetric API. We argue that in order to be able to evaluate wider non-academic impact we need to mine information from a much wider set of resources, including social media posts, press releases, news articles and political debates stemming from academic work. We also provide our data as a free and reusable collection for further analysis, including the PubMed citation network and the correspondence between REF case studies, grant applications and the academic literature.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper tests whether citation-based and altmetric metrics correlate with REF impact case study scores as a proxy for scientific impact beyond academia. It finds negligible correlation for MNCS, author h-index, and Altmetric, with only a weak relationship for per-case-study h-index, and regression models fail to predict REF scores.

Key findings

  • Existing citation-based metrics do not correlate well with REF impact: overall Pearson r = 0.035 for MNCS and r = −0.005 for average author h-index.
  • A newly calculated per-REF-case-study h-index showed a stronger but still weak correlation, r = 0.141 overall.
  • Altmetric scores also showed little relationship with REF impact, r = −0.0803, and approximately 40% of papers had no Altmetric score.
  • A linear OLS regression using per-study h-index, per-author h-index, total citations, and mean Altmetric score as features had very poor predictive capability, R² = −0.113; SVR was also unsuccessful.

Methods

  • Collected all 6,637 REF 2014 impact case studies and built Experimental Citation Networks from CiteSeerX/RefSeer (~5.3M papers), arXiv/Paperscape (~903K papers), and PubMed Central (1.2M papers).
  • Linked REF case studies to papers using regular expressions, Freecite CRF parsing, and SOLR search with conservative exact title and author-surname matching; this produced 1,052 linked papers across 647 unique REF studies and 235 UoA-per-institution result groups.
  • Calculated MNCS, author h-index, per-case-study h-index, and Altmetric scores, then plotted and correlated them against REF impact scores and trained OLS and SVR regression models with leave-one-out cross-validation.

Limitations

  • The paper states that only a minority of REF studies could be linked to ECN papers because REF bibliographies were not machine readable and had varied styles; it also notes possible missing papers from the open-access ECNs and that Web of Science or Scopus might yield different results.
  • The paper states that per-case-study REF scores are not available, so results are grouped by UoA-per-institution; Altmetric coverage was insufficient, with about 40% of papers missing scores, and author disambiguation remains a challenge.
  • Evident from the text: the conservative matching strategy limits recall, some UoA analyses have very few data points, and the regression dataset was too small to divide into folds, using leave-one-out instead.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0173152

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 20 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Of the 1052 papers linked to REF case studies, 63 were duplicate entries.

    “We found that of the 1052 papers linked to REF case studies, 63 were duplicate entries.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The search pipeline identified links between papers in the available ECNs and 647 unique REF case studies.

    “Our search pipeline was able to identify links between papers in the available ECNs and 647 unique REF case studies.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Internally the ECNs contain no duplicate papers.

    “Internally the ECNs contain no duplicate papers”
    Quote in the paperNot a substantial claim
  • Findingown data

    Existing citation-based metrics for impact measurement do not correlate well with REF impact results.

    “We have demonstrated that existing citation-based metrics for impact measurement do not correlate well with REF impact results.”
    Quote in the paperSubstantial claim
  • Findingown data

    The linear regression OLS model predicting REF score from per-study h-index, per-author h-index, total citations and mean altmetrics score had very poor predictive capability (R2 = -0.113).

    “We found that the predictive capability of the model was very poor (R2 = −0.113).”
    Quote in the paperSubstantial claim
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 24

  • May combine two claims.

    Findingown data

    Neither a simple regression model nor a more complex SVR model was successful in predicting REF scores from the metrics.

    “When predicting REF scores from the set of aforementioned metrics as features, neither a simple regression model nor a more complex SVR model was successful.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There are 647 unique studies linked across the three ECNs, some of which appear multiple times giving a total of 711.

    “Note: *There are 647 unique studies linked across the three ECNs, some of which appear multiple times giving a total of 711.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Physics and Allied Health Professions papers have some of the lower h-index and citation counts but the highest Altmetric scores.

    “The inverse appears to be true for Physics and Allied Health Professions papers, which have some of the lower h-index and citation counts in previous graphs but have the highest Altmetric scores.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Neither Physics nor Allied Health Professions UoAs demonstrate any correlation between Altmetric scores and REF scores.

    “However, neither of these UoAs demonstrate any kind of correlation between Altmetric scores and REF scores.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The regular expression and CRF pipeline returned 6627 references, of which 1052 papers from the ECNs were matched.

    “However our regular expression and CRF pipeline returns 6627 references and of these we are able to match 1052 papers from our ECNs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Altmetric score and REF impact score show little correlation, with a Pearson coefficient of r = -0.0803.

    “Perhaps surprisingly the results show that there is little in the way of correlation between Altmetric score and REF impact. The Pearson coefficient for the overall relationship is r = −0.0803.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Grouping results by UoA-per-Institution leaves 235 result groups, with an average case study population of 2.65 and a standard deviation of 3.2.

    “This leaves us with 235 result groups (with an average case study population of 2.65 and Standard Deviation of 3.2) to be used in experiments and plotted.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    They deliberately use data from close to the REF study period where possible to accurately measure the 10 year window within which the REF report is valid.

    “We deliberately use data from close to the REF study period where possible such that we can accurately measure the 10 year window within which the REF report is valid.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The REF guidelines stipulate that all supporting works must have been published in the ten year window starting in December 2003 and ending in December 2013.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The model was trained using Leave-One-Out cross validation because the data set was too small to divide into folds.

    “The model was trained using Leave-One-Out cross validation since the data set was too small to sensibly divide up into folds.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The paper does not know exactly how many references there are in total or what proportion of them were successfully matched.

    “Therefore, we do not know exactly how many references there are in total and what proportion of them we are able to successfully match.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    REF reports tend to focus on a small number of high impact works from institutions, penalising academics who contribute in small amounts to a large number of projects.

    “Thus reports tend to focus on a small number of high impact works from institutions, penalising academics who contribute in small amounts to a large number of projects.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    It is possible to game the Altmetric system through the generation of false likes and mentions, adding significant noise.

    “In addition to this, it is possible to game the Altmetric system through the generation of false likes and mentions [21], adding significant noise.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [21] L Bornmann, 2014, Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics, Journal of Informetrics source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The ECNs were comprehensive in their combined coverage of STEM subjects and the timespans covered.

    “The ECNs were comprehensive in their combined coverage of STEM subjects and the timespans covered too.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The inability to identify links between ECNs and many REF studies is due to the format of the REF studies, which had very varied bibliography styles that were not machine readable.

    “We believe this is due to the format of the REF studies which had very varied bibliography styles which were not machine readable.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Not all researchers in scientific domains covered by arXiv, CiteSeerX and PubMed Central publish to these repositories.

  • May combine two claims.

    Premisecitation

    The REF assessment process is resource intensive, requiring a committee of academics to evaluate each submission individually, necessitating brief 3-5 page submissions.

    “The assessment process is resource intensive, requiring a committee of academics to evaluate each submission individually, necessitating brief 3–5 page submissions [25, p. 51].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] REF 2014. Assessment framework and guidance on submissions; 2011. Available from: <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="http://www.ref.ac.uk/pubs/2011-02/" xlink:type="simple">http://www.ref.ac.uk/pubs/2011-02/</ext-link>.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The subjective nature of human assessors, combined with flexible guidelines and varying opinions on what makes a good case study, is a potential weakness of the REF system.

    “The subjective nature of human assessors, combined with flexible guidelines and varying opinions on what makes a good case study is also a potential weakness of this system.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Full text features drawn from academic papers are more informative in calculating academic impact.

    “This would support the view taken by McKeown et al. that full text features drawn from academic papers are more informative in calculating academic impact [20].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] K McKeown, 2016, Predicting the impact of scientific concepts using full-text features, Journal of the Association for Information Science and Technology source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The matching strategy is very conservative, estimated to have very few errors but limited recall for citation matches.

    “Since we require an exact match between titles normalising for case and punctuation as well as checking that one or more author surnames match between REF citation and paper metadata record, our matching strategy is very conservative and as such we estimate that there are very few errors but also limits our recall for citation matches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Current altmetric systems are far from mature.

    “However we believe that these systems are far from mature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    They calculated the h-indices of the most prolific authors in the study and compared these against their respective Google h-index as a probable upper limit.

    “Since these are not as comprehensive as those provided by Google Scholar and Web of Science, we have calculated the h-indices of the most prolific authors in our study and compared these against their respective Google h-index as a probable upper limit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The paper checked that titles of RefSeer papers and those suggested by CrossRef matched word for word before accepting the CrossRef suggested DOI.

    “We checked that the titles of the RefSeer papers and those suggested by CrossRef matched word for word before accepting the CrossRef suggested DOI.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Variation of citation behaviour across disciplines may mean that comparisons made via h-index and JIF across scientific domains are invalid.

    “Bornmann (2008) suggests that variation of citation behaviour across disciplines may also mean that comparisons made via h-index and JIF across scientific domains are invalid [10].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Bornmann (2008) source

      Not checked in the free run.

    • cites [10] L Bornmann, 2008, What do citation counts measure? A review of studies on citing behavior, Journal of Documentation source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Computer Science papers are cited many times by academics but are not discussed as frequently on social media.

    “This suggests that although Computer Science papers are cited many times by academics (as previous graphs imply), they are not discussed as frequently on social media.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    REF impact is more of a proxy for comprehensive impact whereas citation-based metrics and altmetrics mostly measure academic impact.

    “We believe that REF impact is more of a proxy for comprehensive impact whereas citation-based metrics and altmetrics mostly measure academic impact.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Per-case-study h-index more succinctly encapsulates the academic impact of the REF case study than the per-author metric.

    “In other words, it more succinctly encapsulates the academic impact of the REF case study than the per-author metric used above.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The 10 principles of the Leiden Manifesto also apply to the use of altmetrics to measure academic impact.

    “Bornmann (2016) suggests that the 10 principles of the Leiden Manifesto, originally written to address citation-based metrics, also apply to the use of altmetrics to measure academic impact [22].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Bornmann (2016) source

      Not checked in the free run.

    • cites [22] L Bornmann, 2016, To what extent does the Leiden manifesto also apply to altmetrics? A discussion of the manifesto against the background of research into altmetrics, Online Information Review source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Citation behaviour may be driven by motivators including the negation or correction of some part of the cited work.

    “Cronin (1984) suggests that citation behaviour may be driven by a number of motivators including the negation or correction of some part of the cited work [9, pp 35–46].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Cronin (1984)

      Not checked in the free run.

    • cites [9] B Cronin, 1984, The role and significance of citations in scientific communication

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    MNCS is usually normalised by year and by scientific discipline.

    “MNCS is usually normalised by year and by scientific discipline [4].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] Waltman L. A review of the literature on citation impact indicators. 2015 jul;Available from: <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="http://arxiv.org/abs/1507.02099" xlink:type="simple">http://arxiv.org/abs/1507.02099</ext-link>.

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Typically REF submissions are only assessed by one Unit-of-Assessment sub-panel, placing importance on employing diverse assessment panels who can fairly judge interdisciplinary work.

    “Whilst REF does contain provisions for interdisciplinary researchers, typically submissions are only assessed by one Unit-of-Assessment sub-panel (a panel that judges work deemed to be in a similar academic discipline e.g. Physics, Mathematics, Biology and so forth) [25, p. 15] placing great importance on employing diverse assessment panels who can fairly judge interdisciplinary work.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] REF 2014. Assessment framework and guidance on submissions; 2011. Available from: <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="http://www.ref.ac.uk/pubs/2011-02/" xlink:type="simple">http://www.ref.ac.uk/pubs/2011-02/</ext-link>.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Large diverse citation networks should allow accurate calculation of citation metrics for the REF studies under examination.

    “Large diverse citation networks should allow us to calculate citation metrics accurately for the REF studies under examination.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    It is to be expected that authors featured in the ECNs also publish in journals and publications that are not present in these networks but are present in Google's indexing platform.

    “Despite our ECNs being large and relatively comprehensive, it is to be expected that authors featured within them also publish in journals and publications that are not present in these networks but that are present in Google’s indexing platform.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Thelwall et al. found that social data coverage was inadequate for drawing any conclusions.

    “They found that social data coverage was inadequate for drawing any conclusions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] M Thelwall, 2013, Do Altmetrics Work? Twitter and Ten Other Social Web Services, PLoS ONE

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    This metric represents the citation-count impact or Academic impact of a REF case study, as opposed to the average per-author metric which describes the average academic impact of an author's entire work.

    “This metric represents the citation-count impact or Academic impact of a REF case study as opposed to the average per-author metric which describes the average academic impact of an author’s entire work.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The finding of little correlation between Altmetric score and REF impact seems to support the experience of Thelwall et al.

    “This finding seems to support the experience of Thelwall et al [28].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] M Thelwall, 2013, Do Altmetrics Work? Twitter and Ten Other Social Web Services, PLoS ONE

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Altmetric would seem to be the existing academic impact metric most closely linked to socio-economic impact because it is modelled upon online social media interactions rather than citations.

    “Among the existing metrics for academic impact explored above, Altmetric would seem to be the one most closely linked to socio-economic impact since it is modelled upon online social media interactions rather than citations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Per-case-study h-index is derived from the pure citation counts depicted in Fig 4, so the graph is very similar.

    “Since per-case-study h-index is a metric derived from the pure citation counts depicted in Fig 4, it should come as no surprise that the graph is very similar.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Their scoped survey does not provide much insight into the extent of duplication across an author's entire back-catalogue of work.

    “Our scoped survey does not provide much insight into the extent of duplication across an author’s entire back-catalogue of work.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Haustein et al. (2014) suggest that papers are more likely to be tweeted if they are "curious or funny, have potential health applications or refer to a catastrophe".

    “Haustein et al. (2014) suggest that papers are more likely to be tweeted if they are “curious or funny, have potential health applications or refer to a catastrophe” [30].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [30] S Haustein, 2016, Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter, Journal of the Association for Information Science and Technology source

      Not checked in the free run.

38 of 38

  • The paper relies on this without support.

    Altmetric scores are a valid measure of social-web-based impact.

    “we chose to use Altmetric as our source as they provide an easy to use API which takes a paper’s PubMed or DOI identifier and produces a continuous score which could be plotted against REF and used in regression without any further manipulation.”

  • The authors state this as an assumption.

    All papers referenced by a REF case study are from within the same UoA as that REF case study.

    “This decision assumes that all papers referenced by a REF case study are from within the same UoA as that REF case study.”

  • The paper relies on this without support.

    The REF impact system currently in place is a very manual and resource intensive process.

    “The REF impact system currently in place is a very manual and resource intensive process.”

  • The paper relies on this without support.

    Existing analysis of policy documents is based on counting references in these documents to scholarly articles.

    “Furthermore, existing analysis of policy documents is based on counting references in these documents to scholarly articles.”

  • The paper relies on this without support.

    Altmetric.com and other altmetrics providers already examine some policy documents as part of their offerings.

    “Altmetric.com and other altmetrics providers already examine some policy documents as part of their offerings.”

  • The paper relies on this without support.

    Not all researchers in scientific domains covered by arXiv, CiteSeerX and PubMed Central publish to these repositories.

    “Not all researchers in scientific domains covered by arXiv, CiteSeerX and PubMed Central publish to these repositories.”

  • The paper relies on this without support.

    Papers need time to accumulate citations.

    “Papers need time to accumulate citations.”

  • The paper relies on this without support.

    h-index for REF case studies has not been calculated before.

    “We do not believe h-index for REF case studies has been calculated before.”

  • The paper relies on this without support.

    The REF guidelines stipulate that all supporting works must have been published in the ten year window starting in December 2003 and ending in December 2013.

    “The REF guidelines stipulate that all supporting works must have been published in the ten year window starting in December 2003 and ending in December 2013.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Stated ECN size conflicts with the reported size of one network

    Minor

    The paper describes all its ECNs together as being of the order of 2 million papers, but reports approximately 5.3 million papers in the CiteSeerX-derived network alone.

  • Reported mean case-study population does not match the group count

    Minor

    The paper reports 647 unique linked case studies and says the 235 result groups are disjoint. That gives 647 ÷ 235 ≈ 2.75 case studies per group, not 2.65.

    Reported 2.65 · recomputed 2.753191

  • Fig 4 is described as showing raw citation counts, but shows MNCS

    Minor

    The discussion of Fig 6 says Fig 4 depicts pure citation counts. The paper's caption identifies Fig 4 as a plot of mean normalised citation scores.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Lane J, Bertuzzi S. The STAR METRICS project: current and future uses for S&amp;E workforce data. In: Science of Science Measurement Workshop, held Washington DC; 2010.

Cited by

Sorted by citations

Related

Authors
Ravenscroft, J., Liakata, M., Clare, A., & Duma, D.
Affiliations
  • University of Warwick (GB)
  • Aberystwyth University (GB)
  • University of Edinburgh (GB)
Corresponding author
Ravenscroft, James
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2017-03-09
Volume
12
Issue
3
Pages
e0173152
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0173152
PMC
PMC5344357
OpenAlex
W2593028313
Semantic Scholar
7c6dd680030a3111409c45dfc28b1d9feebc25e3
Cited by
202
References
29
Citations by year
2026: 18 · 2025: 25 · 2024: 21 · 2023: 18 · 2022: 21 · 2021: 19 · 2020: 30 · 2019: 28 · 2018: 18 · 2017: 4
FWCI
19.56
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Engineering and Physical Sciences Research Council (1535875, 1643924)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
scientific impact measurement, citation metrics, Research Excellence Framework, h-index, I-index, citation networks, altmetrics, online attention, social media, media coverage, science policy, grant applications
MeSH terms
Achievement, Humans, Publishing, Science, Models, Statistical, Biomedical Research, Journal Impact Factor, Social Media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review