CoChat

Over-optimization of academic publishing metrics: observing Goodhart's Law in action

Open access
Fire, Michael, Guestrin, Carlos · 2019 · GigaScience

The DOI matches its Crossref record; 70 of the 77 references it cites were found in a registry or an index.

Abstract

Background: The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, “when a measure becomes a target, it ceases to be a good measure.” Results: In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions: Academic publishing has changed considerably; now we need to reconsider how we measure success.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/gigascience/giz053

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1: The number of papers over time. The total number of papers has surged exponentially over the years.
Figure 1: The number of papers over time. The total number of papers has surged exponentially over the years.CC BY · PubMed Central
Figure 2: Papers with titles in the top 9 non-English languages. An increasing number of papers have non-English titles.
Figure 2: Papers with titles in the top 9 non-English languages. An increasing number of papers have non-English titles.CC BY · PubMed Central
Figure 3: Mean title length over time. A paper’s mean title length increased from 8.71 words to 11.83 words. Moreover, the mean word length increased from 5.95 characters to 6.6 characters per title word.
Figure 3: Mean title length over time. A paper’s mean title length increased from 8.71 words to 11.83 words. Moreover, the mean word length increased from 5.95 characters to 6.6 characters per title word.CC BY · PubMed Central
Figure 4: Percentage of papers with author lists in alphabetical order, grouped by the number of authors. The more authors, the less likely the authors will be listed alphabetically in the byline.
Figure 4: Percentage of papers with author lists in alphabetical order, grouped by the number of authors. The more authors, the less likely the authors will be listed alphabetically in the byline.CC BY · PubMed Central
Figure 5: Distribution over time of the number of words in abstracts. Over time, papers’ abstracts have tended to become longer.
Figure 5: Distribution over time of the number of words in abstracts. Over time, papers’ abstracts have tended to become longer.CC BY · PubMed Central
Figure 6: The number and percentage of multidisciplinary papers over time. Between 1900 and 2010, both the number and percentage of multidisciplinary papers increased.
Figure 6: The number and percentage of multidisciplinary papers over time. Between 1900 and 2010, both the number and percentage of multidisciplinary papers increased.CC BY · PubMed Central
Figure 7: The mean and maximal number of self-citations. Both the mean and maximal number of self-citations increased over time.
Figure 7: The mean and maximal number of self-citations. Both the mean and maximal number of self-citations increased over time.CC BY · PubMed Central
Figure 8: Paper's lengths. Both the papers’ mean and median lengths decreased over time. In the right panel, the horizontal line indicates the median, and the box encloses the interquartile range.
Figure 8: Paper's lengths. Both the papers’ mean and median lengths decreased over time. In the right panel, the horizontal line indicates the median, and the box encloses the interquartile range.CC BY · PubMed Central
Figure 9: Papers with no citations other than self-citations after 5 years. The percentage of papers with no citations after 5 years decreased; nevertheless, 72.1% of all papers published in 2009 had no citations after 5 years.
Figure 9: Papers with no citations other than self-citations after 5 years. The percentage of papers with no citations after 5 years decreased; nevertheless, 72.1% of all papers published in 2009 had no citations after 5 years.CC BY · PubMed Central
Figure 10: Citation distributions over time. The citation distributions of different decades show notable changes.
Figure 10: Citation distributions over time. The citation distributions of different decades show notable changes.CC BY · PubMed Central
Figure 11: Mean number of papers by authors’ academic birth decades. With each decade, the rate of paper publication has increased.
Figure 11: Mean number of papers by authors’ academic birth decades. With each decade, the rate of paper publication has increased.CC BY · PubMed Central
Figure 12: Mean number of co-authors by academic birth decade. The mean number of co-authors has considerably increased over the decades.
Figure 12: Mean number of co-authors by academic birth decade. The mean number of co-authors has considerably increased over the decades.CC BY · PubMed Central
Figure 13: Percentage of times researcher was first author. We can observe that over time on average the percentage of senior researchers as first authors declined. Moreover, in the same time intervals, the percentage of times recent generations of researchers were first authors declined compared to older generations.
Figure 13: Percentage of times researcher was first author. We can observe that over time on average the percentage of senior researchers as first authors declined. Moreover, in the same time intervals, the percentage of times recent generations of researchers were first authors declined compared to older generations.CC BY · PubMed Central
Figure 14: Number of active journals over time. Over a period of 18 years, from 1999 to 2016, both the number of active journals and the papers per journal increased greatly.
Figure 14: Number of active journals over time. Over a period of 18 years, from 1999 to 2016, both the number of active journals and the papers per journal increased greatly.CC BY · PubMed Central
Figure 15: Journals’ quartile number of papers over time. The number of papers published in Q1 journals has vastly increased.
Figure 15: Journals’ quartile number of papers over time. The number of papers published in Q1 journals has vastly increased.CC BY · PubMed Central
Figure 16: The mean number of citations [Citation Number/Documents Number (2 years)] over time. The mean number of citations values have almost doubled in the past 18 years; additionally, their distributions have changed considerably.
Figure 16: The mean number of citations [Citation Number/Documents Number (2 years)] over time. The mean number of citations values have almost doubled in the past 18 years; additionally, their distributions have changed considerably.CC BY · PubMed Central
Figure 17: Top-selected journals’ mean first and last authors ages. Both the first and last authors’ mean ages have increased sharply.
Figure 17: Top-selected journals’ mean first and last authors ages. Both the first and last authors’ mean ages have increased sharply.CC BY · PubMed Central
Figure 18: Percentage of papers with returning first or last authors. The percentage of returning first or last top-journal authors increased considerably.
Figure 18: Percentage of papers with returning first or last authors. The percentage of returning first or last top-journal authors increased considerably.CC BY · PubMed Central
Figure 19: Mean percentage of return authors in top-selected journals over time. In most journals the number of papers with ≥1 author who previously published in the journal increased sharply. In many of the selected journals the percentage of papers with returning authors was >60%, and in some cases >80%.
Figure 19: Mean percentage of return authors in top-selected journals over time. In most journals the number of papers with ≥1 author who previously published in the journal increased sharply. In many of the selected journals the percentage of papers with returning authors was >60%, and in some cases >80%.CC BY · PubMed Central
Figure 20: L0 Fields-of-study number of papers over time. The numbers of papers in each field of study have increased drastically.
Figure 20: L0 Fields-of-study number of papers over time. The numbers of papers in each field of study have increased drastically.CC BY · PubMed Central
Figure 21: L0 Field-of-study median citation number after 5 years. There is notable variance among the L0 fields-of-study median citation numbers.
Figure 21: L0 Field-of-study median citation number after 5 years. There is notable variance among the L0 fields-of-study median citation numbers.CC BY · PubMed Central
Figure 22: Measuring success in academic publishing.
Figure 22: Measuring success in academic publishing.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 70 references

Research works printed but not linked: 6

  1. 2003, Central Banking, Monetary Theory and Practice: Essays in honour of Charles Goodhart, 221
  2. Francescani, 2012, NYPD report confirms manipulation of crime stats, Reuters
  3. Economist, 2016, Why research papers have so many authors, The Economist
  4. Gálvez, 2000, Scientific publication trends and the developing world: What can the volume and authorship of scientific articles tell us about scientific progress in various regions?, American Sci, 88, 526
  5. 2016, KDD Cup 2016: Whose papers are accepted the most: towards measuring the impact of research institutions
  6. Low, 2014, Graphlab: A new framework for parallel machine learning, 1006.4990

Cited by

See all 356

Metadata

Authors
Fire, M., & Guestrin, C.
Affiliations
  • Ben-Gurion University of the Negev (IL)
  • University of Washington (US)
Corresponding author
  • Fire, Michael
  • Guestrin, Carlos
Venue
GigaScience
Published
2019-05-30
Volume
8
Issue
6
ISSN
2047-217X (ISSN-L)
OpenAlex
W2890107153
Cited by
351
References
77
Citations by year
2026: 52 · 2025: 74 · 2024: 65 · 2023: 65 · 2022: 46 · 2021: 18 · 2020: 20 · 2019: 9
FWCI
15.56
Type
journal article
Language
en
Open access
Locations
  • GigaScience (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Washington Research Foundation
  • University of Washington
  • Amazon Web Services
  • Microsoft Azure Research Award
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
Goodhart's law, citation metrics, impact factor, h-index, publication counts, self-citation, author list, research evaluation, biology research, bibliometric analysis
MeSH terms
Publishing, Bibliometrics, Benchmarking, Journal Impact Factor

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Over-optimization of academic publishing metrics: observing Goodhart's Law in action. https://app.cochat.ai/papers/10.1093/gigascience/giz053
The paper
Fire, M., & Guestrin, C. (2019). Over-optimization of academic publishing metrics: observing Goodhart's Law in action. GigaScience. https://doi.org/10.1093/gigascience/giz053

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.