Over-optimization of academic publishing metrics: observing Goodhart's Law in action
Open accessThe DOI matches its Crossref record; 70 of the 77 references it cites were found in a registry or an index.
Abstract
Background: The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, “when a measure becomes a target, it ceases to be a good measure.” Results: In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions: Academic publishing has changed considerably; now we need to reconsider how we measure success.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















![Figure 16: The mean number of citations [Citation Number/Documents Number (2 years)] over time. The mean number of citations values have almost doubled in the past 18 years; additionally, their distributions have changed considerably.](https://figures.cochat.ai/paper-figures/03933677bbea423b8340291d60efccd5/over-optimization-of-academic-publishing-metrics-observing-goodhart-s-law-in-figure-16.jpg)






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As articles that describe miR functions have also been shown to be highly cited (60)”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [60]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, the number of academic papers published per year is more than 7 million (Fire and Guestrin, 2019).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Fire and Guestrin, 2019
In the citing paper
“Scientists likely feel increased pressure to hype their results because productivity metrics have taken on a greater role in scientific advancement (23).”
From Misinformation in and about science (2021) · cites this paper as [23]
In the citing paper
“Much of the present pathology of hype, hyperbole, and publication bias is associated with an overreliance on productivity metrics (23).”
From Misinformation in and about science (2021) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Observation of Gravitational Waves from a Binary Black Hole Merger
Abbott, B. P., Abbott, R., Abbott, T. D., Abernathy, M. R., Acernese, F., Ackley, K., Adams, C., Adams, T., Addesso, P., Adhikari, R. X., et al. (2016) · Physical Review Letters · doi:10.1103/physrevlett.116.061102 · Cited by 14,151 · cited 1×
“For example, the recent Laser Interferome-ter Gravitational-Wave Observatory paper [37] listed > 1,000 authors [38].”
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · cited 2×
“Another common metric to measure a researcher’s output or a journal’s impact is the h-index, which measures an author’s or a journal’s number of papers that have at least h citations each [3].”
BackgroundMethodThe History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · cited 1×
“Even with their many known shortcomings [8, 24, 55, 68– 70], measures such as the impact factor, citation number, and h-index are still widely used.”
ArnetMiner: extraction and mining of academic social networks
Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., & Su, Z. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1402008 · Cited by 2,231 · cited 1×
“The AMiner open academic graph dataset [77] contains data from > 154 million papers.”
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497 · cited 1×
“This research is a large-scale scientometrics study (also referred to as the “science of science” [30]).”
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 2×
“…of the measures can be compromised, and unwelcome and unethical behaviors may develop, such as salami publications [17], ghost author-ships [18], p-hacking [19], metrics manipulation [20], faking research data [21], faking of peer reviews [22], and even plagiariz-ing by a peer reviewer [23].”
Does the h index have predictive power?
Hirsch, J. E. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0707962104 · Cited by 973 · cited 1×
“It has been shown that the h-index can predict academic achievements [57].”
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957 · cited 2×
“Good-hart’s Law, and the closely related Campbell’s Law [11], influence many systems in our everyday life, including educational [11], biological [12], and other decision-making systems [13, 14].”
The “Gender Gap” in Authorship of Academic Medical Literature — A 35-Year Perspective
Jagsi, R., Guancial, E. A., Worobey, C. C., Henault, L. E., Chang, Y., Starr, R., Tarbell, N. J., & Hylek, E. M. (2006) · New England Journal of Medicine · doi:10.1056/nejmsa053910 · Cited by 870 · cited 2×
“Additionally, Gwilym Lockwood observed that “articles with positively-framed titles, interesting phrasing, and no word-play get more attention online” [45].”
Research works printed but not linked: 12
- 2003, Central Banking, Monetary Theory and Practice: Essays in honour of Charles Goodhart, 221
- Top publications. Google Scholar. https://scholar.google.com/citations?view_op=top_venues. Accessed 13 February 2019.
- Journal Citation Reports (JCR) https://jcr.clarivate.com/. Accessed 12 May 2019.
- arXiv Monthly Submission Rates https://arxiv.org/stats/monthly_submissions. Accessed 20 January 2019.
- Gálvez A, Maqueda M, Martinez-Bueno M, et al. Scientific publication trends and the developing world: What can the volume and authorship of scientific articles tell us about scientific progress in various regions?. American Sci. 2000; 88(6): 526– 33.
- Semantic Scholar https://www.semanticscholar.org. Accessed 14 February 2019.
- Google Scholar https://scholar.google.com. Accessed 13 February 2019.
- KDD Cup 2016: Whose papers are accepted the most: towards measuring the impact of research institutions. 2016 http://www.kdd.org/kdd-cup/view/kdd-cup-2016/Data.
- Semantic Scholar - An Overview of Microsoft Academic Service (MAS) and Applications https://www.semanticscholar.org/paper/An-Overview-of-Microsoft-Academic-Service-(MAS)-and-Sinha-Shen/b6b6d2504fd57d27a0467654fa62169cc7dedbdd?navId=citing-papers. Accessed 14 February 2019.
- Scimago Journal & Country Rank https://www.scimagojr.com/journalrank.php. Accessed 14 February 2019.
- Nature: Editorial Criteria and Processes https://www.nature.com/nature/for-authors/editorial-criteria-and-processes. Accessed 15 July 2018.
- Low Y, Gonzalez JE, Kyrola A, et al. Graphlab: A new framework for parallel machine learning. arXiv. 2014: 1006.4990.
Cited by
Items 301–356 of 356
Qussini, S., Hammoudeh, S., Shahbal, S., Singh, R., & Dierickx, K. (2025) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/15562646251395350
“These dynamics have contributed to a notable rise in publication counts and the emergence of hyperauthorship – a phenomenon where research articles are authored by hundreds to thousands of authors (Fire & Guestrin, 2019).”
Introducing the GM-Index: A Metric for Evaluating Research Quality and EfficiencyPreprint
González‐Masip, J. J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5813528
Are peer reviewers influenced by their work being cited?Preprint
Barnett, A. G. (2025) · eLife · doi:10.7554/elife.108748.3
Are peer reviewers influenced by their work being cited?Preprint
Barnett, A. G. (2025) · eLife · doi:10.7554/elife.108748.2
Muller, T., Takano, M., & Gallagher, N. (2024) · Figshare · doi:10.6084/m9.figshare.26616931
Kiparoglou, V., Brown, L. A., McShane, H., Channon, K. M., & Shah, S. G. S. (2021) · Figshare · doi:10.6084/m9.figshare.17088206
L. Macartney, E., M. Drobniak, S., Nakagawa, S., & Lagisz, M. (2023) · Figshare · doi:10.6084/m9.figshare.22604995
L. Macartney, E., M. Drobniak, S., Nakagawa, S., & Lagisz, M. (2023) · Figshare · doi:10.6084/m9.figshare.22604998
Drinking from the Firehose? Write More and Publish Less
Marcum, C. (2024) · doi:10.59350/4qy2c-y7q31
Are peer reviewers influenced by their work being cited?
Barnett, A. G. (2025) · eLife · doi:10.7554/elife.108748.4
Surugue, L., & Rivière, P. (2025) · Droit Santé et Société · doi:10.54695/dsso.121.0038
Brewing Epidemic of Research Publications in Medical Colleges: Critical Commentary
Nagaraja, M. S. (2025) · Healthline · doi:10.51957/healthline_811_2025
Lendvai, G., & Sasvári, P. (2026) · Magyar Tudomány · doi:10.1556/2065.187.2026.2.13
Tracking Scientific Publications in Non-traditional Academic Medical Centers
Paleoudis, E. G. (2026) · Cureus · doi:10.7759/cureus.103708
“For decades, publications have been considered the “universal currency” [1] even if the collected and reported metrics, including the number of publications, citation number, and impact factor, have been widely criticized in more recent times [2].”
Activity and Resilience: How Personality Measures Predict Career Success in Academia
Junker, T. L., Born, M. P., & Butter, R. (2026) · Journal of Business and Psychology · doi:10.1007/s10869-026-10106-2 · Cited by 1
Fang, Z., & Zhao, K. (2026) · International Conference on Human-Computer Interaction, Neural Networks and Deep Learning · doi:10.1145/3795892.3795906
“Therefore, this paper 10].”
Engagement in Open and Closed Research Cultures
Gasson, S., & Wijaya, S. W. (2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83549-089-120251010
On the Vulnerability of Citation Metrics in the Era of Generative Artificial Intelligence
Smarsly, K. (2026) · Publications · doi:10.3390/publications14020023 · Cited by 1
“In particular, Google Scholar, widely favored by researchers due to extensive coverage and rapid indexing, is comparatively susceptible to intentional manipulations compared to curated databases (Fire & Guestrin, 2019).”
Ethics in the evaluation of the quality of scientific research: An AIOM-TJ roundtable
Adani, A. V., Crosetto, P., Galbiati, L., Giglia, E., Scarpa, A., & Perrone, F. (2026) · Tumori Journal · doi:10.1177/03008916261441119
Gatekeepers through time: Peer review and the making of nursing knowledge
Iheduru-Anderson, K. C., Waite, R. L., & Valderama‐Wallace, C. (2026) · Nursing Outlook · doi:10.1016/j.outlook.2026.102788
Yeung, A. W. K., & Tsoi, J. K. H. (2026) · International Dental Journal · doi:10.1016/j.identj.2026.109591
Student Engagement in IT Programs: What NSSE Captures and What It Misses
Wolf, J. R., & Mahatanankoon, P. (2026) · Information Systems Education Journal · doi:10.62273/jwos5934
Panigrahi, S. K., Priyadarshini, V., Pradhan, S., & Behera, S. (2026) · F1000Research · doi:10.12688/f1000research.168615.1
Why aren’t we publishing more? The narrative bottleneck in evolutionary biology
Gossmann, T. I. (2026) · BioScience · doi:10.1093/biosci/biag055
Winter bioenergetics in freshwater fishes: Knowledge gaps and model evaluation
Reeve, C., Zrini, Z. A., Bihun, C. J., Rowsey, L. E., Murphy, M. K., Speers‐Roesch, B., Brownscombe, J. W., Cooke, S. J., & Raby, G. D. (2026) · Fisheries Research · doi:10.1016/j.fishres.2026.107743
Kalita, A., Saikia, A. P., Dutta, A., & Satvat, P. S. (2026) · Discover Education · doi:10.1007/s44217-026-01716-2
“Although easy to monitor, these metrics are susceptible to inflation through volume rather than quality, creating integrity risks and masking true research excellence [4, 7].”
Gómez, P. A. M., Nuñez, J. P. S. C., Paucca-Calla, K. J., Minchón-Medina, C. A., Castro-Delgado, V. L., Sotelo-Llancari, S., Ignacio-Punin, C., Rodríguez-Salazar, D. Y., & Urrutia-Baca, V. H. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1842222
“Numerous studies warn that equating productivity with quality can lead to fragmented publications, inflated co-authorship, or strategic citation practices that undermine scientific integrity (Fire and Guestrin, 2018; Sandström and van den Besselaar, 2016).”
Testing for alphabetical bias in the c-score: Implications for evaluating scientific excellence
D’Angelo, C. A. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101841
Fear of Missing Out in Academic Workers: A Theoretical Framework
Bozzetti, M., Cascio, A. L., & Napolitano, D. (2026) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.70338
“…core concepts and consequences of academic FoMO. to intensify these dynamics by framing nonparticipation in key opportunities as a potential professional liability; similarly, career insecurity has been shown to predict both informational and relational FoMO in work settings [ 10, 20, 25 ].”
Computer vision to attendance recognition: a bibliometric analysis
Folador, J. P., Martins, J. B., & Oliveira, F. H. M. (2026) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2026.6103
On the Curse of Publication Metrics
Thorne, S. E. (2026) · Nursing Inquiry · doi:10.1111/nin.70147
“They have become so pervasive as the lingua franca of academics that we forget how pernicious and perverse they can be (Fire and Guestrin 2019).”
Aneaus, S., Rashid, I., Loan, F. A., & Charoo, S. A. (2026) · Metrics · doi:10.3390/metrics3030016
“The WoS databases represent one of the largest available resources, widely utilized for bibliometric analysis and the examination of research and publishing trends across various fields [61–63].”
VILLA: V ersatile I nformation Retrieval from Scientific L iterature Using Large La nguage Models
Antony, B., Dutta, A., Aggarwal, S., Gatne, V., Gökdemir, O., Grimes, S. L., Lauring, A. S., Wasik, B. R., Karpatne, A., & Murali, T. M. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3819035
Global patterns and regional disparities in library and information science research productivity
Adetayo, A. J. (2026) · IFLA Journal · doi:10.1177/03400352261470844
Existing guidelines and training resources for creating Narrative CVs: a Scoping Review
Desai, P., Alaoui, A., Armond, A. C. V., Moher, D., & Cobey, K. D. (2026) · F1000Research · doi:10.12688/f1000research.187182.1
Science as a global public good: responding to challenges of the modern era
Boulton, G. S. (2026) · Royal Society Open Science · doi:10.1098/rsos.260764
Adeosun, S. O., & Lemmons, F. L. (2026) · American Journal of Pharmaceutical Education · doi:10.1016/j.ajpe.2026.102073
Sawar, K., Mekani, L., Hannoudi, A., Atto, J. P., & Atisha, D. M. (2026) · Journal of surgical education · doi:10.1016/j.jsurg.2026.104128
Indexability: The Citeable Scientific Author-ity in the Making
Hofbauer, S. (2026) · Springer eBooks · doi:10.1007/978-981-92-1483-9_14
Personal Research Ontologies for Personal Open Science
Oppenlaender, J. (2026) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3834581.3838630
Cai, S., & Wang, X. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101878
Estimating Current H-Index Using Explainable Machine Learning and Diverse Bibliometric Features
Ajram, M., Nasser, A., Khalife, D., Zaki, C., Kheirbek, A., & Trocan, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-37936-8_13
A General Model for Solving the Surrogate-Endpoint Problem in Architecture Assessment
Stemmons, R. A. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.08.194
Suarez-Rodríguez, C. H., Garzón-García, A. M., & Largo-Ávila, E. (2026) · Publications · doi:10.3390/publications14030060
Rethinking Academic Impact: A Context-Constrained Framework for Publication Strategy and Evaluation
Currie, G. M. (2026) · Journal of Nuclear Medicine Technology · doi:10.2967/jnmt.126.273072
Reis, E., Paulino, A. A., & Albertini, M. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_10
Cavero, J. M., & Vela, B. (2026) · Research Square · doi:10.21203/rs.3.rs-10983491/v1
Davidovitch, N., & Shiller, Z. (2026) · Education Sciences · doi:10.3390/educsci16101654
Suspected distortion of citations in high-impact cancer journals
Scancar, B., Byrne, J. A., Causeur, D., & Barnett, A. G. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2618402123 · Cited by 3
Research Funding, Costs, and Integrity in Scientific Publication
Sannita, W. G. (2026) · Journal of Psychophysiology · doi:10.1027/0269-8803/a000366
Ogunyemi, A. A., Bauters, M., & Orisadare, E. A. (2026) · Research Square · doi:10.21203/rs.3.rs-11120204/v1
Scientific slop: generative AI, research incentives, and the integrity of the genomic knowledge base
Greenbaum, D. (2026) · Frontiers in Genetics · doi:10.3389/fgene.2026.1894137
Wang, Y., Cao, X., Li, T., Zhang, X., Wang, F., Xu, Z., Xu, Y., Yang, H., & Liang, W. (2026) · Chinese general practice journal. · doi:10.1016/j.cgpj.2026.100135
Metadata
- Authors
- Fire, M., & Guestrin, C.
- Affiliations
- Ben-Gurion University of the Negev (IL)
- University of Washington (US)
- Corresponding author
- Fire, Michael
- Guestrin, Carlos
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-05-30
- Volume
- 8
- Issue
- 6
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC6541803
- OpenAlex
- W2890107153
- Cited by
- 351
- References
- 77
- Citations by year
- 2026: 52 · 2025: 74 · 2024: 65 · 2023: 65 · 2022: 46 · 2021: 18 · 2020: 20 · 2019: 9
- FWCI
- 15.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Washington Research Foundation
- University of Washington
- Amazon Web Services
- Microsoft Azure Research Award
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Goodhart's law, citation metrics, impact factor, h-index, publication counts, self-citation, author list, research evaluation, biology research, bibliometric analysis
- MeSH terms
- Publishing, Bibliometrics, Benchmarking, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.