Over-optimization of academic publishing metrics: observing Goodhart's Law in action
Open accessThe DOI matches its Crossref record; 70 of the 77 references it cites were found in a registry or an index.
Abstract
Background: The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, “when a measure becomes a target, it ceases to be a good measure.” Results: In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions: Academic publishing has changed considerably; now we need to reconsider how we measure success.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















![Figure 16: The mean number of citations [Citation Number/Documents Number (2 years)] over time. The mean number of citations values have almost doubled in the past 18 years; additionally, their distributions have changed considerably.](https://figures.cochat.ai/paper-figures/03933677bbea423b8340291d60efccd5/over-optimization-of-academic-publishing-metrics-observing-goodhart-s-law-in-figure-16.jpg)






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As articles that describe miR functions have also been shown to be highly cited (60)”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [60]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, the number of academic papers published per year is more than 7 million (Fire and Guestrin, 2019).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Fire and Guestrin, 2019
In the citing paper
“Scientists likely feel increased pressure to hype their results because productivity metrics have taken on a greater role in scientific advancement (23).”
From Misinformation in and about science (2021) · cites this paper as [23]
In the citing paper
“Much of the present pathology of hype, hyperbole, and publication bias is associated with an overreliance on productivity metrics (23).”
From Misinformation in and about science (2021) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Observation of Gravitational Waves from a Binary Black Hole Merger
Abbott, B. P., Abbott, R., Abbott, T. D., Abernathy, M. R., Acernese, F., Ackley, K., Adams, C., Adams, T., Addesso, P., Adhikari, R. X., et al. (2016) · Physical Review Letters · doi:10.1103/physrevlett.116.061102 · Cited by 14,151 · cited 1×
“For example, the recent Laser Interferome-ter Gravitational-Wave Observatory paper [37] listed > 1,000 authors [38].”
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · cited 2×
“Another common metric to measure a researcher’s output or a journal’s impact is the h-index, which measures an author’s or a journal’s number of papers that have at least h citations each [3].”
BackgroundMethodThe History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · cited 1×
“Even with their many known shortcomings [8, 24, 55, 68– 70], measures such as the impact factor, citation number, and h-index are still widely used.”
ArnetMiner: extraction and mining of academic social networks
Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., & Su, Z. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1402008 · Cited by 2,231 · cited 1×
“The AMiner open academic graph dataset [77] contains data from > 154 million papers.”
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497 · cited 1×
“This research is a large-scale scientometrics study (also referred to as the “science of science” [30]).”
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 2×
“…of the measures can be compromised, and unwelcome and unethical behaviors may develop, such as salami publications [17], ghost author-ships [18], p-hacking [19], metrics manipulation [20], faking research data [21], faking of peer reviews [22], and even plagiariz-ing by a peer reviewer [23].”
Does the h index have predictive power?
Hirsch, J. E. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0707962104 · Cited by 973 · cited 1×
“It has been shown that the h-index can predict academic achievements [57].”
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957 · cited 2×
“Good-hart’s Law, and the closely related Campbell’s Law [11], influence many systems in our everyday life, including educational [11], biological [12], and other decision-making systems [13, 14].”
The “Gender Gap” in Authorship of Academic Medical Literature — A 35-Year Perspective
Jagsi, R., Guancial, E. A., Worobey, C. C., Henault, L. E., Chang, Y., Starr, R., Tarbell, N. J., & Hylek, E. M. (2006) · New England Journal of Medicine · doi:10.1056/nejmsa053910 · Cited by 870 · cited 2×
“Additionally, Gwilym Lockwood observed that “articles with positively-framed titles, interesting phrasing, and no word-play get more attention online” [45].”
Research works printed but not linked: 12
- 2003, Central Banking, Monetary Theory and Practice: Essays in honour of Charles Goodhart, 221
- Top publications. Google Scholar. https://scholar.google.com/citations?view_op=top_venues. Accessed 13 February 2019.
- Journal Citation Reports (JCR) https://jcr.clarivate.com/. Accessed 12 May 2019.
- arXiv Monthly Submission Rates https://arxiv.org/stats/monthly_submissions. Accessed 20 January 2019.
- Gálvez A, Maqueda M, Martinez-Bueno M, et al. Scientific publication trends and the developing world: What can the volume and authorship of scientific articles tell us about scientific progress in various regions?. American Sci. 2000; 88(6): 526– 33.
- Semantic Scholar https://www.semanticscholar.org. Accessed 14 February 2019.
- Google Scholar https://scholar.google.com. Accessed 13 February 2019.
- KDD Cup 2016: Whose papers are accepted the most: towards measuring the impact of research institutions. 2016 http://www.kdd.org/kdd-cup/view/kdd-cup-2016/Data.
- Semantic Scholar - An Overview of Microsoft Academic Service (MAS) and Applications https://www.semanticscholar.org/paper/An-Overview-of-Microsoft-Academic-Service-(MAS)-and-Sinha-Shen/b6b6d2504fd57d27a0467654fa62169cc7dedbdd?navId=citing-papers. Accessed 14 February 2019.
- Scimago Journal & Country Rank https://www.scimagojr.com/journalrank.php. Accessed 14 February 2019.
- Nature: Editorial Criteria and Processes https://www.nature.com/nature/for-authors/editorial-criteria-and-processes. Accessed 15 July 2018.
- Low Y, Gonzalez JE, Kyrola A, et al. Graphlab: A new framework for parallel machine learning. arXiv. 2014: 1006.4990.
Cited by
Items 101–200 of 356
Kang, P., Kim, S. J., Park, H., Han, S.-J., Kim, I.-C., Lee, H., & Yim, J. (2025) · Plants · doi:10.3390/plants14030447 · Cited by 5
“This exponential growth in research output has necessitated the development of more efficient methods for processing and extracting information from the ever-expanding corpus of published text data [40].”
Altmetrics in the evaluation of scholarly impact: a systematic and critical literature review
González, P., Fors, M. M., & Torres, A. (2025) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2025.1693304 · Cited by 22
Jódar, L., & de la Poza, E. (2020) · Sustainability · doi:10.3390/su12156064 · Cited by 12
Kiparoglou, V., Brown, L. A., McShane, H., Channon, K. M., & Shah, S. G. S. (2021) · Journal of Translational Medicine · doi:10.1186/s12967-021-03149-x · Cited by 13
“63), but it was clear that this figure was inflated by outliers, a few mega-author papers [17, 52].”
BackgroundMethodInfluentialHood, A. S. C., Shackelford, G. E., Christie, A. P., Usieta, H. O., Martin, P. A., & Sutherland, W. J. (2023) · Ecological Solutions and Evidence · doi:10.1002/2688-8319.12249 · Cited by 10
“(FAO, 2019), or it may follow the wider pattern of increasing publication rates in academia (Fire & Guestrin, 2019; Figure 4).”
BackgroundPerry, G. L. W., & McGlone, M. S. (2021) · New Zealand Journal of Ecology · doi:10.20417/nzjecol.45.12 · Cited by 11
“Across the biological sciences, there has been a trend for longer titles and abstracts (Fire & Guestrin 2019), which is also evident in the NZES journals (Figs 1 c, d).”
BackgroundRoss, C., Stewart, J., Nichol, L., Elliott, C. J., & Sambrook, S. A. (2022) · Human Resource Development International · doi:10.1080/13678868.2022.2121016 · Cited by 11
“These laws are also used by Fire and Guestrin (2019) in their analysis and critique of the metrics of academic publishing.”
MethodAnalysis of predatory emails in early career academia and attempts at prevention
Tomlinson, O. W. (2022) · Learned Publishing · doi:10.1002/leap.1500 · Cited by 11
“Academic publishing is evolving at an unprecedented rate, with increasing numbers of authors, publications and journals, as well as metrics to quantitively compare outputs, individuals and institutions (Fire & Guestrin, 2019).”
BackgroundA network-driven study of hyperprolific authors in computer science
Vieira, V. D. F., Ferreira, C. H. G., Almeida, J. M., Moreira, E., Laender, A. H. F., Meira, W., & Gonçalves, M. A. (2024) · Scientometrics · doi:10.1007/s11192-024-04940-5 · Cited by 10
“Recently observed phenomena, such as long author lists, short papers, increasing flood of papers, self-cita-tion and long lists of references, have affected the metrics for academic success and, as a consequence, the whole scientific environment (Fire & Guestrin, 2019).”
InfluentialDo you want your paper published? – Then referee others! The view from seasoned editors
Borja, Á., & Elliott, M. (2024) · Estuarine Coastal and Shelf Science · doi:10.1016/j.ecss.2024.109006 · Cited by 9
Hype in science communication: exploring scientists’ attitudes and practices in quantum physics
Soto-Sanfiel, M. T., Chong, C.-W., & Latorre, J. I. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05200-4 · Cited by 17
“In today ’ s competitive academic environment, scientists often hype their fi ndings to capture the attention of various stakeholders (Fire and Guestrin, 2019; Bucher, 2020), employing techniques such as simpli fi cation, exaggeration, and sensationalization (Intemann, 2022; Roberson, 2020; Brown,…”
Citation beneficiaries of discipline-specific mega-journals: who and how much
Li, J., Long, Q., Lu, X., & Wu, D. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-02050-w · Cited by 8
da Silva, J. A. T., Nazarovets, S., Daly, T. P., & Kendall, G. (2024) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2024.102898 · Cited by 12
Search engine optimization for scientific publications: How one can find your needle in the haystack
Brouwer, W. P., & Hollenbach, M. (2022) · United European Gastroenterology Journal · doi:10.1002/ueg2.12311 · Cited by 8
“Moreover, there is a growing number of publications each year with an accompanying growth in the number of scientific journals.(1,3) Still, when adjusted for a stable number of scientific journals, the number of publication per journal per year keeps rising.”
BackgroundAlón, I., Chebance, Z., Massucci, F. A., Bounartzi, T., & Ravitsky, V. (2023) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-023-02834-8 · Cited by 12
“At the same time, with the global growth in academic publications [5, 6], the volume of research output about both technical and nontechnical aspects of ART has expanded substantially [7].”
BackgroundA new framework for assessing the contributions of professionals in the natural sciences
Esposito, L. A., Daly, M., Fujita, M. K., Gorneau, J. A., Rapacciuolo, G., Rocha, L. A., Scheinberg, L. A., Ware, J. L., Welch, C. K., Young, A. N., et al. (2022) · Bulletin of the Society of Systematic Biologists · doi:10.18061/bssb.v1i1.8332 · Cited by 7
“…an inclusive model of science with an expanded view of impact that contributes broadly to a more equitable and accessible science (as suggested by Davies et al. 2021; see also discussions on drawbacks/limitations of existing academic ranking/metric systems in Gruber 2014; Fire and Guestrin 2019).”
BackgroundDiscovering AI adoption patterns from big academic graph data
Kim, S.-Y., Lee, W. K., Jee, S. J., & Sohn, S. Y. (2025) · Scientometrics · doi:10.1007/s11192-024-05228-4 · Cited by 11
“These advantages have led to the active use of the MAG database in recent studies (e.g.,AlShebli et al., 2018; Betancourt et al., 2023; Bu et al., 2021; Fire & Guestrin, 2019; Frank et al., 2019a, 2019b; Marx & Fuegi, 2020).”
English Language Education Policy and EMI in Higher Education in Taiwan
Lin, C.-Y. (2023) · Springer eBooks · doi:10.1007/978-981-19-9040-3_11 · Cited by 8
Mittas, N., Gkika, D. A., Georgiou, K., Alodhayb, A. N., AbdelAll, N., Khouqeer, G. A., & Kyzas, G. Z. (2023) · Environmental Science and Pollution Research · doi:10.1007/s11356-023-30752-w · Cited by 7
“The growing number of academic journals (Fire and Guestrin 2019) has further contributed to a “publish or perish” mentality among researchers, giving rise to the dilemma of “quality or quantity” in their studies.”
BackgroundThe National Academy of Kinesiology 2023 Evaluation of Doctoral Programs in Kinesiology
Knudson, D. V., Mahar, M. T., & Myers, N. D. (2023) · Kinesiology Review · doi:10.1123/kr.2023-0069 · Cited by 7
Afiaz, A., Ivanov, A. A., Chamberlin, J., Hanauer, D. A., Savonen, C., Goldman, M., Morgan, M. T., Reich, M. M., Getka, A., Holmes, A., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae469 · Cited by 11
“…but one way to minimize this effect is to continue to evaluate metrics over time, to consider if our metrics are truly measuring what we think they are, to consider if our metrics are actually fair to a diverse range of projects, and to consider new metrics as needed (Fire and Guestrin 2019).”
BackgroundZobel, J. (2022) · SIGIR Forum · doi:10.1145/3582524.3582540 · Cited by 8
“Some examples were given above: teaching scores for lecturers, manipulation (by governments) of the prices that are used to calculate inflation, or citation indices for researchers, an instance that has been explicitly examined as demonstrating Goodhart’s law [Fire and Guestrin, 2019].”
Coercive Citation: Understanding the Problem and Working Toward a Solution
Basil, D. Z., Burton, S., Soboleva, A., & Nesbit, P. (2023) · Academy of Management Perspectives · doi:10.5465/amp.2022.0081 · Cited by 10
Effective publication strategies in clinical research
Deutz, D. B., Vlachos, E., Drongstrup, D., Dorch, B. F., & Wien, C. N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0228438 · Cited by 8
“Research decisions may be influenced by the effect on a researcher’s publication metrics, rather than by scientific merit [2].”
Background彭晓芸, Li, F., Ma, Q., Tan, S., Jiang, H., Ye, X., Dong, J., You, F., Ren, Y., & Zheng, C. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1611304 · Cited by 11
pyBiblioNet: a Python library for a comprehensive network-based bibliometric analysis
Lai, M., Vilella, S., Cena, F., & Ruffo, G. (2025) · Scientometrics · doi:10.1007/s11192-025-05458-0 · Cited by 9
“…is becoming crucial, considering the ever-increasing growing rate of papers published every year, that saw an exponential surge since the 1990s Fire and Guestrin (2019), making it considerably harder for scientists to keep up-to-date in their field of research, and even harder for them to…”
Forero, D., Castro‐Huertas, V., Morales-Devia, H., Barão, K. R., Bianchi, F. M., Campos, L. A., Dellapé, P. M., Melo, M. C., & SCHWERTNER, C. F. (2024) · Anais da Academia Brasileira de Ciências · doi:10.1590/0001-3765202420230218 · Cited by 6
Trends in Publishing Muography Related Research: The Situation at the End of 2020
Holma, M., Joutsenvaara, J., & Kuusiniemi, P. K. P. (2022) · Journal of advanced instrumentation in science. · doi:10.31526/jais.2022.292 · Cited by 6
Dougherty, M. R. P., & Horne, Z. (2019) · doi:10.31234/osf.io/9g5wk · Cited by 6
Papucci, E., Valbuena, R., Roberge, C., Mensah, A. A., & Ståhl, G. (2026) · Forestry An International Journal of Forest Research · doi:10.1093/forestry/cpag007 · Cited by 11
Guía para escribir un artículo científico
Arias-Carrión, Ó. (2023) · Revista Española de Geriatría y Gerontología · doi:10.1016/j.regg.2023.101424 · Cited by 21
Clark, L. A., Clark, R., Raffo, D. M., & Williams, R. I. (2024) · Journal of Management Inquiry · doi:10.1177/10564926241235850 · Cited by 5
“In a detailed analysis of scienti fi c academic publishing, Fire and Guestrin (2019) found that the number of published academic articles in the Microsoft Academic Graph dataset now exceeds seven million yearly.”
Semantic Search Using LLM-Aided Topic Generation on Knowledge Graphs for Paper Discovery
Chow, S., Guo, L., Chow, J., Chia, C., Li, S., & Huang, D. (2024) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp63861.2024.10800417 · Cited by 8
“The number of academic papers has surged from 3 million at the turn of the century to 7 million in 2014 [13].”
Bibliometric Profile of an Emerging Journal: Participatory Educational Research
Arslan, R., Orbay, K., & Orbay, M. (2022) · Participatory Educational Research · doi:10.17275/per.22.84.9.4 · Cited by 9
“Today, advances in information technologies have made it possible and cheaply to access information easily and have increased the amount of information available by doubling every day (Fire & Guestrin, 2019).”
BackgroundMeier, F., Dixon, T. A., Williams, T. C., & Paulsen, I. T. (2023) · ACS Synthetic Biology · doi:10.1021/acssynbio.3c00192 · Cited by 5
A bibliometric analysis of publications on special education between 2011 and 2020
Arslan, R., Orbay, K., & Orbay, M. (2023) · The Hungarian Educational Research Journal · doi:10.1556/063.2023.00212 · Cited by 8
“” in terms of the conducted research (Civera, Lehmann, Paleari, & Stockinger, 2020; Fire & Guestrin, 2019; McGrail, Rickard, & Jones, 2006; Orbay et al., 2020, 2021; Van Dalen, 2021).”
Clubb, J., Allen, S. V., & Yung, K. K. (2023) · Strength and conditioning journal · doi:10.1519/ssc.0000000000000813 · Cited by 5
“Therefore, if indicator is imperfectly correlated inefficient application of resources. has been exemplified elsewhere in such as academic publishing (21), higher education (19), and government spending (29) to name a few.”
BackgroundTeamTree analysis: A new approach to evaluate scientific production
Pfrieger, F. W. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253847 · Cited by 6
A Multimodal Move Analysis of Graphical Abstracts in Medicine and Chemistry
Florek, C. S., & Hendges, G. R. (2023) · ESP Today · doi:10.18485/esptoday.2023.11.2.3 · Cited by 5
Faculty publication trends in a Japanese national university: a diachronic document analysis
Muller, T., Takano, M., & Gallagher, N. (2023) · Journal of Pharmaceutical Health Care and Sciences · doi:10.1186/s40780-023-00302-0 · Cited by 5
“[18] However, the met-rification of higher education [3] is also evident through increases in other publication types, such as conference and ‘other’ papers, illustrating how 2019–2020 faculty felt more expectations to publish than their 1979–1980 counterparts.”
BackgroundInfluentialDrinking from the Firehose? Write More and Publish Less
Marcum, C. S. (2024) · doi:10.54900/r8zwg-62003 · Cited by 5
Science as a global public good: A necessary responsibility for science
Boulton, G. S. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-23655-6.00001-0 · Cited by 4
Awais, M. (2025) · Communication & Sport · doi:10.1177/21674795251384300 · Cited by 4
“This approach aligns with arguments that academic visibility is linked to national and institutional contexts often overlooked in conventional evaluations (Fire & Guestrin, 2019; Tucker, 2018).”
Changing the Landscape of Archaeological Publishing
Beck, J., Flad, R. K., Alex, B., Caramanica, A., Costopoulos, A., Kawa, N. C., Marwick, B., Warinner, C., Bardolph, D. N., Borck, L., et al. (2026) · Current Anthropology · doi:10.1086/739789 · Cited by 4
“Over the past 60 years, the number of academic papers, scholarly journals, and published scholars has increased exponentially (Fire and Guestrin 2019), and in recent years, science megajournals such as PLoS One have churned out more than 10,000 publications annually (Brainard 2019).”
Mapping the composite cattle worldwide using bibliometric analysis
McManus, C., Pimentel, F., Ferraz, J. B. S., Núñez-Domínguez, R., Guimarães, R. F., Pimentel, D., Gama, L. T., Costa, N. D. S., & Peripolli, V. (2024) · Livestock Science · doi:10.1016/j.livsci.2024.105593 · Cited by 5
Short, But How Short? Analysis of Educational Research Titles
Orbay, K., Fernando, A. M., & Orbay, M. (2025) · SAGE Open · doi:10.1177/21582440251320538 · Cited by 6
“Every year, as mentioned above, the number of articles increases exponentially compared to the previous years (Fire & Guestrin, 2019; X. Hu et al., 2020; Ioannidis et al., 2018).”
Library science literature, 2019–2025: An exploration using critical bibliometric methods
Szydlowski, N. (2025) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2025.103142 · Cited by 8
Ranking journals through the lens of active visibility
Giri, R., & Chaudhuri, S. K. (2021) · Scientometrics · doi:10.1007/s11192-020-03850-6 · Cited by 6
Automatic literature screening using the PAJO deep-learning model for clinical practice guidelines
Lin, Y., Li, J., Xiao, H., Zheng, L., Xiao, Y., Song, H., Fan, J., Xiao, D., Ai, D., Fu, T., et al. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02328-8 · Cited by 4
“However, the number of articles is witnessing an exponentially growth, it is reported that over 120 million papers have been published so far [3].”
BackgroundHierarchical Catalogue Generation for Literature Review: A Benchmark
Zhu, K., Feng, X., Feng, X., Wu, Y., & Qin, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.453 · Cited by 26
Must we lose our biological connection to nature to endure changing times?
Berger, J., Bleich, V. C., & Bowyer, R. T. (2024) · Journal of Wildlife Management · doi:10.1002/jwmg.22639 · Cited by 4
Xue, H. (2024) · Information · doi:10.3390/info15100597 · Cited by 8
D’Angelo, C. A. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0005 · Cited by 9
Faghri, A., & Bergman, T. L. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00291 · Cited by 6
“…accounted for the adverse distorting effects of including self-citations (Pranckut ė , 2021; Vîiu, 2016) or giving full credit to each author for publications and citations (Batista, Campiteli et al., 2006; Fire & Guestrin, 2019; Liu, Yu et al., 2021; Van Hooydonk, 1997; Zeng, Shen et al., 2017).”
BackgroundIncacutipa-Limachi, D. J., Estrada-Araoz, E. G., Quispe-Mamani, Y. A., Chayña, E. T., & Mamani-Flores, A. (2024) · Data & Metadata · doi:10.56294/dm2024301 · Cited by 4
EPB turns 50 years old: An analytical tour of the last five decades
Crooks, A. T., Jiang, N., See, L., Alvanides, S., Arribas‐Bel, D., Wolf, L. J., & Batty, M. (2024) · Environment and Planning B Urban Analytics and City Science · doi:10.1177/23998083241246928 · Cited by 4
“This is due to a notable increase in submissions to the journal, a trend that is happening more generally (Fire and Guestrin, 2019) but was also accommodated by Sage in the form of publishing larger volumes and issues along with a more fl exible page count (See, 2024, this issue).”
BackgroundFischer, T. B. (2023) · Impact Assessment and Project Appraisal · doi:10.1080/14615517.2023.2207269 · Cited by 4
“over 7 million just 15 years later (Fire and Guestrin 2019).”
BackgroundRosetta Statements: simplifying FAIR knowledge graph construction with a user-centred approach
Vogt, L., Farfar, K. E., Karanth, P., Konrad, M., Oelen, A., Prinz, M., & Strömert, P. (2026) · Database · doi:10.1093/database/baag030 · Cited by 4
Sanabria, J., Egan, S. L., Masuda, R., Lee, A. J., Gibson, G. R., Nicholson, J. K., Wist, J., & Holmes, E. C. (2022) · Journal of Agricultural and Food Chemistry · doi:10.1021/acs.jafc.2c03597 · Cited by 4
Gendrin, A., Harrison, E., Noveras, J., Souliotis, L., Vince, H., Smit, I. A., Costa, F., Milward, D., Dimitrievska, S., Metcalfe, P., et al. (2024) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2024.100152 · Cited by 4
Private, Public, Personal: Shifting Patterns in Geospatial Data Sources in Geographic Research
Appiah, G., Kaufman, M., Cooney, B., & Andris, C. (2024) · Annals of the American Association of Geographers · doi:10.1080/24694452.2024.2394078 · Cited by 3
“Our study shows an increase in publications and funding opportunities for academic research (echoing Fire and Guestrin 2019) and that researchers continue to use government data sources, but their use of private-sector data sources has increased substantially in recent times.”
Smart Building Technologies for Fire Rescue: A QR Code-Enabled Notification System
Kuo, T.-W., & Lin, C. (2025) · Fire · doi:10.3390/fire8030114 · Cited by 3
Research on Contamination of Foods with Mercury Mining: A Ten-Year (2011-2020) Bibliometric Analysis
Zuas, O., Elishian, C., Darmayanti, N. T. E., Budiman, H., & Yaumidin, U. K. (2021) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/926/1/012058 · Cited by 3
Accounting for time dependency in meta‐analyses of concordance probability estimates
Schmid, M. C., Friede, T., Klein, N., & Weinhold, L. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1655 · Cited by 3
“The solid black line refers to the true C-index according to the data-generating process (Equation (1)) whereas the horizontal gray line refers to the pooled C-index estimate that would have been obtained from a standard random effects meta-analysis ignoring time dependency.”
BackgroundMethodInfluentialBreathing Life Into Meta-Analytic Methods
Allbritton, D. W., Gómez, P., Angele, B., Vasilev, M. R., & Perea, M. (2024) · Journal of Cognition · doi:10.5334/joc.389 · Cited by 5
The Citation Economy as a Site of Extraction for Surveillance Publishing
Colbert‐Lewis, D., Maminta, L., McElroy, K., Slaght, G., & Swartz, M. (2024) · Canadian Journal of Academic Librarianship · doi:10.33137/cjal-rcbu.v10.43293 · Cited by 4
An Approach to Trustworthy Article Ranking by NLP and Multi-Layered Analysis and Optimization
Li, C., Zhang, J., Chen, W., & Ma, X. (2025) · Algorithms · doi:10.3390/a18070408 · Cited by 3
Technologiemanagement in produzierenden Unternehmen: Grundlagen und aktuelle Lösungsansätze
Warschat, J., Ardilio, A., & Riedel, O. (2023) · Springer eBooks · doi:10.1007/978-3-642-45370-0_14-1 · Cited by 3
Enhancing Literature Review with LLM and NLP Methods. Algorithmic Trading Case.Preprint
Łaniewski, S., & Ślepaczuk, R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4996945 · Cited by 5
Drinking from the firehose? Write more and Publish Less (Version 2)
Marcum, C. S. (2025) · doi:10.54900/vr8ax-nz653 · Cited by 3
Are peer reviewers influenced by their work being cited?Preprint
Barnett, A. G. (2025) · eLife · doi:10.7554/elife.108748 · Cited by 3
“Reviewers should be equally happy with any citation to their work; however, some reviewers may prefer citations to indexed articles, as these are more likely to count towards their h-indices ( Fire and Guestrin, 2019 ).”
Joelving, F., Cabanac, G., & Labbé, C. (2025) · The Conversation · doi:10.64628/aai.339cv4r5a · Cited by 6
Pressure to publish introduces large‐language model risks
Johnson, T. F., Simmons, B. I., Millard, J. W., Strydom, T., Danet, A., Sweeny, A. R., & Evans, L. C. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14397 · Cited by 3
Bellantuono, L., Sasso, A. L., Amoroso, N., Monaco, A., Tangaro, S., & Bellotti, R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0323356 · Cited by 4
The Use of Technology in Primary School Science Education From 1974 to 2025: A Bibliometric Analysis
Uyanık, G. (2025) · European Journal of Education · doi:10.1111/ejed.70156 · Cited by 7
Mapping Lesson Study as a research field: a bibliometric review
Aas, H. K., Alstad, K., & Vasseljen, N. A. (2025) · International Journal for Lesson and Learning Studies · doi:10.1108/ijlls-09-2024-0218 · Cited by 3
Trends in Medical Student-Authored Publications, 2012-2022
Elliott, B. P., Gottschall, Z., Suppa, E., & Carmody, J. B. (2025) · Researchers Journal of Internal Medicine · doi:10.63495/7106384 · Cited by 3
Zhang, R., & Horvát, E.-Á. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757510 · Cited by 4
“Anyhow, increased visibility and the possibility of higher citation counts becomes salient considering the increasing vitality of publication metrics to a researcher’s success over the last century [27].”
The academic ecosystems: A systematic literature review
Lenart-Gansiniec, R. A., & Sułkowski, Ł. (2025) · Economics & Sociology · doi:10.14254/2071-789x.2025/18-3/14 · Cited by 3
Explosion behavior of hybrid mixtures
Spitzer, S. H., Amyotte, P. R., & Salzano, E. (2026) · Progress in Energy and Combustion Science · doi:10.1016/j.pecs.2025.101276 · Cited by 2
Mamedov, V. Y., Kovalevsky, D. A., Morozov, D. A., Stolyarov, S. S., & Ospichev, S. (2025) · Automatic Control and Computer Sciences · doi:10.3103/s0146411625700440 · Cited by 2
“The acceleration of this growth is significant: in 1980 a total of about one million articles were published, in the years leading to 2020, more than seven million articles were being published annually [2].”
Marcelino, R. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00197-z · Cited by 2
“Investigations of citation metrics and authorship practices demonstrate that current assessment systems create both opportunities and incentives for strategic manipulation [10].”
Improving acknowledgments sections to better credit research contributors
Holcombe, A. O., Pedersen, R., Lagisz, M., Pollo, P., Kovacs, M., & Hosseini, M. (2026) · Accountability in Research · doi:10.1080/08989621.2026.2681050 · Cited by 2
“Fire and Guestrin (2019) suggest that part of the solution to Goodhart’s Law with regard to authorship and citations is to develop new metrics.”
An Outsider’s Perspective on Why We Climb Mountains and Why Projects Like the i5k Matter
Molik, D. C. (2022) · Journal of Insect Science · doi:10.1093/jisesa/ieac038 · Cited by 2
Trend Summary by SCOPUS Literature Analysis on Machine Learning Assisted BIPV Research
Lee, J.-H., You, S., Kim, C. K., Oh, M., Kim, B., Kang, Y.-H., & Kim, H. (2022) · Journal of the Korean Solar Energy Society · doi:10.7836/kses.2022.42.3.013 · Cited by 2
A perspective on new dilemmas in scientific publishing in the time of a pandemic
Arjuman, A. (2023) · Indian Journal of Medical Microbiology · doi:10.1016/j.ijmmb.2023.100366 · Cited by 3
“A large-scale scientometric study [20] recently provided substantial evidence pertaining to an observable decline in the validity of academic metrics which have been routinely used in scientific publishing to indicate academic success, either at an individual level, in the capacity of an author or as an academic entit…”
BackgroundMeasured in a context: making sense of open access book data
Snijder, R. C. (2023) · Insights the UKSG journal · doi:10.1629/uksg.627 · Cited by 2
Recent Advances in the Behavioral Ecology of European Plethodontid Salamanders
Costa, A., Lunghi, E., Rosa, G., & Salvidio, S. (2023) · Animals · doi:10.3390/ani13233667 · Cited by 2
“Initially, we evaluated the existence of a recent trend in publications concerning the European genus Speleomantes , both in their absolute number and in relation to the continuous growth of global scientific production [39].”
Braet, F., & Taatjes, D. J. (2024) · Microscopy Today · doi:10.1093/mictod/qaae067 · Cited by 3
The complex ecosystem of hyperprolific authors
Almeida, J. M., Antelmi, A., Gonçalves, M. A., & Pellegrino, M. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05563-8 · Cited by 2
Transformative choices towards a sustainable academic publishing system
Kayal, M., Ballard, J., & Kayal, E. (2022) · Ideas in Ecology and Evolution · doi:10.24908/iee.2021.14.3.f · Cited by 2
“…on this highly profitable business model thanks to decades of growing public investments in science and globalization of scientific literature (Odlyzko 1997, Walker 1998, Smith 2006, Van Noorden 2013, Schmitt 2014, Larivière et al. 2015, Buranyi 2017, Copiello 2018, Fire and Guestrin 2019).”
Collaboration in Research for Rare Disease: How Nonprofits can be the Change Makers
(2022) · British Journal of Cancer Research · doi:10.31488/bjcr.174 · Cited by 2
Aydemir, G., Orbay, K., & Orbay, M. (2023) · Shanlax International Journal of Education · doi:10.34293/education.v11i2.4483 · Cited by 5
“…under a Creative Commons Attribution-ShareAlike 4.0 International License http://www.shanlaxjournals.com2 Advances in information technologies have made it possible to access information easily and have increased the amount of information available by doubling every day (Fire & Guestrin, 2019).”
BackgroundInfluentialVillar, R. N. (2019) · Journal of Hip Preservation Surgery · doi:10.1093/jhps/hnz034 · Cited by 2
“His words may have been focussed on the world of economics, but a recent publication on the metrics of academic publishing, by Fire and Guestrin [1], more than grabbed my attention and has shown me how Goodhart’s Law applies just as much to academia.”
BackgroundInfluentialCharacteristics of high‐impact agronomic journals
Zhang, Y., & Yu, Q. (2020) · Agronomy Journal · doi:10.1002/agj2.20320 · Cited by 2
“New methods, agricultural models, and software platforms continue to be developed regarding various agronomic applications and goals (Folberth et al., 2019; Maes & Steppe, 2019; Shelia et al., 2019).”
MethodMeasuring serendipity with altmetrics and randomness
Pacher, A. (2022) · Journal of Librarianship and Information Science · doi:10.1177/09610006221124338 · Cited by 2
“Cf. discussions about the role of Goodhart’s Law—“when a measure becomes a target, it ceases to be a good measure”— in academic publishing by Fire and Guestrin (2019).”
BackgroundProxyeconomics, the inevitable corruption of proxy-based competitionPreprint
Braganza, O. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1803.00345 · Cited by 5
“The current revolution in data-science promises to provide ample testing ground for the present theory [9].”
BackgroundPerbal, B. (2023) · Journal of Cell Communication and Signaling · doi:10.1007/s12079-023-00796-1 · Cited by 2
“Since the pros and cons of the various tools developed over the past years, have been widely discussed in the scientific literature (see for instance, Fire and Guestrin 2019, we will only focus here on two different types of evaluation.”
BackgroundLim, W. M., & Manrai, A. K. (2026) · Journal of Global Marketing · doi:10.1080/08911762.2026.2678025 · Cited by 2
Designing for Audience Engagement Exploring the Use of Online Metrics in the GLAM Sector
Charlesworth, E., Warwick, C., Impett, L., & Beresford, A. (2023) · 4 | 1 | 2023 Relations · doi:10.30687/mag/2724-3923/2023/07/005
Metadata
- Authors
- Fire, M., & Guestrin, C.
- Affiliations
- Ben-Gurion University of the Negev (IL)
- University of Washington (US)
- Corresponding author
- Fire, Michael
- Guestrin, Carlos
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-05-30
- Volume
- 8
- Issue
- 6
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC6541803
- OpenAlex
- W2890107153
- Cited by
- 351
- References
- 77
- Citations by year
- 2026: 52 · 2025: 74 · 2024: 65 · 2023: 65 · 2022: 46 · 2021: 18 · 2020: 20 · 2019: 9
- FWCI
- 15.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Washington Research Foundation
- University of Washington
- Amazon Web Services
- Microsoft Azure Research Award
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Goodhart's law, citation metrics, impact factor, h-index, publication counts, self-citation, author list, research evaluation, biology research, bibliometric analysis
- MeSH terms
- Publishing, Bibliometrics, Benchmarking, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.