Over-optimization of academic publishing metrics: observing Goodhart's Law in action
Open accessThe DOI matches its Crossref record; 70 of the 77 references it cites were found in a registry or an index.
Abstract
Background: The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart's Law, according to which, “when a measure becomes a target, it ceases to be a good measure.” Results: In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal's impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions: Academic publishing has changed considerably; now we need to reconsider how we measure success.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















![Figure 16: The mean number of citations [Citation Number/Documents Number (2 years)] over time. The mean number of citations values have almost doubled in the past 18 years; additionally, their distributions have changed considerably.](https://figures.cochat.ai/paper-figures/03933677bbea423b8340291d60efccd5/over-optimization-of-academic-publishing-metrics-observing-goodhart-s-law-in-figure-16.jpg)






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As articles that describe miR functions have also been shown to be highly cited (60)”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [60]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, the number of academic papers published per year is more than 7 million (Fire and Guestrin, 2019).”
From ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025) · cites this paper as Fire and Guestrin, 2019
In the citing paper
“Scientists likely feel increased pressure to hype their results because productivity metrics have taken on a greater role in scientific advancement (23).”
From Misinformation in and about science (2021) · cites this paper as [23]
In the citing paper
“Much of the present pathology of hype, hyperbole, and publication bias is associated with an overreliance on productivity metrics (23).”
From Misinformation in and about science (2021) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Observation of Gravitational Waves from a Binary Black Hole Merger
Abbott, B. P., Abbott, R., Abbott, T. D., Abernathy, M. R., Acernese, F., Ackley, K., Adams, C., Adams, T., Addesso, P., Adhikari, R. X., et al. (2016) · Physical Review Letters · doi:10.1103/physrevlett.116.061102 · Cited by 14,151 · cited 1×
“For example, the recent Laser Interferome-ter Gravitational-Wave Observatory paper [37] listed > 1,000 authors [38].”
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · cited 2×
“Another common metric to measure a researcher’s output or a journal’s impact is the h-index, which measures an author’s or a journal’s number of papers that have at least h citations each [3].”
BackgroundMethodThe History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · cited 1×
“Even with their many known shortcomings [8, 24, 55, 68– 70], measures such as the impact factor, citation number, and h-index are still widely used.”
ArnetMiner: extraction and mining of academic social networks
Tang, J., Zhang, J., Yao, L., Li, J., Zhang, L., & Su, Z. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1402008 · Cited by 2,231 · cited 1×
“The AMiner open academic graph dataset [77] contains data from > 154 million papers.”
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497 · cited 1×
“This research is a large-scale scientometrics study (also referred to as the “science of science” [30]).”
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 2×
“…of the measures can be compromised, and unwelcome and unethical behaviors may develop, such as salami publications [17], ghost author-ships [18], p-hacking [19], metrics manipulation [20], faking research data [21], faking of peer reviews [22], and even plagiariz-ing by a peer reviewer [23].”
Does the h index have predictive power?
Hirsch, J. E. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0707962104 · Cited by 973 · cited 1×
“It has been shown that the h-index can predict academic achievements [57].”
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957 · cited 2×
“Good-hart’s Law, and the closely related Campbell’s Law [11], influence many systems in our everyday life, including educational [11], biological [12], and other decision-making systems [13, 14].”
The “Gender Gap” in Authorship of Academic Medical Literature — A 35-Year Perspective
Jagsi, R., Guancial, E. A., Worobey, C. C., Henault, L. E., Chang, Y., Starr, R., Tarbell, N. J., & Hylek, E. M. (2006) · New England Journal of Medicine · doi:10.1056/nejmsa053910 · Cited by 870 · cited 2×
“Additionally, Gwilym Lockwood observed that “articles with positively-framed titles, interesting phrasing, and no word-play get more attention online” [45].”
Research works printed but not linked: 6
- 2003, Central Banking, Monetary Theory and Practice: Essays in honour of Charles Goodhart, 221
- Francescani, 2012, NYPD report confirms manipulation of crime stats, Reuters
- Economist, 2016, Why research papers have so many authors, The Economist
- Gálvez, 2000, Scientific publication trends and the developing world: What can the volume and authorship of scientific articles tell us about scientific progress in various regions?, American Sci, 88, 526
- 2016, KDD Cup 2016: Whose papers are accepted the most: towards measuring the impact of research institutions
- Low, 2014, Graphlab: A new framework for parallel machine learning, 1006.4990
Cited by
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., et al. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102642 · Cited by 4,584
Misinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
“Much of the present pathology of hype, hyperbole, and publication bias is associated with an overreliance on productivity metrics (23).”
BackgroundThe strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00327 · Cited by 345
“The difficulty is that “quality” is hard to define (17, 18, 23), and some metrics are at risk of abuse per Goodhart’s law : “when a measure becomes a target, it ceases to be a good measure” (24).”
BackgroundThe strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2023) · arXiv · doi:10.48550/arxiv.2309.15884 · Cited by 345
“The difficulty is that “quality” is hard to define (17, 18, 23), and some metrics are at risk of abuse per Goodhart’s law : “when a measure becomes a target, it ceases to be a good measure” (24).”
BackgroundThe sustainable development goals of the United Nations: A comparative midterm research review
Sorooshian, S. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142272 · Cited by 205
Research Trends, Challenges, and Emerging Topics in Digital Forensics: A Review of Reviews
Casino, F., Dasaklis, T. K., Spathoulas, G. P., Anagnostopoulos, M., Ghosal, A., Borocz, I., Solanas, A., Conti, M., & Patsakis, C. (2022) · IEEE Access · doi:10.1109/access.2022.3154059 · Cited by 165
“are rapidly expanding, thus allowing today’s researchers to publish their work in a multitude of venues [16].”
BackgroundFundamental research questions in subterranean biology
Mammola, S., Amorim, I. R., Bichuette, M. E., Borges, P. A. V., Cheeptham, N., Cooper, S. J., Culver, D. C., Deharveng, L., Eme, D., Ferreira, R. L., et al. (2020) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12642 · Cited by 148
“1872 I. INTRODUCTION In the era of the Internet, social media, and open-access megajournals, the amount of accessible scientific information is overwhelming (Landhuis, 2016; Wakeling et al., 2016; Fire & Guestrin, 2019; Jari c et al., 2020).”
BackgroundByrne, J. A., & Christopher, J. (2020) · FEBS Letters · doi:10.1002/1873-3468.13747 · Cited by 133
Färber, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30796-7_8 · Cited by 126
“So far, the MAG has been used, among other ways, for citation analysis [21] and for impact analysis of papers and researchers [22,23].”
MethodDougherty, M. R. P., & Horne, Z. (2022) · Royal Society Open Science · doi:10.1098/rsos.220334 · Cited by 156
Vlase, I., & Lähdesmäki, T. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01582-5 · Cited by 114
“The increase of publications on cultural heritage across a wider range of publication fora corresponds to the general increase of publication output volume in academia (see for a general tendency, e.g., Kyvik and Aksnes, 2015; Fire and Guestrin, 2019).”
BackgroundEmmer, A., Allen, S., Carey, M., Frey, H., Huggel, C., Korup, O., Mergili, M., Sattar, A., Veh, G., Chen, T. Y., et al. (2022) · Natural hazards and earth system sciences · doi:10.5194/nhess-22-3041-2022 · Cited by 138
Evans, T. R., Burns, C., Essex, R., Finnerty, G., Hatton, E., Clements, A. J., Breau, G., Quinn, F., Elliott, H., Smith, L. D., et al. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1078797 · Cited by 118
Impact of the reference list features on the number of citations
Mammola, S., Fontaneto, D., Martínez, A., & Chichorro, F. (2020) · Scientometrics · doi:10.1007/s11192-020-03759-0 · Cited by 80
Quantitative research assessment: using metrics against gamed metrics
Ioannidis, J. P. A., & Maniadis, Z. (2023) · Internal and Emergency Medicine · doi:10.1007/s11739-023-03447-w · Cited by 75
Schuman, M. C. (2023) · Annual Review of Plant Biology · doi:10.1146/annurev-arplant-040121-114908 · Cited by 64
Chemometrics in food science and technology: A bibliometric study
Aleixandre-Tudó, J. L., Castelló‐Cogollos, L., Aleixandre, J. L., & Aleixandre‐Benavent, R. (2022) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2022.104514 · Cited by 66
PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Lála, J., O'Donoghue, O., Shtedritski, A., Cox, S., Rodriques, S. G., & White, A. D. (2023) · arXiv · doi:10.48550/arxiv.2312.07559 · Cited by 55
“The rate of papers published yearly grows at an exponential rate, with over 5 million academic articles published in 2022 [1], and over 200 million articles in total [2].”
BackgroundScientific Publishing in Biomedicine: A Brief History of Scientific Journals
Ghasemi, A., Mirmiran, P., Kashfi, K., & Bahadoran, Z. (2022) · Intern J Endocrinol Metab · doi:10.5812/ijem-131812 · Cited by 88
“51 in 2014) and the number of hyper-authorship papers (papers with hundreds or even thousands of authors) have increased (26).”
BackgroundInfluentialOesef, K., Cranston, E. D., & Abdin, Y. (2024) · Materials & Design · doi:10.1016/j.matdes.2024.113417 · Cited by 64
Pottier, P., Lagisz, M., Burke, S., Drobniak, S. M., Downing, P. A., Macartney, E. L., Martinig, A. R., Mizuno, A., Morrison, K., Pollo, P., et al. (2024) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2024.1222 · Cited by 97
"When a Measure Becomes a Target, It Ceases to be a Good Measure".
Mattson, C. D., Bushardt, R. L., & Artino, A. (2021) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-20-01492.1 · Cited by 32
“Finally, the fixation in academia on ‘‘number of publications’’ and journal impact factor is also felt in GME research environments.(8) Department chairs and promotion committees use these numbers to help make appointment and promotion decisions.”
BackgroundScientometric trends for coronaviruses and other emerging viral infections
Kagan, D., Moran‐Gilad, J., & Fire, M. (2020) · GigaScience · doi:10.1093/gigascience/giaa085 · Cited by 56
“By analyzing the results presented in the Results section, the following can be noted: First, the surge in infectious disease publications (Figure 1) supports the results of Fire and Guestrin [36] that found there has been a general escalation of scientific publications.”
InfluentialSelf-citationBaek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.342 · Cited by 46
Ullrich, A., Vladova, G., Eigelshoven, F., & Renz, A. (2022) · Discover Artificial Intelligence · doi:10.1007/s44163-022-00031-7 · Cited by 41
“On the one hand, this goes hand in hand with a general increase in the number of research outputs due to the increasing number of outlets [43, 44].”
BackgroundSezgin, A., Orbay, K., & Orbay, M. (2022) · SAGE Open · doi:10.1177/21582440221141628 · Cited by 53
“The competitive environment created by the increasing number of academic journals (Fire & Guestrin, 2019) has brought along the ‘‘publish or perish!’’ philosophy among researchers, and thus, the ‘‘quality or quantity?’’ dilemma has arisen in the studies conducted (Civera et al., 2020; McGrail et…”
BackgroundWho Counts? Ghanaian Academic Publishing and Global Science
Mills, D., Kingori, P., Branford, A., Chatio, S. T., Robinson, N., & Tindana, P. O. (2023) · African Minds eBooks · doi:10.47622/9781928502647 · Cited by 51
Petrović, E. K., & Thomas, C. A. (2024) · Sustainability · doi:10.3390/su16041561 · Cited by 54
Identification of human gene research articles with wrongly identified nucleotide sequences
Park, Y., West, R. A., Pathmendra, P., Favier, B., Stoeger, T., Capes-Davis, A., Cabanac, G., Labbé, C., & Byrne, J. A. (2022) · Life Science Alliance · doi:10.26508/lsa.202101203 · Cited by 37
“As articles that describe miR functions have also been shown to be highly cited (60), miR’s and other ncRNA’s could represent attractive target genes for paper mills.”
PeerQA: A Scientific Question Answering Dataset from Peer Reviews
Baumgärtner, T., Briscoe, T., & Gurevych, I. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.22 · Cited by 39
Equity in climate scholarship: a manifesto for action
Schipper, E. L. F., Ensor, J., Mukherji, A., Mirzabaev, A., Fraser, A., Harvey, B., Totin, E., Garschagen, M., Pathak, M., Antwi‐Agyei, P., et al. (2021) · Climate and Development · doi:10.1080/17565529.2021.1923308 · Cited by 39
In defense of quantitative metrics in researcher assessments
Ioannidis, J. P. A., & Maniadis, Z. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002408 · Cited by 45
“By Goodheart’s law, when metrics acquire power, they will be gamed [4].”
BackgroundThe gaming of citation and authorship in academic journals: a warning from medicine
Macdonald, S. (2022) · Social Science Information · doi:10.1177/05390184221142218 · Cited by 59
“The measures of performance quickly come to bear no relationship at all to actual performance and become a measure of gaming proficiency instead (Fire and Guestrin, 2019).”
BackgroundFactors affecting the citations of papers in tribology journals
Liskiewicz, T. W., Liśkiewicz, G., & Paczesny, J. (2021) · Scientometrics · doi:10.1007/s11192-021-03870-w · Cited by 38
Tautorat, P., Lalin, B., Schmidt, T. S., & Steffen, B. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.137055 · Cited by 40
Open Minded: Searching for Truth about the Unconscious Mind
Newell, B. R., & Shanks, D. R. (2023) · doi:10.7551/mitpress/14922.001.0001 · Cited by 17
The Influence of International Collaboration on the Scientific Impact in V4 Countries
Kohus, Z., Demeter, M., Szigeti, G. P., Kun, L., Lukács, E., & Czakó, K. F. (2022) · Publications · doi:10.3390/publications10040035 · Cited by 42
Embryo Selection and Mate Choice: Can 'Honest Signals' Be Trusted?
McCoy, D. E., & Haig, D. (2020) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2019.12.002 · Cited by 27
“In academic publishing, metrics such as impact factor over time become less informative of quality and depend heavily on field [71].”
BackgroundJohn, Y. J., Caldwell, L., McCoy, D. E., & Braganza, O. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23002753 · Cited by 51
Territorial bias in university rankings: a complex network approach
Bellantuono, L., Monaco, A., Amoroso, N., Aquaro, V., Bardoscia, M., Loiotile, A. D., Lombardi, A., Tangaro, S., & Bellotti, R. (2022) · Scientific Reports · doi:10.1038/s41598-022-08859-w · Cited by 48
Mason, S., & Singh, L. (2022) · Scientometrics · doi:10.1007/s11192-022-04402-w · Cited by 32
A scoping review of seismic risk communication in Europe
Musacchio, G., Saraò, A., Falsaperla, S., & Scolobig, A. (2023) · Frontiers in Earth Science · doi:10.3389/feart.2023.1155576 · Cited by 36
Meho, L. I. (2025) · Scientometrics · doi:10.1007/s11192-025-05480-2 · Cited by 27
“These developments exemplify Goodhart’s Law : once bibliometric measures become targets, they lose diagnostic value and invite optimization over substance (Fire & Guestrin, 2019).”
Malekpour, M., Abbasi‐Kangevari, M., Azadnajafabad, S., Ghamari, S., Rezaei, N., Rezazadeh-Khadem, S., Rezaei, N., Aminorroaya, A., Abdolhamidi, E., Fateh, S. M., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258064 · Cited by 26
“In addition, the time-trends in publishing research articles on SARS and MERS indicate that the scientific community has shown fluctuating interest in the subject, which was associated with the incidence dynamics of the outbreaks [1, 9], rather than the annual list of priority diseases published by World Health Organi…”
BackgroundCitation manipulation through citation mills and pre-print servers
Ibrahim, H., Liu, F., Zaki, Y., & Rahwan, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-88709-7 · Cited by 84
Public Policy and Universities: The Interplay of Knowledge and Power
Gunn, A. S., & Mintrom, M. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108645867 · Cited by 27
Does monetary support increase citation impact of scholarly papers?
Tonta, Y., & Akbulut, M. (2020) · Scientometrics · doi:10.1007/s11192-020-03688-y · Cited by 32
Meho, L. I., & Akl, E. A. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00339 · Cited by 34
Creation and dissemination of hospitality and tourism research outputs in the new millennium
Wong, A. K. F., Köseoğlu, M. A., & Kim, S. (2021) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-05-2020-0493 · Cited by 26
Mapping the landscape of medical research in the Arab world countries
AlMuhaidib, S., Alqahtani, R. S., Alotaibi, H. F., Saeed, A., Alnasrallah, S., Alshamsi, F. E., Alqahtani, S. A., & Alhazzani, W. A. (2024) · Saudi Medical Journal · doi:10.15537/smj.2024.45.4.20230968 · Cited by 20
“23,24 Our study explored the top medical journals that Arab researchers frequently publish in, and having Saudi Arabian journals at the top of the list is a significant milestone both for the Saudi context and the Arab region.”
BackgroundThe Role of Visual Abstracts in the Dissemination of Medical Research.
Millar, B. C., & Lim, M. (2022) · PubMed Central · Cited by 20
Subfield prestige and gender inequality among U.S. computing faculty
LaBerge, N., Wapman, K. H., Morgan, A. C., Zhang, S., Larremore, D. B., & Clauset, A. J. (2022) · Communications of the ACM · doi:10.1145/3535510 · Cited by 34
“14 We use an alternative, output-based ranking, based on institutional placement power, which quantifies prestige according to how well an institution can place its graduates as faculty at other institutions.”
MethodNew Attending Surgeons Hired by Their Training Institution Exhibit Greater Research Productivity
Bajaj, S. S., Wang, H., Williams, K. M., Pickering, J. M., Heiler, J. C., Manjunatha, K., O’Donnell, C. T., Sanchez, M., & Boyd, J. H. (2020) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2020.09.026 · Cited by 20
Emmer, A., Allen, S., Carey, M., Frey, H., Huggel, C., Korup, O., Mergili, M., Sattar, A., Veh, G., Chen, T. Y., et al. (2022) · doi:10.5194/nhess-2022-143 · Cited by 20
The current state of graphical abstracts and how to create good graphical abstracts
Lee, J.-E., & Yoo, J. (2023) · Science Editing · doi:10.6087/kcse.293 · Cited by 25
“First, the number of papers published online has rapidly increased, especially in SCIE (Science Citation Index Expanded) journals [1].”
Vogt, L., Strömert, P., Matentzoglu, N., Karam, N., Konrad, M., Prinz, M., & Baum, R. (2025) · Scientific Data · doi:10.1038/s41597-025-05011-x · Cited by 22
The changing roles of scientific journals
Casadevall, A., Clark, L. F., & Fang, F. C. (2024) · mBio · doi:10.1128/mbio.02515-24 · Cited by 25
Nyberg, Y., Mackay, H., Wilson, M., Samkunde, M., & Wetterlind, J. (2025) · International Journal of Agricultural Sustainability · doi:10.1080/14735903.2025.2505387 · Cited by 40
“…articles had been published during the last ten years 2012–2022 and continuously increasing in number (Figure 3), suggesting a possible growing interest in this topic, even when comparing with the overall increase in scientific publications in the web of science database (Fire & Guestrin, 2019).”
Bibliometric Analysis of Financial and Economic Implications during the COVID-19 Pandemic Crisis
Obrenovic, B., Oblaković, G., & Romeo, A. (2024) · Sustainability · doi:10.3390/su16072897 · Cited by 30
Biosphere Reserves as model regions for transdisciplinarity? A literature review
Dabard, C. H., Gohr, C., Weiss, F., von Wehrden, H., Neumann, F., Hordasevych, S., Arieta, B., Hammerich, J., Meier, C., Jargow, J., et al. (2024) · Sustainability Science · doi:10.1007/s11625-024-01542-1 · Cited by 25
“However, the number of publications on Biosphere Reserves can be expected to increase substantially, as the number of scientific publications published annually is generally increasing (Fire and Guestrin 2019).”
Consumer Behaviour towards Pork Meat Products: A Literature Review and Data Analysis
Jiménez, J. A. M., Sánchez‐Cubo, F., & Jiménez, J. M. (2022) · Foods · doi:10.3390/foods11030307 · Cited by 23
Proxyeconomics, a theory and model of proxy-based competition and cultural evolution
Braganza, O. (2022) · Royal Society Open Science · doi:10.1098/rsos.211030 · Cited by 13
Kıvanç, E., Tuzkaya, G., & Vayvay, Ö. (2024) · Safety Science · doi:10.1016/j.ssci.2024.106755 · Cited by 17
The editor: A demanding but underestimated role in scientific publishing
Vuong, Q. (2022) · Learned Publishing · doi:10.1002/leap.1466 · Cited by 17
“They also note that not only did the number of journals increase dramatically in the 21st century, but the number of publications per journal also increased greatly from a mean of 74.2 papers per journal in 1999 to 99.6 papers per journal in 2016 (Fire & Guestrin, 2019).”
BackgroundInfluentialAn epistemology for democratic citizen science
Jaeger, J., Masselot, C., Tzovaras, B. G., Hidalgo, E. S., Haklay, M., & Santolini, M. (2023) · Royal Society Open Science · doi:10.1098/rsos.231100 · Cited by 26
Identifying prize-winning scientists by a competition-aware ranking
Zhou, Y., Wang, R., Zeng, A., & Zhang, Y. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101038 · Cited by 18
“Meanwhile, this also means that traditional academic indicators may have lost their meaning when evaluating scholars from different periods (Fire & Guestrin, 2019).”
BackgroundInfluentialPreprints in Medicine: Useful or Harmful?
Bonnechère, B. (2020) · Frontiers in Medicine · doi:10.3389/fmed.2020.579100 · Cited by 17
“Therefore, there has been an exponential increase in the number of scientific papers published (4).”
BackgroundTackling overpublishing by moving to open-ended papers
Akbashev, A. R., & Kalinin, S. V. (2023) · Nature Materials · doi:10.1038/s41563-023-01489-1 · Cited by 19
The limitations of evidence-based medicine compel the practice of personalized medicine
Einav, S., & O'Connor, M. C. (2024) · Intensive Care Medicine · doi:10.1007/s00134-024-07528-y · Cited by 20
“Up-to-Date reviews cover approximately 440 journals while more than 30,000 biomedical journals are being published nowadays [5].”
da Silva, J. A. T., & Nazarovets, S. (2025) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-025-04651-5 · Cited by 25
“…rendering them less effective since they are subject to abuse, i.e., a derivative of Goodhart’s law (Grisemer 2020), forms one basis of POP culture (Fire and Guestrin 2019; Csiszar 2020), in some ways leading to a fracturing of the publishing industry (Fassin and Teixeira da Silva 2025), themes…”
Don’t follow the leader: how ranking performance reduces meritocracy
Livan, G. (2019) · Royal Society Open Science · doi:10.1098/rsos.191255 · Cited by 43
Mapping, evolution, and application trends in co-citation analysis: a scientometric approach
Robledo, S., Figueroa-Camargo, J. G., Zuluaga, M., Escobar, S. B. V., & Duque, P. (2023) · Revista de Investigación Desarrollo e Innovación · doi:10.19053/20278306.v13.n1.2023.16070 · Cited by 22
“Researchers produce around 3 million academic documents every year (Fire & Guestrin, 2019), and this value is increasing every time.”
BackgroundSurvive and Thrive in the Time of Changes: A Bibliometric Review of Teacher Resilience, 1998–2023
Li, X., & Chen, J. (2024) · Review of Educational Research · doi:10.3102/00346543241293786 · Cited by 22
“Nonetheless, some researchers also warn that the flood of hyperauthorship might adversely hamper the value of publication numbers as a good metric (Fire & Guestrin, 2019).”
Pathmendra, P., Park, Y., Enguita, F. J., & Byrne, J. A. (2024) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-023-02846-2 · Cited by 14
“Editor awareness that papers about ncRNAs attract high numbers of citations [63] could also lead journal editors to prioritize these manuscripts, as highly cited papers will boost their journal’s IF.”
BackgroundAn extended state of uncertainty: A snap-shot of expressions of concern in neuroscience
da Silva, J. A. T., & Yamada, Y. (2021) · Current Research in Behavioral Sciences · doi:10.1016/j.crbeha.2021.100045 · Cited by 17
Nabavi, A., Safari, F., Shmoury, A. H., Tabet, S., Perdomo-Luna, C., & Celi, L. A. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106418 · Cited by 24
A practical guide to data management and sharing for biomedical laboratory researchers
Fouad, K., Vavrek, R., Surles-Zeigler, M. C., Huie, J. R., Radabaugh, H. L., Gurkoff, G. G., Visser, U., Grethe, J. S., Martone, M. E., Ferguson, A. R., et al. (2024) · Experimental Neurology · doi:10.1016/j.expneurol.2024.114815 · Cited by 14
Probationary or Second-Class Citizens? Postdoctoral Experiences in the Swiss Context
Feuvre, N. L., Bataille, P., & Sautier, M. (2020) · Springer eBooks · doi:10.1007/978-3-030-52600-9_3 · Cited by 17
ChatGPT and its application in the field of mental health.
Bhattacharyya, R., Chakraborty, K., & Neogi, R. (2023) · Journal of SAARC Psychiatric Federation · doi:10.4103/jspf.jspf_9_23 · Cited by 12
Lees, A. F., Beni, C. E., Lee, A., Wedgeworth, P., Dzara, K., Joyner, B. D., Tarczy-Hornoch, P. Z., & Leu, M. G. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005288 · Cited by 11
Bonnet, S., & Teuteberg, F. (2025) · International Journal of Knowledge Management · doi:10.4018/ijkm.368223 · Cited by 16
Features and signals in precocious citation impact: A meta-research study
Ioannidis, J. P. A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0328531 · Cited by 14
“Given that the Hirsch h-index in particular has acquired inappropriately large influence, citations may be placed strategically so as to maximize the h-index [9–11].”
InfluentialTrends in authorship characteristics and collaboration in pharmacy practice publications: 2011–2020
Adeosun, S. O. (2022) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2022.11.002 · Cited by 15
Gaming the Metrics? Bibliometric Anomalies and the Integrity Crisis in Global University RankingsPreprint
Meho, L. I. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5249082 · Cited by 13
A simple model suggesting economically rational sample-size choice drives irreproducibility
Braganza, O. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229615 · Cited by 12
The rise of hyperprolific authors in computer science: characterization and implications
Moreira, E., Meira, W., Gonçalves, M. A., & Laender, A. H. F. (2023) · Scientometrics · doi:10.1007/s11192-023-04676-8 · Cited by 17
“Recent years have witnessed a widespread increase in scientific publications in many research areas (Biagioli & Lippman, 2020; Bornmann & Mutz, 2015; Fire & Guestrin, 2019).”
BackgroundMethodInfluentialInvestigating Infectious Organisms of Public Health Concern Associated with Wild Meat
Moloney, G. K., Gaubert, P., Gryseels, S., Verheyen, E., & Chaber, A. (2023) · Transboundary and Emerging Diseases · doi:10.1155/2023/5901974 · Cited by 11
“Publications studying wild meat increased from 2001 to 2021, in-line with the general increase in scienti fi c publications per year [36], and subsequently demonstrated diverse infectious organism screening methodologies.”
How co-authorship affects the H-index?
Tzitzikas, Y., & Dovas, G. (2024) · Scientometrics · doi:10.1007/s11192-024-05088-y · Cited by 14
“Some interesting findings about the number and order of authors, as well as discipline-specific measurements over time, are given in Fire and Guestrin (2019).”
Sociological Solutions: Building Communities of Hope, Justice, and Joy
Misra, J. (2025) · American Sociological Review · doi:10.1177/00031224241302828 · Cited by 12
“…practiced in other global locations by decen-tering conventional academic metrics, part-nering with communities, and focusing on creating a better world (Burawoy 2005, 2014; Fire and Guestrin 2019; Gans 1989; Green-berg, London, and McKay 2022; London et al. forthcoming; Romero 2020; Smith 2022).”
Bi, W., Chan, H. F., & Torgler, B. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00608-6 · Cited by 11
Macartney, E. L., Drobniak, S. M., Nakagawa, S., & Lagisz, M. (2023) · Environmental Evidence · doi:10.1186/s13750-022-00290-y · Cited by 12
“…and for disciplines beyond biomedical science [33, 54–57], as well as a general increase in the number of articles published per year across fields [58] it is apt that the evidence syntheses of the non-genetic inheritance literature have only started to accumulate just over the last decade.”
BackgroundCampbell, H., & Gustafson, P. (2019) · The American Statistician · doi:10.1080/00031305.2018.1555101 · Cited by 14
“This may not be an entirely unreasonable assumption considering the influence of Goodhart’s Law (“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”) on academia, see Fire and Guestrin (2018) and van Dijk, Manor, and Carey (2014).”
BackgroundSuter, S., Barrett, B., & Welden, N. A. C. (2023) · Environmental Monitoring and Assessment · doi:10.1007/s10661-022-10887-y · Cited by 19
“The results here support the notion that scientific research is often based on quantity not quality, where the number of publications and citations is seen as a measure of success (Fire & Guestrin, 2019).”
ResultTwo-level Graph Caching for Expediting Distributed GNN Training
Zhang, Z., Luo, Z., & Wu, C. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228911 · Cited by 10
“, the citation graph has more than 120 million nodes (published papers before 2019) [7] with sophisticated citation connections.”
BackgroundEncoding the citation life-cycle: the operationalization of a literature-aging conceptual model
Gou, Z., Meng, F., Chinchilla‐Rodríguez, Z., & Bu, Y. (2022) · Scientometrics · doi:10.1007/s11192-022-04437-z · Cited by 16
“The citation based-system is affected by the flood of mediocre papers, self-citation cartels, and lengthy reference lists, among other factors (Fire & Guestrin, 2019).”
BackgroundPublication games: In the web of reciprocity
Barta, Z. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270618 · Cited by 12
“As, among others, Charles Goodhart observed, if a metric is used as a target then it becomes a bad metric [2, 6].”
BackgroundInfluentialUtilization of—Omic technologies in cold climate hydrocarbon bioremediation: a text-mining approach
Abdullah, K., Wilkins, D., & Ferrari, B. C. (2023) · Frontiers in Microbiology · doi:10.3389/fmicb.2023.1113102 · Cited by 10
“Academic papers are being produced at an unprecedented rate, with annually published research articles increasing exponentially (Fire and Guestrin, 2019).”
BackgroundHimanen, L., & Nykyri, S. (2024) · Research Evaluation · doi:10.1093/reseval/rvae008 · Cited by 16
Cite me! Perspectives on coercive citation in reviewing
Burton, S., Basil, D. Z., Soboleva, A., & Nesbit, P. (2024) · Journal of Services Marketing · doi:10.1108/jsm-08-2024-0387 · Cited by 11
Metadata
- Authors
- Fire, M., & Guestrin, C.
- Affiliations
- Ben-Gurion University of the Negev (IL)
- University of Washington (US)
- Corresponding author
- Fire, Michael
- Guestrin, Carlos
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-05-30
- Volume
- 8
- Issue
- 6
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC6541803
- OpenAlex
- W2890107153
- Cited by
- 351
- References
- 77
- Citations by year
- 2026: 52 · 2025: 74 · 2024: 65 · 2023: 65 · 2022: 46 · 2021: 18 · 2020: 20 · 2019: 9
- FWCI
- 15.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Washington Research Foundation
- University of Washington
- Amazon Web Services
- Microsoft Azure Research Award
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Goodhart's law, citation metrics, impact factor, h-index, publication counts, self-citation, author list, research evaluation, biology research, bibliometric analysis
- MeSH terms
- Publishing, Bibliometrics, Benchmarking, Journal Impact Factor
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.