Emerging Trends: SOTA-Chasing
Open accessAbstract
Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Important parts of present-day NLP and machine learning could be described as finding clever ways of gaming metrics, whether automated or human.
“indeed important parts of present-day NLP and machine learning could be described as finding clever ways of gaming metrics, whether automated or human [10].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [10]
Section 2 says NLP and machine-learning publications increasingly pursue better numbers with less insight, and §2.1 criticizes leaderboard-driven competition. Section 2.3 also warns that inappropriate metrics can produce misleading results. These passages support the metric-chasing characterization, but they do not establish that researchers find clever ways to game both automated and human metrics. (LLM, conf=0.84)
Experimental — accuracy not yet measuredMetrics can be gamed.
“Metrics can be gamed; indeed important parts of present-day NLP and machine learning could be described as finding clever ways of gaming metrics, whether automated or human [10].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [10]
Section 2.1 says that measurement shapes what researchers pursue, and Section 2.3 warns that benchmark gains can reflect inappropriate metrics rather than real progress. Those passages support the point that metrics can be gamed. They do not establish the sentence’s broader characterization of important parts of NLP and machine learning as gaming either automated or human metrics. (LLM, conf=0.82)
Experimental — accuracy not yet measuredEvaluation tables in the release-by-blogpost model allow releases to retain the veneer of scientific work while avoiding the fine-grained accounting and the scrutiny of peer review that comes with actual scientific publication.
“These evaluation tables, clearly modelled after NLP's coveted SOTA tables [10], allow releases to retain the veneer of scientific work while at the same time avoiding the fine-grained accounting and the scrutiny of peer review that comes with actual scientific publication.”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 1× · see citing works
“…of the literature is wrong: Amgen researchers declared that they had been unable to reproduce the findings in 47 of 53 “landmark” cancer papers (Begley and Ellis 2012; Baker 2016 b ) More than 70% of researchers have tried and failed to reproduce another scientist’s experiments ce (Baker 2016 a ).”
BackgroundThe mathematics of statistical machine translation: parameter estimation
Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., & Mercer, R. L. (1993) · Cited by 4,019 · see citing works
Speech and language processing
Politt, R., Pollock, J., & Waller, E. (2010) · Taylor & Francis eBooks · doi:10.4324/9780203461891_chapter_3 · Cited by 3,594 · see citing works
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,110 · see citing works
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2× · see citing works
“Amgen researchers declared that they had been unable to reproduce the findings in 47 of 53 “landmark” cancer papers (Begley and Ellis 2012; Baker 2016b)”
BackgroundPrediction and Entropy of Printed English
Shannon, C. E. (1951) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1951.tb01366.x · Cited by 2,686 · see citing works
Statistical methods for speech recognition
Jelinek, F. (1997) · Journal of the American Statistical Association · doi:10.2307/2670189 · Cited by 2,376 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
A statistical approach to machine translation
Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J., Mercer, R. L., & Roossin, P. S. (1990) · doi:10.5555/92858.92860 · Cited by 1,691 · see citing works
Building Watson: An Overview of the DeepQA Project
Ferrucci, D. A., Brown, E. W., Chu‐Carroll, J., Fan, J., Gondek, D. C., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J. M., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,523 · see citing works
Toward Causal Representation Learning
Scholkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3058954 · Cited by 1,170 · see citing works
Continuous speech recognition by statistical methods
Jelinek, F. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10159 · Cited by 922 · see citing works
Information Science and Statistics
Andersen, E. B. (1986) · COMPSTAT · doi:10.1007/978-3-642-46890-2_1 · Cited by 910 · see citing works
The Oxford handbook of computational linguistics
Mitkov, R. (2003) · Choice Reviews Online · doi:10.5860/choice.41-0758 · Cited by 718 · see citing works
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
Watts, T. W., Duncan, G. J., & Quan, H. (2018) · Psychological Science · doi:10.1177/0956797618761661 · Cited by 522 · see citing works
Introduction to “This is Watson”
Ferrucci, D. A. (2012) · IBM Journal of Research and Development · doi:10.1147/jrd.2012.2184356 · Cited by 488 · see citing works
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa‐jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5301 · Cited by 461 · see citing works
On Self-Organizing Systems and Their Environments
von Foerster, H. (2003) · Springer eBooks · doi:10.1007/0-387-21722-3_1 · Cited by 460 · see citing works
SELF-ORGANIZED LANGUAGE MODELING FOR SPEECH RECOGNITION
Jelinek, F. (1990) · Elsevier eBooks · doi:10.1016/b978-0-08-051584-7.50045-0 · Cited by 432 · see citing works
Introduction to the special issue on computational linguistics using large corpora
Church, K., & Mercer, R. L. (1993) · OLAC - Open Language Archives Community · doi:10.5555/972450.972452 · Cited by 296 · see citing works
Self-citationThe TREC question answering track
Voorhees, E. M., & Tice, D. M. (2000) · Natural Language Engineering · doi:10.1017/s1351324901002789 · Cited by 293 · see citing works
Simulation of self-organizing systems by digital computer
Farley, B. G., & Clark, W. A. (1954) · Transactions of the IRE Professional Group on Information Theory · doi:10.1109/tit.1954.1057468 · Cited by 276 · see citing works
Language and Machines: Computers in Translation and Linguistics
Pierce, J. R., & Carroll, J. B. (1966) · National Academy of Sciences-National Research Council eBooks · Cited by 229 · see citing works
The Handbook of Computational Linguistics and Natural Language Processing
Clark, A., Fox, C., & Lappin, S. (2010) · Wiley eBooks · doi:10.1002/9781444324044 · Cited by 229 · see citing works
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
He, W. P., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H. R., She, Q., et al. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2605 · Cited by 227 · see citing works
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 220 · see citing works
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210 · see citing works
Handbook of Natural Language Processing
(2000) · CRC Press eBooks · doi:10.1201/9780824746346 · Cited by 189 · see citing works
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
Bengio, Y., Deleu, T., Rahaman, N., Ke, R., Lachapelle, S., Bilaniuk, O., Goyal, A., & Pal, C. (2019) · arXiv · doi:10.48550/arxiv.1901.10912 · Cited by 182 · see citing works
An estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179 · see citing works
AI and the Everything in the Whole Wide World Benchmark
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021) · arXiv · doi:10.48550/arxiv.2111.15366 · Cited by 127 · see citing works
Goldratt, E. M. (2016) · Routledge eBooks · doi:10.4324/9781315270456 · Cited by 109 · see citing works
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning ResearchPreprint
Koch, B., Denton, E., Hanna, A., & Foster, J. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01716 · Cited by 88 · see citing works
Pierce, J. R. (1969) · The Journal of the Acoustical Society of America · doi:10.1121/1.1911801 · Cited by 77 · see citing works
Overview of the TREC 2019 deep learning track
Craswell, N., Mitra, B., Yilmaz, E., Campos, D., & Voorhees, E. M. (2020) · arXiv · doi:10.48550/arxiv.2003.07820 · Cited by 68 · see citing works
Church, K. (2011) · Linguistic Issues in Language Technology · doi:10.33011/lilt.v6i.1245 · Cited by 66 · see citing works
Self-citationAn Introduction Natural Language Processing
(2019) · doi:10.46632/rmc/1/1/014 · Cited by 62 · see citing works
Ioannidis, J. P. A. (2013) · Biostatistics · doi:10.1093/biostatistics/kxt036 · Cited by 60 · see citing works
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark
Nangia, N., & Bowman, S. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1449 · Cited by 57 · see citing works
Biotech giant publishes failures to confirm high-profile science
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.19269 · Cited by 47 · see citing works
Taking the load off the conference chairs-towards a digital paper-routing assistant
Yarowsky, D., & Florian, R. (1999) · Cited by 46 · see citing works
He Said, She Said: Gender in the ACL Anthology
Vogel, A. P., & Jurafsky, D. (2012) · Cited by 44 · see citing works
The Evolution of Benchmarking as a Computer Performance Evaluation Technique
Lewis, B. C., & Crews, A. E. (1985) · MIS Quarterly · doi:10.2307/249270 · Cited by 38 · see citing works
Super-Human Performance in Online Low-Latency Recognition of Conversational Speech
Nguyen, T., Stüker, S., & Waibel, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1114 · Cited by 32 · see citing works
Liberman, M. (2010) · Computational Linguistics · doi:10.1162/coli_a_00032 · Cited by 29 · see citing works
Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019Preprint
Toral, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05738 · Cited by 24 · see citing works
Liberman, M. Y., & Wayne, C. L. (2020) · AI Magazine · doi:10.1609/aimag.v41i2.5297 · Cited by 22 · see citing works
Emerging trends: I did it, I did it, I did it, but. . .
Church, K. (2017) · Natural Language Engineering · doi:10.1017/s1351324917000067 · Cited by 16 · see citing works
Self-citationVoorhees, E. M. (2020) · ACM SIGIR Forum · doi:10.1145/3483382.3483384 · Cited by 9 · see citing works
Emerging trends: A tribute to Charles Wayne
Church, K. (2017) · Natural Language Engineering · doi:10.1017/s1351324917000389 · Cited by 8 · see citing works
Self-citationBenchmarking: Past, Present and Future
Church, K., Liberman, M. Y., & Kordoni, V. (2021) · doi:10.18653/v1/2021.bppf-1.1 · Cited by 7 · see citing works
Self-citationEmerging trends: Deep nets for poets
Church, K., Yuan, X., Guo, S. N., Wu, Z., Yang, Y., & Chen, Z. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000231 · Cited by 7 · see citing works
Self-citationJager, L. R., & Leek, J. T. (2013) · Biostatistics · doi:10.1093/biostatistics/kxt038 · Cited by 7 · see citing works
Emerging trends: APIs for speech and machine translation and more
Church, K. (2018) · Natural Language Engineering · doi:10.1017/s1351324918000335 · Cited by 6 · see citing works
Self-citationEmerging trends: Reviewing the reviewers (again)
Church, K. W. (2020) · Natural Language Engineering · doi:10.1017/s1351324920000030 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 7
- Anderson A., Jurafsky D. and McFarland D.A. (2012). Towards a computational history of the ACL: 1980-2008. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Korea: Association for Computational Linguistics, pp. 13-21.
- Baker M. (2016b). Biotech giant posts negative results. Nature 530(7589), 141.
- Banchs R.E. (ed) (2012). Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Korea: Association for Computational Linguistics.
- Fillmore C.J. (1968). In Emmon Bach & R. Harms (eds.), Universals in Linguistic Theory. Holt, Rinehart, and Winston.
- Goldratt E.M. and Cox J. (2016). The Goal: A Process of Ongoing Improvement. Routledge.
- Goodfellow I., Bengio Y. and Courville A. (2016). Deep Learning. MIT Press.
- Jurafsky D. (2000). Speech & Language Processing. Upper Saddle River, NJ, USA: Pearson Education.
Cited by
Sorted by citations
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Rethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 67 · see citing works
Machine learning for small molecule drug discovery in academia and industry
Volkamer, A., Riniker, S., Nittinger, E., Lanini, J., Grisoni, F., Evertsson, E., Rodríguez-Pérez, R., & Schneider, N. (2023) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2022.100056 · Cited by 48 · see citing works
Bauer, C., Carterette, B., Ferro, N., Fuhr, N., Beel, J., Breuer, T., Clarke, C. L. A., Crescenzi, A., Demartini, G., Nunzio, G. M. D., et al. (2023) · ACM SIGIR Forum · doi:10.1145/3636341.3636351 · Cited by 27 · see citing works
Distributionally-Informed Recommender System Evaluation
Ekstrand, M. D., Carterette, B., & Díaz, F. (2023) · ACM Transactions on Recommender Systems · doi:10.1145/3613455 · Cited by 19 · see citing works
Artificial intelligence in drug discovery — what it is, where we stand and the path forward
Bender, A., Thomas, M. C., Scannell, J. W., Shaywitz, D. A., Ghiandoni, G. M., Greener, J. G., Pruteanu, L.-L., Jacobson, R. D., Handa, K., Hirano, M., et al. (2026) · Nature Reviews Drug Discovery · doi:10.1038/s41573-026-01496-2 · Cited by 22 · see citing works
An Approach Based on Semantic Relationship Embeddings for Text Classification
Lezama-Sánchez, A. L., Vidal, M. T., & Reyes‐Ortiz, J. A. (2022) · Mathematics · doi:10.3390/math10214161 · Cited by 15 · see citing works
Grill, G. (2022) · doi:10.31219/osf.io/zekqv · Cited by 15 · see citing works
Deanthropomorphising NLP: Can a language model be conscious?
Shardlow, M., & Przybyła, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307521 · Cited by 16 · see citing works
“The large companies behind them were able to invest ever-increasing amounts of time, money and carbon emissions to obtain the stateof-the-art results in standard benchmarks – the process known as SOTA-chasing (Church and Kordoni, 2022).”
Making Intelligence: Ethical Values in IQ and ML Benchmarks
Blili-Hamelin, B., & Hancox-Li, L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593996 · Cited by 12 · see citing works
Integrating Text Classification into Topic Discovery Using Semantic Embedding Models
Lezama-Sánchez, A. L., Vidal, M. T., & Reyes‐Ortiz, J. A. (2023) · Applied Sciences · doi:10.3390/app13179857 · Cited by 9 · see citing works
Exploring Fake News Detection with Heterogeneous Social Media Context Graphs
Donabauer, G., & Kruschwitz, U. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_29 · Cited by 9 · see citing works
Chugg, B., Rothbacher, N., FENG, A. S., Long, X., & Ho, D. E. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557104 · Cited by 6 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · SIGDIAL Conferences · doi:10.18653/v1/2023.sigdial-1.45 · Cited by 18 · see citing works
TAPE: Assessing Few-shot Russian Language Understanding
Taktasheva, E., Shavrina, T., Fenogenova, A., Shevelev, D., Katricheva, N., Tikhonova, M., Akhmetgareeva, A., Zinkevich, O., Bashmakova, A., Iordanskaia, S., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.183 · Cited by 10 · see citing works
Fostering Coopetition While Plugging Leaks
Lin, J., Campos, D., Craswell, N., Mitra, B., & Yılmaz, E. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531725 · Cited by 4 · see citing works
A Gentle Introduction to Deep Nets and Opportunities for the Future
Church, K., Kordoni, V., Marcus, G., Davis, E., Ma, Y., & Chen, Z. (2022) · doi:10.18653/v1/2022.acl-tutorials.1 · Cited by 4 · see citing works
Self-citationVerifying the robustness of automatic credibility assessment
Przybyła, P., Shvets, A., & Saggion, H. (2024) · Natural language processing. · doi:10.1017/nlp.2024.54 · Cited by 8 · see citing works
Emerging trends: Deep nets thrive on scale
Church, K. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000365 · Cited by 3 · see citing works
Self-citationEvaluating the Evaluations: A Perspective on Benchmarks
Alonso, O., & Church, K. (2024) · ACM SIGIR Forum · doi:10.1145/3722449.3722467 · Cited by 2 · see citing works
Self-citationAn Example of (Too Much) Hyper-Parameter Tuning In Suicide Ideation Detection
Schoene, A. M., Ortega, J. E., Amir, S., & Church, K. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22227 · Cited by 2 · see citing works
Self-citationEmerging trends: General fine-tuning (gft)
Church, K., Cai, X., Ying, Y., Chen, Z., Xun, G., & Bian, Y. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000237
Self-citationParry, B. R., Vermeulen, G., & Deetlefs, S. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251376195
Li, Q., Gan, J., Zhu, H., & Gao, X. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00225
A Video Sentiment Analysis Method Based on Improved YOLOv8 and MAE-DFER
Han, J., Ma, S., Lu, Q., & Qi, Z. (2025) · International Conference on Advanced Computer Theory and Engineering · doi:10.1109/icacte66284.2025.11412081
Highleyman’s Data: Pattern Recognition, Bell Labs and the Birth of Reference Datasets
Goujon, V. (2026) · Digital Society · doi:10.1007/s44206-026-00258-2
Emerging trends: Too many submissions and too few qualified reviewers
Church, K. (2026) · Natural language processing. · doi:10.1017/nlp.2026.10038
Self-citationKrysztoforska, M. (2026) · Cultural inquiry · doi:10.37050/ci-40_mk
- Authors
- Church, K. W., & Kordoni, V.
- Affiliations
- Baidu (China) (CN)
- Humboldt-Universität zu Berlin (DE)
- Corresponding author
- Church, Kenneth Ward
- Venue
- Natural Language Engineering
- Publisher
- Cambridge University Press (CUP)
- Published
- 2022-02-08
- Volume
- 28
- Issue
- 2
- Pages
- 249-269
- ISSN
- 1351-3249 (ISSN-L)
- 1469-8110
- OpenAlex
- W4210913687
- Cited by
- 28
- References
- 60
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 4 · 2023: 9 · 2022: 8
- FWCI
- 25.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Natural Language Engineering (publishedVersion, cc-by)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- research evaluation, peer review, AI winter, replication crisis, research leadership, coopetition, interdisciplinary collaboration, AI research trends
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
