Mapping global dynamics of benchmark creation and saturation in artificial intelligence
Open accessAbstract
Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, we introduce methodologies for creating condensed maps of the global dynamics of benchmark creation and saturation. We curate data for 3765 benchmarks covering the entire domains of computer vision and natural language processing, and show that a large fraction of benchmarks quickly trends towards near-saturation, that many benchmarks fail to find widespread utilization, and that benchmark performance gains for different AI tasks are prone to unforeseen bursts. We analyze attributes associated with benchmark popularity, and conclude that future benchmarks should emphasize versatility, breadth and real-world utility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing benchmarks have become saturated and fail to capture the frontier of what state-of-the-art LMs can and cannot do.
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Ott et al., 2022
Dependence on stationary evaluation constrains self-improvement when a target task has no direct benchmark, when evaluation is slow or weakly informative, and when static benchmarks saturate or become vulnerable to reward hacking.
“This dependence constrains self-improvement when a target task has no direct benchmark, as for paper and proof writing, unlike paper reviewing and proof grading [5-7], when evaluation is slow or weakly informative [810], and when static benchmarks saturate or become vulnerable to reward hacking [11, 12].”
From The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators (2026) · cites this paper as [12]
These systems hold their evaluation criterion fixed outside the loop, exposing them to benchmark saturation, judge biases, and reward hacking.
“These systems hold their evaluation criterion fixed outside the loop, exposing them to benchmark saturation [12], judge biases [15, 16], and reward hacking [17].”
From The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators (2026) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Protein function prediction via graph kernels
Borgwardt, K. M., Ong, C. S., Schönauer, S., Vishwanathan, S. V. N., Smola, A. J., & Kriegel, H.-P. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti1007 · Cited by 1,103 · see citing works
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman, S. R., & Dahl, G. E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.385 · Cited by 117 · see citing works
Research community dynamics behind popular AI benchmarks
Martínez-Plumed, F., Barredo, P., hÉigeartaigh, S. Ó., & Hernández-Orallo, J. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00339-6 · Cited by 86 · see citing works
Blagec, K., Barbosa-Silva, A., Ott, S., & Matthias Samwald, A. (2022) · Scientific Data · doi:10.1038/s41597-022-01435-x · Cited by 57 · see citing works
Self-citationBlagec, K., Kraiger, J., Frühwirt, W., & Samwald, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104274 · Cited by 47 · see citing works
Self-citationA global analysis of metrics used for measuring performance in natural language processing
Blagec, K., Dorffner, G., Moradi, M., Ott, S., & Samwald, M. (2022) · doi:10.18653/v1/2022.nlppower-1.6 · Cited by 43 · see citing works
Self-citationBeyond Accuracy: Behavioral Testing of NLP Models with Checklist (Extended Abstract)
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/659 · Cited by 33 · see citing works
WebProtégé: A Cloud-Based Ontology Editor
Horridge, M., Gonçalves, R. S., Nyulas, C. I., et al. (2019) · doi:10.1145/3308560.3317707 · Cited by 32 · see citing works
Evaluation Gaps in Machine Learning PracticePreprint
Hutchinson, B., Rostamzadeh, N., Greer, C., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05256 · Cited by 1 · see citing works
Intelligence Task Ontology and Knowledge Graph (ITO)
Samwald, M., & Blagec, K. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5561990 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- [4]Zhang, D. et al. The AI Index 2021 Annual Report. Prepint at arXiv10.48550/arxiv.2103.06312 (2021).
- [9]Koch, B., Denton, E., Hanna, A. & Foster, J. G. Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research. in Proc. Neural Information Processing Systems Track on Datasets and Benchmarks (eds. Vanschoren, J. & Yeung, S.) vol. 1 (2021).
- [10]Dehghani, M. et al. The Benchmark Lottery. Preprint at arXiv10.48550/arxiv.2107.07002 (2021).
- [23]Barbosa-Silva, A., Ott, S., Blagec, K., Brauner, J. & Samwald, M. Supplementary data for “Mapping global dynamics of benchmark creation and saturation in artificial intelligence.” https://zenodo.org/record/6782721 (2022). PMC9649641 36357391
- [24]Soomro, K., Zamir, A. R. & Shah, M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. Preprint at arXiv10.48550/arxiv.1212.0402 (2012).
Cited by
Sorted by citations
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · see citing works
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
BackgroundA benchmark of expert-level academic questions to assess AI capabilities
Center for AI Safety, Phan, L., Gatti, A., Li, N., Khoja, A., Kim, R., Ren, R., Hausenloy, J., Zhang, O., Mazeika, M., et al. (2026) · Nature · doi:10.1038/s41586-025-09962-4 · Cited by 23 · see citing works
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35 · see citing works
The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators
Iacob, A., Jovanović, A., Shen, W. F., Burkhardt, D., Kurmanji, M., Tastan, N., Sani, L., Venanzi, N. A. E., Nie, J., Odonnat, A., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.26294 · Cited by 1 · see citing works
“Let R elig ⊆ R be the roles eligible at 𝑎 ★ , D 𝑟, elig ⊆ D 𝑟 the eligible tasks of a role 𝑟 , and 𝑛 𝑟 ( 𝑎 ) , 𝑛 𝑑 ( 𝑎 ) ∈ N the evaluations assigned to role 𝑟 and task 𝑑 at node 𝑎 .”
- Authors
- Ott, S., Barbosa-Silva, A., Blagec, K., Brauner, J., & Samwald, M.
- Affiliations
- Medical University of Vienna (AT)
- University of Oxford (GB)
- Venue
- Nature Communications
- Published
- 2022-11-10
- Volume
- 13
- Issue
- 1
- Pages
- 6793-6793
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC9649641
- OpenAlex
- W4308885736
- Cited by
- 73
- References
- 18
- Citations by year
- 2026: 19 · 2025: 10 · 2024: 9 · 2023: 33 · 2022: 2
- FWCI
- 8.15
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Cancer Research UK (S_4017)
- Engineering and Physical Sciences Research Council (2242671, EP/S024050/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- AI benchmarks, benchmark saturation, computer vision benchmarks, natural language processing benchmarks, performance trends
- MeSH terms
- Artificial Intelligence, Ecosystem, Benchmarking, Physical Phenomena
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
