The DOI matches its Crossref record; 109 of the 115 references it cites were found in a registry or an index.
Dynabench: Rethinking Benchmarking in NLP
Open accessAbstract
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These benchmarks, however, are limited to single-turn or short-context scenarios, do not require sequential decision-making or adaptation to changing environments and have been saturating rapidly”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Kiela et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
From Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Kiela et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Self-citation
Research works printed but not linked: 4
- Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP , pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Victor Sánchez Valencia. 1991. Studies on natural logic and categorial grammar, University of Amsterdam Ph. D . Ph.D. thesis, thesis.
- Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data: Garbage In, Garbage Out. arXiv:2004.01670 , pages 1-26. · arXiv
Cited by
Sorted by citations
Items 101–200 of 301
Automatic Construction of Evaluation Suites for Natural Language Generation DatasetsPreprint
Mille, S., Dhole, K., Mahamood, S., Perez-Beltrachini, L., Gangal, V., Kale, M., van Miltenburg, E., & Gehrmann, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09069 · Cited by 13
Robust Conversational Agents against Imperceptible Toxicity Triggers
Mehrabi, N., Beirami, A., Morstatter, F., & Galstyan, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.204 · Cited by 21
Jin, Z., Wang, X., Cheng, F., Sun, C., Liu, Q., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3236380 · Cited by 12
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616244 · Cited by 16
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline
Liu, X., Sun, T., He, J., Wu, J., Wu, L., Zhang, X., Jiang, H., Cao, Z., Huang, X., & Qiu, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.240 · Cited by 18
Philosophy of cognitive science in the age of deep learning
Millière, R. (2024) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1684 · Cited by 18
Comparative Analysis of Chatbots Using Large Language Models for Web Development Tasks
Smutný, P., & Bojko, M. (2024) · Applied Sciences · doi:10.3390/app142110048 · Cited by 14
Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge
Raju, R., Jain, S., Li, B., Li, J. L., & Thakker, U. (2024) · Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U) · doi:10.18653/v1/2024.customnlp4u-1.14 · Cited by 13
The ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11
What Makes Reading Comprehension Questions Difficult?
Sugawara, S., Nangia, N., Warstadt, A., & Bowman, S. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.479 · Cited by 11
CiteBench: A Benchmark for Scientific Citation Text Generation
Funkquist, M., Kuznetsov, I., Hou, Y., & Gurevych, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.455 · Cited by 11
Deep Emotions Across Languages: A Novel Approach for Sentiment Propagation in Multilingual WordNets
Kocoń, J. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00101 · Cited by 10
Enhance Multi-Domain Sentiment Analysis of Review Texts Through Prompting Strategies
Wang, Y., & Luo, Z. (2023) · International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS) · doi:10.1109/hdis60872.2023.10499502 · Cited by 12
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
Neuro-Symbolic Computing: Advancements and Challenges in Hardware–Software Co-Design
Yang, X., Wang, Z., Hu, X. S., Kim, C. H., Yu, S., Pajić, M., Manohar, R., Chen, Y., & Li, H. H. (2023) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2023.3336251 · Cited by 11
GlobalBench: A Benchmark for Global Progress in Natural Language Processing
Song, Y., Khanuja, S., Liu, P., Faisal, F., Ostapenko, A., Winata, G. I., Aji, A. F., Cahyawijaya, S., Tsvetkov, Y., Anastasopoulos, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.875 · Cited by 9
Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
Lam, M. S., Hohman, F., Moritz, D., Bigham, J. P., Holstein, K., & Kery, M. B. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747680 · Cited by 8
Lenk, A., Vogt, M., & Herrmann, C. A. (2024) · Energies · doi:10.3390/en18010002 · Cited by 8
General scales unlock AI evaluation with explanatory and predictive power
Zhou, L., Pacchiardi, L., Martínez‐Plumed, F., Collins, K. M., Moros-Daval, Y., Zhang, S., Zhao, Q., Huang, Y., Sun, L., Prunty, J. E., et al. (2026) · Nature · doi:10.1038/s41586-026-10303-2 · Cited by 21
Evaluation Paradigms in Question Answering
Rodríguez, P., & Boyd-Graber, J. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.758 · Cited by 12
Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y. R., Yu, C., Tetreault, J., Hovy, E. H., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.799 · Cited by 11
Unsupervised Natural Language Inference Using PHL Triplet Generation
Varshney, N., Banerjee, P., Gokhale, T., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.159 · Cited by 8
Production Deployment of Machine-Learned Rotorcraft Surrogate Models on HPC
Brewer, W. H., Martínez, D., Boyer, M., Jude, D. P., Wissink, A., Parsons, B., Yin, J., & Anantharaj, V. G. (2021) · Workshop on Machine Learning in HPC Environments · doi:10.1109/mlhpc54614.2021.00008 · Cited by 8
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Phang, J., Chen, A., Huang, W., & Bowman, S. R. (2022) · doi:10.18653/v1/2022.dadc-1.8 · Cited by 9
Human-Centered Evaluation and Auditing of Language Models
Liu, Y. L., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Liao, Q. V., Xu, W., Novikova, J., & Xiao, Z. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706729 · Cited by 7
Investigating What Factors Influence Users’ Rating of Harmful Algorithmic Bias and Discrimination
Kingsley, S., Zhi, J., Deng, W. H., Lee, J. E., Zhang, S. Q., Eslami, M., Holstein, K., Hong, J., Li, T., & Shen, H. (2024) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v12i1.31602 · Cited by 7
Generalizability of Large Language Model-Based Agents: A Comprehensive Survey
Zhang, M., Yang, Y., Xie, R., Dhingra, B., ZHOU, S., & Pei, J. (2026) · ACM Computing Surveys · doi:10.1145/3794858 · Cited by 15
Salido, E. S., Gonzalo, J. A., & Marco, G. (2026) · IEEE Access · doi:10.1109/access.2026.3651579 · Cited by 26
Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks
Cui, R., Hershcovich, D., & Søgaard, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.359 · Cited by 9
On the Limitations of Simulating Active Learning
Margatina, K., & Αλέτρας, Ν. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.269 · Cited by 7
Targeted Data Generation: Finding and Fixing Model Weaknesses
He, Z., Ribeiro, M. T., & Khani, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.474 · Cited by 11
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements
von Werra, L., Tunstall, L., Thakur, A., Luccioni, A. S., Thrush, T., Piktus, A., Marty, F., Rajani, N. F., Mustar, V., & Ngo, H. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.13 · Cited by 7
Self-citationDynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views
Margatina, K., Wang, S., Vyas, Y., John, N. A., Benajiba, Y., & Ballesteros, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.211 · Cited by 9
Monolithic 3D stacking for neural network acceleration
Kim, B., & Li, H. H. (2023) · Nature Electronics · doi:10.1038/s41928-023-01098-5 · Cited by 7
Petals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Ryabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2209.01188 · Cited by 6
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark
Hardalov, M., Atanasova, P., Mihaylov, T., Angelova, G., Simov, K., Osenova, P., Stoyanov, V., Koychev, I. K., Nakov, P., & Radev, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.487 · Cited by 7
Practical Takes on Federated Learning with Pretrained Language Models
Agarwal, A., Rezagholizadeh, M., & Parthasarathi, P. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.34 · Cited by 7
Griewing, S., Gremke, N., Wagner, U. A. G., Wallwiener, M., Kühn, S., & Commission Digital Medicine of the German Society for Gynecology and Obstetrics (2024) · Geburtshilfe und Frauenheilkunde · doi:10.1055/a-2335-6122 · Cited by 6
An Empirical Study on the Power Consumption of LLMs with Different GPU Platforms
Chen, Z., Lin, W., Xie, X., Hu, Y., Li, C., Tong, Q., Wu, Y., & Li, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825662 · Cited by 7
The future of open human feedback
Don-Yehiya, S., Burtenshaw, B., Astudillo, R. F., Osborne, C., Jaiswal, M., Kuo, T.-S., Zhao, W., Shenfeld, I., Peng, A., Yurochkin, M., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01038-2 · Cited by 16
Recommendations for the Recommenders: Reflections on Prioritizing Diversity in the RecSys Challenge
Heitz, L., Vrijenhoek, S., & Inel, O. (2024) · ACM Recommender Systems Challenge (RecSys Challenge) · doi:10.1145/3687151.3687155 · Cited by 6
Scaboro, S., Portelli, B., Chersoni, E., Santus, E., & Serra, G. (2022) · Experimental Biology and Medicine · doi:10.1177/15353702221128577 · Cited by 8
Benchmarking Long-tail Generalization with Likelihood Splits
Godbole, A., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.71 · Cited by 7
Self-citationExperimental Standards for Deep Learning in Natural Language Processing Research
Ulmer, D., Bassignana, E., Müller-Eberstein, M., Varab, D., Zhang, M., van der Goot, R., Hardmeier, C., & Plank, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.196 · Cited by 9
Superlim: A Swedish Language Understanding Evaluation Benchmark
Berdičevskis, A., Bouma, G., Kurtz, R., Morger, F., Öhman, J., Adesam, Y., Borin, L., Dannélls, D., Forsberg, M., Isbister, T., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.506 · Cited by 5
An Evaluation System for Large Language Models based on Open-Ended Questions
Cao, Z., Ma, Z., & Chen, M. (2024) · IEEE International Conference on Cyber Security and Cloud Computing (CSCloud) · doi:10.1109/cscloud62866.2024.00019 · Cited by 5
Self-citationUser-defined trade-offs in LLM benchmarking: balancing accuracy, scale, and sustainability
Gjorgjevikj, A., Nikolikj, A., Seljak, B. K., & Eftimov, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114405 · Cited by 5
Not a Number: Identifying Instance Features for Capability-Oriented Evaluation
Burnell, R., Burden, J., Rutar, D., Voudouris, K., Cheke, L. G., & Hernández‐Orallo, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/392 · Cited by 8
Towards robust automated math problem solving: a survey of statistical and deep learning approaches
Saraf, A., Kamat, P. V., Gite, S., Kumar, S., & Kotecha, K. V. (2024) · Evolutionary Intelligence · doi:10.1007/s12065-024-00957-0 · Cited by 6
Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking
Tamari, R., Richardson, K., Kahlon, N., Sar-Shalom, A., Liu, N. F., Tsarfaty, R., & Shahaf, D. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.9 · Cited by 5
Barriers and enabling factors for error analysis in NLG research
van Miltenburg, E., Clinciu, M., Dušek, O., Gkatzia, D., Inglis, S., Leppänen, L., Mahamood, S., Schoch, S., Thomson, C., & Wen, L. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4529 · Cited by 7
Chen, Y., Gao, H., Cui, G., Yuan, L., Kong, D., Wu, H., Shi, N., Yuan, B., Huang, L., Xue, H., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.611 · Cited by 5
A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models
Sun, J., Fernandes, P., Wang, X., & Neubig, G. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.128 · Cited by 7
Experimental Standards for Deep Learning in Natural Language Processing ResearchPreprint
Ulmer, D., Bassignana, E., Müller-Eberstein, M., Varab, D., Zhang, M., van der Goot, R., Hardmeier, C., & Plank, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06251 · Cited by 9
Braga, J., Regateiro, F. S., Stiubiener, I., & Braga, J. C. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/wexjr · Cited by 5
Anchor Points: Benchmarking Models with Much Fewer Examples
Vivek, R., Ethayarajh, K., Yang, D., & Kiela, D. (2023) · arXiv · doi:10.48550/arxiv.2309.08638 · Cited by 4
Self-citationAnchor Points: Benchmarking Models with Much Fewer Examples
Vivek, R., Ethayarajh, K., Yang, D., & Kiela, D. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.95 · Cited by 4
Self-citationEvaluation of Semantic Answer Similarity Metrics
Mustafazade, F., & Ebbinghaus, P. F. (2022) · International Conference on Machine Learning & Applications (CMLA) · doi:10.5121/csit.2022.121109 · Cited by 4
“All that Glitters”: Techniques for Evaluations with Unreliable Model and Human Annotations
Hardy, M. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.120 · Cited by 6
Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets
Dammu, P. P. S., Naidu, H., & Shah, C. S. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730324 · Cited by 5
Resources for Automated Identification of Online Gender-Based Violence: A Systematic Review
Abercrombie, G., Jiang, A., Gerrard-Abbott, P., Konstas, I., & Rieser, V. (2023) · doi:10.18653/v1/2023.woah-1.17 · Cited by 4
Is NLP Ready for Standardization?
Aufrant, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.202 · Cited by 4
Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation
Gupta, S., Singh, S., & Gardner, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.365 · Cited by 6
Ahmed, S. T., Danouchi, K., Hefenbrock, M., Prenat, G., Anghel, L., & Tahoori, M. B. (2024) · IEEE Transactions on Nanotechnology · doi:10.1109/tnano.2024.3445455 · Cited by 4
White-box Testing of NLP models with Mask Neuron Coverage
Sekhon, A., Ji, Y., Dwyer, M. B., & Qi, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.116 · Cited by 4
NLP Techniques and Challenges to Process Social Media Data
Mir, T. A., Lawaye, A. A., & Khanday, A. M. U. D. (2023) · Advances in social networking and online communities book series · doi:10.4018/978-1-6684-6909-5.ch009 · Cited by 4
Vote’n’Rank: Revision of Benchmarking with Social Choice Theory
Rofin, M., Mikhailov, V., Florinsky, M., Kravchenko, A., Shavrina, T., Tutubalina, E. V., Самвелович, К. Д., & Artemova, E. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.48 · Cited by 4
Posthoc Verification and the Fallibility of the Ground Truth
Ding, Y., Botzer, N., & Weninger, T. (2022) · doi:10.18653/v1/2022.dadc-1.3 · Cited by 6
HATE-ITA: Hate Speech Detection in Italian Social Media Text
Nozza, D., Bianchi, F., & Attanasio, G. (2022) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2022.woah-1.24 · Cited by 6
Rawlsian AI fairness loopholes
Jørgensen, A. K., & Søgaard, A. (2022) · AI and Ethics · doi:10.1007/s43681-022-00226-9 · Cited by 4
Bommasani, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.522 · Cited by 7
Widjaja, H., Gashteovski, K., Rim, W. B., Liu, P., Malon, C. D., Ruffinelli, D., Lawrence, C., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.34 · Cited by 6
Cross-functional Analysis of Generalization in Behavioral Learning
de Araujo, P. H. L., & Roth, B. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00590 · Cited by 4
Toward Stronger Textual Attack Detectors
Colombo, P., Picot, M., Noiry, N., Staerman, G., & Piantanida, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.35 · Cited by 5
AutoNLU: Detecting, root-causing, and fixing NLU model errorsPreprint
Sethi, P., Savenkov, D., Arabshahi, F., Goetz, J., Tolliver, M., Scheffer, N., Kabul, I., Liu, Y., & Aly, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06384 · Cited by 4
Exploring possible AI trajectories through 2030
Hobbs, H., Docherty, D., Aranda, L., Perset, K., Sugimoto, K., & Kierzenkowski, R. (2026) · OECD artificial intelligence papers · doi:10.1787/cb41117a-en · Cited by 4
Takahashi, K. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.18954285 · Cited by 4
SciCode: A Research Coding Benchmark Curated by Scientists
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.13168 · Cited by 15
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models’ perceived and actual capabilities [36].”
BackgroundNikitin, N. A. (2024) · Finance: Theory and Practice · doi:10.26794/2587-5671-2024-28-3-206-217 · Cited by 3
Wolf, C. (2024) · Springer eBooks · doi:10.1007/978-3-658-44659-8_2 · Cited by 4
Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon
Cohen-Inger, N., Elisha, Y., Shapira, B., Rokach, L., & Cohen, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1098 · Cited by 14
Adversarial Attacks and Defenses on Large Language Models: A Systematic Review
Li, R., Liu, J., Zhen, J., Bao, Y., Qiang, Y., Xu, Y., Gu, Y., Jin, Y., Yu, D., He, Y., et al. (2025) · IEEE International Conference on Big Data and Artificial Intelligence (BDAI) · doi:10.1109/bdai66031.2025.11325628 · Cited by 3
Social-sum-Mal: A Dataset for Abstractive Text Summarization in Malayalam
Rahul Raj, M., & Pankaj, D. S. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3696107 · Cited by 3
Vipera: Towards systematic auditing of generative text-to-image models at scale
Huang, Y., Deng, W. H., Xiao, S., Eslami, M., Hong, J., & Perer, A. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719757 · Cited by 5
Towards Stronger Adversarial Baselines Through Human-AI Collaboration
You, W., & Lowd, D. (2022) · doi:10.18653/v1/2022.nlppower-1.2 · Cited by 3
Robustness Evaluation of Transformer-Based Form Field Extractors via Form Attacks
Xue, L., Gao, M., Chen, Z., Xiong, C., & Xu, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41679-8_10 · Cited by 3
Kovatchev, V., Chatterjee, T., Govindarajan, V. S., Chen, J., Choi, E., Chronis, G., Das, A., Erk, K., Lease, M., Li, J. J., et al. (2022) · doi:10.18653/v1/2022.dadc-1.5 · Cited by 7
When Combating Hype, Proceed with Caution.Preprint
Bowman, S. R. (2021) · arXiv (Cornell University) · Cited by 3
How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?Preprint
Sen, I., Samory, M., Floeck, F., Wagner, C., & Augenstein, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07022 · Cited by 3
Combining Data-driven Supervision with Human-in-the-loop Feedback for Entity ResolutionPreprint
Yin, W., Heinecke, S., Li, J., Keskar, N., Jones, M. J., Shi, S., Georgiev, S., Milich, K., Esposito, J., & Xiong, C.-M. (2021) · arXiv.org · doi:10.48550/arxiv.2111.10497 · Cited by 3
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AIPreprint
Deng, W. H., Wang, C., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Qeios · doi:10.32388/b1lxnv · Cited by 3
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2
“These benchmarks, however, are limited to single-turn or short-context scenarios, do not require sequential decision-making or adaptation to changing environments and have been saturating rapidly (Kiela et al., 2021).”
Schuster, T., Kalyan, A., Polozov, O., & Kalai, A. T. (2021) · arXiv · doi:10.48550/arxiv.2106.05784 · Cited by 2
Züfle, M., Dankers, V., & Titov, I. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.9 · Cited by 2
CAN: Concept‐Aligned Neurons for Visual Comparison of Deep Neural Network Models
Li, M., Jeong, S., Liu, S., & Berger, M. (2024) · Computer Graphics Forum · doi:10.1111/cgf.15085 · Cited by 2
Grumulaitis, A. (2025) · Teisė · doi:10.15388/teise.2025.134.3 · Cited by 2
LLM Evaluations: A Survey of Programmatic, Human, and LLM-as-Judge Approaches
Senior Solutions Architect, A. W. S., & Krishnan, S. B. P. (2025) · International Journal of Innovative Research in Science Engineering and Technology · doi:10.15680/ijirset.2025.1406011 · Cited by 2
The PacifAIst Benchmark: Do AIs Prioritize Human Survival over Their Own Objectives?
Herrador, M. (2025) · AI · doi:10.3390/ai6100256 · Cited by 2
Sermpezis, P., Karamanidis, S., Paraschou, E., Dimitriadis, I., Yfantidou, S., Kouskouveli, F. I. I., Troboukis, T., Kiki, K., Galanopoulos, A., & Vakali, A. I. (2026) · Journal of Computational Social Science · doi:10.1007/s42001-026-00469-0 · Cited by 4
Metadata
- Authors
- Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., & Williams, A.
- Affiliations
- Meta (United Kingdom) (GB)
- ORCID
- Venue
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2021
- Pages
- 4110-4124
- OpenAlex
- W3157225713
- Cited by
- 254
- References
- 115
- Citations by year
- 2026: 34 · 2025: 29 · 2024: 37 · 2023: 71 · 2022: 58 · 2021: 25
- FWCI
- 21.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-4031, N66001-19)
- Office of Naval Research (N000141812871, N00014-18, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- benchmarking, natural language processing, benchmark design, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
