The DOI matches its Crossref record; 109 of the 115 references it cites were found in a registry or an index.
Dynabench: Rethinking Benchmarking in NLP
Open accessAbstract
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These benchmarks, however, are limited to single-turn or short-context scenarios, do not require sequential decision-making or adaptation to changing environments and have been saturating rapidly”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Kiela et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
From Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Kiela et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Self-citation
Research works printed but not linked: 4
- Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP , pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Victor Sánchez Valencia. 1991. Studies on natural logic and categorial grammar, University of Amsterdam Ph. D . Ph.D. thesis, thesis.
- Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data: Garbage In, Garbage Out. arXiv:2004.01670 , pages 1-26. · arXiv
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationA Metaverse: Taxonomy, Components, Applications, and Open Challenges
Park, S., & Kim, Y. (2022) · IEEE Access · doi:10.1109/access.2021.3140175 · Cited by 2,046
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
“Notably, DynaBench [21], a research platform dedicated to dynamic data collection and benchmarking, aligns with our spirit.”
BackgroundSelf-citationUnleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D. E., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 617
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420
Sustainable AI: Environmental Implications, Challenges and Opportunities
Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Behram, F. A., Huang, J., Bai, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.00364 · Cited by 405
Lee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
HateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 287
Self-citationPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 188
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.806 · Cited by 324
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T. T. H., Adler, S., Jiang, A., & Weng, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26752 · Cited by 264
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 178
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Bean, A. M., Payne, R. E., Parsons, G., Kirk, H. R., Ciro, J., Mosquera-Gómez, R., Sara Hincapié, M., Ekanayaka, A. S., Tarassenko, L., Rocher, L., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04074-y · Cited by 130
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 145
Colombo, M., & Piccinini, G. (2023) · doi:10.1017/9781009183734 · Cited by 106
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97
Self-citationMeasure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., & Li, B. (2021) · arXiv · doi:10.48550/arxiv.2111.02840 · Cited by 88
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604712 · Cited by 105
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74
Self-citationMapping global dynamics of benchmark creation and saturation in artificial intelligence
Ott, S., Barbosa-Silva, A., Blagec, K., Brauner, J., & Samwald, M. (2022) · Nature Communications · doi:10.1038/s41467-022-34591-0 · Cited by 73
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
Bartolo, M., Thrush, T., Jia, R., Riedel, S., Stenetorp, P., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.696 · Cited by 94
Self-citationPhan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.14249 · Cited by 65
Self-citationNL-Augmenter 🦎 → 🐍 A Framework for Task-Sensitive Natural Language Augmentation
Dhole, K., Gangal, V., Gehrmann, S., Gupta, A., Li, Z., Mahamood, S., Mahadiran, A., Mille, S., Shrivastava, A., Tan, S., et al. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4725 · Cited by 97
Robust Visual Question Answering: Datasets, Methods, and Future Challenges
Ma, J., Wang, P., Kong, D., Wang, Z., Liu, J., Pei, H., & Zhao, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3366154 · Cited by 64
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
BackgroundPetals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52
Verlingue, L., Boyer, C., Olgiati, L., Brutti-Mairesse, C., Morel, D., & Blay, J. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 50
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47
Self-citationThe Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail
Bowman, S. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.516 · Cited by 60
Kirk, H. R., Vidgen, B., Röttger, P., Thrush, T., & Hale, S. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.97 · Cited by 47
Self-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 81
Adaptive Testing and Debugging of NLP Models
Ribeiro, M. T., & Lundberg, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.230 · Cited by 59
ReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42
What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 48
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
Thrush, T., Tirumala, K., Gupta, A., Bartolo, M., Rodríguez, P., Kane, T., Rojas, W. G., Mattson, P. R., Williams, A., & Kiela, D. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.17 · Cited by 43
Self-citationGENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.787 · Cited by 62
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
Phan, L., Gatti, A., Han, Z., & Li, N. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i1.13973 · Cited by 36
AI Ethics: A Bibliometric Analysis, Critical Issues, and Key Gaps
Gao, D., Haverly, A., Mittal, S., Wu, J., & Chen, J.-D. (2024) · International Journal of Business Analytics · doi:10.4018/ijban.338367 · Cited by 14
AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications
Radharapu, B., Robinson, K., Aroyo, L. M., & Lahoti, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.37 · Cited by 61
An Explainable Toolbox for Evaluating Pre-trained Vision-Language Models
Zhao, T., Zhang, T., Zhu, M., Shen, H., Lee, K., Lu, X., & Yin, J. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.4 · Cited by 34
Human-Adversarial Visual Question AnsweringPreprint
Sheng, S., Singh, A., Goswami, V., Magana, J. A. L., Galuba, W., Parikh, D., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02280 · Cited by 37
Self-citationRevisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing
Ghaddar, A., Wu, Y., Bagga, S., Rashid, A., Bibi, K., Rezagholizadeh, M., Xing, C., Wang, Y., Duan, X., Wang, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.205 · Cited by 28
ChatGPT and Global Higher Education: Using Artificial Intelligence in Teaching and Learning
Lin, X., Chan, R. Y., Sharma, S., & Bista, K. (2024) · doi:10.32674/rh27qv16 · Cited by 29
Kaushik, D., Kiela, D., Lipton, Z. C., & Yih, W.-T. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.517 · Cited by 30
Self-citationScatterShot: Interactive In-context Example Curation for Text Transformation
Wu, T., Shen, H., Weld, D. S., Heer, J., & Ribeiro, M. T. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584059 · Cited by 34
Testing the limits of natural language models for predicting human language judgements
Golan, T., Siegelman, M., Kriegeskorte, N., & Baldassano, C. A. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00718-1 · Cited by 29
Investigating Youth AI Auditing
Solyst, J., Peng, C., Deng, W. H., Pratapa, P., Ogan, A., Hammer, J., Hong, J., & Eslami, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732142 · Cited by 25
A simple neural vector space model for medical concept normalization using concept embeddings
Xu, D., & Miller, T. A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104080 · Cited by 26
Xu, J., Ung, M., Komeili, M., Arora, K., Boureau, Y.-L., & Weston, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.758 · Cited by 37
Adaptive Testing of Computer Vision Models
Gao, I., Ilharco, G., Lundberg, S., & Ribeiro, M. T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00370 · Cited by 35
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Romanou, A., Ibrahim, M., Ross, C., Oktar, K., Shaib, C., Bell, S. J., Ovalle, A., Bosselut, A., Dodge, J., Sinha, K., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.13285 · Cited by 24
Self-citationPythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs
Yadav, A. K., Beniwal, H., & Singh, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.996 · Cited by 28
Nibbling at the Hard Core of Word Sense Disambiguation
Maru, M., Conia, S., Bevilacqua, M., & Roberto Navigli, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.324 · Cited by 23
Interpreting mental state decoding with deep learning models
Thomas, A. W., Ré, C., & Poldrack, R. A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.07.003 · Cited by 24
How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?
Sen, I., Samory, M., Flöck, F., Wagner, C., & Augenstein, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.28 · Cited by 24
Designing Ocean Vision AI: An Investigation of Community Needs for Imaging-based Ocean Conservation
Crosby, A., Orenstein, E. C., Poulton, S. E., Bell, K. L. C., Woodward, B., Ruhl, H. A., Katija, K., & Forbes, A. G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580886 · Cited by 21
Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
Robertson, S., Wang, Z. J., Moritz, D., Kery, M. B., & Hohman, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580790 · Cited by 22
NEWTON: Are Large Language Models Capable of Physical Reasoning?
Wang, Y., Duan, J., Fox, D., & Srinivasa, S. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.652 · Cited by 52
Human-Centered Evaluation and Auditing of Language Models
Xiao, Z., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Lee, M., & Liao, Q. V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636302 · Cited by 26
AI-Driven Virtual Mock Interview Development
Mishra, P. K., Arulappan, A., Ra, I.-H., Mariappan, L. T., Gina Rose, G., & Lee, Y. (2024) · Joint International Conference on Soft Computing and Intelligent Systems and International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · doi:10.1109/scisisis61014.2024.10760210 · Cited by 19
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
Bartolo, M., Thrush, T., Riedel, S., Stenetorp, P., Jia, R., & Kiela, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.275 · Cited by 28
Self-citationA Survey of Dataset Refinement for Problems in Computer Vision Datasets
Wan, Z., Wang, Z., Chung, C., & Wang, Z. (2023) · ACM Computing Surveys · doi:10.1145/3627157 · Cited by 20
Quaye, J., Parrish, A., Inel, O., Rastogi, C., Kirk, H. R., Kahng, M., van Liemt, E. M., Bartolo, M., Tsang, J., White, J., et al. (2024) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3630106.3658913 · Cited by 34
Self-citationDynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation BenchmarkingPreprint
Ma, Z., Ethayarajh, K., Thrush, T., Jain, S., Wu, L., Jia, R., Potts, C., Williams, A., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06052 · Cited by 21
Self-citationData-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages
Röttger, P., Nozza, D., Bianchi, F., & Hovy, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.383 · Cited by 18
To what extent do human explanations of model behavior align with actual model behavior?
Prasad, G., Nie, Y., Bansal, M., Jia, R., Kiela, D., & Williams, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.1 · Cited by 22
Self-citationGokhale, T., Mishra, S. R., Luo, M., Sachdeva, B., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.213 · Cited by 22
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2022) · arXiv · doi:10.48550/arxiv.2211.07819 · Cited by 17
A Survey of Large Language Model for Drug Research and Development
Guo, H., Xing, X., Zhou, Y., Jiang, W., Chen, X., Wang, T., Jiang, Z., Wang, Y., Hou, J., Jiang, Y., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3552256 · Cited by 18
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.240 · Cited by 19
Moriniello, F., Testón, A. M., Muñoz, A., Jasaui, D. S., Gracia, L., & Solanes, J. E. (2024) · Applied Sciences · doi:10.3390/app14051994 · Cited by 17
Neural Methods for Data-to-text Generation
Sharma, M., Gogineni, A. K., & Ramakrishnan, N. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3660639 · Cited by 23
What’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 25
Metaethical perspectives on ‘benchmarking’ AI ethics
LaCroix, T., & Luccioni, A. S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00703-x · Cited by 29
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.230539 · Cited by 23
OpenML: Insights from 10 years and more than a thousand papers
Bischl, B., Casalicchio, G., Das, T., Feurer, M., Fischer, S., Gijsbers, P., Mukherjee, S., Müller, A., Németh, L., Oala, L., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101317 · Cited by 15
Supporting User Engagement in Testing, Auditing, and Contesting AI
Deng, W. H., Lam, M. S., Cabrera, Á. A., Metaxa, D., Eslami, M., & Holstein, K. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3611279 · Cited by 17
Li, J., Zhu, S., Yang, H. H., & Xu, J. (2024) · International Symposium on Educational Technology · doi:10.1109/iset61814.2024.00013 · Cited by 15
Analyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 25
Self-citationRecommendations for machine learning benchmarks in neuroimaging
Leenings, R., Winter, N. R., Dannlowski, U., & Hahn, T. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119298 · Cited by 17
Ocampo, N. B., Cabrio, E., & Villata, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.173 · Cited by 21
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
Deng, W. H., Claire, W., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757702 · Cited by 21
Martin, A., & Jenkins, K. (2024) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2024.101840 · Cited by 14
Chiu, Y. Y., Jiang, L., Lin, B. Y., Park, C. Y., Li, S. S., Ravi, S., Bhatia, M., Antoniak, M., Tsvetkov, Y., Shwartz, V., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1247 · Cited by 13
SEAL: Interactive Tool for Systematic Error Analysis and Labeling
Rajani, N. F., Liang, W., Chen, L., Mitchell, M., & Zou, J. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.36 · Cited by 19
Metadata
- Authors
- Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., & Williams, A.
- Affiliations
- Meta (United Kingdom) (GB)
- ORCID
- Venue
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2021
- Pages
- 4110-4124
- OpenAlex
- W3157225713
- Cited by
- 254
- References
- 115
- Citations by year
- 2026: 34 · 2025: 29 · 2024: 37 · 2023: 71 · 2022: 58 · 2021: 25
- FWCI
- 21.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-4031, N66001-19)
- Office of Naval Research (N000141812871, N00014-18, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- benchmarking, natural language processing, benchmark design, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
