Dynabench: Rethinking Benchmarking in NLP
Open accessAbstract
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Datasets are central to evaluating language models' generative capabilities in NLP.
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
The Background (p. 2) says task-level datasets have traditionally been used to measure progress in NLP, supporting the general point about datasets and evaluation. But Caveats and Objections (p. 8) says Dynabench currently focuses on classification and span extraction, while evaluating generative tasks requires further research. The paper therefore does not establish the claim specifically about evaluating language models’ generative capabilities. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThese benchmarks are limited to single-turn or short-context scenarios, do not require sequential decision-making or adaptation to changing environments and have been saturating rapidly.
“These benchmarks, however, are limited to single-turn or short-context scenarios, do not require sequential decision-making or adaptation to changing environments and have been saturating rapidly”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Kiela et al., 2021
The Introduction (p. 1) says newer benchmarks routinely reach estimated human performance within a few years, supporting the rapid-saturation point. The discussion of initial tasks (p. 5) describes a constrained question-answering setting and plans for longer passages, but our copy does not establish that benchmarks generally are single-turn or short-context, or that they do not require sequential decision-making or adaptation to changing environments. (LLM, conf=0.86)
Experimental — accuracy not yet measuredDynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
From Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) · cites this paper as [21]
The Introduction (p. 1) presents Dynabench as dynamic data collection with humans and models in the loop, in contrast to static benchmarking. Caveats and Objections (p. 8) explicitly says it anticipates dataset saturation. The copy does not identify overfitting as a challenge of static benchmarks; its discussion of annotators ‘overfitting’ on models (p. 7) describes a potential risk of the dynamic approach instead. (LLM, conf=0.84)
Experimental — accuracy not yet measuredDatasets have long been useful for evaluating language models in natural language processing research.
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
Existing benchmarks have become saturated and fail to capture the frontier of what state-of-the-art LMs can and cannot do.
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Kiela et al., 2021
Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated, leading to discrepancies between the models' perceived and actual capabilities.
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Self-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Self-citationUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · see citing works
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
Self-citationA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Self-citationBidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Self-citationNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
Annotating Expressions of Opinions and Emotions in Language
Wiebe, J., Wilson, T., & Cardie, C. (2005) · Computers and the Humanities · doi:10.1007/s10579-005-7880-9 · Cited by 1,763 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Self-citationAdversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
Self-citationFEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · see citing works
Designing games with a purpose
von Ahn, L., & Dabbish, L. (2008) · Communications of the ACM · doi:10.1145/1378704.1378719 · Cited by 1,156 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Self-citationHotFlip: White-Box Adversarial Examples for Text Classification
Ebrahimi, J., Rao, A., Lowd, D., & Dou, D. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2006 · Cited by 948 · see citing works
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · see citing works
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · see citing works
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890 · see citing works
Alm, C. O., Roth, D., & Sproat, R. (2005) · doi:10.3115/1220575.1220648 · Cited by 781 · see citing works
Hypothesis Only Baselines in Natural Language Inference
Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., & Durme, B. V. (2018) · International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-2023 · Cited by 685 · see citing works
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Self-citationUniversal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642 · see citing works
(2012) · Cited by 618 · see citing works
The pascal recognising textual entailment challenge
Dagan, I., & Glickman, O. (2005) · Cited by 579 · see citing works
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550 · see citing works
A model of textual affect sensing using real-world knowledge
Liu, H., Lieberman, H. A., & Selker, T. (2003) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/604045.604067 · Cited by 533 · see citing works
Synthetic and Natural Noise Both Break Neural Machine TranslationPreprint
Belinkov, Y., & Bisk, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02173 · Cited by 533 · see citing works
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Ettinger, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00298 · Cited by 511 · see citing works
Mitchell, T. M., Cohen, W. W., Junior, E. R. H., Talukdar, P. P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B. B., Gardner, M., Kisiel, B., et al. (2018) · Communications of the ACM · doi:10.1145/3191513 · Cited by 508 · see citing works
Understanding Abuse: A Typology of Abusive Language Detection Subtasks
Waseem, Z., Davidson, T., Warmsley, D., & Weber, I. G. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3012 · Cited by 486 · see citing works
Self-citationSentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2gscircifffd · Cited by 477 · see citing works
Self-citationOn Measuring Social Biases in Sentence Encoders
May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1063 · Cited by 461 · see citing works
The Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448 · see citing works
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446 · see citing works
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · see citing works
Stress Test Evaluation for Natural Language InferencePreprint
Naik, A., Ravichander, A., Sadeh, N., Rosé, C., & Neubig, G. (2018) · International Conference on Computational Linguistics · doi:10.48550/arxiv.1806.00692 · Cited by 431 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 2× · see citing works
“…QA, and are looking to expand this to include unanswerable questions (Rajpurkar et al., 2018), longer and more complex passages, Yes/No questions and multi-span answers (Kwiatkowski et al., 2019), and numbers, dates and spans from the question (Dua et al., 2019) as model performance progresses.”
BackgroundResources and benchmark corpora for hate speech detection: a systematic review
Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., & Patti, V. (2020) · Language Resources and Evaluation · doi:10.1007/s10579-020-09502-8 · Cited by 387 · see citing works
Learning the Difference that Makes a Difference with Counterfactually-Augmented DataPreprint
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12434 · Cited by 346 · see citing works
Self-citationDirections in abusive language training data, a systematic review: Garbage in, garbage out
Vidgen, B., & Derczynski, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243300 · Cited by 335 · see citing works
Self-citationDataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.746 · Cited by 320 · see citing works
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Glockner, M., Shwartz, V., & Goldberg, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2103 · Cited by 319 · see citing works
(2013) · Cited by 290 · see citing works
(2018) · International Conference on Learning Representations · Cited by 285 · see citing works
Evaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 265 · see citing works
HateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 235 · see citing works
Self-citationEveringham, M., Zisserman, A., Williams, C. K. I., Gool, L. V., Allan, M., Bishop, C., Chapelle, O., Dalal, N., Deselaers, T., Dorkó, G., et al. (2006) · Lecture notes in computer science · doi:10.1007/11736790 · Cited by 226 · see citing works
Challenges and frontiers in abusive content detection
Vidgen, B., Harris, A., Nguyen, D., Tromble, R., Hale, S. A., & Margetts, H. Z. (2019) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w19-3509 · Cited by 206 · see citing works
Self-citationLearning The Difference That Makes A Difference With Counterfactually-Augmented Data
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2020) · International Conference on Learning Representations · Cited by 174 · see citing works
Self-citationLearning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection
Vidgen, B., Thrush, T., Waseem, Z., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.132 · Cited by 174 · see citing works
Self-citationOn Adversarial Examples for Character-Level Neural Machine TranslationPreprint
Ebrahimi, J., Lowd, D., & Dou, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09030 · Cited by 172 · see citing works
On the Impact of Various Types of Noise on Neural Machine Translation
Khayrallah, H., & Koehn, P. (2018) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 168 · see citing works
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Hancock, B., Bordes, A., Mazare, P.-E., & Weston, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1358 · Cited by 161 · see citing works
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
Dinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1461 · Cited by 161 · see citing works
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Kim, N., & Linzen, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.731 · Cited by 153 · see citing works
How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
Linzen, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.465 · Cited by 148 · see citing works
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147 · see citing works
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Lewis, P., Stenetorp, P., & Riedel, S. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.86 · Cited by 145 · see citing works
Self-citationCrowdsourcing and language studies: the new generation of linguistic data
Munro, R., Bethard, S. J., Kuperman, V., Lai, V. T., Melnick, R., Potts, C., Schnoebelen, T., & Tily, H. J. (2010) · Max Planck Institute for Plasma Physics · Cited by 144 · see citing works
Self-citationPerformance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
Tsuchiya, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/26mjmiipm2qf · Cited by 130 · see citing works
Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study
Sankar, C., Subramanian, S., Pal, C., Chandar, S., & Bengio, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1004 · Cited by 120 · see citing works
Dangerous Speech and Dangerous Ideology: An Integrated Model for Monitoring and Prevention
Maynard, J. L., & Benesch, S. (2016) · Genocide Studies and Prevention · doi:10.5038/1911-9933.9.3.1317 · Cited by 116 · see citing works
Beat the Machine: Challenging Humans to Find a Predictive Model's “Unknown Unknowns”
Attenberg, J., Ipeirotis, P., & Provost, F. (2015) · Journal of Data and Information Quality · doi:10.1145/2700832 · Cited by 103 · see citing works
Beat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102 · see citing works
Self-citationInvestigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs
Warstadt, A., Cao, Y. H., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1286 · Cited by 100 · see citing works
Self-citationInoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets
Liu, N. F., Schwartz, R., & Smith, N. A. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1225 · Cited by 88 · see citing works
Adversarially Regularising Neural
Minervini, P., & Riedel, S. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1007 · Cited by 85 · see citing works
Self-citationWith Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Self-citationAnalyzing Compositionality-Sensitivity of NLI Models
Nie, Y., Wang, Y., & Bansal, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016867 · Cited by 83 · see citing works
Self-citationAre Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
Jeretič, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.768 · Cited by 83 · see citing works
Self-citationRecognition of Affect, Judgment, and Appreciation in Text
Neviarouskaya, A., Prendinger, H., & Ishizuka, M. (2010) · Cited by 79 · see citing works
Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task
Ettinger, A., Rao, S., Daumé, H., & Bender, E. M. (2017) · doi:10.18653/v1/w17-5401 · Cited by 78 · see citing works
Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
Belinkov, Y., Poliak, A., Shieber, S., Van Durme, B., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1084 · Cited by 73 · see citing works
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents
Shuster, K., Da, J. U., Roller, S., Dinan, E., Boureau, Y.-L., & Weston, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.222 · Cited by 72 · see citing works
A brief history of natural logic
van Benthem, J. (2008) · UvA-DARE (University of Amsterdam) · Cited by 67 · see citing works
An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
Hossain, M. M., Kovatchev, V., Dutta, P., Kao, T. H., Wei, E., & Blanco, E. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.732 · Cited by 66 · see citing works
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Sugawara, S., Stenetorp, P., Inui, K., & Aizawa, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6422 · Cited by 62 · see citing works
Self-citationSyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.10 · Cited by 61 · see citing works
A model of textual affect sensing using real-world knowledge
Liu, H., Lieberman, H. A., & Selker, T. (2003) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/604050.604067 · Cited by 60 · see citing works
Assessing Phrasal Representation and Composition in Transformers
Yu, L., & Ettinger, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.397 · Cited by 58 · see citing works
DynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56 · see citing works
Self-citationNijkamp, P. J., & Suzuki, S. (2009) · Spatial Economic Analysis · doi:10.1080/17421770903114687 · Cited by 52 · see citing works
Towards Ecologically Valid Research on Language User InterfacesPreprint
de Vries, H., Bahdanau, D., & Manning, C. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14435 · Cited by 49 · see citing works
Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets
Rozen, O., Shwartz, V., Aharoni, R., & Dagan, I. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1019 · Cited by 34 · see citing works
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
Zhou, X., Nie, Y., Tan, H., & Bansal, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.659 · Cited by 32 · see citing works
Self-citationLexicosyntactic Inference in Neural Models
White, A. S., Rudinger, R., Rawlins, K., & Durme, B. V. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1501 · Cited by 29 · see citing works
Huang, W., Liu, H., & Bowman, S. R. (2020) · doi:10.18653/v1/2020.insights-1.13 · Cited by 29 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
ConjNLI: Natural Language Inference Over Conjunctive Sentences
Saha, S., Nie, Y., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.661 · Cited by 26 · see citing works
Self-citationHarnessing the linguistic signal to predict scalar inferences
Schuster, S., Chen, Y., & Degen, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.479 · Cited by 25 · see citing works
Mastering the Dungeon: Grounded Language Learning by Mechanical Turker DescentPreprint
Yang, Z., Zhang, S., Urbanek, J., Feng, W., Miller, A., Szlam, A. D., Kiela, D., & Weston, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07950 · Cited by 22 · see citing works
Self-citationStress-Testing Neural Models of Natural Language Inference with Multiply-Quantified SentencesPreprint
Geiger, A., Cases, I., Karttunen, L., & Potts, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13033 · Cited by 21 · see citing works
Self-citationSentiment expression conditioned by affective transitions and social forces
Sudhof, M., Emilsson, A. G., Maas, A. L., & Potts, C. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623687 · Cited by 19 · see citing works
Self-citationThe Universal Decompositional Semantics Dataset and Decomp ToolkitPreprint
White, A. S., Stengel-Eskin, E., Vashishtha, S., Govindarajan, V. S., Reisinger, D. A., Vieira, T., Sakaguchi, K., Zhang, S., Ferraro, F., Rudinger, R., et al. (2019) · arXiv (Cornell University) · doi:10.13016/m2fhz1-knec · Cited by 14 · see citing works
Huang, W., Liu, H., & Bowman, S. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04762 · Cited by 5 · see citing works
Flagged first, then by citations
6 of the 115 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP , pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Victor Sánchez Valencia. 1991. Studies on natural logic and categorial grammar, University of Amsterdam Ph. D . Ph.D. thesis, thesis.
- Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data: Garbage In, Garbage Out. arXiv:2004.01670 , pages 1-26. · arXiv
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationA Metaverse: Taxonomy, Components, Applications, and Open Challenges
Park, S., & Kim, Y. (2022) · IEEE Access · doi:10.1109/access.2021.3140175 · Cited by 1,926 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
“Notably, DynaBench [21], a research platform dedicated to dynamic data collection and benchmarking, aligns with our spirit.”
BackgroundSelf-citationUnleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D. E., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 582 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
Sustainable AI: Environmental Implications, Challenges and Opportunities
Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Behram, F. A., Huang, J., Bai, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.00364 · Cited by 405 · see citing works
Lee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345 · see citing works
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
HateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 235 · see citing works
Self-citationPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178 · see citing works
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 165 · see citing works
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.806 · Cited by 160 · see citing works
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Nekoul, F. E., Lee, T. T. H., Adler, S., Jiang, A., & Weng, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26752 · Cited by 153 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 131 · see citing works
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Bean, A. M., Payne, R. E., Parsons, G., Kirk, H. R., Ciro, J., Mosquera-Gómez, R., Sara Hincapié, M., Ekanayaka, A. S., Tarassenko, L., Rocher, L., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04074-y · Cited by 130 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 121 · see citing works
Colombo, M., & Piccinini, G. (2023) · doi:10.1017/9781009183734 · Cited by 102 · see citing works
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97 · see citing works
Self-citationMeasure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., & Li, B. (2021) · arXiv · doi:10.48550/arxiv.2111.02840 · Cited by 88 · see citing works
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604712 · Cited by 87 · see citing works
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 78 · see citing works
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
Self-citationMapping global dynamics of benchmark creation and saturation in artificial intelligence
Ott, S., Barbosa-Silva, A., Blagec, K., Brauner, J., & Samwald, M. (2022) · Nature Communications · doi:10.1038/s41467-022-34591-0 · Cited by 73 · see citing works
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
Bartolo, M., Thrush, T., Jia, R., Riedel, S., Stenetorp, P., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.696 · Cited by 72 · see citing works
Self-citationNL-Augmenter 🦎 → 🐍 A Framework for Task-Sensitive Natural Language Augmentation
Dhole, K., Gangal, V., Gehrmann, S., Gupta, A., Li, Z., Mahamood, S., Mahadiran, A., Mille, S., Shrivastava, A., Tan, S., et al. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4725 · Cited by 57 · see citing works
Robust Visual Question Answering: Datasets, Methods, and Future Challenges
Ma, J., Wang, P., Kong, D., Wang, Z., Liu, J., Pei, H., & Zhao, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3366154 · Cited by 57 · see citing works
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · see citing works
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
BackgroundPetals: Collaborative Inference and Fine-tuning of Large Models
Borzunov, A., Baranchuk, D., Dettmers, T., Riabinin, M., Belkada, Y., Chumachenko, A., Samygin, P., & Raffel, C. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2023.acl-demo.54 · Cited by 52 · see citing works
Verlingue, L., Boyer, C., Olgiati, L., Brutti-Mairesse, C., Morel, D., & Blay, J. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 50 · see citing works
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48 · see citing works
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47 · see citing works
Self-citationThe Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail
Bowman, S. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.516 · Cited by 45 · see citing works
Kirk, H. R., Vidgen, B., Röttger, P., Thrush, T., & Hale, S. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.97 · Cited by 44 · see citing works
Self-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 43 · see citing works
Adaptive Testing and Debugging of NLP Models
Ribeiro, M. T., & Lundberg, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.230 · Cited by 43 · see citing works
ReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42 · see citing works
What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 41 · see citing works
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
Thrush, T., Tirumala, K., Gupta, A., Bartolo, M., Rodríguez, P., Kane, T., Rojas, W. G., Mattson, P. R., Williams, A., & Kiela, D. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.17 · Cited by 40 · see citing works
Self-citationGENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.787 · Cited by 38 · see citing works
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 37 · see citing works
Phan, L., Gatti, A., Han, Z., & Li, N. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i1.13973 · Cited by 36 · see citing works
AI Ethics: A Bibliometric Analysis, Critical Issues, and Key Gaps
Gao, D., Haverly, A., Mittal, S., Wu, J., & Chen, J.-D. (2024) · International Journal of Business Analytics · doi:10.4018/ijban.338367 · Cited by 14 · see citing works
AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications
Radharapu, B., Robinson, K., Aroyo, L. M., & Lahoti, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.37 · Cited by 34 · see citing works
An Explainable Toolbox for Evaluating Pre-trained Vision-Language Models
Zhao, T., Zhang, T., Zhu, M., Shen, H., Lee, K., Lu, X., & Yin, J. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.4 · Cited by 31 · see citing works
Human-Adversarial Visual Question AnsweringPreprint
Sheng, S., Singh, A., Goswami, V., Magana, J. A. L., Galuba, W., Parikh, D., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02280 · Cited by 30 · see citing works
Self-citationRevisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing
Ghaddar, A., Wu, Y., Bagga, S., Rashid, A., Bibi, K., Rezagholizadeh, M., Xing, C., Wang, Y., Duan, X., Wang, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.205 · Cited by 27 · see citing works
ChatGPT and Global Higher Education: Using Artificial Intelligence in Teaching and Learning
Lin, X., Chan, R. Y., Sharma, S., & Bista, K. (2024) · doi:10.32674/rh27qv16 · Cited by 26 · see citing works
Kaushik, D., Kiela, D., Lipton, Z. C., & Yih, W.-T. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.517 · Cited by 26 · see citing works
Self-citationScatterShot: Interactive In-context Example Curation for Text Transformation
Wu, T., Shen, H., Weld, D. S., Heer, J., & Ribeiro, M. T. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584059 · Cited by 26 · see citing works
Testing the limits of natural language models for predicting human language judgements
Golan, T., Siegelman, M., Kriegeskorte, N., & Baldassano, C. A. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00718-1 · Cited by 26 · see citing works
Investigating Youth AI Auditing
Solyst, J., Peng, C., Deng, W. H., Pratapa, P., Ogan, A., Hammer, J., Hong, J., & Eslami, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732142 · Cited by 25 · see citing works
A simple neural vector space model for medical concept normalization using concept embeddings
Xu, D., & Miller, T. A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104080 · Cited by 25 · see citing works
Xu, J., Ung, M., Komeili, M., Arora, K., Boureau, Y.-L., & Weston, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.758 · Cited by 25 · see citing works
Adaptive Testing of Computer Vision Models
Gao, I., Ilharco, G., Lundberg, S., & Ribeiro, M. T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00370 · Cited by 24 · see citing works
A benchmark of expert-level academic questions to assess AI capabilities
Center for AI Safety, Phan, L., Gatti, A., Li, N., Khoja, A., Kim, R., Ren, R., Hausenloy, J., Zhang, O., Mazeika, M., et al. (2026) · Nature · doi:10.1038/s41586-025-09962-4 · Cited by 23 · see citing works
Self-citationPythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs
Yadav, A. K., Beniwal, H., & Singh, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.996 · Cited by 23 · see citing works
Nibbling at the Hard Core of Word Sense Disambiguation
Maru, M., Conia, S., Bevilacqua, M., & Roberto Navigli, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.324 · Cited by 22 · see citing works
Interpreting mental state decoding with deep learning models
Thomas, A. W., Ré, C., & Poldrack, R. A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.07.003 · Cited by 22 · see citing works
How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?
Sen, I., Samory, M., Flöck, F., Wagner, C., & Augenstein, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.28 · Cited by 21 · see citing works
Designing Ocean Vision AI: An Investigation of Community Needs for Imaging-based Ocean Conservation
Crosby, A., Orenstein, E. C., Poulton, S. E., Bell, K. L. C., Woodward, B., Ruhl, H. A., Katija, K., & Forbes, A. G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580886 · Cited by 21 · see citing works
Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
Robertson, S., Wang, Z. J., Moritz, D., Kery, M. B., & Hohman, F. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580790 · Cited by 20 · see citing works
NEWTON: Are Large Language Models Capable of Physical Reasoning?
Wang, Y., Duan, J., Fox, D., & Srinivasa, S. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.652 · Cited by 20 · see citing works
Human-Centered Evaluation and Auditing of Language Models
Xiao, Z., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Lee, M., & Liao, Q. V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636302 · Cited by 20 · see citing works
AI-Driven Virtual Mock Interview Development
Mishra, P. K., Arulappan, A., Ra, I.-H., Mariappan, L. T., Gina Rose, G., & Lee, Y. (2024) · Joint International Conference on Soft Computing and Intelligent Systems and International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · doi:10.1109/scisisis61014.2024.10760210 · Cited by 19 · see citing works
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
Bartolo, M., Thrush, T., Riedel, S., Stenetorp, P., Jia, R., & Kiela, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.275 · Cited by 19 · see citing works
Self-citationA Survey of Dataset Refinement for Problems in Computer Vision Datasets
Wan, Z., Wang, Z., Chung, C., & Wang, Z. (2023) · ACM Computing Surveys · doi:10.1145/3627157 · Cited by 19 · see citing works
Quaye, J. A., Parrish, A., Inel, O., Rastogi, C., Kirk, H. R., Kahng, M., van Liemt, E., Bartolo, M., Tsang, J., White, J., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658913 · Cited by 19 · see citing works
Self-citationDynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation BenchmarkingPreprint
Ma, Z., Ethayarajh, K., Thrush, T., Jain, S., Wu, L., Jia, R., Potts, C., Williams, A., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06052 · Cited by 19 · see citing works
Self-citationData-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages
Röttger, P., Nozza, D., Bianchi, F., & Hovy, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.383 · Cited by 18 · see citing works
To what extent do human explanations of model behavior align with actual model behavior?
Prasad, G., Nie, Y., Bansal, M., Jia, R., Kiela, D., & Williams, A. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.1 · Cited by 18 · see citing works
Self-citationGokhale, T., Mishra, S. R., Luo, M., Sachdeva, B., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.213 · Cited by 18 · see citing works
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2022) · arXiv · doi:10.48550/arxiv.2211.07819 · Cited by 17
A Survey of Large Language Model for Drug Research and Development
Guo, H., Xing, X., Zhou, Y., Jiang, W., Chen, X., Wang, T., Jiang, Z., Wang, Y., Hou, J., Jiang, Y., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3552256 · Cited by 17 · see citing works
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.240 · Cited by 17 · see citing works
Moriniello, F., Testón, A. M., Muñoz, A., Jasaui, D. S., Gracia, L., & Solanes, J. E. (2024) · Applied Sciences · doi:10.3390/app14051994 · Cited by 17 · see citing works
Neural Methods for Data-to-text Generation
Sharma, M., Gogineni, A. K., & Ramakrishnan, N. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3660639 · Cited by 17 · see citing works
What’s the Meaning of Superhuman Performance in Today’s NLU?
Tedeschi, S., Bos, J., Declerck, T., Hajič, J., Hershcovich, D., Hovy, E. H., Koller, A., Krek, S., Schockaert, S., Sennrich, R., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.697 · Cited by 16 · see citing works
Metaethical perspectives on ‘benchmarking’ AI ethics
LaCroix, T., & Luccioni, A. S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00703-x · Cited by 16 · see citing works
General intelligence requires rethinking exploration
Jiang, M., Rocktäschel, T., & Grefenstette, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.230539 · Cited by 16 · see citing works
OpenML: Insights from 10 years and more than a thousand papers
Bischl, B., Casalicchio, G., Das, T., Feurer, M., Fischer, S., Gijsbers, P., Mukherjee, S., Müller, A., Németh, L., Oala, L., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101317 · Cited by 15 · see citing works
Supporting User Engagement in Testing, Auditing, and Contesting AI
Deng, W. H., Lam, M. S., Cabrera, Á. A., Metaxa, D., Eslami, M., & Holstein, K. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3611279 · Cited by 15 · see citing works
Li, J., Zhu, S., Yang, H. H., & Xu, J. (2024) · International Symposium on Educational Technology · doi:10.1109/iset61814.2024.00013 · Cited by 15 · see citing works
Analyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 15 · see citing works
Self-citationRecommendations for machine learning benchmarks in neuroimaging
Leenings, R., Winter, N. R., Dannlowski, U., & Hahn, T. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119298 · Cited by 15 · see citing works
Ocampo, N. B., Cabrio, E., & Villata, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.173 · Cited by 15 · see citing works
WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
Deng, W. H., Claire, W., Han, H. Z., Hong, J., Holstein, K., & Eslami, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757702 · Cited by 14 · see citing works
Martin, A., & Jenkins, K. (2024) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2024.101840 · Cited by 14 · see citing works
Chiu, Y. Y., Jiang, L., Lin, B. Y., Park, C. Y., Li, S. S., Ravi, S., Bhatia, M., Antoniak, M., Tsvetkov, Y., Shwartz, V., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1247 · Cited by 13 · see citing works
SEAL: Interactive Tool for Systematic Error Analysis and Labeling
Rajani, N. F., Liang, W., Chen, L., Mitchell, M., & Zou, J. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.36 · Cited by 13 · see citing works
Automatic Construction of Evaluation Suites for Natural Language Generation DatasetsPreprint
Mille, S., Dhole, K., Mahamood, S., Perez-Beltrachini, L., Gangal, V., Kale, M., van Miltenburg, E., & Gehrmann, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09069 · Cited by 13 · see citing works
- Authors
- Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., & Williams, A.
- Affiliations
- Meta (United Kingdom) (GB)
- ORCID
- Venue
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2021
- Pages
- 4110-4124
- OpenAlex
- W3157225713
- Cited by
- 254
- References
- 115
- Citations by year
- 2026: 34 · 2025: 29 · 2024: 37 · 2023: 71 · 2022: 58 · 2021: 25
- FWCI
- 21.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-4031, N66001-19)
- Office of Naval Research (N000141812871, N00014-18, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- benchmarking, natural language processing, benchmark design, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
