The DOI matches its Crossref record; 109 of the 115 references it cites were found in a registry or an index.
Dynabench: Rethinking Benchmarking in NLP
Open accessAbstract
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Datasets are central to evaluating LMs' generative capabilities in NLP [12,25,28,32,49,55].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These benchmarks, however, are limited to single-turn or short-context scenarios, do not require sequential decision-making or adaptation to changing environments and have been saturating rapidly”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Kiela et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
From Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) · cites this paper as [21]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Datasets have long been useful in evaluating LMs in Natural Language Processing (NLP) research [25,32,47,56].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [32]
The cited source says what the claim attributes to it.
In the citing paper
“At the same time, existing benchmarks have become saturated (Kiela et al., 2021; Ott et al., 2022) and fail to capture the frontier of what state-of-the-art LMs can and cannot do.”
From SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) · cites this paper as Kiela et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“Many established benchmarks struggle to keep pace with the advancements in LM performance and have quickly become saturated [91, 14, 70, 58], leading to discrepancies between the models' perceived and actual capabilities [36].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Self-citation
Research works printed but not linked: 4
- Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP , pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Victor Sánchez Valencia. 1991. Studies on natural logic and categorial grammar, University of Amsterdam Ph. D . Ph.D. thesis, thesis.
- Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data: Garbage In, Garbage Out. arXiv:2004.01670 , pages 1-26. · arXiv
Cited by
Sorted by citations
Items 201–300 of 301
Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data
Guo, M., Ye, Z., Xu, W., Zhu, X., Hua, W., & Metaxas, D. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809868 · Cited by 6
A resampling-based method to evaluate NLI models
Salvatore, F. D. S., Finger, M., Hirata, R., & Patriota, A. G. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000268 · Cited by 2
Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlines
Sung, Y. Y., Boyd-Graber, J. L., & Hassan, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1010 · Cited by 2
Testing Cross-Database Semantic Parsers With Canonical Utterances
Lent, H., Yavuz, S., Yu, T., Niu, T., Zhou, Y., Radev, D., & Lin, X. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.8 · Cited by 2
Adversarial Text Normalization
Bitton, J., Pavlova, M., & Evtimov, I. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.30 · Cited by 2
Bridging the Training-Inference Gap for Dense Phrase Retrieval
Kim, G., Lee, J., Oğuz, B., Xiong, W., Zhang, Y., Mehdad, Y., & Wang, W. Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.272 · Cited by 2
Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks
Rodríguez, P., Azab, M., Silvert, B., Sanchez, R., Labson, L., Shah, H., & Moon, S. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.3 · Cited by 4
Measuring Adversarial Datasets
Bai, Y., Huang, R., Viswanathan, V., Kuo, T.-S., & Wu, T. (2023) · doi:10.18653/v1/2023.artofsafety-1.4 · Cited by 2
Mullins, J., & Iskandarova, S. (2024) · IEEE Global Conference on Artificial Intelligence and Internet of Things (GCAIoT) · doi:10.1109/gcaiot63427.2024.10833574 · Cited by 2
Tradeoffs Between Richness and Bias of Augmented Data in Long-Tail Recognition
Ma, Y., Dai, W., Chen, J., Chen, X., & Li, S. (2025) · Entropy · doi:10.3390/e27020201 · Cited by 2
From apologies to insights: extracting topics from ChatGPT apologetic responses
Hnich, B., Mrad, A. B., Thiombiano, A. M. O., & Mkaouer, M. W. (2025) · Journal of Decision System · doi:10.1080/12460125.2024.2438610 · Cited by 2
Evaluation of Semantic Answer Similarity Metrics
Mustafazade, F., & Ebbinghaus, P. F. (2022) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2022.11305 · Cited by 2
infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information
Kim, J., Kim, Y., de Langis, K., Shin, J., & Kang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.547 · Cited by 2
The 7th Workshop on Online Abuse and Harms (WOAH)
(2023) · doi:10.18653/v1/2023.woah-1 · Cited by 2
Morandé, S. (2023) · Qeios · doi:10.32388/znkj97 · Cited by 2
Chan, R., Amini, A., & El‐Assady, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.44 · Cited by 1
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1
Beyond Bias Detection: Community Auditors and Normative Reasoning in AI Oversight CSCW006
Jackson, C., Ahmad, T., Raj, S. D., & Wu, N. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3788042 · Cited by 1
Rohl, A., Roughan, M., White, M., & Chambers, A. J. (2024) · ACM SIGCOMM Conference Posters and Demos · doi:10.1145/3672202.3673718 · Cited by 1
Benchmarking Generative AI Performance Requires a Holistic Approach
Dholakia, A., Ellison, D. W., Hodak, M., Dutta, D., & Binnig, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-68031-1_3 · Cited by 1
Gretscher, P., Keck, A., Wolff, J., Konig, A. S., & Graf‐Vlachy, L. (2026) · Springer eBooks · doi:10.1007/978-3-658-36874-6_26-1 · Cited by 1
EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer
Zhao, P., Wang, L., Wang, M., Chen, C., Zhou, F., & Huang, H. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i43.40979 · Cited by 5
Ching, K. C. H., Ng, S. K. P., Jiang, C. Q., Ching, H. C. W., Cheung, R. C. C., Li, H., & Lam, A. C. L. (2026) · Future Transportation · doi:10.3390/futuretransp6050182 · Cited by 1
iBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1
The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review
Yang, E., Ko, S., & Woo, H. (2026) · Journal of Medical Internet Research · doi:10.2196/98184 · Cited by 1
The OPUS-MT Dashboard – A Toolkit for a Systematic Evaluation of Open Machine Translation Models
Tiedemann, J., & de Gibert, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.30 · Cited by 1
Adversarial Benchmark Evaluation Rectified by Controlling for Difficulty
Mehrbakhsh, B., Martínez‐Plumed, F., & Hernández‐Orallo, J. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230454 · Cited by 1
McGill BabyLM Shared Task Submission: The Effects of Data Formatting and Structural Biases
Cheng, Z., Aralikatte, R., Porada, I., Spinoso-Di Piano, C., & Cheung, J. C. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.18 · Cited by 1
A Bibliometric View of AI Ethics Development
Gao, D. K., Haverly, A., Mittal, S., & Chen, J. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487710 · Cited by 2
Research on the Architecture Design Based on Large Language Model
Li, X., Li, M., Xia, X., Song, J., & Bu, X. (2024) · International Conference on Interactive Intelligent Systems and Techniques (IIST) · doi:10.1109/iist62526.2024.00145 · Cited by 1
GenCeption: Evaluate vision LLMs with unlabeled unimodal data
Cao, L., Buchner, V., Senane, Z., & Yang, F. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101785 · Cited by 3
Clustering Examples in Multi-Dataset Benchmarks with Item Response Theory
Rodríguez, P., Htut, P. M., Lalor, J., & Sedoc, J. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.14 · Cited by 3
LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity
Arunkumar, A., Sharma, S., Agrawal, R., Chandrasekaran, S., & Bryan, C. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14840 · Cited by 1
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 1
Pseudointelligence: A Unifying Lens on Language Model Evaluation
Murty, S., Paradise, O., & Sharma, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.485 · Cited by 3
Meaningfully Explaining Model Mistakes Using Conceptual Counterfactuals.Preprint
Abid, A., Yüksekgönül, M., & Zou, J. (2021) · arXiv (Cornell University) · Cited by 1
The Three Terms Task - an open benchmark to compare human and artificial semantic representations
Borghesani, V., Armoza, J., Hebart, M. N., Bellec, P., & Brambati, S. M. (2023) · Scientific Data · doi:10.1038/s41597-023-02015-3 · Cited by 1
Bian, Z., & Mou, L. (2025) · Preprints.org · doi:10.20944/preprints202511.0001.v1 · Cited by 1
Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation
Mustahsan, Z., Lim, A., Anand, M., Jain, S., & McCann, B. (2025) · arXiv · doi:10.48550/arxiv.2512.06710
“Infrastructure projects such as the lm-eval-harness Gao et al. (2021), OpenAI Evals Chen et al. (2023), and Dynabench Kiela et al. (2021) emphasize standardization and robustness, though they rarely quantify uncertainty.”
Meaningfully Explaining a Model's Mistakes.Preprint
Abid, A., & Zou, J. (2021) · arXiv (Cornell University)
Overconfidence in the Face of Ambiguity with Adversarial Data
Li, M., & Michael, J. (2022) · doi:10.18653/v1/2022.dadc-1.4
Diaz, D. Y. R., Anioł, M., & Culnan, J. (2022) · doi:10.18653/v1/2022.dadc-1.6
6 Questions for Socially Aware Language Technologies
Yang, D. (2021) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2021.3874
Robust and Efficient Claim Retrieval for Online Fact-Checking ApplicationsPreprint
Boland, K., Hövelmeyer, A., Fafalios, P., Todorov, K., Mazhar, U., & Dietze, S. (2023) · Research Square · doi:10.21203/rs.3.rs-3007151/v1
NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?
Gabriel, S., Palangi, H., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.413
Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow
Arunkumar, A., Mishra, S. R., Sachdeva, B., Baral, C. R., & Bryan, C. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.212
Atanasova, P. (2024) · Springer eBooks · doi:10.1007/978-3-031-51518-7_1
Contextual Forensic Analysis of Emails Using Machine Learning Algorithms
Apoorva, K. A., & Sangeetha, S. (2024) · IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT) · doi:10.1109/ic2pct60090.2024.10486397
Sentence-Space Metrics (SSM) for the Evaluation of Sentence Comprehension
Lin, J., Chen, H., & Ding, N. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_29
Koay, L. (2025) · Routledge eBooks · doi:10.4324/9781003391333-14
A semi supervised framework for human and machine collaboration in computer assisted text refinement
Sun, Y., Wang, Y., Yang, H., & Suen, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-10085-z
Specifics Matter: An Analysis of Mutual Fund ESG DisclosuresPreprint
Shi, H., Han, X., Lee, J. B., & Lu, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5355991
ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition
Alyahya, H. A., Khan, H., Alnumay, Y., Bari, M. S., & Yener, B. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-demo.33
CausalLink: An Interactive Evaluation Framework for Causal Reasoning
Feng, J., & Rudzicz, F. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1147
Language Models Lack Temporal Generalization and Bigger is Not Better
Verkijk, S., Vossen, P. T. J. M., & Sommerauer, P. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1060
Mind the Gap: Static and Interactive Evaluations of Large Audio Models
Li, M., Held, W., Ryan, M. J., Pipatanakul, K., Manakul, P., Zhu, H., & Yang, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.428
MELLM - An Automatic Evaluation Framework for LLM Without Human Supervision
Cui, C., Zhu, L., Ning, H., Sheng, X., & Wu, H. (2024) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie63306.2024.11116990
Intelligent Generative Design: A New Mechanical Design ConceptPreprint
Ning, F., Wang, Y., Li, H., Shi, Y., & Lu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5393870
Muteract: Interactive and Iterative Prompt Mutation Interface for LLM Developers and Evaluators
Meena, Y. K., Mondal, S., & Potta, M. P. (2025) · International Conference of Human-Computer Interaction Design & Research · doi:10.1145/3768633.3770129
Williams, E., & Iskandarova, S. (2025) · IEEE Global Conference on Artificial Intelligence and Internet of Things (GCAIoT) · doi:10.1109/gcaiot68269.2025.11275539
AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models
Xing, H., Tian, C., Zhao, L., Ma, Z., Wang, W., Zhang, N., Huang, C., & Duan, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3783995
Self-citationEvaluating Robustness and Generalization in LLMs under Adversarial and Real-World Conditions
Demirsan, Y., & Yıldız, O. T. (2026) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp68355.2026.00058
The UNLP 2024 Shared Task on Fine-Tuning Large Language Models for Ukrainian
Romanyshyn, M., Syvokon, O., & Kyslyi, R. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/57pbivswb65o
Design and Application of an Evaluation System for Large Language Models
Shen, Y., Cao, Z., & Chen, M. (2025) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai67447.2025.11412740
MetaThink: Empowering Large Reasoning Models with Adaptive Self-Correction at Inference TimePreprint
Qi, J., & Bian, H. (2026) · Preprints.org · doi:10.20944/preprints202603.0750.v1
Analyzing AI Evaluation Benchmarks Through Information Retrieval and Network Science
Simeoni, G., Soprano, M., Lunardi, R., Roitero, K., & Mizzaro, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_25
Önden, A., & Önden, İ. (2026) · Argumentation Based Systems Journal · doi:10.59543/avhexs22
Contextualization and Generalization in Entity and Relation Extraction
Taillé, B. (2022) · theses.fr (ABES) · doi:10.70675/72fea173z5549z4bd5z8332zf81f26d3c2b9
Semi-supervised deep learning for robust anomaly detection in smart manufacturing systems
Le, T. (2024) · theses.fr (ABES) · doi:10.70675/19d23228zf030z4df7za3abz596d36577818
Third Workshop on Human-Centered Evaluation and Auditing of Language Models: AI Agents-in-the-Loop
van der Maden, W., Deng, W. H., Liu, Y. L., Jiang, H., Chen, V., Li, H., Kim, J., Liao, Q. V., Xu, W., Eslami, M., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778709
Palani, S., & Setlur, V. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790735 · Cited by 3
Bissoli, A., Mollaeefar, M., Landuyt, D. V., & Ranise, S. (2026) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2026.104489 · Cited by 2
BasqueGLUE: A Natural Language Understanding Benchmark for Basque
Urbizu, G., Vicente, I. S., Saralegi, X., Agerri, R., & Soroa, A. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3f53mtyzdxwr
Integrating Neural and Symbolic Paradigms for Reflective Ethical Reasoning and Belief Modeling
Eswarawaka, R., Naveen, L., & A, V. K. (2026) · International Conference on Next-Gen Quantum and Advanced Computing: Algorithms, Security, and Beyond (NQComp) · doi:10.1109/nqcomp68334.2026.11497759
AGWAN, S., Tambe, A., Shah, S., & Pole, G. S. (2026) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-026-05091-6
LLM Orchestration with Geometry-Aware Memory
Mojila, D. (2026) · International Conference Big Data and Machine Learning · doi:10.1109/icbdml68582.2026.11544690
The meek shall inherit the Earth: personality, empathy, and the human shaping of LLMs
Mangalam, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01215-y
The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing
Shergadwala, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806536 · Cited by 1
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
Baack, S., Buschek, C., & Boháček, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812240
Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South
Rastogi, C., Bhutani, M., Kahng, M., Muhammad, S. H., Razumovskaia, E., Suresh, P., Ahmad, I. S., Kalia, C., Mahomed, Y., Maji, M., et al. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812296 · Cited by 2
BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks
Sokol, A. A., Vejsbjerg, I., Daly, E. M., Piorkowski, D., Hind, M., Moniz, N., & Chawla, N. V. (2026) · Workshop on Evaluating Evaluations (EvalEval) · doi:10.18653/v1/2026.evaleval-1.29
Cabrera, L. M., D'Arcy, J., & Saxton-Knight, I. (2026) · Workshop on Evaluating Evaluations (EvalEval) · doi:10.18653/v1/2026.evaleval-1.20
Comparative Analysis of Ahmad-Yusoff and Jaro-Winkler Approaches for Javanese Language Stemming
Andira, A. B. A., Ahda, F. A., & Sulistyo, D. A. (2026) · Journal of Information System Research (JOSH) · doi:10.47065/josh.v7i2.7879
Çulha, D. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30801-6_9
Dammu, P. P. S., Palkhiwala, A., Roosta, T. G., & Shah, C. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808627 · Cited by 1
Recipes for Agents: Understanding Skills and Their Open Questions
Xing, H., Zhuang, H., Zhao, X., Huang, Y., Tang, Z., & Zhang, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818652 · Cited by 2
Data and Knowledge Dual-Driven Text Embedding Approaches: A Survey
Chen, X., 詹佳青, Lin, J., Liu, H., Zhang, Q., Chen, J., & Chen, H. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3721883
The Impact of Sarcastic Features on Sentiment Analysis of Russian-Language Texts
Мухин, И. С., Avksentieva, E. Y., & Kharkovchuk, N. A. (2026) · International Conference on Neural Networks and Neurotechnologies (NeuroNT) · doi:10.1109/neuront71829.2026.11650780
Evidence and Judgment: AI in mission-oriented work without hype or dismissal
USDOE Office of Science (SC), & Blakely, B. (2026) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/3484700
BOTTER, P. D. (2026) · Measurement Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2026.2720515
PersonaBase: A Formative Evaluation Study of Computational Resources for Data-Driven Personas
Salminen, J. O., Amin, D., Kaate, I., & Jansen, B. J. J. (2026) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3800935.3830833
A roadmap to make geoscience data AI-ready
Guo, Y., Feng, M., Li, G., Wang, J., Bai, Y., 李麓含之, Ran, Y., Pan, X., & Li, X. (2026) · Information Geography · doi:10.1016/j.infgeo.2026.100061
Abla, M. M., Abu-Elkheir, M., & Mashaly, M. E. (2026) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt69916.2026.11688484
Kotte, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39028-8_40
Performance within the foundation paradox: the case for small qualitative evaluations
Simeone, M. P., Hyatt, J., Bienenstock, E. J., Babu, R., & Solís, P. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12328-0
Bakhtary, P. (2026) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.66834/0azb3w20
MedEvalArena: A Self-Generated, Peer-Judged Benchmark for Medical Reasoning (Preprint)
Prem, P., Kuppa, V., Shidara, K., Wheeler, E., Chang, G., Liu, F., Alaa, A. M., & Bernardo, D. (2026) · Journal of Medical Internet Research · doi:10.2196/96885
Agyemang, J. O., Agyekum, K. O.-B. O., Agyekum, K. A.-P., Acheampong, F. A., & Kponyo, J. J. (2026) · Array · doi:10.1016/j.array.2026.101371
Metadata
- Authors
- Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., & Williams, A.
- Affiliations
- Meta (United Kingdom) (GB)
- ORCID
- Venue
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2021
- Pages
- 4110-4124
- OpenAlex
- W3157225713
- Cited by
- 254
- References
- 115
- Citations by year
- 2026: 34 · 2025: 29 · 2024: 37 · 2023: 71 · 2022: 58 · 2021: 25
- FWCI
- 21.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-4031, N66001-19)
- Office of Naval Research (N000141812871, N00014-18, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- benchmarking, natural language processing, benchmark design, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
