The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
HellaSwag: Can a Machine Really Finish Your Sentence?
Open accessAbstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys."With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached.Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves di cult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset.Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle ( †48%).We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers.AF proves to be surprisingly robust.The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting di culty, sheds light on the inner workings of deep pretrained models.More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no 'ground-truth' implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been 'retrofitted' for compatibility with autoregressive LMs.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zellers et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Traditional language model benchmarks, such as HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), and BIG-Bench (Srivastava et al., 2023), primarily assess LLMs on tasks like commonsense reasoning and standardized exams.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAI version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Zellers et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Self-citationDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Research works printed but not linked: 2
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for e cient text classification. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427-431.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
Items 401–500 of 908
Exploring the Curious Case of Code Prompts
Zhang, L., Dugan, L. D., Xu, H., & Callison-Burch, C. (2023) · doi:10.18653/v1/2023.nlrse-1.2 · Cited by 6
Cross-functional Analysis of Generalization in Behavioral Learning
de Araujo, P. H. L., & Roth, B. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00590 · Cited by 4
A simple yet effective self-debiasing framework for transformer models
Wang, X., Liu, X., Wang, L., Wu, S., Su, J., & Wu, H. R. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104258 · Cited by 5
Establishing vocabulary tests as a benchmark for evaluating large language models
Martínez, G., Conde, J., Merino-Gómez, E., Margaretto, B. B., Hernández, J. A., Reviriego, P., & Brysbaert, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308259 · Cited by 6
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 27
Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities
Ma, N., Mazumder, S., Politowicz, A. L., Liu, B., Robertson, E., & Grigsby, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.627 · Cited by 4
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4
Gpt-4 Passes Most of the 297 Written Polish Board Certification ExaminationsPreprint
Pokrywka, J., Gorzelańczyk, E. J., & Kaczmarek, J. I. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4861528 · Cited by 12
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, Xu, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.19348 · Cited by 4
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3
“Early model-based efforts like BERTScore (Zhang et al., 2020) sought to improve on traditional n-gram metrics by recognizing high-quality responses that differ from the reference.”
Bit-slice Architecture for DNN Acceleration with Slice-level Sparsity Enhancement and Exploitation
Choi, I., Yoon, Y.-S., & Yang, J.-S. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00067 · Cited by 4
“2 1B [14] and Gemma-2 2B [74] are evaluated using the HellaSwag benchmark [88], as detailed in Section V-G.”
Intermediate-task transfer learning for Indonesian NLP tasks
Ekakristi, A. S., Wicaksono, A. F., & Mahendra, R. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100161 · Cited by 17
Abdelkarim, S., Lu, D., Flores, D., Jaeggi, S. M., & Baldi, P. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100170 · Cited by 6
“HellaSwag tests adversarial commonsense narrative completion (Zellers et al., 2019), whereas GSM8K concentrates on multi-step arithmetic word problems (Cobbe et al., 2021).”
Selective fine-tuning for large language models via matrix nuclear norm
Xia, T., Li, Y., Wu, Y., & Chang, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104259 · Cited by 3
PILs of Knowledge: A Synthetic Benchmark for Evaluating Question Answering Systems in Healthcare
Lunardi, R., Soprano, M., Coppola, P., Mea, V. D., Mizzaro, S., & Roitero, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730283 · Cited by 3
“The evaluation process consists of prompting each model with the question and requiring it to select the correct answer from the four possible choices, exactly as done to assess the effectiveness of LLMs in popular benchmarks [11, 13, 21, 32, 35, 60, 63].”
Xu, W., Choi, H., Hsu, P.-K., Yu, S., & Simunic, T. (2025) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3750727 · Cited by 3
“We evaluate the zero-shot accuracy for LLMs using the following benchmarks: commonsense reasoning (WinoGrande [47], HellaSwag [70], PIQA [6], BoolQ [61], and OpenBookQA [38]) and reading comprehension (Arc-Easy and Arc-Challenge [15]).”
Tao, R., Chen, H., Guo, Y., Wang, J., Wang, B., Ni, R., & Zhao, Y. J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107909 · Cited by 3
Kojima, T., Matsuo, Y., & Iwasawa, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.21 · Cited by 4
Zhang, Y., Yu, J., Li, G., He, Z., & Yen, G. G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3596257 · Cited by 3
“…on single-domain data may su ff er from catastrophic forgetting, where performance on general tasks (e.g., common-sense reasoning on HellaSwag [60], knowledge acquisition on MMLU [61], or reading comprehension on SQuAD [62]) degrades due to the absence of diverse, broader-domain instruction…”
Li, J., Huang, S., Xu, J., Liu, J., Xu, N., & Dai, G. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3624278 · Cited by 4
“Since the randomly generated tokens lack logical coherence, we utilize common LLM datasets ( e.g. , WikiText [14], Lambada [15], Piqa [16], HellaSwag [17], WinoGrande [18], and Arc-easy [19]) as input tokens to profile the SSM value distribution.”
MixPQ: Joint Pruning and Quantization for Speech and Language Foundation Models Compression
Liu, B., Gu, T., Wang, H., & Qian, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3613948 · Cited by 3
Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference
Abstreiter, M., Tarkoma, S., & Morabito, R. (2026) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3788870 · Cited by 12
“However, it currently supports only a limited selection of models.”
S 2 Mamba: An Efficient Mamba Accelerator With Word-Importance SSM Sparsity
Sun, K., Zhou, J., Wang, M., & Wang, Z. (2026) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3649880 · Cited by 3
“We evaluate zero-shot accuracy on six benchmark tasks, including WinoGrande [35], Piqa [36], OpenbookQA [37], HellaSwag [38] and Arc-Easy / Challenge [39].”
Kim, S., Choi, Y., Kim, B., Oh, J. H., & Yoo, H. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408453 · Cited by 3
“For task-level evaluation, we conduct zero-shot inference on various benchmarks, including PIQA [2], ARC-e, ARC-c [6], BoolQ [5], HellaSwag [52], and WinoGrande [34], which collectively test commonsense reasoning, factual understanding, and logical inference.”
Self-citationM 2 XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
Hu, W., Zhang, Z., Zhang, H., Zhang, C., Guo, C., Feng, Y., Hu, T., Li, G., Hu, G., Wang, J., et al. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790185 · Cited by 11
“LLM benchmarks include Wikitext v2 and common sense QA tasks such as Arc-challenge, Arc-easy, HellaSwag, PIQA, Wino-Grande, and BoolQ [4, 6, 7, 57, 69].”
Lee, G., Kim, S., Lee, D., Goh, K., & Kim, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108900 · Cited by 4
Dynamic nested hierarchies: self-evolving machine learning architectures for lifelong learning
Jafari, A. A., Ozcinar, C., & Anbarjafari, G. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1804338 · Cited by 4
“Commonsense reasoning uses zero-shot accuracy on PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-easy/challenge (Clark et al., 2018), Socialiqa (Sap et al., 2019), and BoolQ (Clark et al., 2019).”
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization
Gu, Y., Ke, P., Zhu, X., & Huang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.105 · Cited by 8
Koala: An Index for Quantifying Overlaps with Pre-training Corpora
Vu, T.-T., He, X., Haffari, G., & Shareghi, E. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.7 · Cited by 19
Crystal: Introspective Reasoners Reinforced with Self-Feedback
Liu, J., Pasunuru, R., Hajishirzi, H., Choi, Y., & Çelikyılmaz, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.708 · Cited by 4
Self-citationJoaquin, A. S., Wang, B., Liu, Z., Müller, P., Asher, N. M., Lim, B. Y., & Chen, N. F. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.604 · Cited by 9
Li, Z., Xia, T., Chang, Y., & Wu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130711 · Cited by 5
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanovic, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 3
ICP: Immediate Compensation Pruning for Mid-to-high Sparsity
Luo, X., Lu, X., Jiang, Z., & Zhou, S. K. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00886 · Cited by 3
“We also evaluate the pruned Llama 2-7B model using accuracy on seven Zero-Shot tasks: BoolQ [2], RTE [32], HellaSwag [35], Wino-Grande [26], ARC Easy and Challenge [3], and Open-bookQA [22], following the experimental setup introduced by Sun et al. [27] in their work on Wanda.”
Where You Say Matters: A Study of Positional Bias of Small LLMs
Vella, S., Sharieh, S., Hussain, F. A., & Ferworn, A. (2025) · IEEE World AI IoT Congress (AIIoT) · doi:10.1109/aiiot65859.2025.11105359 · Cited by 3
“HellaSWAG [21] is a more challenging version which uses grounded truth from WikiHow and ActivityNet instead of LLM-generated incorrect answers.”
Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
Gafni, T., Karnieli, A., & Hanani, Y. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00308 · Cited by 3
“Quantization times are similar to the timings reported in [12].”
Efficient LLMs with AMP: Attention Heads and MLP Pruning
Mugnaini, L. G., Yamamoto, B. L., de Alcantara, L. L., Zacarias, V., Bollis, E., Pellicer, L. F. A. O., Costa, A. H. R., & Jordão, A. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227985 · Cited by 6
“We use the EleutherAI LM Harness framework [35] to perform zero-shot task classification on common sense reasoning datasets: WinoGrande [36], HellaSwag [37], ARC-e / ARC-c [38] and PIQA [39].”
Liquid: Language Models are Scalable and Unified Multi-Modal Generators
Wu, J., Jiang, Y., Ma, C., Liu, Y., Zhao, H., Yuan, Z., Bai, S., & Bai, X. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02639-5 · Cited by 90
“…large language models and multi-modal language models across a suite of popular benchmarks that measure commonsense reasoning and reading comprehension capabilities: HellaSwag [80], WinoGrande [56] , ARC-Easy [10], ARC-Challenge [10], OpenBookQA [44], PIQA [3], SIQA [57], and BoolQ [9].”
Adaptive Pruning for Large Language Models With Structural Importance Awareness
Zheng, H., Ren, J., Han, Y., Sun, Y., Zhang, R., Zhang, W., Li, Z., Niyato, D. T., & Cui, S. R. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3654102 · Cited by 3
“The HellaSwag dataset [41] is employed to evaluate the model’s understanding and reasoning capabilities in daily scenarios.”
Relation-aware Bidirectional Path Reasoning for Commonsense Question Answering
Wang, J., Li, X., Tan, Z., Zhao, X. H., & Xiao, W. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.35 · Cited by 3
Improving Abstractive Summarization with Commonsense Knowledge
Nair, P. A., & Singh, A. K. (2021) · Recent Advances in Natural Language Processing · doi:10.26615/issn.2603-2821.2021_019 · Cited by 3
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning
Maharana, A., & Bansal, M. (2022) · doi:10.18653/v1/2022.dlg4nlp-1.6 · Cited by 3
SLR: A million-scale comprehensive crossword dataset for simultaneous learning and reasoning
Wang, C., Zhu, T., Li, Z., & Liu, J. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126591 · Cited by 3
Causal interventions expose implicit situation models for commonsense language understanding
Yamakoshi, T., McClelland, J. L., Goldberg, A. E., & Hawkins, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.839 · Cited by 7
Kavumba, P., Brassard, A., Heinzerling, B., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.162 · Cited by 4
Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
Zhu, L., Zhao, R., Gui, L., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.677 · Cited by 13
Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
Faysse, M., Viaud, G., Hudelot, C., & Colombo, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.559 · Cited by 9
NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation
West, P., Bras, R. J. L., Sorensen, T., Lin, B. Y., Jiang, L., Lu, X., Chandu, K. R., Hessel, J., Baheti, A., Bhagavatula, C., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.80 · Cited by 4
Self-citationShukla, U., Singh, S., Pundir, A., & Saxena, G. J. (2023) · IEEE Conference on Information and Communication Technology (CICT) · doi:10.1109/cict59886.2023.10455408 · Cited by 4
Leveraging the Potential of Large Language Models
Prasad, S., Gupta, H., & Ghosh, A. (2024) · Informatica · doi:10.31449/inf.v48i8.5635 · Cited by 7
Operating Conversational Large Language Models (LLMs)in the Presenceof Errors
Gao, Z., Deng, J., Reviriego, P., Liu, S., Pozo, A., & Lombardi, F. (2024) · IEEE Nanotechnology Magazine · doi:10.1109/mnano.2024.3513112 · Cited by 3
“There are benchmarks for example for solving mathematical problems [8], general reasoning [9] or to evaluate the capability of LLMs to answer questions on many topics [10].”
Self-evaluation of LLMs on challenging LLM-generated STEM MCQs
Soliman, G., Zaki, H., & Lotfy, M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11164-y · Cited by 4
“Inspired by the work of Zellers et al. [67], the current benchmarks have proven to be trivial for the state-of-the-art models as they primarily emphasize Natural Language Generation (NLG) rather than being built on the capabilities of LLMs.”
Methodology for Quality Assurance Testing of LLM-based Multi-Agent Systems
Shamim, I., & Singhal, R. (2024) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3703412.3703439 · Cited by 6
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
Vania, C., Chen, R., & Bowman, S. R. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.68 · Cited by 5
Intrinsic Probing through Dimension Selection
Hennigen, L. T., Williams, A., & Cotterell, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.15 · Cited by 41
Towards Flow Graph Prediction of Open-Domain Procedural Texts
Shirai, K., Kameko, H., & Mori, S. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.8 · Cited by 4
Shi, H., Wang, W., Fang, T., Xu, B., Ding, W., Liu, X., & Song, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1023 · Cited by 4
Zafar, A., Parthasarathy, V. B., Van, C. L., Shahid, S., Khan, A. I., & Shahid, A. (2023) · doi:10.36227/techrxiv.24009351 · Cited by 3
LLMEra: Impact of Large Language ModelsPreprint
Vavekanand, R., & Kumar, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4857084 · Cited by 4
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 24
“• Single task : we employ 9 datasets spanning knowledge (MMLU (Hendrycks et al., 2021), MMLU-pro (Wang et al., 2024e), Hellaswag (Zellers et al., 2019)), reasoning (GSM8k (Cobbe et al., 2021), Knowledge Crosswords (Ding et al., 2024), NLGraph (Wang et al., 2024a; Zhang et al., 2024a)), and safety (TruthfulQA (Lin et a…”
Self-citationThink before you speak: Training Language Models With Pause Tokens
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., & Nagarajan, V. (2023) · arXiv · doi:10.48550/arxiv.2310.02226 · Cited by 2
“HellaSwag: Next-sentence prediction task based on common-sense inference (Zellers et al., 2019).”
BackgroundQuantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2
“In Tables 3 and 4, we report various metrics on the discrete performance measure for GSM8k, Hellaswag, and ARC-C.”
InfluentialBenchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish
Uzunoğlu, A., & Şahin, G. G. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.52 · Cited by 6
Dynamic Low-Rank Adaptation Based Pruning Algorithm for Large Language Models
Li, L., & Guo, L. (2024) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai62207.2024.10826600 · Cited by 3
Beyond KV caching: Shared attention for efficient LLMs
Liao, B., & Vargas, D. V. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130587 · Cited by 2
Jin, X., Zheng, H., Nie, M., Wang, J., & Shi, C. ‐. R. (2025) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3716368.3735187 · Cited by 2
An Efficient White-box LLM Watermarking for IP Protection on Online Market Platforms
Yuan, S., Su, X., Lv, P., Xue, W., Yu, J., Zhu, X., & Chen, C. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736841 · Cited by 2
Zhao, J., Zeng, C., Wang, M., Han, L., Shang, Y., Zhang, M., & Nie, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3599479 · Cited by 2
“Weselectseveralpopularbenchmarksincluding PIQA[55],ARC[56],BoolQ[57],HellaSwag[58]andWino-grande[59]usinglm-evaluation-harness[60].”
A new fixed-point simulation methodology for on-device AI based on large language models
Kim, J.-W., Yoon, S.-H., Kang, D.-K., Lim, S.-W., Lee, H.-B., Oh, S.-M., & Seo, Y.-H. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103548 · Cited by 2
Grand Challenges in AI and Education Beyond 2030
Woolf, B. P., Allessio, D., Arroyo, I., Gattupalli, S., & Zhang, B. (2025) · Interaction design & architecture(s)/ID&A Interaction design & architecture(s) · doi:10.55612/s-5002-064-001sp · Cited by 2
Exploring the Use and Misuse of Large Language Models
Valdez, H. P. D., Abri, F., Webb, J. B., & Austin, T. H. (2025) · Information · doi:10.3390/info16090758 · Cited by 5
“Two of the datasets discussed in Section 2.1, HellaSwag [26] and HumanEval [28], were used, with GPT-4 scoring 95.3% and 67% and GPT-3.”
InfluentialRising From Pieces: Effective Inference at the Edge via Robust Split ML
Weng, Y., Zheng, T., Chen, Z., Hu, M., & Luo, J. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3605382 · Cited by 3
“For transformer-based LLMs, we use HellaSwag [60] for sentence completion, where the model selects the most plausible sentence ending from four options; MMLU [61] for multitask question answering, which spans 57 subjects from elementary to expert level; and ARC-c [62] for science reasoning, which…”
Fajčík, M., Dočekal, M., Doležal, J., Ondrej, K., Beneš, K., Kapsa, J., Smrž, P., Polok, A., Hradiš, M., Nevěřilová, Z., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.32 · Cited by 14
Finding the Pareto Frontier of Low-Precision Data Formats and MAC Architecture for LLM Inference
Crafton, B., Peng, X., Sun, X., Lele, A. S., Zhang, B., Khwa, W.-S., & Akarvardar, K. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132989 · Cited by 2
“2-1B [38] on the Hellaswag dataset [40] and BERT-Large [25] and RoBERTa-Large [25] on the STSB dataset [6].”
Intent-Based Kubernetes Configuration via LLMs: Current Trends and Open Challenges
Sacco, A., Zilli, C., & Marchetto, G. (2025) · Conference on Local Computer Networks · doi:10.1109/lcn65610.2025.11146359 · Cited by 2
“Most benchmarks are designed to assess a specific dimension of model capability, such as natural language understanding ( e.g., SuperGLUE [27]), reasoning ( e.g., HellaSWAG [29]), or code generation ( e.g., HumanEval [28] and CodeXGLUE [30]).”
Zhao, W., Hu, Y., Sui, X., Li, Z., Deng, Y., Zhao, Y., Qin, B., & Che, W. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-41099-x · Cited by 2
ccAI: A Compatible and Confidential System for AI Computing
Wang, C., Tang, D., Ci, C., Huang, J., Xu, Y., Zhang, F., Cao, J., Song, J., Yan, S., Wei, T., et al. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756104 · Cited by 2
Efficient Low-Rank Adaptation for Sparse Large Language Model
Hu, Y., Tian, T., Chen, X., Zhao, Z., Tao, T., Zhang, W., Li, Y., Liang, Y., Li, C., Chen, H., et al. (2025) · Tsinghua Science & Technology · doi:10.26599/tst.2025.9010174 · Cited by 2
HybridMoE: LoRA-Based LLMs Fine-Tune With Hybrid Mixture of Experts
Lin, S., Ma, Y., Liu, S., Shi, J., Jin, L., Gao, D., Yu, S., Xuan, Q., Cai, X., & Yang, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3637555 · Cited by 2
Grouped query attention supported with graph-based query clustering
Zheng, L., Zhang, Y., Shen, L., Miao, C., & Shen, Q. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115311 · Cited by 2
A large-scale evaluation of commonsense knowledge in humans and large language models
Nguyen, T. D., Watts, D. J., & Whiting, M. E. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag029 · Cited by 6
“…method to evaluate progress in this area, we argue that the notion of invariant ground truth — widely assumed but rarely articulated in benchmarking practices — has likely misguided AI researchers, creating an illusion of success as LLMs continue to pass a growing number of tests ( 47, 65 – 67).”
Retrofitting language models to operate over bytes
Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. D., Soldaini, L., & Hofmann, V. (2026) · Nature · doi:10.1038/s41586-026-11111-4 · Cited by 19
Down and Across: Introducing Crossword-Solving as a New NLP Benchmark
Kulshreshtha, S., Kovaleva, O., Shivagunde, N., & Rumshisky, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.189 · Cited by 4
DiscoSense: Commonsense Reasoning with Discourse Connectives
Bhargava, P., & Ng, V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.703 · Cited by 2
CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model
Zhang, K., Ding, N., Qi, B., Zhu, X., Long, X., & Zhou, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.597 · Cited by 5
Empower Nested Boolean Logic via Self-Supervised Curriculum Learning
Wu, H., Liu, L., Zhao, H. Y., & Zhang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.847 · Cited by 5
Comparative Analysis for Open-Source Large Language Models
Schur, A., & Groenjes, S. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-49215-0_7 · Cited by 2
A Comprehensive Overview of CFN From a Commonsense Perspective
Li, R., Zhao, Y., Wang, Z., Su, X., Guo, S., Guan, Y., Han, X., & Zhao, H. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1450-8 · Cited by 2
Heterogeneous-Graph Reasoning With Context Paraphrase for Commonsense Question Answering
Wang, Y., Hu, Z., Liang, J., & Li, R. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434469 · Cited by 2
Testing Stimulus Equivalence in Transformer-Based Agents
Carrillo, A., & Betancort, M. (2024) · Future Internet · doi:10.3390/fi16080289 · Cited by 2
“Commonly evaluated tasks are factual knowledge [43], commonsense inference [44] and reasoning [45], model’s ability to apply knowledge across various domains [46], tendency to generate falsehoods [47], or mathematical reasoning [48].”
BackgroundOptimizing Fine-Tuning in Quantized Language Models: An In-Depth Analysis of Key Variables
Shen, A., Lai, Z., Li, D., & Hu, X. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.057491 · Cited by 2
Enhancing DNN Computational Efficiency via Decomposition and Approximation
Schweitzer, O., Weiser, U. C., & Gabbay, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521980 · Cited by 2
HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
Fan, H., Zhou, H., Huang, G., Raman, P., Fu, X., Gupta, G., Ram, D., Wang, Y., & Huan, J. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825098 · Cited by 9
Effective Diversity Optimizations for High Accuracy Deep Ensembles
Jin, H., Aghdam, M. A., Medikonduru, S. N. C., Wei, W., Wang, X., Zhang, W., & Wu, Y. (2024) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi62246.2024.00044 · Cited by 2
Mazzone, S. B., Forden, J., & Brylow, D. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892816 · Cited by 2
“The benchmarks include the AI2 Reasoning Challenge [5], HellaSwag [39], MMLU [13], TruthfulQA [21], Winogrande [34], and GSM8k [6], with evaluations in both 0-shot and few-shot settings to assess models’ reasoning, inference, and knowledge capabilities.”
Yang, J.-W., Kim, H., Ji, J., & Kim, Y. (2025) · Future Internet · doi:10.3390/fi17040185 · Cited by 17
Hallee, L., Kapur, R., Patel, A., Gleghorn, J. P., & Khomtchouk, B. B. (2025) · Scientific Reports · doi:10.1038/s41598-025-98185-8 · Cited by 7
Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning
Puccioni, L., Farshin, A., Scazzariello, M., Wang, C., Chiesa, M., & Kostić, D. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00028 · Cited by 2
“For CSR, following prior studies, we use a straightforward zero-shot evaluation across six datasets [44], [48], [58]–[60].”
Shen, A., Lai, Z., Wang, Q., Li, X., Zhang, L., Li, D., & Li, J. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.23 · Cited by 5
Metadata
- Authors
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4791-4800
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2946609015
- Cited by
- 830
- References
- 20
- Citations by year
- 2026: 200 · 2025: 263 · 2024: 109 · 2023: 93 · 2022: 50 · 2021: 61 · 2020: 44 · 2019: 10
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1703166, 1256082, 1637479, 1524371)
- Nvidia
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, natural language inference, AI-generated text, BERT, benchmark design, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
