The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
HellaSwag: Can a Machine Really Finish Your Sentence?
Open accessAbstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys."With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached.Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves di cult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset.Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle ( †48%).We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers.AF proves to be surprisingly robust.The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting di culty, sheds light on the inner workings of deep pretrained models.More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no 'ground-truth' implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been 'retrofitted' for compatibility with autoregressive LMs.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zellers et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Traditional language model benchmarks, such as HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), and BIG-Bench (Srivastava et al., 2023), primarily assess LLMs on tasks like commonsense reasoning and standardized exams.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAI version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Zellers et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Self-citationDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Research works printed but not linked: 2
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for e cient text classification. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427-431.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
Items 201–300 of 908
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
“• HellaSwag (Zellers et al., 2019) A novel benchmark for commonsense Natural Language Inference (NLI) which is build by adversarial filtering against transformer models.”
BackgroundIlić, D., & Gignac, G. E. (2024) · Intelligence · doi:10.1016/j.intell.2024.101858 · Cited by 43
Kaiser, J., Lauscher, A., & Eichler, A. (2025) · Science Advances · doi:10.1126/sciadv.adr4173 · Cited by 28
“These include the LMSYS Chatbot Arena ELO rating [Zheng et al., 2023], the MT-bench score [Zheng et al., 2023], the Massive Multitask Language Understanding (MMLU) score [Hendrycks et al., 2021] and the HellaSwag score [Zellers et al., 2019].”
AiF: Accelerating On-Device LLM Inference Using In-Flash Processing
Lee, J., Kim, H., Oh, S., Chun, M., Kim, M., & Kim, J. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731073 · Cited by 19
PROST: Physical Reasoning about Objects through Space and Time
Aroca-Ouellette, S., Paik, C., Roncone, A., & Kann, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.404 · Cited by 23
Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study
Martínez, G., Hernández, J. A., Conde, J., Reviriego, P., & Merino-Gómez, E. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3696459 · Cited by 29
Knowledge-guided large language model for material science
Wang, G., Hu, J., Zhou, J., Liu, S., Li, Q., & Sun, Z. (2025) · Review of Materials Research · doi:10.1016/j.revmat.2025.100007 · Cited by 16
Downstream Datasets Make Surprisingly Good Pretraining Corpora
Krishna, K., Garg, S., Bigham, J. P., & Lipton, Z. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.682 · Cited by 27
LLMEffiChecker : Understanding and Testing Efficiency Degradation of Large Language Models
Feng, X., Han, X., Chen, S. H., & Yang, W. M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664812 · Cited by 20
The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models
Renze, M., & Guven, E. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852493 · Cited by 69
Edge large language models: a comprehensive survey
Jiang, S., Zhou, X., Zhang, M., Xu, C., Liao, G., Chen, J., & Cao, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-025-00227-7 · Cited by 18
“For example, in HellaSwag (Zellers et al. 2019 ), the model is presented with a context and must choose the most logical and natural-sounding completion from four options; Com - monsenseQA (Talmor et al. 2019 ) provides a multiple-choice question format that requires a model to draw upon…”
Coherence boosting: When your pretrained language model is not paying enough attention
Malkin, N., Wang, Z., & Jojić, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.565 · Cited by 30
DictPrompt: Comprehensive dictionary-integrated prompt tuning for pre-trained language model
Cao, R., Wang, Y., Gao, L. C., & Yang, M. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110605 · Cited by 15
Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU
Koto, F., Aisyah, N., Li, H., & Baldwin, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.760 · Cited by 36
How Well Do Large Language Models Perform on Faux Pas Tests?
Shapira, N., Zwirn, G., & Goldberg, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.663 · Cited by 21
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14
Shen, K., & Kejriwal, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0295925 · Cited by 17
Can LLMs Generate Competency Questions?
Rebboud, Y., Tailhardat, L., Lisena, P., & Raphaël Troncy, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78952-6_7 · Cited by 14
Goal-Oriented Script Construction
Lyu, Q., Zhang, L., & Callison-Burch, C. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.19 · Cited by 19
WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering
Bolotova-Baranova, V., Blinov, V., Филиппова, С. В., Scholer, F., & Sanderson, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.290 · Cited by 15
SafeText: A Benchmark for Exploring Physical Safety in Language Models
Levy, S., Allaway, E., Subbiah, M., Chilton, L. B., Patton, D. U., McKeown, K. R., & Wang, W. Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.154 · Cited by 30
JASMINE: Arabic GPT Models for Few-Shot Learning
Nagoudi, E. M. B., Abdul-Mageed, M., Elmadany, A., Inciarte, A. A., & Khondaker, M. T. I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1040 · Cited by 25
Rethinking Learning Rate Tuning in the Era of Large Language Models
Jin, H., Wei, W., Wang, X., Zhang, W., & Wu, Y. (2023) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi58952.2023.00025 · Cited by 26
Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models
Li, L., Li, Q., Zhang, B., & Chu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29815 · Cited by 25
Shaikh, T. A., Rasool, T., & Mir, W. A. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104005 · Cited by 18
LightMamba: Efficient Mamba Acceleration on FPGA with Quantization and Hardware Co-design
Wei, R., Xu, S., Zhong, L., Yang, Z., Guo, Q., Wang, Y., Wang, R., & Li, M. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993079 · Cited by 17
“We evaluate the perplexity and the zero-shot accuracy on six tasks: LAMBADA [25], HellaSwag [26], PIQA [27], Arc (Easy and Challenge) [28], Winogrande [29], and OpenbookQA [30] using lm-eval-harness [31].”
EntEval: A Holistic Evaluation Benchmark for Entity Representations
Chen, M., Chu, Z., Chen, Y., Stratos, K., & Gimpel, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1040 · Cited by 12
Leveraging Symbolic Knowledge Bases for Commonsense Natural Language Inference using Pattern Theory
Aakur, S. N., & Sarkar, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3287837 · Cited by 17
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation
Zhu, W., Yan, A., Lu, Y., Xu, W., Wang, X. E., Eckstein, M. P., & Wang, W. Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.5 · Cited by 35
Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-Aware Cache Compression
Cheng, F., Guo, C., Wei, C., Zhang, J., Zhou, C., Hanson, E., Zhang, J., Liu, X., Li, H. H., & Chen, Y. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731024 · Cited by 22
Shen, K., & Kejriwal, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112253 · Cited by 13
Zhong, W., Wang, S., Tang, D., Xu, Z., Guo, D., Chen, Y., Wang, J., Yin, J., Zhou, M., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.177 · Cited by 18
A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues
Li, Y., Hu, B., Chen, X., Ding, Y., Ma, L., & Zhang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.601 · Cited by 20
Navigating Confidentiality in Test Automation: A Case Study in LLM Driven Test Data Generation
Karmarkar, H., Agrawal, S., Chauhan, A., & Shete, P. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00041 · Cited by 15
Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models
Zhang, Y., Zhao, L., Cao, S., Zhang, S., Wang, W., Cao, T., Yang, F., Yang, M., Zhang, S., & Xu, N. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688089 · Cited by 13
Weakly supervised veracity classification with LLM-predicted credibility signals
Leite, J. A., Razuvayevskaya, O., Bontcheva, K., & Scarton, C. (2025) · EPJ Data Science · doi:10.1140/epjds/s13688-025-00534-0 · Cited by 26
Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., & Roth, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.660 · Cited by 47
SmolLM2: When Smol Goes Big - Data-Centric Training of a Small Language Model
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, A. P., Srivastav, V., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.02737 · Cited by 13
EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark
Li, M., Zhong, J., Chen, T., Lai, Y., & Psounis, K. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01245 · Cited by 12
“5 [96], (b) 7 closed-source LMMs: GPT-4o [62], GPT-4o-mini [63], GPT-4V [61], Gemini 1.5 Pro, Gemini 1.5 Flash [69, 76], Qwen-VL-Max and Qwen-VL-Plus [4].”
InfluentialZero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective
Yang, P., Wang, J., Gan, R., Zhu, X., Zhang, L., Wu, Z., Gao, X., Zhang, J., & Sakai, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.474 · Cited by 14
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models
Xia, M., Artetxe, M., Du, J., Chen, D., & Stoyanov, V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.780 · Cited by 15
Kassem, A. M., Mahmoud, O., & Saad, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.265 · Cited by 25
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 13
Koto, F., Mahendra, R., Aisyah, N., & Baldwin, T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00726 · Cited by 29
“English translations are for illustrative purposes. on the English datasets, such as PIQA (Bisk et al., 2020a), HellaSwag (Zellers et al., 2019), Social IQA (Sap et al., 2019), often overlook geographi-a cal aspects, thereby risking cultural bias.”
Piecing Together Teamwork: A Responsible Approach to an LLM-based Educational Jigsaw Agent
Doherty, E., Perkoff, E. M., von Bayern, S., Zhang, R., Dey, I., Bodzianowski, M., Puntambekar, S., & Hirshfield, L. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713349 · Cited by 12
“Mistral-7B outperforms other LLMs of comparable size, including Meta’s Llama 2 [92], on a variety of language task benchmarks such as Hellaswag for commonsense reasoning and GSM8K for solving math word problems [96, 100].”
The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation
He, J., Wang, T., Xiong, D., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.327 · Cited by 17
Analyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 25
Adversarial Self-Attention for Language Understanding
Wu, H., Ding, R., Zhao, H. Y., Xie, P., Huang, F., & Zhang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26608 · Cited by 15
Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements
Liu, J., Wang, W., Wang, D., Smith, N. A., Choi, Y., & Hajishirzi, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.81 · Cited by 38
Self-citationSemantic Novelty Detection in Natural Language Descriptions
Ma, N., Politowicz, A. L., Mazumder, S., Chen, J., Liu, B., Robertson, E., & Grigsby, S. S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.66 · Cited by 13
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145
“We test the following: 5-shot MMLU [14] and 20-shot HellaSwag [61] benchmarks.”
Gjorgjevikj, A., Martinc, M., Cenikj, G., Stojanov, R., Drole, J., Ispirova, G., Menichetti, G., Ogrinc, N., Trajanov, D., Džeroski, S., et al. (2026) · Current Research in Food Science · doi:10.1016/j.crfs.2026.101351 · Cited by 14
“…the Llama 3 8B model represented one of the most advanced open-weight large language models available, offering competitive performance on standard benchmarks such as ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and TruthfulQA (Lin et al., 2022).”
From text to multimodal: a survey of adversarial example generation in question answering systems
Yiğit, G., & Amasyalı, M. F. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02199-z · Cited by 13
The ethical evaluation of large language models and its optimization
Lyu, Y., & Du, Y. (2025) · AI and Ethics · doi:10.1007/s43681-024-00654-9 · Cited by 11
“Specifically, techniques such as length control [ 88], adversarial filtering to improve reasoning capabilities [ 89], detection task disambiguation [90], the introduction of unbiased indicators [91], addressing the spurious correlation of LLMs towards predefined attributes [ 92], employing diverse…”
Beyond words: evaluating large language models in transportation planning
Ying, S., Li, Z., & Yu, M. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2493073 · Cited by 17
MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization
Ramachandran, A., Kundu, S., & Krishna, T. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3730989 · Cited by 24
“For the single ReCoN variant of Micro-ScopiQ, as the design scales to higher array sizes, the area overhead of ReCoN decreases (128 × 128 has overhead of 3%), with the on-chip area being dominated by the large PE array and buffers.”
Suong, T. P. M., Nghiem, L. H., & Tran, T. K. (2025) · Enterprise Information Systems · doi:10.1080/17517575.2025.2520213 · Cited by 11
“Common sense reasoning allows the model to identify these contradictions, facilitating the detection of false information (Zellers et al. 2019).”
Semantic Data Set Construction from Human Clustering and Spatial Arrangement
Majewska, O., McCarthy, D., van den Bosch, J. J. F., Kriegeskorte, N., Vulić, I., & Korhonen, A. (2021) · Computational Linguistics · doi:10.1162/coli_a_00396 · Cited by 11
Jiang, J., Zhou, K., Wen, J.-R., & Zhao, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.131 · Cited by 19
Feature-Level Debiased Natural Language Understanding
Lyu, Y., Li, P., Yang, Y., de Rijke, M., Ren, P., Zhao, Y., Yin, D., & Ren, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26567 · Cited by 13
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?
Qin, C., Joty, S., Li, Q., & Zhao, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.659 · Cited by 14
On the Role of Bidirectionality in Language Model Pre-Training
Artetxe, M., Du, J., Goyal, N., Zettlemoyer, L., & Stoyanov, V. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.293 · Cited by 19
Gubelmann, R., Katis, I., Niklaus, C., & Handschuh, S. (2023) · Journal of Logic Language and Information · doi:10.1007/s10849-023-09410-4 · Cited by 12
Task Compass: Scaling Multi-task Pre-training with Task Prefix
Zhang, Z., Wang, S., Yi‐chong, X., Fang, Y., Yu, W., Liu, Y., Zhao, H. Y., Zhu, C., & Zeng, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.416 · Cited by 17
ThriftLLM: On Cost-Effective Selection of Large Language Models for Classification Queries
Huang, K., Shi, Y., Ding, D., Li, Y., Yang, F., Lakshmanan, L. V. S., & Xiao, X. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749702 · Cited by 10
“For a fair comparison, we modify the budget constraint of FrugalGPT on testing queries to align with this per-query approach in our experiments.”
Pacis, F. J., Alyaev, S., Pelfrene, G., & Wiktorski, T. (2024) · SPE/IADC Drilling Conference · doi:10.2118/217671-ms · Cited by 10
TB-STC: Transposable Block-wise N:M Structured Sparse Tensor Core
Liu, J., Zeng, S., Zhao, J., Ding, L., Wang, Z., Li, J., Zhu, Z., Ning, X., Zhang, C., Wang, Y., et al. (2025) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00075 · Cited by 10
“7B [79] and Llama2-7B [41], [63] across four commonsense reasoning benchmarks, namely Piqa [2], HellaSwag [76], WinoGrande [52], and Arc-e [7].”
Moëll, B., Farestam, F., & Beskow, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1557920 · Cited by 15
How coherent are neural models of coherence?
Pishdad, L., Fancellu, F., Zhang, R., & Fazly, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.539 · Cited by 11
On Curriculum Learning for Commonsense Reasoning
Maharana, A., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.72 · Cited by 10
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Ivison, H., Smith, N. A., Hajishirzi, H., & Dasigi, P. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.576 · Cited by 23
Zafar, A., Parthasarathy, V. B., Van, C. L., Shahid, S., Khan, A. I., & Shahid, A. (2023) · doi:10.36227/techrxiv.24009351.v1 · Cited by 11
MAGE: Machine-generated Text Detection in the Wild
Li, Y., Li, Q., Cui, L., Bi, W., Wang, Z., Wang, L., Yang, L., Shi, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.13242 · Cited by 10
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00838 · Cited by 10
“…evaluation suite (see Table 3) consists of: arc (both arc_easy and arc_challenge ) (Clark et al., 2018), boolq (Clark et al., 2019), openbookqa (Mihaylov et al., 2018), sciq (Welbl et al., 2017), hellaswag (Zellers et al., 2019), piqa (Bisk et al., 2020), and winogrande (Sakaguchi et al., 2021).”
BackgroundDecker: Double Check with Heterogeneous Knowledge for Commonsense Fact Verification
Zou, A., Zhang, Z., & Zhao, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.752 · Cited by 13
Contrastive variational information bottleneck for aspect-based sentiment analysis
Chang, M., Yang, M., Jiang, Q., & Xu, R. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111302 · Cited by 11
Benchmarking Large Language Model (LLM) Performance for Game Playing via Tic-Tac-Toe
Topsakal, O., & Harper, J. B. (2024) · Electronics · doi:10.3390/electronics13081532 · Cited by 9
User Centric Evaluation of Code Generation Tools (Invited Paper)
Miah, T., & Zhu, H. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00022 · Cited by 18
Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models
Sicari, S., Moreno, J. F. C., Rizzardi, A., & Coen‐Porisini, A. (2024) · ACM Computing Surveys · doi:10.1145/3703454 · Cited by 13
“The authors of HellaSwag concluded that once a powerful model solves a particular benchmark for natural language inference, then such a model should be used alongside adversarial techniques like those used in [336, 337] to try to craft a more challenging benchmark.”
A Generalize Hardware Debugging Approach for Large Language Models Semi-Synthetic, Datasets
Fu, W., Li, S., Zhao, Y., Yang, K., Zhang, X., Jin, Y., & Guo, X. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3487486 · Cited by 11
“…used, context window size, and average scores across six benchmarks indicating the model’s intelligence: AI2 Reasoning Challenge [72] ( 25 − shot ) , HellaSwag [73] ( 10 − shot ) , MMLU [74] ( 5 − shot ) , TruthfulQA [75] ( 0 − shot ) , Winogrande [76] ( 5 − shot ) , and GSM8k [77] ( 5 − shot ) .”
Is Open Source the Future of AI? A Data-Driven Approach
Vake, D., Šinik, B., Vičič, J., & Tošić, A. (2025) · Applied Sciences · doi:10.3390/app15052790 · Cited by 9
A Flexible Template for Edge Generative AI With High-Accuracy Accelerated Softmax and GELU
Belano, A., Tortorella, Y., Garofalo, A., Benini, L., Rossi, D., & Conti, F. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3562734 · Cited by 13
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
Zhang, Y., Aggarwal, S., & Mitra, T. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992741 · Cited by 9
“All methods aim to maximize GPU memory utiliza-01234”
HAPE: Hardware-Aware LLM Pruning For Efficient On-Device Inference Optimization
Zhao, W., Zou, L., Wang, Z., Yao, X., & Yu, B. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3744244 · Cited by 12
“For evaluation, we follow the same setting as Llama [2] to perform zero-shot task classification on seven common sense reasoning datasets to test the model performance on accuracy (%): BoolQ [31], PIQA [32], HellaSwag [33], WinoGrande [34], ARC-easy [35], ARC-challenge [35], and OpenbookQA [36].”
Mucciaccia, S. S., Paixão, T. M., Mutz, F. W., Souza, A. F. D., Badue, C. S., & Oliveira-Santos, T. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5801 · Cited by 10
Recent Advances of Multimodal Continual Learning: A Comprehensive Survey
Yu, D., Zhang, X., Chen, Y., Liu, A., Zhang, Y., Yu, P. S., & King, I. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3658485 · Cited by 52
Ghosal, D., Majumder, N., Mihalcea, R. F., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.691 · Cited by 10
Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Shashidhar, S., Chinta, A., Sahai, V., Wang, Z., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.608 · Cited by 9
Zero-Shot Learners for Natural Language Understanding via a Unified Multiple-Choice Perspective
Wang, J., Yang, P., Gan, R., Zhang, Y., Zhang, J., & Sakai, T. (2023) · IEEE Access · doi:10.1109/access.2023.3343123 · Cited by 10
Li, Y., Guérin, F., & Lin, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29822 · Cited by 38
Asking Crowdworkers to Write Entailment Examples: The Best of Bad OptionsPreprint
Vania, C., Chen, R., & Bowman, S. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06122 · Cited by 9
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8
“Then, we kept all models that had scores for all six scenarios: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2021), Winogrande (Sakaguchi et al., 2021), and GSM8K (Cobbe et al., 2021).”
MethodMore, A. B. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8557-9.ch002 · Cited by 9
A Survey on LLMs: Evolution, Applications, and Future Frontiers
Harsha, K., Kumar, K. T., Sumathi, D., & Jubilson, E. A. (2024) · Studies in computational intelligence · doi:10.1007/978-981-97-8460-8_14 · Cited by 11
cosmosage: A natural-language assistant for cosmology
de Haan, T. (2025) · Astronomy and Computing · doi:10.1016/j.ascom.2025.100934 · Cited by 11
Position: LLM Unlearning Benchmarks are Weak Measures of Progress
Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., & Smith, V. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00035 · Cited by 47
“To evaluate model quality on non-Harry Potter -related information, the authors run a number of standard benchmarks such as HellaSwag [39] and OpenBookQA [40].”
Demystifying the Resilience of Large Language Model Inference: An End-to-End Perspective
Sun, Y., Coalson, Z., Chen, S., Liu, H., Zhang, Z., Hong, S., Fang, B., & Yang, L. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759803 · Cited by 13
“We consider two sets of downstream tasks: 1) multiple-choice style tasks to test LLM”
From computation to environmental cost the resource burden of artificial intelligence
Falk, S., Corrêa, N. K., Luccioni, A. S., Biber‐Freudenberger, L., & van Wynsberghe, A. (2026) · Communications Earth & Environment · doi:10.1038/s43247-026-03537-5 · Cited by 8
Maximum Entropy Loss, the Silver Bullet Targeting Backdoor Attacks in Pre-trained Language Models
Liu, Z., Shen, B., Lin, Z., Wang, F., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.237 · Cited by 8
Metadata
- Authors
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4791-4800
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2946609015
- Cited by
- 830
- References
- 20
- Citations by year
- 2026: 200 · 2025: 263 · 2024: 109 · 2023: 93 · 2022: 50 · 2021: 61 · 2020: 44 · 2019: 10
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1703166, 1256082, 1637479, 1524371)
- Nvidia
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, natural language inference, AI-generated text, BERT, benchmark design, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
