The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
HellaSwag: Can a Machine Really Finish Your Sentence?
Open accessAbstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys."With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached.Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves di cult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset.Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle ( †48%).We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers.AF proves to be surprisingly robust.The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting di culty, sheds light on the inner workings of deep pretrained models.More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no 'ground-truth' implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been 'retrofitted' for compatibility with autoregressive LMs.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zellers et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Traditional language model benchmarks, such as HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), and BIG-Bench (Srivastava et al., 2023), primarily assess LLMs on tasks like commonsense reasoning and standardized exams.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAI version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Zellers et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Self-citationDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Research works printed but not linked: 2
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for e cient text classification. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427-431.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
Items 501–600 of 908
NUPES: Non-Uniform Post-Training Quantization via Power Exponent Search
Yvinec, E., Dapogny, A., & Bailly, K. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3593987 · Cited by 13
Huang, B., Tang, S., Shen, G., Li, G., Wang, X., & Zhu, W. (2020) · International Workshop on Human-centric Multimedia Analysis (HCMA) · doi:10.1145/3422852.3423484 · Cited by 2
Language modeling and bidirectional coders representations: an overview of key technologies
Kachkou, D. I. (2021) · Informatics · doi:10.37661/1816-0301-2020-17-4-61-72 · Cited by 2
Zhou, R., Tian, K., Lai, H., & Yin, J. P. (2021) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd49262.2021.9437862 · Cited by 2
Xie, Y., Xing, L., Peng, W., & Hu, Y. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.22 · Cited by 3
Test Harder than You Train: Probing with Extrapolation Splits
Kunz, J., & Kuhlmann, M. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.2 · Cited by 3
CEG: A joint model for causal commonsense events enhanced story ending generation
Zhang, Y., Yang, Y., Gu, M., Gao, F., Chen, C., & He, L. (2023) · PLoS ONE · doi:10.1371/journal.pone.0286049 · Cited by 2
Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer
Lv, X., Ding, N., Qin, Y., Liu, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.24 · Cited by 3
Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates
Li, D., Hu, B., & Chen, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.178 · Cited by 3
The CLEF 2024 Monster Track: One Lab to Rule Them All
Ferro, N., Gonzalo, J. A., Jussi Karlgren, C., & Müller, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56072-9_2 · Cited by 2
Razeghi, Y., Mekala, R. R., Logan, R. L., Gardner, M., & Singh, S. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.39 · Cited by 2
Learning Stable Classifiers by Transferring Unstable FeaturesPreprint
Bao, Y., Chang, S., & Barzilay, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07847 · Cited by 2
An Empirical Exploration in Quality Filtering of Text DataPreprint
Gao, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.00698 · Cited by 9
Metadata Conditioning Accelerates Language Model Pre-trainingPreprint
Gao, T., Wettig, A., He, L., Dong, Y., Malladi, S., & Chen, D. (2025) · Qeios · doi:10.32388/d226qh · Cited by 22
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00656 · Cited by 2
Self-citationIvanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.09605 · Cited by 2
“…evaluate are closely related to the notion of commonsense knowledge, an area that has numerous text-based benchmarks (e.g., Levesque et al., 2012; Sakaguchi et al., 2021; Zellers et al., 2019), including targeted evaluations of physical (Bisk et al., 2020) and social commonsense (Sap et al., 2019).”
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“…evaluate it on common language modeling and common-sense reasoning tasks with the setup of: • Datasets: We evaluate H OPE and baselines on Wikitext [69], LMB [70], PIQA [71], Hel-laSwag [72], WinoGrande [73], ARC-easy (ARC-e) and ARC-challenge (ARC-c) [74], SIQA [75], and BoolQ [117] benchmarks.”
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“We evaluate MiMo-7B-Base on a series of benchmarks, encompassing natural language understanding and reasoning, scientific question answering, reading comprehension, mathematics reasoning, coding, Chinese understanding, and long-context comprehension capabilities: Language understanding and reasoning : BBH (Suzgun et a…”
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3086 · Cited by 21
Berges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 46
“Our evaluations cover factual question answering (NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017)), multi-hop question answering (HotpotQA (Yang et al., 2018)), scientific and common sense world knowledge (MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), OBQA (Mihaylov et al.…”
Mamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
Prompt Engineering 101 Prompt Engineering Guidelines from a Linguistic Perspective
Han, W., Wei, X., Cui, X., Cheng, N., Jiang, G., Qian, W., & Zhang, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_34 · Cited by 3
Adaptive Bias Discovery for Learning Debiased Classifier
Bae, J.-H., Lee, M., & Jung, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0966-6_3 · Cited by 1
Assisting Hearing Impaired Children by Personalized Gamification of Auditory Verbal Therapy
Fernando, S., Leo, C., Tharushika, C., Mawaththa, R., Perera, J., Vidhanaarachchi, S., & Kasthurirathna, D. (2024) · International Conference on Advancements in Computing (ICAC) · doi:10.1109/icac64487.2024.10850970 · Cited by 1
“ HumanEval Benchmark: Evaluate a model’s ability to generate code by analyzing the caliber of output generated in response to a particular natural language prompt [25].”
Topsakal, O., Edell, C., & Jackson, H. (2025) · The Journal of Cognitive Systems · doi:10.52876/jcs.1611181 · Cited by 37
Temporal-based graph reasoning for Visual Commonsense Reasoning
Wu, S., Liu, K., Li, J., Chen, P., Zhang, X., & Feng, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113214 · Cited by 1
Towards effective continued pre-training of EU institutional LLMs on EuroHPC supercomputers
Rausch, I., Bhaskar, B., Safont-Andreu, A., Ewetz, H., Kolovratník, D., Oravecz, C., & Runonen, M. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.02.256 · Cited by 1
WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
Liu, J., Ning, K., Su, Y., Han, W., Xu, J., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889153 · Cited by 21
“The general datasets contain BoolQ [14], RACE [15], PIQA [16], RTE [21], and HellaSwag [17], focusing on reading comprehension, commonsense reasoning, and sentence completion.”
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1
“Small variations in prompt phrasing can significantly impact a model's ability to make the correct commonsense inferences required [39].”
Mishra, V., Duvvuri, V., & Chintalapati, R. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6400-9_26 · Cited by 1
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
Uncovering Large Language Model Weaknesses in Character and Word Understanding and Manipulating
Zhang, Y., He, Z., & Yen, G. G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3575818 · Cited by 1
“Most evaluations concentrate on specialized tasks in NLP, leveraging datasets, e.g., CommonsenseQA [19], Wino-grande [17], and HellaSwag [16] for commonsense reasoning assessments, SQuAD [7] for gauging reading comprehension, as well as ARC [18] for testing scientific knowledge and reasoning…”
KAFA-Merge: Model Merging Method With Layer-Based Bayes+Linear Search
Yuce, M. K., & Amasyalı, M. F. (2025) · IEEE Access · doi:10.1109/access.2025.3582015 · Cited by 2
Multimodal AI for Romanian University Support: An LLM, RAG and Voice Approach
Joldea, A.-R., Cernăzanu-Glăvan, D., Sârbu, V., & Bulzan, A.-Ş. (2025) · IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI) · doi:10.1109/saci66288.2025.11030141 · Cited by 1
FlashDecoding++Next: High Throughput LLM Inference With Latency and Memory Optimization
Dai, G., Hong, K., Mao, Q., Li, X., Xu, J., Huang, H., Xia, H., Ning, X., Yan, S., Liang, Y. E., et al. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3585339 · Cited by 1
“The frequency of recomputation is statistically obtained to be 0.45% on average across datasets including ARC [40], HellaSwag [41] and Winogrande [42].”
InfluentialTowards enhanced LLM pretraining: Dynamic checkpoint merging via generation quality
Wang, Z., Liu, D., Li, C., Chu, D., Chen, W., Wang, B., & Sui, D. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103415 · Cited by 1
Research on enhancing model performance by merging with Korean language models
Cho, T., Kim, R., & Choi, A. J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111686 · Cited by 1
Modelling and Predicting User Actions in Conversational Information Retrieval
Fu, X., Lipani, A., & Kando, N. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744622 · Cited by 1
BitWeaver: Read-Time Truncation in Memory
Gagnon, G., Malla, S., Kang, Y., & Liu, L. (2025) · International Conference on Supercomputing · doi:10.1145/3721145.3725779 · Cited by 1
“First, some tasks, such as Hellaswag [52], exhibit strong correlation to the adjustment term.”
Pay more attention to the robustness of LLMs on adversarial prompt for instruction data mining
Wang, Q., Feng, D., Zhang, X., Shen, A., Xu, Y., Ding, B., & Wang, H. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107989 · Cited by 2
DuoQ: A DSP Utilization-aware and Outlier-free Quantization for FPGA-based LLMs Acceleration
Yu, Z., Ji, H., Cao, Y., Wu, J., Yan, X., Zheng, L., & Zou, Z. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132816 · Cited by 1
A Critical Review of Benchmarking LLMs for Real-World Applications: Trends and Limitations
Ashqar, H. I. (2025) · International Conference on Ubiquitous and Future Networks · doi:10.1109/icufn65838.2025.11169931 · Cited by 1
ProFNSE: propagation dynamics-derived fake news detection in social networks
You, F., Ding, M., Luo, H., Ma, Y., & Li, H. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07860-0 · Cited by 1
“Complex network theory reveals that online social networks generally exhibit scale-free property [12]; a small number of hub nodes have super-linear connectivity (e.g., the top 1% of users in Twitter control 83% of the information forwarding links) [13], which leads to the exponential proliferation…”
Leveraging teleological explanation to support general-purpose AI assessment
Fumagalli, M., & Ferrario, R. (2025) · AI & Society · doi:10.1007/s00146-025-02587-1 · Cited by 1
“…across multiple dimensions, such as response time , memory usage , and accuracy in supporting specific tasks like solving math problems , 8 handling conversations (Zheng et al. 2023), generating code (Jain et al. 2022), or performing reasoning tasks (Clark et al. 2018; Zellers et al. 2019).”
Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges
Islam, M. R., Deng, B., Dhar, N., Nguyen, T. N., He, S., Shi, Y., & Suo, K. (2025) · IEEE International Conference on Mobile Ad-Hoc and Sensor Systems (MASS) · doi:10.1109/mass66014.2025.00059 · Cited by 1
“The evaluation of our models was conducted using the MMLU benchmark [5], HellaSwag [6], and Winogrande [7].”
Bit-Serial Acceleration of LLM Inference With Mixture-of-Datatype Quantization
Chen, Y., Chang, C.-C., Dai, X., AbouElhamayed, A. F., Andronic, M., Constantinides, G. A., & Abdelfattah, M. S. (2025) · IEEE Transactions on Computers · doi:10.1109/tc.2025.3628193 · Cited by 1
“For discriminative tasks, we choose three datasets: HellaSwag [39], WinoGrande [40], and Piqa [41], and evaluate their accuracy under zero-shot setting.”
Uncertainty Quantification for Multiple-Choice Questions is Just One-Token Deep
Zeng, Q., Jin, M., Yu, Q., Wang, Z. G., Hua, W., Sun, G., Meng, Y., Ma, S., Wang, Q., Juefei-Xu, F., et al. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760887 · Cited by 1
“Specifically, we include: MMLU [4] for general knowledge question answering, CosmosQA [7] for reading comprehension, HellaSwag [25] for commonsense inference, HaluDial [10] for dialogue response selection, and both HaluSum [10] and CNN/Daily Mail [16] for document summarization.”
LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
Hu, H., Xiao, B., Sun, S., Yin, J., Zhang, Z., Luo, X., Jiang, C., Xu, W., Jia, X., Liu, X., et al. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759852 · Cited by 10
Spanish and LLM Benchmarks: Is MMLU Lost in Translation?
Ortiz, I. P., Melero, N., del Pozo, C., Conde, J., Reviriego, P., Mayor-Rocher, M., & Grandury, M. (2025) · International Generative AI and Computational Language Modelling Conference (GACLM) · doi:10.1109/gaclm67198.2025.11232191 · Cited by 1
“For example, there are tests to evaluate the capabilities of LLMs to solve common sense reasoning problems Zellers et al. (2019) or to answer mathematical questions Hendrycks et al. (2021b).”
Ou, J., Guo, J., Jiang, S., Wang, Z., Chen, Y., Xue, R., & Tian, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229416 · Cited by 1
“Large Language Models (LLMs) such as GPT-4 [1], LLaMA-1/2 [2], [3], Codex [4], and ChatGPT [5] have demonstrated remarkable proficiency in various natural language processing benchmarks, such as language understanding and problem-solving tasks [6]–[8].”
Emotion in Context: Human and AI Perspectives on the Affective Landscape of Severance
Ortiz, A. M., & García-Gámez, M. (2025) · Corpus Pragmatics · doi:10.1007/s41701-025-00220-6 · Cited by 4
“Reasoning benchmarks include HellaSwag, which tests commonsense reasoning through plausible sentence completions (Zellers et al., 2019), and BBH (Big-Bench Hard), a subset of particularly challenging tasks from the larger Big-Bench collection (Suzgun et al., 2023).”
Pang, T., Mao, Y., Pan, Z., Qi, R., Zhang, B., He, X., & Wu, J. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3642866 · Cited by 1
“To access the performance on zero-shot tasks, we select several popular benchmarks, including ARC [39], Hellaswag [40], PIQA [41], and Winogrande [42] using the lm-evaluation-harness.”
FP2: A 2-bit Floating-Point Format for Edge-AI Inference and Fine-Tuning
Dang, Q., Ma, C., Huang, H., Fu, G., Huo, Z., Yang, G., Zong, P., Xia, T., Zhao, W., & Ren, P. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3638831 · Cited by 1
“We also conducted a comparative analysis of the zero-shot performance for the proposed FP 2 format based on the LLAMA [38] model across three benchmark datasets (namely Winogrande, Winograd, and Hellaswag [39], [40], [41]).”
Language models for the analysis of and interaction with climate change documents
Volkanovska, E. (2025) · Environmental Data Science · doi:10.1017/eds.2025.10024 · Cited by 1
“The general-domain evaluation datasets include HellaSwag 40 (Zellers et al., 2019), PIQA 41 (Bisk et al., 2020), OpenBookQA 42 (Mihaylov et al., 2018), WinoGrande 43 (Sakaguchi et al., 2021), and the MMLU dataset 44 (Hendrycks et al., 2020).”
Wang, J., & Shi, Y. (2025) · IEEE International Conference on Social Sciences and Intelligence Management (SSIM) · doi:10.3390/engproc2025098033 · Cited by 1
Robin: RWKV Accelerator using Block Circulant Matrices based on FPGA
Li, Z., Li, S., Dai, C., Ma, C., Liang, J., Li, X., & Zhang, W. (2025) · 2025 IEEE/ACM International Conference On Computer Aided Design (ICCAD) · doi:10.1109/iccad66269.2025.11240845 · Cited by 1
Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
Li, Y., Liu, Z., Hu, B., Wang, W. W., Ding, Y., Cao, X., & Zhang, M. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3649356 · Cited by 9
Phase transitions in large language model compression
Ma, Z., Li, Z. M., Zhang, L., Xia, G.-S., Du, B., Zhang, L., & Tao, D. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00072-8 · Cited by 1
Hou, S., Liu, T., Wang, L., Hou, L., & Yan, B. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131637 · Cited by 1
SpinOut: Enhanced Rotation-Based Quantization for LLM by Outlier Injection
Park, S., & Chung, K. (2026) · IEEE Access · doi:10.1109/access.2026.3664084 · Cited by 1
QAQ: Quality Adaptive Quantization for LLM KV Cache
Cheng, W. M., Dong, S., Qin, J., & Wang, W. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00267 · Cited by 29
KnitLoRA: bridging low-rank adaptation as interwoven layers for deeper semantic reasoning
Qiu, H., Ning, Y., Li, J., Zhu, X. F., Gong, Y., Yan, L., Guan, F., Zhang, J., Feng, F., XU, Z., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-47668-3 · Cited by 1
“Systematic evaluation against comparative LoRA frameworks utilizes six datasets, assessing model performance across mathematical reasoning (GSM8K 27 , SVAMP 7 , MetaMathQA 28 , MathQA 29 ) and commonsense reasoning (HellaSwag 8 and our proposed HSCoT).”
Exploring question answering: metric analysis and evaluation framework for enhanced interpretability
Navarro, L. C., Mucciaccia, S. S., Mutz, F. W., Paixão, T. M., Badue, C. S., Souza, A. F. D., & Oliveira-Santos, T. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12053-8 · Cited by 1
“HellaSwag [79] targets commonsense reasoning, requiring models to select the most plausible continuation among distractor options generated to be deceptively similar.”
Enhancing Post-Training Quantization via Future Activation Awareness
Lv, Z., Fan, Z., Tian, Q., Zhang, W., & Zhuang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465076 · Cited by 1
Cross-layer Attention Sharing for Pre-trained Large Language Models
Mu, Y., Wu, Y., Fan, Y., Wang, C. Q., Li, H., Zeng, J., He, Q., Yang, M., Meng, F., Zhou, J., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.616 · Cited by 25
“We reported the 0-shot accuracy on PIQA, BoolQ (Clark et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC easy (ARC-E) (Bhakthavat-salam et al., 2021), 5-shot accuracy on OBQA (Mihaylov et al., 2018) and MMLU, 10-shot accuracy on HellaSwag (Zellers et al., 2019), 25-shot accuracy on ARC challenge (ARC-C).”
AIvaluateXR: An Evaluation Framework for On-Device AI in XR With Benchmarking Results
Khan, D., Liu, X., Mena, O., Jia, D., Kouyoumdjian, A., & Viola, I. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3693354 · Cited by 1
“We evaluate each model using six benchmarks, assessing various reasoning and knowledge capabilities: Hel-laswag [48] (commonsense reasoning), MMLU [49] (multi-subject knowledge), ARC [50] (complex science questions), TruthfulQA [51] (truthfulness in responses), Winogrande [52] (pronoun resolution),…”
Gómez, R., Miranda, C. E., Romero-González, J.-A., Esparza, D. M. C., Alfonso-Francia, G., Chávez‐Urbiola, E. A., Ramírez-Pedraza, A., & Terven, J. R. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8060141 · Cited by 2
Edge-LLM-Bench: Characterising LLM Inference on Raspberry Pi 5 with Hailo-10H NPU Acceleration
Le, V. D., & İncel, Ö. D. (2026) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3812836.3814998 · Cited by 1
PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization
Chen, M., Liu, Y., Wang, J., Bin, Y., Shao, W., & Luo, P. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3711802 · Cited by 33
“We also evaluate on 5 zero-shot reasoning tasks, including PIQA [42], ARC [43], HellaSwag [44], and Wino-Grande [45].”
BitNet v2: Low-Bit and Sparsified Activations for 1-Bit Large Language Models
Wang, H., Ma, S., & Wei, F. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3717905 · Cited by 1
“We report the zero-shot accuracy across a range of language tasks, including ARC-Easy [31], ARC-Challenge [31], Hellaswag [32], Winogrande [33], PIQA [34], and Lambada [38].”
Peykani, P., Charles, V., Emrouznejad, A., Ghanidel, S., & Sabour, D. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10720-6 · Cited by 1
EAIRA: Establishing a methodology for evaluating LLMs as scientific research assistants
Cappello, F., Madireddy, S., Underwood, R., Getty, N., Chia, N., Ramachandra, N., Nguyen, T. D., Keçeli, M., Mallick, T., Li, Z., et al. (2026) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420261467786 · Cited by 21
Towards Stable Natural Language Understanding via Information Entropy Guided Debiasing
Du, L., Ding, X. H., Sun, Z., Liu, T., Qin, B., & Liu, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.161 · Cited by 1
Yi, J., Chen, W., Qin, Y., Lin, Y., Ding, N., Xu, H., Liu, Z., Sun, M., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.244 · Cited by 1
Adversarial Benchmark Evaluation Rectified by Controlling for Difficulty
Mehrbakhsh, B., Martínez‐Plumed, F., & Hernández‐Orallo, J. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230454 · Cited by 1
Raithubot: An RLHF-Fine-Tuned Telugu Chatbot for Farmers
Jagirdar, S., Prashanth, U. S., Malapaka, A., & Amulya, V. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-7820-5_32 · Cited by 1
Neural Reasoning about Agents’ Goals, Preferences, and Actions
Bortoletto, M., Shi, L., & Bulling, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27800 · Cited by 2
Augmenting NLP Models with Commonsense Knowledge
Jiang, M., Lin, B. Y., Wang, S., Xu, Y., Yu, W., & Zhu, C. (2024) · SpringerBriefs in computer science · doi:10.1007/978-981-97-0747-8_5 · Cited by 1
Knowledge-augmented Methods for Natural Language Understanding
Jiang, M., Lin, B. Y., Wang, S., Xu, Y., Yu, W., & Zhu, C. (2024) · SpringerBriefs in computer science · doi:10.1007/978-981-97-0747-8_3 · Cited by 1
Distilling Multi-Step Reasoning Capabilities into Smaller Language Model
Yim, Y., & Wang, Z. (2024) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3651671.3651737 · Cited by 1
Curating Naturally Adversarial Datasets for Learning-Enabled Medical Cyber-Physical Systems
Pugh, S., Ruchkin, I., Weimer, J. E., & Lee, I. (2024) · ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS) · doi:10.1109/iccps61052.2024.00026 · Cited by 1
“Several works have explored filtration by removing examples solved with simple spurious cues in the image domain [13] and natural language processing (NLP) domain [24], [25], [26], [27], [28], [29].”
BackgroundTan, J., Yu, G., Li, J., Ma, X., Bao, F., Pan, E., Bian, D., Li, Y., Du, Y., Du, L., et al. (2024) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas59952.2024.10595886 · Cited by 1
Boosting Long-Tail Data Classification with Sparse Prototypical Networks
Figueroa, A., Papaioannou, J.-M., Fallon, C., Bekiaridou, Α., Bressem, K. K., Zanos, S. P., Gers, F. A., Nejdl, W., & Löser, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70368-3_26 · Cited by 1
Zhuang, X., Mao, X. H., Jiang, Y.-H., Wu, H., Zhao, S., Cai, L., Liu, S., Chen, Y., Song, Y., Jia, C., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_18 · Cited by 1
Adaptive Composite Accuracy Scoring for Domainspecific LLM Evaluation
Teh, J. S. R., Koay, E. K., Lim, S. W., Lee, K. H., Lai, M. S., Lee, M. S., & Nee, Y. K. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852484 · Cited by 3
“…questions related to extensive general knowledge across 57 subjects [6]; HellaSwag which includes multiple-choice commonsense reasoning dataset [7] and TruthfulQA is to test model’s truthfulness in answering general knowledge questions spanning across 38 different categories designed to elicit…”
Learning to Prune and Low-Rank Adaptation for Compact Language Model Deployment
Ali, A. H., Zhang, F., Yang, L., & Fan, D. (2025) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3658617.3697648 · Cited by 2
“For the LLaMa-7B model, we employ common sense reasoning datasets, including BoolQ [42], PIQA [43], HellaSwag [44], WinoGrande [45], ARC-easy [46], ARC-challenge [46], and OpenbookQA [47].”
LBPE: Long-token-first Tokenization to Improve Large Language Models
Lian, H., Xiong, Y., Lin, Z., Niu, J., Mo, S., Chen, H., Liu, P., & Ding, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889733 · Cited by 8
“We incorporate 7 benchmarks recognized for evaluating LLMs, including BoolQ [25], HellaSwag [26], OpenBookQA [27], PIQA [28], SIQA [29], StoryCloze [30], and Winogrande [31].”
Guo, L., Zhu, Z., Liu, T., Ning, X., Li, S., Dai, G., Yang, H., Fu, W., & Wang, Y. (2024) · doi:10.1145/3676536.3676776 · Cited by 1
Liu, C., Wang, S., Kang, Y., Zhao, F., Kuang, K., Lü, W., Sun, C., & Wu, F. Q. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3595143 · Cited by 1
Cao, D., & Aref, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00891-6_28 · Cited by 3
“Also, consistent with the literature, we assessed the zero-shot accuracy on five common-sense reasoning tasks, namely WinoGrande [30], PIQA [1], HellaSwag [40], Arc-Easy [5], and Arc-Challenge [5].”
Wei, Y., & Fan, J. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01236-0 · Cited by 1
“…resource data extraction task as a starting point, we hope that the LLM we use can have better language comprehension, logic, and reasoning ability, so we mainly focus on the model’s scores on the MMLU [107], HellaSwag [108], BBH [109], GSM8K [110], and MATH [111] when selecting the model we use.”
GATmath and GATLc: Comprehensive benchmarks for evaluating Arabic large language models
AlBallaa, S., Al-Twairesh, N., Al‐Salman, A. S., & Alfarhood, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329129 · Cited by 3
Enabling Real-Time Conversations with Minimal Training Costs
Wang, X., Wang, H., Wang, S., Zhao, W., Han, X., Yan, Y., Zhao, H., Zhang, Y., Tao, Z., Liu, Z., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-2725-0_21 · Cited by 6
LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models
Su, Y., Guan, Z., Liu, X., Jin, T., Wu, D., Chen, Z., Chesi, G., Wong, N., & Yu, H. (2025) · doi:10.1109/iccad66269.2025.11240704 · Cited by 3
“Zero-shot accuracy is assessed using the EleutherAI LM Harness [23] across six benchmarks: BoolQ [24], RTE [25], HellaSwag [26], ARC Easy and Challenge [27], and OpenbookQA [28].”
Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense
Chang, T.-Y., Liu, Y., Gopalakrishnan, K., Hedayatnia, B., Zhou, P., & Hakkani-Tür, D. Z. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383453 · Cited by 1
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge
Hagström, L., & Johansson, R. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.19 · Cited by 2
Language-Based Syllogistic Reasoning Using Deep Neural Networks
Aghahadi, Z., & Talebpour, A. (2022) · Cognitive Semantics · doi:10.1163/23526416-bja10026 · Cited by 2
Large Language Models and Their Current Use Cases
Özçift, A. (2023) · doi:10.59287/pcse.346 · Cited by 1
On Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 1
Metadata
- Authors
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4791-4800
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2946609015
- Cited by
- 830
- References
- 20
- Citations by year
- 2026: 200 · 2025: 263 · 2024: 109 · 2023: 93 · 2022: 50 · 2021: 61 · 2020: 44 · 2019: 10
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1703166, 1256082, 1637479, 1524371)
- Nvidia
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, natural language inference, AI-generated text, BERT, benchmark design, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
